MultiModalQA: complex question answering over text, tables and images

When answering complex questions , people can seamlessly combine information from visual , textual and tabular sources . While interest in models that reason over multiple pieces of evidence has surged in recent years , there has been relatively little work on question answering models that reason across multiple modalities . In this paper , we present MULTIMODALQA ( MMQA ) : a challenging question answering dataset that requires joint reasoning over text , tables and images . We create MMQA using a new framework for generating complex multi-modal questions at scale , harvesting tables from Wikipedia , and attaching images and text paragraphs using entities that appear in each table . We then define a formal language that allows us to take questions that can be answered from a single modality , and combine them to generate cross-modal questions . Last , crowdsourcing workers take these automatically generated questions and rephrase them into more fluent language . We create 29,918 questions through this procedure , and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task : our multi-hop model , ImplicitDecomp , achieves an average F1 of 51.7 over cross-modal questions , substantially outperforming a strong baseline that achieves 38.2 F1 , but still lags significantly behind human performance , which is at 90.1 F1 . 1 INTRODUCTION . When presented with complex questions , people often do not know in advance what source ( s ) of information are relevant for answering it . In general scenarios , these sources can encompass multiple modalities , be it paragraphs of text , structured tables , images or combinations of those . For instance , a user might ponder “ When was the famous painting with two touching fingers completed ? ” , if she can not remember the exact name of the painting . Answering this question is made possible by integrating information across both the textual and visual modalities . Recently , there has been substantial interest in question answering ( QA ) models that reason over multiple pieces of evidence ( multi-hop questions ( Yang et al. , 2018 ; Talmor & Berant , 2018 ; Welbl et al. , 2017 ) ) . In most prior work , the question is phrased in natural language and the answer is in a context , which may be a paragraph ( Rajpurkar , 2016 ) , a table ( Pasupat & Liang , 2015 ) , or an image ( Antol et al. , 2015 ) . However , there has been relatively little work on answering questions that require integrating information across modalities . Hannan et al . ( 2020 ) created MANYMODALQA : a dataset where the context for each question includes information from multiple modalities . However , the answer to each question can be derived from a single modality only , and no cross-modality reasoning is needed . Thus , the task is focused on identifying the relevant modality . Recently , Chen et al . ( 2020b ) presented HYBRIDQA , a dataset that requires reasoning over tabular and textual data . While HYBRIDQA requires cross-modal reasoning , it does not require visual inference , limiting the types of questions that can be represented ( See Table 1 for a comparison between the datasets ) . ∗ The authors contributed equally In this work , we present MMQA , the first large-scale ( 29,918 examples ) QA dataset that requires integrating information across free text , semi-structured tables , and images , where 35.7 % of the questions require cross-modality reasoning . Figure 1 shows an example question : “ Which B.Piazza title came earlier : the movie S. Stallon ’ s son starred in , or the movie with half of a lady ’ s face on the poster ? ” . Answering this question entails ( i ) decomposing the question into a sequence of simpler questions , ( ii ) determining the modalities for the simpler questions and answering them , i.e. , information on the poster is in an image , the information on “ S . Stallon ’ s son ” is in free text , and the years of the movies are in the table , ( iii ) combining the information from the simpler questions to compute the answer : “ Tell Me that you love me , Junie Moon ” . Our methodology for creating MMQA involves three high-level steps . ( a ) Context construction : we harvest tables from Wikipedia , and connect each table to images and paragraphs that appear in existing Reading Comprehension ( RC ) datasets ( Kwiatkowski et al. , 2019 ; Clark et al. , 2019 ; Yang et al. , 2018 ) ; ( b ) Question generation : Following past work ( Talmor & Berant , 2018 ) , we use the linked structure of the context to automatically generate questions that require multiple reasoning operations ( composition , conjunction , comparison ) across modalities in pseudo-language ; ( c ) Paraphrasing : we use crowdsourcing workers to paraphrase the pseudo-language questions into more fluent English . To tackle MMQA , we introduce ImplicitDecomp , a new model that predicts a program that specifies the required reasoning steps over different modalities , and executes the program with dedicated text , table , and image models . ImplicitDecomp performs multi-hop multimodal reasoning without the need for an explicit decomposition of the question . We empirically evaluate MMQA by comparing ImplicitDecomp to strong baselines that do not perform cross-modal reasoning and to human performance . We find that on multimodal questions , ImplicitDecomp improves F1 from 38.2→ 51.7 over a single-hop approach . Humans are able to reach 90.1 F1 , significantly outperforming our best model . Because automatic evaluation is non-trivial , we also manually analyze human performance and find humans correctly answer 94.5 % of the questions in MMQA . Finally , our dataset can be used in an open-domain setup over all of Wikipedia . In this setup , the F1 of humans is 84.8 . To summarize , our key contributions are : • MMQA : a dataset with 29,918 questions and answers , 35.7 % of which require cross-modal reasoning . • A methodology for generating multimodal questions over text , tables and images at scale . • ImplicitDecomp , A model for implicitly decomposing multimodal questions , which improves on a single-hop model by 13.5 absolute F1 points on questions requiring cross-modal reasoning . • Our dataset and code are available at https : //allenai.github.io/multimodalqa . 2 DATASET GENERATION . Our goal is to develop a method that allows generating complex questions over multiple modalities at scale . An overview of the methodology is captured in Figure 2 . We first select a Wikipedia table as an anchor , to which we add images and texts paragraphs and obtain a context . Single modality questions are generated based on these contexts , and used to automatically create multimodal , multihop questions . AMT workers rephrase the questions into natural language , and finally distractor paragraphs and images are selected for each question . We now elaborate on the 6 steps of the process . 2.1 Wikipedia tables as anchors The 01-01-2020 English Wikipedia dump contains roughly 3M tables . We extracted all tables and selected those that meet the following criteria : ( a ) The tables contain 10-25 rows ( b ) At least 3 images are associated with the table . This results in a total of 700k tables . ( see supp . material for more information ) . These tables are the anchors of our contexts , which we enrich with images and text for multimodal question generation . A key element of the tables are Wikipedia Entities ( WikiEntities ) that appear in them , i.e. , concepts linked to other Wikipedia entries . We use them to connect different modalities , bridge questions , and solve ambiguities ( details below ) . 2.2 Connecting Images and Text to Tables Images . We consider two cases : ( a ) in-table images and ( b ) images from pages of linked WikiEntities . In the former , the images are featured inside the table cells . In the latter , the table contains a column of WikiEntities that potentially have images , e.g . a table describing the filmography of an actor often contains a column of film names , which may have posters in their respective pages . To associate entities with their representative image , we map entities and their profile images in their Wikipedia pages . Overall , we obtain 57,713 images , with 889 in-table images and 56,824 WikiEntities images . Text . We build on texts from contexts appearing in existing reading comprehension datasets . We elaborate on this process next . 2.3 Generating Single-Modality Questions Tables . We generate pseudo-language table questions in the following form “ In [ table title ] of [ Wikipedia page title ] which cells in [ column X ] have the [ value Y ] in [ column Z ] ? ” . We additionally support numeric computations over columns classified as dates or numbers , such as min and max values , e.g. , “ In [ Doubles ] of [ WCT Tournament of Champions ] , what was the MOST RECENT [ Year ] ( s ) where the [ Location ] was [ Forest Hills ] ” . Images . We use crowdsourcing to generate single-modality questions about images . We generated two types of image questions , based on the images we retrieved from the previous step : ( i ) questions over a single image , ( ii ) questions over a list of images . When generating single-image questions , we show Amazon Mechanical Turk ( AMT ) crowd workers an image alongside its WikiEntity , and ask them to phrase a question about the image with the entity being the focus of the question . E.g , if the entity is “ Roger Federer ” , a potential question is “ What ’ s the hair color of Roger Federer ? ” . For questions to have meaning in an open-domain setting , we primed AMT workers to ask questions that correspond to “ stable ” features , i.e. , features that are unlikely to change in different images and are thus appropriate in an open-domain setting . For questions with a list of images , we use images that appear in the same column of a table . To generate these questions , AMT workers were given the images and asked to phrase a binary question about a distinctive feature of the entities that a subset of the images share . E.g. , given a list of statues , the worker could ask “ Which of the statues features a horse ? ” This process results in 2,764 single image questions and 7,773 list image questions that are later used to create multimodal questions . Text . To obtain questions answerable over text paragraphs we build on existing reading comprehension datasets : Natural Questions ( NQ ) ( Kwiatkowski et al. , 2019 ) consists of about 300K questions issued to the Google search engine . This dataset mostly contains simple questions where a single paragraph suffices to answer each question . BoolQ ( Clark et al. , 2019 ) contains 15 , 942 yes/no questions , gathered using the same pipeline as NQ . HotpotQA ( Yang et al. , 2018 ) contains 112K training questions , where crowd workers were shown pairs of related Wikipedia paragraphs and were asked to author questions that require multi-hop reasoning over the paragraphs . To use questions from the above datasets as building blocks for multi-hop multimodal questions , we unified them into a corpus that consists of triples of ( i ) a text question , ( ii ) an answer and ( iii ) 1-2 gold paragraphs from Wikipedia . We link a question to a table , by matching WikiEntities in the table to entities in the text of the question ( see supplementary material for further details ) . Overall , we retrieved 6,644 questions from NQ , 1,246 from BoolQ and 4,733 from HotpotQA . 2.4 Generating multimodal complex questions We present an automatic method for creating at scale multimodal compositional questions ( i , e. , questions that require answering a sequence of subquestions to conclude the final answer ) . Our first step is to introduce a formal language that allows to combine questions answerable from a single modality . Below we introduce the logical operations that allow to generate such pseudo-language ( PL ) questions , while keeping a formal representation of how they were constructed . In Table 2 , we illustrate this process with all 16 different compositional templates used for question generation . We now describe our logical operations . Logical Operations Functions in our formal language take arguments and return a PL partial question , as well as answers that can be a list of one or more strings , or a list of one or more WikiEntities . All operations have access to the full context . In addition , we prepend a prefix containing the Wikipedia table name and page title—e.g . “ In the Filmography of Brad Pitt , ” —to all our PL questions to support an open-domain QA setup . Our set of logical operations are : 1 . TABLEQ : Returns a question from the table questions generated in §2.3 , as well as a list of WikiEntities or a list of strings as answers . 2 . TEXTQ : Returns a text corpus question ( see §2.3 ) and a list of WikiEntities or strings as answers . 3 . IMAGEQ : Returns a question about a single image associated with a WikiEntity and a single token answer from a fixed vocabulary ( see §2.3 ) . 4 . IMAGELISTQ : Returns a question about a list of images and a list of WikiEntities corresponding to the images that answer the question ( see §2.3 ) . 5 . COMPOSE ( · , · ) : Takes a PL question containing a single WikiEntity as a first argument , and a PL question that produces that WikiEntity as the output answer as its second argument . E.g. , COMPOSE ( “ Where was Barack Obama born ? ” , “ Who was the 44th president of the USA ? ” ) . The function replaces the WikiEntity in the first-argument PL question with the second-argument PL question and returns the resulting PL question ( “ Where was the 44th president of the USA born ? ” ) . 6 . INTERSECT ( · , · ) : Takes two PL questions that return lists of more than one WikiEntity , and returns their intersection as the answer . The resulting PL question is of the form “ PL1 and PL2 ” omitting PL2 ’ s first word ( “ Who was born in Hawaii and is the parent of Sasha Obama ? ” ) . 7 . COMPARE ( · , · ) : Takes two PL questions each returning one WikiEntity that can be linked to one cell in the table , denoted by Ans1 , Ans2 . We first choose a numeric or date column in the table , if such exists . We then compare the values of this column corresponding to the rows of Ans1 and Ans2 . Depending on the comparison outcome , output one of ( Ans1 , Ans2 ) as the operation answer . The PL question created is of the form “ What has compare-op numeric-column-name , PL1 or PL2 ? ” omitting PL1 and PL2 ’ s first word . E.g . “ What has most recent creation year , the rocket of Appolo program , or the rocket of Gemini program ? ” 2.5 Paraphrasing using AMT We used English-speaking AMT workers to paraphrase automaticallygenerated PL questions into natural language ( NL ) . Each question was paraphrased by 1 worker and validated by 1-3 other workers . To avoid annotator bias ( Geva et al. , 2019 ) , the number of annotators who worked on both the training and evaluation set was kept to a minimum . We also deployed a feedback mechanism , where workers receive a bonus if a baseline model correctly answered the question after their first paraphrasing attempt , but incorrectly after they refined the paraphrase . See supp . material for print-screens of the AMT annotator interface . To generate diversity , workers got a bonus if the normalized edit distance of a paraphrase compared to the PL question was higher than 0.7 . A total of 971 workers were involved , and 29,918 examples were produced with an average cost of 0.33 $ per question . We split the dataset into 23,817 training , 2,441 development ( dev . ) , and 3,660 test set examples . Context components in the dev . and test sets are disjoint , and were constructed from a disjoint set of single-modality questions . A shortcoming of our method for automatically generating examples is that the question distribution does not come from a “ natural ” source . We argue that developing models that are capable of performing reasoning over multiple modalities is an important direction and MMQA provides an opportunity to develop and evaluate such models . Moreover , this method allows to control the compositional questions created , proving effective in creating a cheap and scalable dataset . 2.6 Adding distractors to the context Images . Questions from the IMAGELISTQ operator require reasoning over a list of images from the same column , and hence do not require additional distractors . For IMAGEQ questions ( single-image ) , we randomly add images that are associated with the WikiEntities that appear in the table , setting a maximum of 15 distractors per question . Text . We used DPR ( Karpukhin et al. , 2020 ) , a neural information retrieval model , to retrieve distractors for all questions . Each context includes exactly 10 paragraphs , where 1-2 are gold paragraphs and the rest are distractors . Specifically , we encode the first 2 paragraphs of each Wikipedia article with the DPR encoder , and use as distractors the paragraphs with the highest dot product between their encoding and the question encoding . We do not allow : ( a ) an overlap between the distractors in the training and evaluation sets , ( b ) distractors originating from the gold article , ( c ) distractors containing an exact match to the gold answer . To summarize , each of our examples contains a question , an answer , the formal representation of the PL question ( ignored by our models ) , and all distractors and gold context for all modalities . This renders MMQA useful for both open-domain multimodal QA , as well as context-dependant QA .

Summary: The authors present a new dataset, MultiModalQA, with the intent of measuring a model’s ability to reason across different modalities (free text, structured tables, and images) in question answering, and in which a large percentage of the questions requires cross-modal reasoning. The authors provide a detailed look at the framework used to generate questions in the context of several modalities. Finally, the authors propose a multimodal model that performs multi-hop reasoning (removing the need for explicit question decomposition), which outperforms strong baselines, but is still far behind human performance, indicating that the task is nontrivial and would benefit the research community.