MultiModalQA: complex question answering over text, tables and images
Authors: Alon Talmor
Ori Yoran
Amnon Catav
Dan Lahav
Yizhong Wang
Akari Asai
Gabriel Ilharco
Hannaneh Hajishirzi
Jonathan Berant
Venue: ICLR 2021 Poster (2021)
Keywords: NLP
Question Answering
Dataset
Multi-Modal
Multi-Hop

Abstract: When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. 
While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities.
In this paper, we present MultiModalQA (MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. 
We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define a formal language that allows us to take questions that can be answered from a single modality, and combine them to generate cross-modal questions. Last, crowdsourcing workers take these automatically generated questions and rephrase them into more fluent language.
We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, ImplicitDecomp, achieves an average F1 of 51.7  over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which is at 90.1 F1.

Published as a conference paper at ICLR 2021
MULTIMODAL COMPLEX QUESTION ANSWERING OVER TEXT, TABLES AND IMAGES
Alon T Ori Yoran 1.2 Amnon Catav*.2 Dan Lahav Yizhong Wang'
Akari Asai Gabriel Ilharco Hannaneh Hajishirzi23 Jonathan Berant
'The Allen Institute for AI, Tel-Aviv University, University of Washington (alont, oriy, jonathan (acnnn@alllee @allenai.org aamonncatav, lahav] @mail ttt.tauaccc ac. (yi zhongw, akari, gamaga, hannaneh } @cs. www.wwgton..e....
ABSTRACT
When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MULTIMODAL QA (MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define formal language that allows us to take questions that can be answered from a single modality, and combine them generate cross-modal questions. Last, crowdsourcing workers take these automatically generated questions and rephrase them into more fluent language. We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, Implicitteccom. achieves an average F1 of 51.7 over cross- modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which 90.1 F1.
INTRODUCTION
When presented with complex questions, people often do not know in advance what source(s) of information are relevant for answering it. In general scenarios, these sources can encompass multiple modalities, be it paragraphs of text, structured tables, images combinations of those. For instance, a user might ponder "When was the famous painting with two touching fingers completed?" she cannot remember the exact name of the painting. Answering this question is made possible by integrating information across both the textual and visual modalities.
Recently, there has been substantial interest in question answering (QA) models that reason
multiple pieces of evidence (multi-hop questions (Yang et al., 20118 Talmor & Berant 2018 Welbl al. 2017)). In most prior work, the question is phrased in natural language and the answer in context, which may be a paragraph (Rajpurkar) (016). table (Pasupat & Liang] 2015). or an image (Antol al. 2015). However, there has been relatively little work on answering questions that require integrating information across modalities. [Hannan et al. (2020) created MAAYYODAAAQA a dataset where the context for each question includes information from multiple modalities. However, the answer to each question can be derived from single modality only, and no cross-modality reasoning needed. Thus, the task is focused on identifying the relevant modality. Recently, Chen et al.) (2020b) presented HYBRIDQA, dataset that requires reasoning over tabular and textual data. While HyBRIdQA requires cross cross-modal modal reasoning, it does not require visual inference. limiting the types of questions that can be represented (See Table T for comparison between the datasets).
The authors contributed equally

---

Published as a conference paper at ICLR 2021
Multimodal Context
Ben Piazza Fimography
The lile kilve nuffnarts
Lallecale (ntern Litullly
Role
prosemel maneemmp with and thoir "eweltcring" and selliig aort ixteersion toan
renase Teasors
1957
Dangerous
David
"Saahorree Swingess and
1959
Hanging Tree
Rune
"Teasen1 a 1971 commeci
1962
No Exit
Camarero
listrann 1903
Michele Masrimm Tarnntin
cou caltru-atuuu padkibcck "Stell
1970
Tell That
Jesse
Rcaale del rprawat
was
love Jenle
Moon
Walliww beting sebut
Perire
prics
1972
The Outside Man
Desk Clerk
eonpade tither in Vlocyyy V
as "Ceiirable Teachert a
[1502 mutallrrat
ceready Birs direcisd
Resert auikan ir the onsriwen chora nee the Ein Rarias con of fatherss
fic.
Alxciss raasaani The
1985
Mask
Mr. Simmss
Son Are
1988
Clean and Sober
Kramer
Afte that, les lester
paa- darn in which the pike
of livel i sulgeene of
1990
RockyVV
Doctor
pixced side.
1991
Guilty by Suspicion
Darryl Zanuck
Q: Which B. Plazza title came earller: the movie S. Stallone's son starred In or the movie with half of lady's face on the poster?
A: Tell Me That You Love Ma, Junie Moon
Figure I: Example of a MMQA question, answer and context. In green are the text modality question and answer, and in red the image modality. The table is used to perform the year comparison between the answers of the text and image question parts.
In this work, we present MMQA, the first large-scale (29,918 examples) QA dataset that requires integrating information across free text, scmi-structured tables, and images, where 35.7% of the questions require mrosstmmdality reasoning.
example question: "Which B. Piazaa title came earlier: the movie S. Stallon's 's son starred in, or the movie with half of a lady's 's face on the poster?". Answering this question entails (i) decomposing the question into a sequence of simpler questions, (ii) determining the modalities for the simpler questions and answering them, i.e. information on the poster an image, the information on "S. Stallon's son" is in free text, and the years of the movies are in the table, (iii) combining the information from the simpler questions to compute the answer: "Tell Me that you love me, Junie Moon".
[
Our methodology for creating MMQ4 involves three high-level steps. (a) Context construction: we harvest tables from Wikipedia, and connect each table to images and paragraphs that appear existing Reading Comprehension (RC) datasets (Kwiatkowski et al. 2019; Clark et al.) 2019 Yang et al.) 2018); (b) Question generation: Following past work (Talmor & Berant 2018), we use the linked structure of the context automatically generate questions that require multiple reasoning operations (composition, conjunction, comparison) across modalities in pseudo- pseudo-language language (c) Paraphrasing- we crowdsourcing workers to paraphrase the pseudo-languse questions into more fluent English.
To tackle MMQA, we introduce ImplicitDecomp, new model that predicts program that spec- ifies the required reasoning steps over different modalities, and executes the program with dedi- cated text, table, and image models. ImplicitDe- comp performs multi-hop multimodal reasoning without the need for an explicit decomposition of the question.
Dataset
Size
Full-
Uses
Multi- hop
wiki
images
MANYMODALQA
10K
/
x
HYBRIDQA
70K
x
/
MULTIMODALQA
30K
/
Table comparison of MULTIMODALQA MANY- MODALQA and HYBRIDQA. We compare dataset size, use of images, and whether the dataset supports multi- hop questions and an open-domain full-wiki setup.
We empirically evaluate MMQA by compar- ing ImplicitDeccmm to strong baselines that do not perform cross-modal reasoning and to hu- man performance, We find that on multimodal questions, ImplicitDecomp improves F, from 38.2 + 51.7 over a single-hop approach. Humans are able to reach 90.1 F1, significantly outperform- ing our best model. Because automatic evaluation is trivtaivial, we also manually analyze human performance and find humans correctly answer 94.5% of the questions in MMOA. Finally, our dataset can be used in an open-domain setup over all of Wikipedia. In this setup, the F, of humans is 84.8.
2

---

Published as a conference paper at ICLR 2021
To summarize, our key contributions are:
MMQA: a dataset with 29,918 questions and answers, 35.7% of which require cross-modal reasoning.
A methodology for generating multimodal questions over text, tables and images at scale. Implicit(eecomp, A model for implicitly decomposing multimodal questions, which improves single-hop model by 13.5 absolute Fj points on questions requiring cross- reasoning. Our dataset and code are available at htttps:/sallenai.co wiweaeeeaiiiiiuu.b.......
imoda
2 DATASET GENERATION
Our goal is to develop method that allows generating complex questions over multiple modalities at scale. An overview of the methodology is captured in Figure 2 We first select Wikipedia table anchor, to which add images and texts paragraphs and obtain context. Single modality questions are generated based on these contexts, and used to automatically create multimodal, multi- hop questions. AMT workers rephrase the questions into natural language, and finally distractor paragraphs and images are selected for each question. We now elaborate on the 6 steps of the process.
1) Select anchor lables from Wikipedia
2) Connect and image contexts
3) Generate single modallty questions
4) Ganerate muli modal complex quesiions
5) AMT paraphrasing
6) Add distractors
Which country has Neal flag"
Composs/ CCanade WW2: 'Witch
"When did the country with a mapie leal WW2?"
leaf
Figure 2: An overview of MMQA dataset generation process.
2.1 Wikipedia tables as anchors The 01-01-2020 English Wikipedia dump contains roughly 3M tables. We extracted all tables and selected those that meet the following criteria: (a) The tables contain 10-25 rows (b) At least 3 images are associated with the table. This results in total of 700k tables. (see supp. material for more information). These tables are the anchors of our contexts, which we enrich with images and text for multimodal question generation. A key element of the tables are Wikipedia Entities (WikiEntities) that appear in them, i.e., concepts linked to other Wikipedia entries. We use them to connect different modalities, bridge questions, and solve ambiguities (details below).
2.2 Connecting Images and Text to Tables Images. We consider two cases: (a) in-table images and (b) images from pages of linked WikiEntities. In the former, the images are featured inside the table cells. In the latter, the table contains column of WikiEntities that potentially have images, e.g. a table describing the filmography of an actor often contains a column of film names, which may have posters in their respective pages. To associate entities with their representative image, we map entities and their profile images in their Wikipedia pages. Overall, we obtain 57,713 images, with 889 in-table images and 56,824 WikiEntities images. Text. We build texts from contexts appearing in existing reading comprehension datasets. We elaborate on this process next.
2.3 Generating Single-Modality Questions Tables. We generate sseudo-Language table questions in the following form "In [table title] of (Wikipedia page title] which cells in [column X] have the fvalue Y] in [column Z]?". We additionally support numeric computations over columns classified as dates numbers, such as min and max values, e.g., "In [Doubles] of (WCT Tournament of Champions], what was the MOST RECENT [Year](s) where the [Location] was [Forest Hills]".
Images. We use crowdsourcing to generate single-modality questions about images. We generated two types of image questions, based on the images we retrieved from the previous step: (i) questions over single image. (ii) questions a list of images.
When generating single-image questions, we show Amazon Mechanical Turk (AMT) crowd workers an image alongside its WikiEntity, and ask them to phrase question about the image with the entity being the focus of the question. E.g. if the entity is "Roger Federer", a potential question is "What's the hair color of Roger Federer?". For questions to have meaning in open-domain setting, primed AMT workers to ask questions that correspond to "stable" features, i.c., features that are unlikely to change in different images and thus appropriate in open-domain setting.
3

---

Published as a conference paper at ICLR 2021
Type TexiQ
Q&A
%
What was the territorial capital of the territory opposing Ohio the Toledo War? Detrolt
31.0
TableQ
Does the Gernun state Balen- Warttemberr Tharingia have
18.3
residents? Baden Württemberg
ImageQ
What weapon the statue Nottinghan holding? bow
8.9
At what age did the Cleveland Cavuliers player with 6190
7.8
rebounds enter the NBA? 19
Imagel.istQ
the common name of the bush warbler Thailand that has an orange stripe above its eye? Chesmut-crowned bush warbler
6.1
Compose((aaree..maagsssss)))
The film that storred Chris Ellison where a man was holding a
5,4
newpppper on the poster, was releused what yeur? 1988
Composse((mggeQ/TTaleee)))
On the poster for the show in which Tom Mison played Dorian
4.5
Crane, what kind of structure can be seen behind the two men? castle
Which manufacturer has fower wins
First Data Buick or brand with cross for logo? Buick
3.5
Compose(([ablee.Texxx
On what date did the original artist who sang Sweet Child of Mine have concert at US Bank Stadium? July 30, 2017
3.2
Inleesect((ablle(TText())
Who was the artist for Doman Fox in 2006 who also sings "You got the moves like Jugger"? Christina Aguilera
2.6
Compose((ex(Q_.mmgeeaaaa)))))
the poster for movie based on the book "Aer like a Lady, Think Like Man," " hho many people are there in total? nine
2.4
What covers of the Chandler Conterbury films from 2009 has more than Couch, LiferLife
2.3
one person? Powder Balls Out, Gary
Did Chelsea club that
You 'I Never Walk Alone rank higher in Deloitte Football Money League 2007! Chelsea
2.1
(omp
Did Gary Oldman take part in the movie whose poster feutures two men writer? no Was the film that features a giant eve on its poster or the first Wolverine Dream
1.0
holding handguns, and which had Mark L. Smith
0.8
Composo(Tabloo..exxQ))
movie the earller film that Scott Silver worked on? Requiem
What common law state with cagle the flag has institation the North region of Division II of the NCCAA? lowa
0.2
Table 2: All 16 compositional templates in MMQA with an example and their relative frequency.
For questions with a list of images, we use images that appear in the same column of a table. To generate these questions, AMT workers were given the images and asked to phrase binary question about a distinctive feature of the entities that a subset of the images share. E.g., given a list of statues, the worker could ask "Which of the statues features horse?" This process results in 2,764 single image questions and 7,773 list image questions that are later used to create multimodal questions.
Text. To obtain questions answerable over text paragraphs we build on existing reading comprehension datasets: Natural Quextions (NQ) (Kwiatkowski et al. 20199) consists of about 300K questions issued to the Google search engine. This dataset mostly contains simple questions where single paragraph suffices to answer each question. BoolQ (Clark et al.] (019) contains 15, 942 yes/no questions, gathered using the same pipeline as NQ. HotporQA (Yang et al., 2018) contains 1 112K training questions, where crowd workers were shown pairs of related Wikipedia paragraphs and were asked to author questions that require multi-hop reasoning over the paragraphs.
To use questions from the above datasets as building blocks for multi-hop multimodal questions, we unified them into corpus that consists of triples of (i) text question, (ii) an answer and (iii) 1-2 gold paragraphs from Wikipedia. We link a question to a table, by matching WikiEntities in the table to entities in the text of the question (see supplementary material for further details). Overall, we retrieved 6,644 questions from NQ. 1,246 from BoolQ and 4,733 from HotpotQA.
2.4 Generating multimodal complex questions We present an automatic method for creating at scale multimodal compositional questions (i.e., questions that require answering a sequence of sub- questions conclude the final answer). Our first step is to introduce formal language that allows to combine questions answerable from single modality. Below we introduce the logical operations that allow to generate such greudo-language (PI.) questions, while keeping formal representation of we illustrate this process with all 16 different compositional
how they were constructed. In Table 2 templates used for question generation. We now describe our logical operations.
Logical Operations Functions in our formal language take arguments and return a PL partial question, as well as answers that can be a list of one or more strings, or a list of one or more WikdEntittes. All operations have access to the full context. In addition, we prepend prefix containing the Wikipedia table name and page title-e.g "In the Filmography of Brad Pitr," " tto our PL questions to support an open-domain QA setup. Our set of logical operations are:
4

---

Published as a conference paper at ICLR 2021
1. TABLEQ: Returns question from the table questions generated in $2.3, as well list of WikiEntities or a list of strings as answers.
2. TEXTQ: Returns a text corpus question (see $2.3) and list of WikiEnntities or strings answers.
3. IMAGEQ: Returns a question about a single image associated with a WikiEntity and a single token
answer from fixed vocabulary (see $2.3).
4. IMAGELIST Q: Returns a question about list of images and a list of WikiEntities corresponding the images that answer the question (see $2.3).
5. COMPOSE(:.)): Takes a PL question containing a single WikiEntity as first argument, and PL question that produces that WikiEntity as the output answer as its second argument. E.g.. COMPOSE(" ("eeee(""WWWrrr""" Wheee was Barack Obama born?". "Woho" was the 44th president of the USA?"). The function replaces the WikiEntity in the first-argument PL question with the second-argument PL. question and returns the resulting PL. question (" ("Where was the 44th president of the USA born?"). 6. INTERSECT(; :): Takes two PL questions that return lists of more than one WikiEntity, and returns their intersection the answer. The resulting PL question is of the form "PL, and PL2" " ommiting PLg's 's first word (" ("Who was born in Hawaii and is the parent of Sasha Ohama?").
7. COMPARE(:->) -): Takes two PL questions each returning
WikiEntity that can be linked to one cell in the table, denoted by Ans Ans2. We first choose a numeric or date column in the table, if such exists. We then compare the values of this column corresponding the rows of Ans and Ansg. Depending on the comparison outcome, output one of (Ans1, Ans²) as the operation answer. The PL question created of the form "What has compare-op cameric-column-nan PL omitting PL, and PL2's 's first word. E.g. "What has most recent creation year, the rocket of Appolo program, or the rocket of Gemini program?"
PL1 or
2.5 Paraphrasing using AMT We used roglish-speaking AMT workers to paraphrase automatically- generated PL questions into natural language (NL). Each question was paraphrased by worker and validated by 1-3 other workers. To avoid annotator bias (Geva et al. 2019). the number of annotators who worked on both the training and evaluation set was kept to a minimum. We also deployed a feedback mechanism, where workers receive bonus if baseline model correctly answered the question after their first paraphrasing attempt, but incorrectly after they refined the paraphrase. See supp. material for print-screens of the AMT annotator interface.
To generate diversity, workers got bonus the normalized edit distance of a paraphrase compared to the PL question was higher than 0.7. A total of 971 workers were involved, and 29,918 examples were produced with an average cost of 0.33S per question. We split the dataset into 23,817 training, 2,441 development (dev.), and 3,660 test set examples. Context components in the dev. and test sets are disjoint, and were constructed from a disjoint set of single-modality questions.
A shortcoming of our method for automatically generating examples is that the question distribution does not come from a "natural" source. We argue that developing models that are capable of performing reasoning over multiple modalities is an important direction and MMQA provides an opportunity to develop and evaluate such models. Moreover, this method allows to control the compositional questions created, proving effective in creating a cheap and scalable dataset.
2.6 Adding distractors to the context Images. Questions from the IMAGELISTQ operator require reasoning over a list of images from the same column, and hence do not require additional distrac- tors. For IMAGEQ questions (single-image), we randomly add images that are associated with the WikiEntities that appear in the table, setting a maximum of 15 distractors per question.
Text. We used DPR (Karpukhin et al., 2020), neural information retrieval model, to retrieve distractors for all questions. Each context includes exactly 10 paragraphs, where 1-2 are gold paragraphs and the rest are distractors. Specifically, we encode the first 2 paragraphs of each Wikipedia article with the DPR encoder, and use as distractors the paragraphs with the highest dot product between their encoding and the question encoding. We do not allow: (a) an overlap between the distractors in the training and evaluation sets, (b) distractors originating from the gold article, (c) distractors containing exact match to the gold answer.
To summarize, each of our examples contains a question, an answer, the formal representation of the PL question (ignored by our models), and all distractors and gold context for all modalities. This renders MMQA useful for both open-domain multimodal QA, as well as context-dependant QA.
5

---

Published as a conference paper at ICLR 2021
Films
Transportation
Video games
Industry
Theater
John Morris the
Is the film with
In what year
the W. Lee
oo pargary made
Ansag locaridee
Chemies
with
did Mark Moor
produce
the your of
What
Great Morthern Ralkwer
What antmal
Ph halan paree Stikus
role of Santa Bay halloween or christmas movie
movie bitle featuring child wearing hat on the poster?
during the 201 was Rupert
Gregeon-villamss
midds thor
ow Earth
qame Troa with the
Music
Pictures studio?
with Calumbia
other
year old Frankle
In what
Stones relessed
The Rollin
Where
was the title of
the Rocky
The film written by Karika Dhillon that froetures two people sticking their tongues out its postar come out year?
Fernell movie did producar first: Bookomart or the movie whose poctor features
Hund in the
Television
with photo of him slaying live
December of 1969 with what food
Carroll movie
povord popelrr
poster
filmed where he
lelevisiion
cover!
played Willie?
now?
showss thet Kandy Musgrave was involved in, who plays on the show whise screen has an hourglass?
snow
Day appeared
singers wha played the song "oeleer Man" in
Was Mannix
Which film did Jon
201L what it?
Michael Drayer
was played by
Turtelaub direct
day
6tth Arnual Grommmy
the film with
man holding guitar on the cover?
Which 2013 move,, which shows cheracters wearing a cowboy het on its postet did Dariel Chong
directed Viinsonn Stone acted D Darey's Kid?
Sports
Who like
each other?
margan 2010-13
Geography
Serwoon Payk vissdd logo nas rase Natter points the Scasin loagee table?
Quelifiing Molson Indy Toronto, who has blond hair and was on team RuSPOKI
2005
episoce title the most wetched show Nick was
Emmet: J. Scorian TV
the clses, which has hosted Congress fawer omes: Londonn or the city that horseshoeshapps
show?
the city World Experanto
arimator?
1977, when wse thatr clock tower buil?
Science
There E chappdd tawer ss the round cap whech city aded 1945-50 La
History
river?
Politics
represents which planets
symbol
trident
Whtcch manufactura
har lover wine sit with cross for
or
Phrom Penh pool water was used by the Combodian League in 2016?
um located
When did the country at
Economy
lsttd in Sub-raatonal
such reg on has shi
Legalletive
Which authors wear eyeglasoes Use than Buukee Prioe 2014 novdl??
Literature
What eversoe Curessan Ue Jemooaay' in diediion?
municcal
Doppler spectroscopy?
Canada build Frs last refinery?
import trading with
this ieghants
the origin
namne?
Figure 3: Domain diversity in MMQA. The area of each color corresponds to the topic frequency in the dataset.
3 DATASET ANALYSIS
To highlight the diversity of MMQA we analyze its key statistics, domains, and lexical richness.
Key Statisties MMQA contains 29,918 questions, and their main statistics are in Table 3 Since we focus on multimodality, we upsample the number of multimodal questions in the dev. and test sets compared to the training set. Also, about 60% of the questions in MMQA are com- positional. Questions are relatively long (18.2 words), but answers tend to be short (2.1 words). The answer for each question can be single answer or a list of answers. While list answers comprise only 7.4% of the data, when consid- ering compositional questions that contain an intermediate question within them, the proportion of list answers in intermediate questions is higher (18.9%).
Measunement
Value
# Distinct Questions Train multimodal questions
29,918
34.6%
Dev.+lest multimodal questions
40.1%
Train compositional questions
58.8%
Dev. Hest compositional questions
62.3%
Average question length (words)
18.2
Average of answers per question
1.16
List
7.4%
List answers per intermediate question
18.95
Average answer length (words)
2.1
# of distinct words in questions
49,649
# of distinct words in answers # of distinct context tables
20,820
11.022
Domain Diversity Figure 3 shows a sample of questions
Table 3: Key statistics for MMQA.
from MMQA categorized to different domains. While
entertainment categories occupy large portion of our dataset (Films 36%, TV 19%), we observe questions represent wide variety of topics.
Lexical Richness Workers received a bonus when substantially modifying the PL questions. We observe that the average normalized edit distance between the NL questions and the PL questions is high (0.7), that NL questions are shorter (avg. length of 20.02 vs. 22.16 words for PL questions), and use richer vocabulary (Hunique words 39, 319 vs 37, 108).
MODELS
Here present our baseline models. We first train models that interact with single modality given a question ($4.1), and use those as building blocks in our multimodal approaches (54.2). We denote the question by Q. context paragraphs by P. Table by T and context images by I.

---

Published as a conference paper at ICLR 2021
4.1
QA MODULES
Text QA Module Following prior work (Min al. 2019a Asai et al. 2020), our text QA module takes as input a question Q and a paragraph p E P and answers Q by selecting a span in each paragraph p independently, predicting the start and end positions (Devlin et al. 2019). Additionally, the model returns four scores for for every paragraph corresponding to: if the answer is (i) a span in pt (ii) "yes"; (iii) "no"; or (iv) ) not in p. At inference time, the model selects the paragraph that has the lowest score for iv the answer is not the paragraph. Our model is based on pre-trained RoBERTa-large model (Liu et al. 2019). fine-tuned on MMQA.
Table QA Module Following prior work (Herzig et al. 2020), our table QA module takes as input the question Q and the table T,, and selects a subset of the table cells and an aggregation operation to compute the final answer. Specifically, we linearize the table T by rows, with column names prepended to the corresponding cells (Chen et al., 2019). For example, this converts the table in to the following text: "Row 1: year is 1957; title is a dangerous age; role is David, Row 2...". Next, we concatenate the question to the linearized table, and encode them using RoBERTa-large. We then pass the contextualized representation of every token in the table cell to linear classifier that computes the probability of the token being selected. The score for cell is the average of its tokens. Cells with probability > 0.5 are selected. Finally, another linear multi-class classifier predicts an aggregation operation from SUM, MEAN, COUNT, YES, NO, and NONE. Aggregation operations are applied on the selected cells, YES and NO operations output "yes" " "no". and the NONE operation outputs all selected cells.
Figure D
Image QA Module Questions with visual information are handled by a multimodal transformer that processes the text question and pre-computed image features. For a question Q and a set of images I. we feed the model the question and the visual features (() extracted from each image i E I, along with the name of the WikiEntity associated with the image. For each image and question, the model predicts an answer from a fixed vocabulary determined by the answers in the training set and special tokens: Yäir- Up and Yn- In questions where the expected answer is a phrase (e.g., Image(),, we return the answer from the image where p(yar) is lowest (similar to text QA). In questions where the expected answer is subset of the images (c.g.. where p(zp) > p(y.). Our model is based on the pre-trained model VILBERT-MT (Lu et al. 2020). Visual features are extracted by a vision network Φ comprised of Faster R-CNN (Ren al. 2015
we return all images
pre-trained on Visual Genome (Krishna et al. 2017).
4.2 MULTIMODALITY QA MODELS
We turn to models that interact with multiple modalities.
Multi-Hop Implicit Decomposition (ImpliciDDecomp) Our dataset designed to test reasoning across modalities. As first attempt towards this goal, we introduce a 2-hop implicit decomposition baseline, capable of combining information scattered across modalities (illustrated in Figure
We first train a question-type classifier, based on RoBERTa-large, that takes a question Q as input, and predicts one of the 16 possible question types (Table 2 The question type can be viewed as a program, specifying the relevant modalities, their order, and the logical operations. For example, if TaapooseeeTeee.aabbbia
the question type is second hop the paragraphs P. In each hop, we feed the model with the question Q, the question type, the hop number, and the context of the corresponding modality. The model automatically identifies which part of the question is relevant at the current hop and does not explicitly decompose the question into sub-questions (hence the name implicit decomposition). In the second hop, answers from the first hop are also given as input so that the model can leverage this information and conduct cross-modal reasoning to output the final answer. For all single- single-modality modality question types (such as TextQ and TableQ), the model uses only the first hop to get the answer.
the first hop should be conducted on the table T, and the
Single-Hop Routing (AutoRouting) A simple approach for answering questions without modal reasoning is to first determine the modality where the answer is expected to occur, and then run the corresponding single-modalit module. We use the aforementioned question type classifier to
'The multi-task version of VILBERT is used, since it was shown in Lu et al. 2020 that fine-tuning task-specific models from the multi-task model is generally beneficial for performance on single tasks.
7

---

Published as a conference paper at ICLR 2021
determine the modality where the answer will appear, route the question and the context for the predicted modality into the corresponding module, and use the output as the final answer.
Question-only and Context-only baselines We run the question-only and context-only basclines, suggested by Kaushik & Lipton| (2018). Our question-only baseline is
(Fiow And]
BART-large (Lewis et al. 2 a sequence-to- -oeennce-t---uuenceeee model that directly generales the answer given the question. For the context-only baseline, we first predict the question type using the classifier described above to pick target mod- ule. We then feed the relevant context to the target module, replacing the question with an empty string.
Text GA
Table QI
Image QA
Hop Iepur
10 Tyd
HDPez
[[f Hop And
[Quest on)
[Context]
Hop Prodiction:
Table
Image
Hop Inpul:
t0 Type
HOP-1
[Question]
[Conted]]
4.3 TRAINING AND SUPERVISION
Modules with the same color share parameters. In this example, the text QA module is activated to produce the sst-hop answer, and this intermediate answer is fed into the Image QA model to produce the final answer. Question type ([Q Type]) is determined by separate classifier.
Figure 4: ImplicisDecomp:
Our dataset provides rich supervision including not only the final answer but also question types and intermediate results. Therefore, train the pipeline modules in supervised fashion. Specifically, we train the question type classifier using cross entropy loss w.r.t the gold question type. For AutoRouting, each QA module is trained with the subset of samples whose final answer corresponding modality. For ImplicttDecomp, only one model trained per modality, which used answer both the first-hop and second-hop questions. The question-only and context-only baselines are trained in the corresponding format.
be extracted from the
5 EXPERIMENTS
We evaluate models in three different setups: (1) questions that require single modality to answer (Single Modality); (2) questions that require reasoning over multiple modalities (Multi Modality); (3) and all questions (All). Our evaluation metrics need to support lists of answers, and thus we use average F, and Exact Match (EM), as described in Dua et al. (2019), where answers on the gold and predicted lists are aligned. Human performance is estimated with 9 expert annotators, who answered 145 questions. Test results are are reported using a single run (one random seed).
We show results in Table Implic- itDecomp achieves significantly higher performance (55.9 F,) compared to the other baselines, but lower than human performance (91.2 F, with provided con- text, and 84.8 F, in the open-domain set- ting over all of Wikipedia), suggesting ample room for improvement. On the Multi Modality subset, ImplicitDecomp
Single Modality
Mutli Modality
All
EM
Fj
EM
Fi
EM
Fj
Question-only"
14.2
17.0
16.9
19.5
15.3
18.0
Context-only
8.0
10.2
6.6
8.5
7.4
9.5
AutoRouting
48.9
57.1
32.0
38.2
42.1
49.5
ImplicitDecomp
51.1
58.8
46.5
51.7
49.3
55.9
Human
87.9
92.5
84.8
90.1
86.2
91.2
Table 4: Test set results
substantially improves performance compared to AutoRouting (38.2 + 51.7), emphasizing the superiority of our approach multi-hop questions, while on single-hop questions this gap is smaller.
Since automatic evaluation of performance is non-trivial in our setup, we also manually evaluate human performance. In 94.5% of the cases, answers either identical or semantically equivalent to the gold answer, 0.7% have an error in the question, and 4.8% are human errors. Human errors are owing the length of the context, resulting in human fatigue (which models do not suffer from).
Analysis To demonstrate that Implicit indeed performs multi-hop reasoning, successfully answering intermediate questions, we analyze ImplicitDecomp predictions for multimodal questions generated using the Compose, Compare and Intersect operations (Table 5 For these questions, we find that when the sst-hop answer is correct, the model achieves an F1 of 63.9, whereas when the hst-hop prediction is incorrect, the Fj drops to 37.4. This suggests that the model relies on the lst-hop
We conjecture that the higher performance exhibited by the question-onl baseline compared to the context- only baseline is due to the fact that in comparison questions the model needs to choose one answer from two candidates, of which usually appears in the question, allowing the model obtain 50% accuracy by guessing.
8

---

Published as a conference paper at ICLR 2021
Type
Question
Ist-hop prediction
Final prediction
1st-hop F;
F;
Compose
What part did Rym Karath play the TV show
Katby Vaughn
62.3
50.8
whose poster features a dog?
Compure
Which video game
Wes Johason involved corber:
Hammer Stekle
Hammer Sickle
55.7
61.1
Fallout or the gume whose cover shows gun gun-wieldin man? Which album, released December 2011, I. has wearing
Intersect
TYO,
Back to Love
33.5
55.1
sunglasses on its cover, and was released under the RCA label?
Back Love
Table 5: Examples where Decomn correctly answers both intermediate and entire question, and a breakdown of the 1st-hop F and final F1 for the three logical operations: Compose, Compare, and Intersect.
answer, effectively performing multi-ho reasoning. Last, question type classifier obtains a high accuracy of 91.5% on the test set.
To test whether the compositional questions created are indeed multi-hop, conducted qualitative analysis over 50 questions as suggested by (Min et al., 2019b). We find that 6% are of the Weak Distractors category, that is, questions such "Whar year... when there is only one year appearing in the context, making the question easy. 2% have Redundant evidence, that is, questions such as "Which Donald Trump TV show has ... where there only one TV show starring Donald Trump, making the rest of the question redundant. The remaining 92% indeed require multi-hop reasoning.
RELATED WORK
Visual question answering- i... the task of answering questions about images has been widely
explored in previous work (Antol al. 2015 Zhang et al. 2016; Goyal et al. 2017 Johnson et al.) 2017, Hudson & Mamning, 2019: et al. 2019; Singh et al. 2019 Methani et al. 2020). ranging from synthetic images to scientific plots. Our work differs significantly from those, by including more complex, multi-hop questions that require reasoning over text, tables and images. Currently, the most successful paradigm in VQA is fine-tuning models pre-trained large amounts of image captioning data (Tan & Bansal 2019 Lu et al.) 2019 2020) Su et al. (000) CCen et al., 2020cc Li
al. 2020), an approach we follow for answering image-related questions.
MANYMODALQA (Hannan et al. 2020 move beyond directing the question to an image-only context, choosing between an image, text, and a table. Their work focuses on routing the question to the correct context modality. Our question-type type classifier, based on RoBERTa-late reaches an accuracy of 91.4% on our 16 possible question types, showing that the main challenge MMQA is reasoning over the context rather than identifying the question type.
HYBRIDQA (Chen et al. 2020b) presents a cross-modality reasoning challenge over tabular and textual data. A fundamental difference is that our setup offers cross- modslttyyl reasoning over images as well. addition, our approach cheaper to annotate since requires only paraphrasing, and the question type distribution is more controllable (we offer 16 major question types vs. 6 in HYBRIDQA), Moreover, our text passages are chosen using the question, answer and table, while in HYBRIDQA only WikiEntities from the table are used to find text passages.
The model proposed in HYBRIDQA introduces a heuristic for linking the text passage to the table cells, which may lead to performance degradation. Conversely, model uses (automatically- annotated) intermediate multi-hop answers, to perform reasoning and linking implicitly over the full table and text, which should lead to more robust reasoning, in partieular when reasoning over multiple table cells, as well as for narrative tracking and co-reference over the full text. In parallel to this work, new open-domain variant of HYBRIDQA has been released by Chen al. (020a).
7 CONCLUSION
We present MMQA, new QA dataset that contains 29,918 examples, 35.7% of which require cross- modality reasoning. We describe a novel framework for generating complex multimodal questions at scale, and showcase the diversity and multimodal properties of the resulting dataset. We evaluate MMQA using a variety of models, and confirm that the best model exploits the multimodality of the dataset and takes into account multi-hop reasoning via implicit decomposition. However, human performance substantially exceeds the best model, establishing the need for further research involving multiple modalities in question answering systems, which we hope that our work will drive.

---

Published as a conference paper at ICLR 2021
8 ACKNOWLEDGMENTS
We thank our colleagues at The Allen Institute of AI, James Ferguson and Amir Globerson. This research was partially supported by The Blavatnik Computer Science Research Fund and The Yandex Initiative for Machine Learning, and the European Union's Seventh Framework Programme (FP7) under grant agreement no, 102800-DELPH Special thanks Carrot Search which allowed use to use their Foam visualization.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. ArXiv, abs//991.1000 2020.
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint PeXiiv:9999..m 644 2019.
Wenhu Chen, Ming-Wai Wei Chang, Eva Schlinger, William Wang, and William W Cohen. Open question answering over tables and text. arXiv preprint arXiv:2010. 10439, 2020a.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong. Hong Wang, and William Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data, 2020b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. In Proceedings of the 2020 European Conference on Computer Vision 2020c. URL http::tt/:///// /ttp://wwwwww.... httpw/wwccaa..vmmm .tt.:eeeaaeettpppssss//////
@ecvv Srtov_2020/paae Sttpgeers_EcC///ppeeeeee /tCC///paprrs/2233000..3 .3.pff .pdf
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1955.00444 2019.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transform- for language understanding. In North American Association for Computational Linguistics (NAACL), 2019.
D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL), 2019.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint 1oXiv::908.07898, 2019.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Darryl Hamnan, Akshay Jain, and Mohit Bansal. Manymodalqa: Modality disambiguation and qa diverse inputs, 2020.
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In ACI., 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
10

---

Published as a conference paper at ICLR 2021
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR), 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004. 04906, 2020.
Divyansh Kaushik and Zachary C Lipton, How much reading does reading comprehension require? a critical investigation of popular benchmarks. In EMNLP, 2018.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising requence-to-sequaeee pre-training for natural language generation, translation, and comprehension, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostic visiolinguistic representations for vision-and-language tasks, In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. Compositional questions do necessitate multi-hop reasoning. In ACL, 2019a.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. Compositional questions do not necessitate multi-hop reasoning. arXiv preprint 1eX.vv1906.02900 2019b.
Pasupat and P. Liang. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics 2015.
2015.
P. Rajpurkar. SQuAD. https: http:///ppuura..iiituu...u...b/a http://wwww...........oo///////a explooee/e 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91- 2015.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8317-8326, 2019.

---

Published as a conference paper at ICLR 2021
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre- Pre-training of generie lisual-lngguisic representations. In Proceedings of the 2020 International Conference on Learning Representations, 2020. URL /LLtttpss///aaa....r////////// 08530
A. Talmor and J. Berant. The web knowledge-base for answering complex questions. North American Association for Computational Linguistics (NAACL), 2018.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 5100-5111, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: D19-1514
URL https: www awwweb.wwb....... org/ anthology.
J. Welbl, P. Stenetorp, and S. Riedel. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprini
2017.
Yang. P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6720-6731, 2019.
Peng Zhang, Yash Goyal, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
A
APPENDIX
Please see separate file for supplementary material.
12


Reviews:
[{"review_id": "PoeQEt0QltS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a question answering dataset that needs up-to reasoning over three distinct modalities — text, wikipedia tables, and images of wikipedia entities. The dataset generation step consists of starting with a wiki table, finding entities in the table, followed by finding images associated with the entities and text from existing reading comprehension datasets such as Natural Questions, HotpotQA etc. Next, they generate single-modality question that can be answered from each mode. Additionally the paper also introduces a grammar for generating, compositional questions from the single-mode questions that needs reasoning across modalities, which I believe would be widely useful for creating future datasets. The grammar lets them scale fairly easily. Lastly, the questions generated from the grammar are then paraphrased by annotators. Incentives were given to workers to produce diverse questions (such as bonus if the second-paraphrase of a question was not answered by a baseline model). The dataset comes in both an open-setting as well as closed setting, in which distractors are chosen carefully (e.g. distractor images for other entities in the table or using a state-of-the-art retriever to get paragraphs).\n\nThe dataset is tested on a model that uses a pretrained model for each modality. A classifier is used to predict the type of question (which is easy to do) and then each model is applied following the grammar. Following cautionary related work which show that models often take advantage of unwanted bias in the data, they also have a context-only and question-only baseline. They also did a manual analysis revealing that 86% of the questions actually need strong multi-hop reasoning. There is significant gap as expected between human performance and the correct model\n\nStrengths\n\n1. I think this dataset would be a useful test-bed for multi-modal models and several models will be build around it.\n2. The grammar used for generating compositional question will also be helpful for building future datasets, so I think that is also an important contribution \n3. It looks like the authors have taken sufficient caution to weed out unwanted biases from the dataset\n4. The paper is very clearly written. The analysis were also helpful.\n\nWeakness:\n1. As rightfully acknowledged in the paper, the distribution of questions is quite different from the question human (currently) ask to a system. However, ability to reason over multiple modalities is important and this dataset is important wrt that goal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AGk16XSE6Q7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds a new large-scale QA dataset for multiple modality reasoning including table, text and images. \n\nThere are several related works: 1) HYBRIDQA, a dataset that requires reasoning over tabular and textual data. The difference between the proposed dataset and this one is that HYBRIDQA does not require visual inference. 2) MANYMODALQA: a dataset where the context for each question includes information from multiple modalities. The difference between the proposed dataset and this one is that\nThe answer to each question in MANYMODALQA can be derived from a single modality. Thus, the main difference from others is that the proposed method adds the table information into the framework while the necessary of table for QA task is not clear in this paper. I think almost all our cases of QA have not have the table in our daily life.\n\nThe methodology for creating MMQA includes three steps: Context construction, Question generation and Paraphrasing. \n\nThis paper also introduces one model to deal with the problem of multiple modality reasoning defined by this new dataset. It seems that these models are not new, which are deeply related to several prior works.\n\nOverall, I think this dataset is new but the necessary of this dataset to QA community is not clear. Moreover, since I am not expert in this area, I am not sure the contribution of this dataset meets the standard of ICLR.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "92WxaJGCyid", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors present a new dataset, MultiModalQA, with the intent of measuring a model’s ability to reason across different modalities (free text, structured tables, and images) in question answering, and in which a large percentage of the questions requires cross-modal reasoning. The authors provide a detailed look at the framework used to generate questions in the context of several modalities. Finally, the authors propose a multimodal model that performs multi-hop reasoning (removing the need for explicit question decomposition), which outperforms strong baselines, but is still far behind human performance, indicating that the task is nontrivial and would benefit the research community.\n\n### Positives\n- The dataset is interesting and comprehensive, covering multiple modalities with a significant portion covering compositional questions as well. \n- The authors describe in detail their method of dataset construction, leveraging existing datasets in the literature and providing insight for others should they so desire to construct their own multimodal qa dataset. \n- In the process, the authors consider and justify shortcomings of the dataset (e.g., that the automatic generation of examples may be seen as limiting the question distribution to a non “natural” source)\n- The authors consider a baseline multi-hop reasoning model that highlights the need for cross-modal reasoning to achieve good performance on the dataset.\n\n### Negatives\n- The paper could benefit from further comparison and exploration of how the methods and data relates to others in the literature.\n- I would have liked to have seen a little bit more comparison between this dataset and others in terms of raw statistics - the authors mention HybridQA and ManyModalQA in the introduction, so comparing to those would be a start.\n- Additionally, the authors could have provided some analysis of their proposed model’s effectiveness on other datasets - does the model still work well for single-modality QA datasets when compared to other pre-existing models? Does the “question classifier” generalize to questions not within the 16-construction PL framework?\n\n### Decision\nI think this paper is marginally above the acceptance threshold. The dataset is unique and well-constructed, and the multimodal/QA community would benefit from its use. The authors consider solid baseline models for the task, though the paper would have benefitted from having more exploration of how well these models generalize to other tasks, and additionally how this task compares to other similar ones in the domain.\n\n### Questions\nPlease address negatives; in addition:\n\n1. There are multiple image-based QA datasets out there (e.g. VQA) with several image-based questions already - did you consider automatic matching of images from WikiEntities to these datasets? As it would eliminate the need for crowdsourcing the image single-modality questions.\n2. The authors mentioned a feedback mechanism for AMT workers such that they were given bonuses if a baseline model answered the question correctly before & after rephrasing; at first glance this seems to have introduced some bias in that it makes the questions easier for models to answer. Was this considered when adding this feedback mechanism?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "53uRQcuy5bT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces MultiModalQA, a dataset that requires joint reasoning over table, text and images. The dataset has been created in a semi-automatic way through Wikipedia tables, the Wikientities in them, their related images and related textual question answer pairs from known text QA datasets. For the collection of the dataset, authors collect single modality questions and then use a programmatic way to generate the multimodality versions. The paper also introduces a multi-hop baseline that guesses the question type and then does two hops over the different single modality modules to generate the final answer. \n\nThe dataset would be a useful resource for multimodal QA advancement but the manuscript in the current form disregards all of the prior work that has happened in the space including the work on VQA [1], TextVQA [2] and PlotQA[3]. There has been quite a lot of work on multimodality QA and the paper should address those and take learnings from those papers. \nSpecifically, the model introduced in the paper is too crude and specific to the dataset generation scheme (like depending on the question type) which limits the applicability of the dataset as well as the further approaches that will be developed on this dataset. For example, M4C [4] a model on TextVQA uses transformers by projecting different modalities into the same space and concatenating them as a single sequence. Comparison against similar approaches on MultiModalQA would be more useful compared to hardcoded approaches like ImplicitDecomp. This approach reminds me of the approaches that were developed on CLEVR using program synthesis and weren’t useful in the long run. Similar to TableQA modules, tables + text + images can be input to the M4C transformer as a single sequence and the classification accuracy can be calculated. \n\nSome other questions to the authors:\n- Is there any specific reason to use ViLBERT-MT instead of ViLBERT as ViLBERT finetuned on your task would be more powerful - compared to using ViLBERT-MT.\n\n- How many runs were done to compute the metrics in Table 3.\n\n- Have you done other ablations on 2-hop setting by using the TextQA module twice, TableQA module twice etc. It would allow us to understand better which module matters most.\n\nOverall, this is an interesting and useful dataset. I would recommend acceptance if the concerns and writing about previous works on multimodality are fixed and my other questions and concerns are considered.\n\n\n[1] Goyal, Yash, et al. \"Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n\n[2] Singh, Amanpreet, et al. \"Towards vqa models that can read.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. \n\n[3] Methani, Nitesh, et al. \"PlotQA: Reasoning over Scientific Plots.\" The IEEE Winter Conference on Applications of Computer Vision. 2020.\n\n[4] Hu, Ronghang, et al. \"Iterative answer prediction with pointer-augmented multimodal transformers for textvqa.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020\n\n\nEdit after rebuttal: I have read the author response and thank the authors for their comments and answers to my questions. I would like to keep my rating as it is. I recommend authors to tone down the claims around being first MultimodalQA dataset and position themself properly with respect to previous related work if accepted.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new dataset for multimodal QA that is deemed interesting, relevant and well executed by all reviewers. Multimodality in NLP (QA included) is an increasingly important topic and this paper provides a potentially impactful benchmark for research in it. All reviewers acknowledge that.

We hence recommend to accept this paper as a poster. We recommend the authors to further improve the draft before camera ready by using the recommendations made by the reviewers with a particular focus on an extended discussion wrt prior work on VQA and other. The paper should also add more precisions on the license(s) related to the images used in the dataset. 
Decision: Accept (Poster)