Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The authors present a new dataset, MultiModalQA, with the intent of measuring a model’s ability to reason across different modalities (free text, structured tables, and images) in question answering, and in which a large percentage of the questions requires cross-modal reasoning. The authors provide a detailed look at the framework used to generate questions in the context of several modalities. Finally, the authors propose a multimodal model that performs multi-hop reasoning (removing the need for explicit question decomposition), which outperforms strong baselines, but is still far behind human performance, indicating that the task is nontrivial and would benefit the research community.

### Positives
- The dataset is interesting and comprehensive, covering multiple modalities with a significant portion covering compositional questions as well. 
- The authors describe in detail their method of dataset construction, leveraging existing datasets in the literature and providing insight for others should they so desire to construct their own multimodal qa dataset. 
- In the process, the authors consider and justify shortcomings of the dataset (e.g., that the automatic generation of examples may be seen as limiting the question distribution to a non “natural” source)
- The authors consider a baseline multi-hop reasoning model that highlights the need for cross-modal reasoning to achieve good performance on the dataset.

### Negatives
- The paper could benefit from further comparison and exploration of how the methods and data relates to others in the literature.
- I would have liked to have seen a little bit more comparison between this dataset and others in terms of raw statistics - the authors mention HybridQA and ManyModalQA in the introduction, so comparing to those would be a start.
- Additionally, the authors could have provided some analysis of their proposed model’s effectiveness on other datasets - does the model still work well for single-modality QA datasets when compared to other pre-existing models? Does the “question classifier” generalize to questions not within the 16-construction PL framework?

### Decision
I think this paper is marginally above the acceptance threshold. The dataset is unique and well-constructed, and the multimodal/QA community would benefit from its use. The authors consider solid baseline models for the task, though the paper would have benefitted from having more exploration of how well these models generalize to other tasks, and additionally how this task compares to other similar ones in the domain.

### Questions
Please address negatives; in addition:

1. There are multiple image-based QA datasets out there (e.g. VQA) with several image-based questions already - did you consider automatic matching of images from WikiEntities to these datasets? As it would eliminate the need for crowdsourcing the image single-modality questions.
2. The authors mentioned a feedback mechanism for AMT workers such that they were given bonuses if a baseline model answered the question correctly before & after rephrasing; at first glance this seems to have introduced some bias in that it makes the questions easier for models to answer. Was this considered when adding this feedback mechanism?

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces MultiModalQA, a dataset that requires joint reasoning over table, text and images. The dataset has been created in a semi-automatic way through Wikipedia tables, the Wikientities in them, their related images and related textual question answer pairs from known text QA datasets. For the collection of the dataset, authors collect single modality questions and then use a programmatic way to generate the multimodality versions. The paper also introduces a multi-hop baseline that guesses the question type and then does two hops over the different single modality modules to generate the final answer. 

The dataset would be a useful resource for multimodal QA advancement but the manuscript in the current form disregards all of the prior work that has happened in the space including the work on VQA [1], TextVQA [2] and PlotQA[3]. There has been quite a lot of work on multimodality QA and the paper should address those and take learnings from those papers. 
Specifically, the model introduced in the paper is too crude and specific to the dataset generation scheme (like depending on the question type) which limits the applicability of the dataset as well as the further approaches that will be developed on this dataset. For example, M4C [4] a model on TextVQA uses transformers by projecting different modalities into the same space and concatenating them as a single sequence. Comparison against similar approaches on MultiModalQA would be more useful compared to hardcoded approaches like ImplicitDecomp. This approach reminds me of the approaches that were developed on CLEVR using program synthesis and weren’t useful in the long run. Similar to TableQA modules, tables + text + images can be input to the M4C transformer as a single sequence and the classification accuracy can be calculated. 

Some other questions to the authors:
- Is there any specific reason to use ViLBERT-MT instead of ViLBERT as ViLBERT finetuned on your task would be more powerful - compared to using ViLBERT-MT.

- How many runs were done to compute the metrics in Table 3.

- Have you done other ablations on 2-hop setting by using the TextQA module twice, TableQA module twice etc. It would allow us to understand better which module matters most.

Overall, this is an interesting and useful dataset. I would recommend acceptance if the concerns and writing about previous works on multimodality are fixed and my other questions and concerns are considered.


[1] Goyal, Yash, et al. "Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[2] Singh, Amanpreet, et al. "Towards vqa models that can read." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. 

[3] Methani, Nitesh, et al. "PlotQA: Reasoning over Scientific Plots." The IEEE Winter Conference on Applications of Computer Vision. 2020.

[4] Hu, Ronghang, et al. "Iterative answer prediction with pointer-augmented multimodal transformers for textvqa." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020


Edit after rebuttal: I have read the author response and thank the authors for their comments and answers to my questions. I would like to keep my rating as it is. I recommend authors to tone down the claims around being first MultimodalQA dataset and position themself properly with respect to previous related work if accepted.

review_title: A new dataset, some concerns about previous related work and model

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a question answering dataset that needs up-to reasoning over three distinct modalities — text, wikipedia tables, and images of wikipedia entities. The dataset generation step consists of starting with a wiki table, finding entities in the table, followed by finding images associated with the entities and text from existing reading comprehension datasets such as Natural Questions, HotpotQA etc. Next, they generate single-modality question that can be answered from each mode. Additionally the paper also introduces a grammar for generating, compositional questions from the single-mode questions that needs reasoning across modalities, which I believe would be widely useful for creating future datasets. The grammar lets them scale fairly easily. Lastly, the questions generated from the grammar are then paraphrased by annotators. Incentives were given to workers to produce diverse questions (such as bonus if the second-paraphrase of a question was not answered by a baseline model). The dataset comes in both an open-setting as well as closed setting, in which distractors are chosen carefully (e.g. distractor images for other entities in the table or using a state-of-the-art retriever to get paragraphs).

The dataset is tested on a model that uses a pretrained model for each modality. A classifier is used to predict the type of question (which is easy to do) and then each model is applied following the grammar. Following cautionary related work which show that models often take advantage of unwanted bias in the data, they also have a context-only and question-only baseline. They also did a manual analysis revealing that 86% of the questions actually need strong multi-hop reasoning. There is significant gap as expected between human performance and the correct model

Strengths

1. I think this dataset would be a useful test-bed for multi-modal models and several models will be build around it.
2. The grammar used for generating compositional question will also be helpful for building future datasets, so I think that is also an important contribution 
3. It looks like the authors have taken sufficient caution to weed out unwanted biases from the dataset
4. The paper is very clearly written. The analysis were also helpful.

Weakness:
1. As rightfully acknowledged in the paper, the distribution of questions is quite different from the question human (currently) ask to a system. However, ability to reason over multiple modalities is important and this dataset is important wrt that goal.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper builds a new large-scale QA dataset for multiple modality reasoning including table, text and images. 

There are several related works: 1) HYBRIDQA, a dataset that requires reasoning over tabular and textual data. The difference between the proposed dataset and this one is that HYBRIDQA does not require visual inference. 2) MANYMODALQA: a dataset where the context for each question includes information from multiple modalities. The difference between the proposed dataset and this one is that
The answer to each question in MANYMODALQA can be derived from a single modality. Thus, the main difference from others is that the proposed method adds the table information into the framework while the necessary of table for QA task is not clear in this paper. I think almost all our cases of QA have not have the table in our daily life.

The methodology for creating MMQA includes three steps: Context construction, Question generation and Paraphrasing. 

This paper also introduces one model to deal with the problem of multiple modality reasoning defined by this new dataset. It seems that these models are not new, which are deeply related to several prior works.

Overall, I think this dataset is new but the necessary of this dataset to QA community is not clear. Moreover, since I am not expert in this area, I am not sure the contribution of this dataset meets the standard of ICLR.

review_title: I think this dataset is new but the necessary of this dataset to QA community is not clear

Meta-review: The paper presents a new dataset for multimodal QA that is deemed interesting, relevant and well executed by all reviewers. Multimodality in NLP (QA included) is an increasingly important topic and this paper provides a potentially impactful benchmark for research in it. All reviewers acknowledge that.

We hence recommend to accept this paper as a poster. We recommend the authors to further improve the draft before camera ready by using the recommendations made by the reviewers with a particular focus on an extended discussion wrt prior work on VQA and other. The paper should also add more precisions on the license(s) related to the images used in the dataset. 
Decision: Accept (Poster)