Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: *  IconQA - a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in the-blank.
* an icon dataset Icon645 which contains 645,687 colored icons on 377 classes.
* user studies, blind experiments and a wide range of advanced VQA methods to benchmark the IconQA task
* a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset.
strengths: * icon question answering, introduced in this paper, is a new attempt in the filed of visual question answering and brings out new challenges. 
* To facilitate algorithms to learn semantic representations for icon images, the authors also collected an icon dataset Icon645.
* Existing advanced VQA methods have been tested on the new IconQA task.
* A strong baseline, Patch-TRM, is also proposed, and a well-conducted ablation study has shown the critical design of the baseline.
weaknesses: I haven't identified any weakness.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Extensive work! Just some formatting feedbacks:
 * Figure 5: if possible, it would be better to make the text and label larger.
* Line #103: there should be more space between the table and the title of section 3.1. Add some space or move the table to the top/bottom of this page.
review_title: Nice work about VQA in an icon image context

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies a new VQA task called Icon Question Answering (IconQA), which considers the VQA of abstract diagrams (i.e., icon images) instead of natural images. To this end, the authors construct an IconQA dataset and an Icon645 dataset. The former focuses on the whole IconQA task while the latter is used to learn the semantic representations of icons during the pre-training phase. Besides, unlike the existing Transformer-based VQA methods, the authors propose a stronger IconQA baseline namely Patch-TRM.
strengths: 1. Rather than the conventional VQA task, the authors consider a new task called Icon Question Answering (IconQA), which requires the abilities of abstract diagram understanding.
2. For IconQA, the authors collect and construct two large-scale datasets: IconQA and Icon645, which focus on question answering and model pre-training separately.
3. To achieve better performance on the IconQA dataset, based on the structure of Transformer, the authors propose a strong baseline method called Patch-TRM.

weaknesses: 1. The authors claim that “each question can be related to up to three skills out of these 13 categories”. However, it is unclear how to determine which question belonging to which categories.
2. In Section 6.4, I am curious about how to divide the Icon645 dataset into three parts (i.e., Head, Medium, and Tail). Randomly? Or with some certain rules?
3. There are some minor issues, e.g., in Figure 2 (a), it seems that the distribution of multi-text-choice is long-tail as well, but the corresponding descriptions are not.

correctness: Most of the claims in this paper are correct. The processing of dataset construction seems reasonable and technical sound. Specifically, to collect and construct the dataset, the authors search for open-source math textbooks with rich icon images and diverse topics, and then divide the collected Q&A into three sub-domains: multi-image-choice, multi-text-choice, and filling-in-the-blank. To balance the complexity of the collected QA, they only consider the problem covering from pre-K to third grade.
clarity: The paper is well written and organised, especially for the descriptions of the data collection part, which makes it easy to follow.
relation to prior work: This paper well discusses the differences between the proposed IconQA dataset and the existing VQA datasets (e.g., VQA [3], CLEVR [21], and VQA-Abstract [3]) in Table 1.
additional feedback: N/A
review_title: This paper proposes an Icon Question Answering (IconQA) task, seeking to answer a question based on icon image context, and construct two datasets: IconQA and Icon645.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces new datasets and models for answering questions over images containing diagrams. Specifically, the paper introduces two new datasets, IconQA and Icon645. IconQA is a visual question answering dataset with three different types of questions; (i) multiple image choice, where models are tasked with choosing which image patch is the answer, among of a set of candidates; (ii) multiple text choice, where models are tasked with choosing which text option is the answer, among a set of candidates; and (iii) filling in the blank, where no options are provided and models must provide a brief, text answer. IconQA is collected based on a compilation of open-source math textbooks, and contains over 100 thousand samples. Additionally, Icon645 is a dataset containing over 0.6 million labeled icons spanning 377 classes such as bed, cake or bucket. Icon645 can be used for training or adapting visual encoders to handle icons. Finally, this work explores the performance of several baselines on their dataset, and introduces a new model, Patch-TRM, that outperforms strong baselines. Both datasets and models are well benchmarked through a number of analyses and experiments. The dataset is made public. Overall, I believe this work would be of interest to many, and I recommend it's acceptance.
strengths: This work has many strengths. Firstly, there is good motivation to introduce this dataset, as question answering over diagrams is still an understudied problem. The authors present extensive analyses of their data, for instance, by labeling each sample with the skills necessary to successfully answer it (e.g. counting, performing algebraic operations, identifying shapes, etc.). Both of the large-scale datasets they introduce are valuable assets for researchers interested in building models able to reason over diagrams. Moreover, their baselines and experiments are solid, and provide a great starting point for the community. The authors present human evaluations on their IconQA dataset, showing significant room for improvement. Finally, the paper is clear and well written.
weaknesses: An important point of comparison---how well models perform when only part of the input is provided (e.g. only the question or only the image)---might be understudied in its current form. In the experiments, the authors only provide numbers in this setting for one of the weakest baselines, Top-Down. To better assess how much performance can be achieved in this setting (an important experiment to understand systematic biases in the dataset), it would be desirable for authors to present these numbers using stronger models, including their Path-TRM.

Moreover, in many points of the paper, authors write unsubstantiated sentences stating that correctly answering questions from their dataset "requires" a "diverse set of human intelligence skills". I'd strongly recommend authors to tone down claims like this unless they can provide more evidence of their veracity.
correctness: Cum grano salis (see wekanesses above), most of the claims in the submission are correct, to the best of my knowledge.
clarity: This paper is very clear and well written.
relation to prior work: To the best of my knowledge, this paper appropriately discusses previous work. 
additional feedback: 1. Fig 2 (a) is missing an y-axis label.

2. It would be great to see how a random baseline performs in Table 6.

3. In Table 6, are the "human" numbers an average of all groups? It would be great if the numerical version of Fig 8 was also provided.
review_title: Official Review

Meta-review: The paper studies VQA over abstract diagrams, and introduces new datasets about abstract icons and question answering over diagrams of icons. All reviewers found the task novel and interesting, felt that the datasets were likely to be useful to the community, and thought that the paper was well-written. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets & Benchmarks Track!
Decision: Accept