Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper introduces a new visual reasoning dataset called Figure-QA which consists of 140K figure images and 1.55M QA pairs. The images are generated synthetically by plotting perturbed sampled data using a visualization tool. The questions are also generated synthetically using 15 templates. Performance of baseline models and humans show that it is a challenging task and more advanced models are required to solve this task.

Strengths:
— FigureQA can help in developing models that can extract useful information from visual representations of data.
— Since performance on CLEVR dataset is already close to 100%, more challenging visual reasoning datasets would encourage the community to develop more advanced reasoning models. One of such datasets can be FigureQA.
— The paper is well written and easy to follow.


Weaknesses:
— Since the dataset is created synthetically, it is not clear if it is actually visual reasoning which is needed to solve this task, or the models can exploit biases (not necessarily language biases) to perform well on this dataset. In short, how do we know if the models trained on this dataset are actually learning something useful? One way to ensure this would be to show that models trained on this dataset can perform well on some other task. The first thing to try to show the usefulness of FigureQA is to show that the models trained on FigureQA dataset perform well on a real (figure, QA) dataset.
— The only advantages mentioned in the paper of using a synthetic dataset for this task are having greater control over task’s complexity and enabling auxiliary supervision signals, but none of them are shown in this paper, so it’s not clear if they are needed or useful.
— The paper should discuss what type of abilities are required in the models to perform well on this task, and how these abilities are currently not studied in the research community. Or in short, what new challenges are being introduced by FigureQA and how should researchers go about solving them on a high level?
— With what goal were these 15 types of questions chosen? Are these the most useful questions analysts want to extract out of plots? I am especially concerned about finding the roughest/smoothest and low/high median. Even humans are relatively bad at these tasks. Why do we expect models to do well on them?
— Why only binary questions? It is probably more difficult for analysts to ask a binary question than to ask non-binary ones such as “What is the highest in this plot?” — Why these 5 types of plots? Can the authors justify that these 5 types of plots are the most frequent ones dealt by analysts?
— Are the model accuracies in Table 3 on the same subset as humans or on the complete test set? Can the authors please report both separately?


Overall: 
The proposed dataset seems reasonable but neither the dataset seems properly motivated (something where analysts actually struggle and models can help) nor it is clear if it will actually be useful for the research community (models performing well on this dataset will need to focus on specific abilities which have not been studied in the research community).
review_title: Poorly motivated and needs more analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper introduces a new dataset (FigureQA) of question answering on figures (line plots, bar graphs, pie charts). The questions involve reasoning about figure elements (e.g., “Is X minimum?”, “Does X have maximum area under curve?”, where X refers to an element in a figure). The images are synthetic and the questions are templated. The dataset consists of over 100,000 images and the questions are from 15 templates. The authors also provide the numerical data associated with each figure and bounding box annotations for all plot elements. The paper trains and evaluates three baselines – question only LSTM, CNN + LSTM and Relation Networks on the proposed FigureQA dataset. The experimental results show that Relation Networks outperform the other two baselines, but still ~30% behind human performance.

Strengths:
1.	The proposed task is a useful task for building intelligent AI agents.
2.	The writing of the paper is clear with enough details about the data generation process.
3.	The idea of using different compositions of color and figure during train and test is interesting.
4.	The baselines experimented with in the paper make sense.

Weaknesses:
1.	The motivation behind the proposed task needs to be better elaborated. As of now, the paper mentions in one line that automatic understanding of figures could help human analysists. But, it would be good if this can be supported with real life examples.
2.	The dataset proposed in the paper comes with bounding box annotations, however, the usage of bounding boxes isn’t clear. The paper briefly mentions supervising attention models using such boxes, but it isn’t clear how bounding boxes for data points could be used.
3.	It would have been good if the paper had the experiments on reconstructing quantitave data from plots and using bounding boxes for providing attention supervision, in order to concretize the usage of these annotations?
4.	The paper mentions the that analyzing the performance of the models trained on FigureQA on real datasets would help extend the FigureQA corupus. So, why didn’t authors try the baselines models on FigureSeer dataset?
5.	It is not clear why did the authors devise a new metric for smoothness and not use existing metrics?
6.	The paper should clarify which CNN is used for CNN + LSTM and Relation Networks models?
7.	The paper should clarify the loss function used to train the models? Is it binary cross entropy loss?
8.	The paper does not mention the accuracy using the quantitative data associated with the plot? Is it 100%? What if two quantities being questions about are equal and the question is about finding the max/min? How much is the error due to such situations?
review_title: Task motivation, experiments on FigureSeer, experiments with quantitative data and bounding box annotations.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new dataset called FigureQA. The images are synthetic scientific style figures and questions are generated from 15 templates which concern various relationships between plot elements. The authors experiment with the proposed dataset with 3 baselines. Text-only baseline, which only considers the questions; CNN+LSTM baseline, which does a late fusion between image and question representation; Relation Network, which follows Santoro et al. (2017). Experiment results show that the proposed task poses a difficult challenge where CNN+LSTM baseline only 2% better than guessing (50%) and relation network which takes spatial reasoning into consideration, performs better on this task (61.54%). 

[Strenghts]

The proposed Figure QA dataset is a first step towards developing models that can recognize the visual representation of data. This is definitely a novel area that requires the machine not only understand the corpus, but also the scientific figure associated with the figure. Traditional VQA methods not working well on the proposed dataset, only 2% better than guessing (50%). The proposed dataset requires the machine understand the spatial arrangement of the object and reason the relations between them.  

[Weaknesses]

1: There are no novel algorithms associated with this dataset. CVPR seems a better place to publish this paper, but I'm open to ICLR accept dataset paper.

2: The generated templated questions of proposed FigureQA dataset is very constraint.  All the question is the binary question, and there is no variation of the template with respect to the same question type. Most of the question type can be represented as a triplet. In this sense, the proposed dataset requires less language understanding compare to previous synthesis dataset such as CLEVER. 

3: Since the generated question is very templated and less variational, a traditional hand-crafted approach may perform much better compared to the end-to-end approach. The paper didn't have any baseline for the hand-crafted approach, thus we don't know how it performs on the proposed dataset, and whether more advanced neural approaches are needed for this dataset. 

[Summary]

This paper introduces a new dataset called FigureQA, which answering the synthetic scientific style figures given the 15 type of questions. The authors experiment the proposed dataset with 3 neural baselines: Question only, LSTM + CNN and relational network. The proposed dataset is the first step towards developing models that can recognize the visual representation of data. However, my major concern about this dataset is the synthesized question is very templated and less variational. From the example, it seems it does not require natural language understanding of the dataset. Triplet representation seems enough to represent the most type of the question. The authors also didn't conduct hand-crafted baselines, which can provide a better intuition about the difficulty lies in the proposed dataset. Taking all these into account, I suggest accepting this paper if the authors could provide more justification on the question side of the proposed dataset. 
review_title: Review from AnonReviewer2

Meta-review: This paper was reviewed by 3 expert reviews. While they all see value in the new task and dataset, they raise concerns (templated language, unclear what exactly are the new challenges posed by this task and dataset, etc) that this AC agrees with. To be clear, the lack of a fundamentally new model is not a problem (or a requirement for every paper introducing a new task/dataset), but make a clear compelling case for why people should work on the task is a reasonable bar. We encourage the authors to incorporate reviewer feedback and invite to the workshop track. 
Decision: Invite to Workshop Track