Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  - The paper presents a Science Question Answering (SQA) dataset, which contains evidence and explanations in multimodalities and covers a diverse category of questions. 
- Authors evaluate standard baselines as well as chain of thoughts (COT) baselines on the established dataset. Empirical result shows that COT can improve the few-shot performance. Due to the introduction of lectures and explanations annotation in SQA, it is also possible to evaluate COT under the supervised finetuning setting. And it could be shown that COT also helps.
- Human performance is measured as the upper bound of the task.
strengths_and_weaknesses: Strengths:

- The paper introduces a novel dataset that fills in the gap of the existing datasets as shown in table 1. Lectures and explanations annotation are introduced to facilitate future research on COT.
- Empirical of SOTA models with and without COT is evaluated to set baseline performance of this dataset
- The paper is clearly written and easy to follow

Weakness:

- See the question section 
questions: - What is the motivation to concatenate evidence and explanation after the answer instead of before it?
- What is the insight that few-shot of GPT-3 is better than finetuned UnifiedQA? It is surprising that a few-shot model outperforms a supervised one, so more analysis is expected.
- For the questions without any context, how were they handled in the evaluation?
limitations: The authors adequately addressed the limitations and potential negative societal impact
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides a novel dataset for science question answering, SQA. Specifically, this paper studies a dataset for evaluating the reasoning/explanation performance of QA models. The authors claimed that existing science question answering datasets either 1) lack annotations for explanations for the answers or 2) has a small data scale/set of topics. SQA consists of 21K examples, a comparably large-scale dataset compared to existing science QA datasets. SQA covers more topics than existing datasets, and SQA is a multi-modal dataset; this dataset provides image context as well as textual context. This paper shows the validity of SQA by showing that the answer prediction performance increases when QA models are fine-tuned on SQA. Also, this paper conducts explanation prediction experiments. In this experiment, QA models predict the annotated explanations/lectures, and humans evaluate the predictions. The experimental results indicate that training QA models on SQA increases the explanation prediction (reasoning) performance of QA models.
strengths_and_weaknesses: Strengths

1. The proposed dataset, SQA, is a large-scale dataset and covers more diverse science topics. Also, SQA consists of more features such as detailed lectures and explanations related to the given question (line 129).
2. This paper showed that predicting explanation increases the overall SQA performance (Table 3).
3. This paper is well written and easy to follow.
4. This paper provides a sufficient analysis of SQA.

Weaknesses

1. This paper claims that SQA is a better dataset for evaluating the reasoning performance of QA models. However, this paper does not compare SQA and other science QA datasets (see Q2 in the “Questions” section for more details).
2. I have left my concerns in the “Questions” section.
questions: Q1: This paper provides SQA for better evaluation of the reasoning performance of QA models (lines 29-30). How can we know the reasoning performance of QA models with this dataset? Does predicting the correct explanation guarantee that the model’s prediction (answer) is based on the correct reasoning processes? Although some dataset papers in this field claim similar arguments (evaluating the reasoning performance of QA models can be done by explanation generation), the explanation prediction task is insufficient to argue that SQA enables evaluating QA models’ reasoning performance.

Q2: Line 58-59: How much improvement can we make with other science QA datasets? Please compare SQA with other datasets to show SQA's efficacy. One possible experiment is fine-tuning QA models on the SQA and TQA datasets, then comparing how much improvement the QA models get.

Q3: The image captioning model also affects the models’ performance. How can you conclude that current VQA models are not generalized to process the challenging questions in SQA (lines 229-230)? Would it be possible that the poor performance came from the poor performance of the image captioning model used in this paper?

Q4: In Table 4, the authors reported the semantic similarity scores computed by Sentence-BERT. However, reporting the absolute similarity scores is not a standard way to show semantic similarity. In semantic textual similarity, ranking-based evaluation metrics are usually used. Please use rank-based metrics such as top-k, recall@K, and MRR.

**Minor questions from here**

Q5: What is the reason for choosing social science and language science?

Q6: What is the difference between “Unique” questions and “Total” questions (line 133)?

Q7: In Table 4, what is the reason for using the BLEU-1 score instead of BLEU-4, a more widely used evaluation metric?
limitations: The authors have addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new dataset ScienceQA which consists of multiple choice questions with knowledge source and explanations. The dataset covers three domains of elementary through high school questions. The authors also conduct experiments using chain-of-thought methods, which prompts GPT3 and unified QA with a concatenation of question context, answer and lecture+explanation.  
strengths_and_weaknesses: Strength: 
1. The dataset looks promising, given that it covers both image and text modality and various science domains. 

Weakness: 
1. I don't understand why the proposed chain-of-thought (QCM-ALE) would be helpful when you generate answer before generating explanations (especially for the few-shot GPT3 examples). I think this might improve the quality of explanations, but given GPT3 is left-to-right autoregressive, the generated answers cannot take advantage of the generated explanations in the right contexts, which seemingly weakens the power of CoT, that looks like  QCM-LEA or QCM-ELA. 

2. Should compare to QCM-LEA or QCM-ELA. I think generating answers conditioned on the explanations would probably yield more performance gains, because the answers can condition on the generated explanations. 

3. The method is not novel enough. Other than my concerns (1) that the chain-of-thought is not used in its most powerful form: the chain-of-thought idea is not novel and nor is explanation/rationale generation.  


questions: Possible improvements directions: 
1. I think the lectures could be retrieved rather than generated, assume that you have a large documents of lecture contents. Using a retrieval based model to retrieve the lectures and use the GPT-3 models to condition on the lecture and generate explanations, and finally the answers could boost the model performance.  

2. Captioning system that converts images to text might be suboptimal, and could also explain why the accuracy for image questions seems to be significantly lower than the text questions. I think you might want to train end-to-end, or find a system that's focused on image description, rather than image captioning. 

3. Report standard deviations in the main table. I think there are high variances for in-context learning, but the table doesn't show it.  
limitations: The authors discussed limitations but in a quite narrow and empirical way, would be nice to see broader, higher-level limitations of this class of methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors propose a new multi-modal multiple-choice question answering dataset for science questions, with explanations. The dataset covers multiple domains, and has questions with text and image contexts.

The authors evaluate several baselines in different setups: VQA, fine-tuning UnifiedQA, prompting GPT-3 and find that the explanations provided in the dataset lead to improved accuracy on the dataset, both when using in a fine-tuning context as well as when used in prompting GPT-3.

strengths_and_weaknesses: ## Strengths:
* (mostly) well-written
* new dataset appears to fill a few gaps from existing datasets
* extensive evaluation with VQA models, as well as text-based finetuned and prompt-based models (UnifiedQA and GPT-3)

## Weaknesses:
* mediocre originality, not clear what really new questions or lessons
* confusing evaluation presentation and confusing to use
* some experimental settings are not explored (QCM→(L)EA)
* a round of proof-reading is needed to fix some mistakes and improve sentence construction
* I think it needs renaming because SQA already exists: https://paperswithcode.com/dataset/sqa

## Detailed review:
The paper is mostly well-written, and presents an interesting dataset for the QA community, and establishes reasonable baselines in various scenarios. Even though the contribution is technically solid (once a few concerns are addressed), there doesn’t appear much new to be learned from this work. It appears to be an interesting dataset for science QA, in particular because it provides explanations. The problem is rather that it’s easy to dismiss this work as showing that chain-of-thought based training works better with some baselines in a similar set-up to what was already explored for different domains. The addition of the visual modality is unfortunately not tied to CoT experiments and appears as another loose dimension of the dataset.
Some concerns that I had while reading the work were that the way evaluation results are presented is quite confusing, which could lead to confusion in practical use by others.
All main results (VQA and text-based) are crammed into Table 3 and there are several issues: (i) the segments in the table are not labeled (ii) when evaluating VQA models, are all the questions used (even the ones without image context, and does this make sense)? (iii) what do the TXT and IMG columns mean? What about questions without either?
I think the main evaluation needs more explanation and improved presentation.
Then, some important settings are missing from evaluation, in particular QCM → (L)EA (generating explanation before answer) hasn’t been included. I think there could be a significant difference between QCM→A(L)E and QCM→(L)EA for auto-regressive decoders because the reasoning steps generated before producing the answer can contribute to predicting the answer more correctly (see also Fig. 5 in [48]).
In addition, it would have been nice to provide some numbers for VQA with CoT. The currently tested VQA models are using a classifier to choose between the multiple choices, but it seems a baseline can be made by replacing that with a decoder? I think improvements in VQA due to CoT could also be an interesting additional selling point for this work.

Conclusion: if the above concerns regarding evaluation presentation and missing evaluation settings are addressed, I think it’s a solid work that presents an interesting dataset for the QA community. However, I wish there were more interesting new questions raised or new lessons learned.

(Note: I tried to formulate all of these concerns in the questions section as well for easier answering)

questions: Q1: what is the evidence that images are actually necessary for answering the questions in VQA experiments? Do you have questions referring to an image without having textual context that contains all the necessary information? Can you include at least one such example in the paper somewhere because I’m not sure the baby example is clear in this regard?

Q1.a: are questions that require an image to be answered included in the evaluation of text-only models and why does that make sense? How do we know the captions capture the necessary information?

Q2: why no QCM → (L)EA?

Q3: questions (i) - (iii) regarding table 3 (see above)

Q4: what is the variance on the reported numbers?

Q5: Are you always using a caption model ? How is the caption added to the context? How is the caption model trained? What is the effect of including generated captions for your text-based experiments? I think the captioning part should be explained in more detail, and evaluated in an ablation. Are these captions provided in the dataset for use by others?

Small notes:
* table 1 columns should be explained
* l. 82: “multi-model”?
* l. 97: “for examples”?
* l. 97-99: sentence incomplete
* l. 219: “will be stopped” → “is stopped”
* l. 246: “on SQA so far (75.17%) on SQA”
* Fig 3,a: the lines are a bit too thick, it is a bit difficult to read this figure
* Fig 3,b: I think this figure is unnecessary because it repeats what was already mentioned in the text, and in table 2, and takes up a lot of space relative to the contained info (unlike Fig. 3,a for example). I would add the % of questions without any context in Table 2 though.

limitations: QCM→(L)EA not evaluated and not clear why.

CoT not evaluated with VQA.

The authors includes an impact statement in appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to aid the development of a reliable model that is capable of generating a coherent chain of thought (CoT) when arriving at the correct answer to reveal the multi-step reasoning process.  Since existing science question datasets either lack annotated explanations for the answers or are restricted to the text modality with small data scales and limited topics, the paper presents Science Question Answering (SQA), a large-scale multi-choice dataset that contains multimodal questions with explanations and features rich domain diversity. It further designs language models to learn to generate lectures and explanations as to the chain of thought to mimic the reasoning process. Experimental results show that CoT benefits large language models in both few-shot and finetuning learning. And analysis shows that CoT helps language models learn from fewer data.

strengths_and_weaknesses: Strengths:
1. The presentation is good. The paper is easy to read.
2. The proposed datasets may aid the development of model reasoning in science QA.
3. Experiments are extensive and enlightening.

Weaknesses:
1. The novelty of the proposed dataset and baselines are limited.
2. The evaluation of the generated CoTs (lectures and explanations) still relies heavily on human judgment. Since the automatic metrics used in the paper are less consistent with human judgment, better automatic evaluation methods specifically for CoT are desired.
3. The correlation between question answering performance and explanation generation performance is not discussed.

questions: 1. Why aren't all examples annotated with lectures and explanations?
2. Why are CoTs lectures and explanations instead of explanations? Shouldn't the lecture be part of the model's input? Are there any experimental results for QCML->A and QCML->AE?
3. In table 3, why does GPT perform worse in the 2-shot setting than in the zero-shot setting?

limitations: The authors have discussed the limitations of this paper.
review_title: null

Meta-review: metareview: The paper introduces a large new multimodal dataset for science question answering, and thoroughly evaluates a range of models, including a version of chain-of-thought. Reviewers agree that the paper is generally solid and well written, and the dataset is potentially useful. The major concerns around the technical novelty of the contributions, which are somewhat incremental extensions to chain of thought (e.g. with fine tuned and multimodal models). Some reviewers are also confused why generating the answer first gives better chain of thought results, because this appears inconsistent with the step-by-step reasoning explanation of chain of thought, and this point could be better explained. Overall I think the submission is borderline, leaning towards acceptance.
confidence: Less certain
Decision: Accept