Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Authors: Pan Lu
Swaroop Mishra
Tony Xia
Liang Qiu
Kai-Wei Chang
Song-Chun Zhu
Oyvind Tafjord
Peter Clark
Ashwin Kalyan
Venue: NeurIPS 2022 Accept (2022)
Keywords: science question answering
multimodal reasoning
chain of thought

Abstract: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Pan Lu 1.3, Swaroop Mishra 2.3 Tony Xia', Liang Qiu', Kai-Wei Chang', Song-Chun Zhu', Oyvind Tafjord', Peter Clark', Ashwin Kalyan² University of California, Los Angeles, Arizona State University, 3 Allen Institute for Al (lupantech, kwchang. cs @mmai com, sczhu@stat. ucla edu, (oyvindt, peterc, ashwinkv)@allensi.
Abstract
When answering a question, humans utilize the information available across differ- ent modalities to synthesize consistent and complete chuin of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diag- nose the multi-hop hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (SCIENCEQA), new benchmark that consists of ~21 multimodal multiple choice questions with diverse science topics and annotations of their answers with corresponding lectures and explanations. We further design language models learn to generate lectures and explanations the chain of thought (Co'T) to mimic the multi-hop reasoning process when answering SCIENCEQA questions. SCIENCEQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few- shot GPT-3 and 3.99% in fine-tuned UnificdQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data.
Introduction
long-standing goal of AI systems is to act reliably and learn complex tasks efficiently like human beings. In the process of reliable decision making. humans follow an explicit bhain-of-thoughh (CoT) reasoning process that is typically expressed as an explanation. However, machine learning models are trained mostly using large number of input-output examples to perform specific task. These black-box models only generate the final decision without reliably revealing the underlying reasoning process. Not surprisingly. it is unclear if they understand the task and can generalize even though they perform well on the benchmark. On the other hand, humans are able to learn from instructions or explanations from past experience and generalize them to novel and unseen problems. This helps them learn more quickly with fewer data. this work, we explore if machines can be endowed with such reasoning abilities in the context of science-based question answering.
Recently, science problem solving benchmarks [18] have been used to diagnose the multi-hop reasoning ability and interpretability of AI systems. To answer science questions, a model needs to
'The data and code are available at https: / stt//scceenn/ enceqa gi thub. io. Work was partially done while Pan Lu and Swaroop Mishra were interns at A12.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Which type of foree from the baby's hand opens the cabinet door?
Question:
Answer: The
BECAUSE:
Options: (A) pull (B) pash
Lecture: A force is push pull that one object applies to second object. The direction of push is away from the object that is pushing. The direction of a pull is toward the object that is pulling.
Context: A baby wants to know what is inside of cabinet. Her hand applies force to the door, and door opens.
The baby's hand applies force to the cabinet door, This force causes the door to open. The direction of this force toward the baby's hand. This force pull.
Explanation:
Figure We construct the SCIENCEQA dataset where a data example consists of multimodal question answering information and the grounded lecture and explanation. We study if QA models can generate a reasonable explanation to reveal the chain-of-thouge reasoning.
not only understand multimodal contents but also extract external knowledge to arrive at the correct answer. Since these tasks require domain-specift knowledge and explicit multi-hop reasoning, model would be not interpretable if fails to provide explanations to reveal the reasoning process. However, current science question datasets [18, 17, 17, 52] mostly lack annotated explanations for the answers. To address this issue, other science datasets annotate the explanations, but they are restricted to the textual only modality and limited to small data scales [13, 6, 37] or small set of topics [20, 14]. Therefore, we collect Science Question Answering (SCIENCEQA), a large-scale multi-choice dataset that contains multimodal science questions with explanations and features rich domain diversity.
SCIENCEQA is collected from elementary and high school science curricula, and contains 21,208 examples along with lectures and explanations. Different from existing datasets [17, 18, 52], SCI- ENCEQA has richer domain diversity from three different subjects: natural science, social science, and language science. A typical example consists of a question, multiple choices, multimodal con- texts, a correct answer, as well as a lecture and an explanation. The lecture and explanation provide general external knowledge and specific reasons, respectively, for arriving the correct answer.
Consider the thoughts one person might have when answering the question in Figure One first recalls the knowledge regarding the definition of a force learned from textbooks: "A force is push or a pull that... ... The direction of a push is. The direction of a pull is ..." then forms line of reasoning: "The baby's ss hand applies force to the cabinet door + This force causes the door to open. - The direction of this force is toward the baby's 's hand.", ", and finally arrives at the correct answer: "This force is pull.". ". Following [41], we formulate the task to output natural explanation alongside the predicted answer. In this paper, we train language models to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop hop reasoning process to answer SCIENCEQA questions.
Our experiments show that current multimodal methods [55, 1, 21, 9, 26, 35] fail to achieve satisfac- tory performance on SCIENCEQA and do not generate correct explanations. Instead, we find that CoT can help large language models not only in the few-shot learning setting but also in the fine-tuning setting. When combined with CoT to generate the lecture and explanation, the fine-tuned UnifiedQA [19] achieves an improvement of 3. 3.99% as opposed to not using CoT in the fine-tuning stage. The few-shot GPT-3 model [4] via chain-of-thought prompting can obtain 75. 75% on SCIENCEQA with an improvement of 1.20% compared to the few-shot GPT-3 without COT. Prompted with CoT, GPT-3 can generate reasonable explanations as evaluated by automated metrics, and promisingly. 65.2% of explanations meet the gold standard of human evaluations. We also investigate the upper bound for models to harness explanations by including them in the input. We find that doing so improves GPT-3's few-shot performance by 18.96% suggesting that explanations do aid models and are currently underutilized in the CoT framework. Further analysis shows that, like humans, language models benefit from explanations to learn with less data: UnifiedQA with CoT obtains the same results as UnifiedQA without CoT with only 40% of the training data.
To sum up, our contributions are three-fold: (a) To bridge the gap in existing datasets in the scientific domain, we build Science Question Answering (SCIENCEQA). new dataset containing 21,208 multimodal science questions with rich domain diversity. To the best of our knowledge, SCIENCEQA is the first scale multimodal dataset that annotates lectures and explanations for the answers.

---

(b) We show that CoT benefits large language models in both few-shot and fine-tuning learning by improving model performance and reliability via generating explanations. (c) We further explore the upper bound of GPT-3 and show that CoT helps language models learn from fewer data.
2 Related Work
Visual question answering. Since the task of visual question answering (VQA) was first proposed in [2], there have been plenty of VQA datasets [56, 58, 23, 11, 15, 12] conducted to facilitate the research work. Although our SCIENCEQA dataset shares some features with VQA, there are several main differences between them. First, SCIENCEQA more challenging than existing VQA datasets because it contains multimodal contexts and diverse topics in the scientific domain. In addition, most answers are annotated with lectures and explanations, which makes SCIENCEQA a suitable dataset for multi-modal question answering and multi-hop reasoning for AI systems. Inspired by the recent remarkable performance achieved for VQA [33, 32, 10, 9, 26, 7, 8]. in this paper, we further extensively benchmark SCIENCEQA with wide range of attention- [1. 33, 21, 9] and Transformer-base based [30, 26, 27, 7] methods.
Datasets for science problems. Science problem solving is a challenging task that requires an AI system not only to understand the multimodal information from the science curriculum but also to reason about how answer the domain-specifee questions. Current science problem datasets such as AI2D [17]. DVQA [16], VLQA [52], and FOODWEDS [24] have contributed to multimodal reasoning in the scientific domain. For example, portion of VLQA contains multimodal questions on science subjects. These datasets, however, lack annotated explanations for the answers to reveal the reasoning steps. Some other datasets annotate the answers in the forms of supporting facts [37, 20J. entailment trees [6], explanation graphs [13], reasoning chains [14]. However, these datasets restricted to the single text modality with small data scales and limited topics. Instead, our SCIENCEQA A annotates the answers with grounded lectures and explanations. Besides, SCIENCEQA features richer domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills.
Learning from explanations and few-shot learning. Explanations help humans understand a task better, and there have been several attempts to show the same for models. For example, the learning from instruction paradigm [40, 43, 53, 39, 45, 25], where the task level explanation is provided in the form of instruction, improves model performance significantly. An example of learning from explanations in the scientific domain is proposed in [51] where the model interprets demonstrative solutions to solve geometry problems. Recently, there has been surge of interest in few-shot learning, where language models learn a specific task from a few examples [46, 3]. For instance, [42, 54, 34] find that explanations in the format of the chain of thought can improve language models' reasoning ability in few-shot learning. In this paper, we show that the chain of thought boosts the performance of large language models like UnifiedQA [19] if the models generate explanations along with the answer in a fine-tuning way. Furthermore, a few-shot GPT-3 model via chain-of-thouge prompting able to improve the reasoning performance SCIENCEQA and generate reasonable explanations.
Dataset
We collect SCIENCEQA, which is multimodal multiple-choice science question dataset containing 21,208 examples. An example in SCIENCEQA is shown in Figure Given the science question and multimodal contexts, the task is to select the correct answer from multiple options. Different from existing datasets [50, 17, 52, 31, 24], SCIENCEQA covers diverse topics across three subjects: natural science, social science, and language science. Moreover, most questions are annotated with grounded lectures and detailed explanations. The lecture provides general knowledge that introduces the background information for solving problems of similar class. The explanation reveals specific reason for the answer. To effectively answer the questions, model often needs to be able to understand the multimodal content in the input and extract external knowledge, similar to how humans do. More importantly, the goal of SCIENCEQA is to aid development of reliable model that capable of generating a coherent chain of thought when arriving at the correct answer to reveal the multi-step reasoning process. For data collection details, see Appendix A.
3

---

Statistic
Number
Total questions
21,208
Questions with text context
10,220 (48.2%) 2%;
Questions with image context
10,332 (48.7%)
Image natural format
R22960 (14.0%)
Image diagram format
R=73372 (34.8%)
Questions with both contexts
6,532 (30.8%)
Egune
Questions without any context
7,188 (33.9%)
word
Questions with a lecture
17,798 (83. (83.9%)
Questions with a explanation
19,202 (90.5%)
Different questions
9,122
Different lectures
261
Topic classes
26
Calegory classes
127
Skill classes
379
PANNNTI
Average question length
12.11
100
Average choice length
4.40
Average lecture length
125.06
Average explanation length
47.66
Table 1:
Main statistics in SCIENCEQA.
Figure 2: Question distribution in SCIENCEQA.
3.1 Data Analysis
Key statisties. We randomly split the dataset into training, validation, and test splits with ratio of 60:20:20. Each split has 12,726, 4,241, and 4,241 examples, respectively. Table shows the main statisties of SCIENCEQA. SCIENCEQA has large set of different questions, totaling up to 9,122. Out of the 21,208 questions in SCIENCEQA, 10,332 (48.7%) have an image context, 10,220 (48.2%) have a text context, and 6,532 (30.8%) have both. 83.9% of the questions are annotated with a lecture, while 90.5% of the questions feature an explanation. The cross-combination of these information sources diversifies the problem scenario: sometimes the model is given a of information from multiple sources, while at other times, the only source of information is the question itself. This level of complexity is very common in grade-level science exams.
AD
Rreqeecy (96)
25
Geometry3K
IconQA
20
ScienceQA (ours)
3,800 (17. 977)
6,532
3, 8.688 (17.39%)
(30 a0%)
Image
Text
10-
7,188 (33.89%)
0-
No context
[image
100
1.5
20
25
30
35
40
of words questions
(b) Question distribution with different con- text formats, 66.11% of the questions SCI- ENCEQA have either an image or text con- text, while 30.80% have both.
(a) Question length distribution of related datasets. SCI- ENCEQA is distributed more evenly in terms of the num- her of question words than other datasets.
Figure 3: Question length distribution (a) and context distribution in SCIENCEQA (b).
Question analysis. SCIENCEQA has diverse set of science questions. Figure shows distribution of the first four words in the question text. A large number of question lengths and formats highlight the diversity of SCIENCEQA. The question lengths range from words to 141 words, and the questions in SCIENCEQA have average length of 12.1 words. The question length distribution visualized against other VQA datasets in Figure (a). As shown the diagram, SCIENCEQA's distribution is flatter than other datasets, spanning more evenly across different question lengths.
Context analysis. Figure 3 (b) shows the number and percentage of questions with either an image context, text context, or both. There are total of 7,803 unique image contexts and 4,651 unique text

---

Biolegy
Physies
Geography
History
Civics
Genes tniss
Materials
State capitals
Colonial America
Social skills
Classification
Magnets
Goography
English colonics in North America The American Revolution
Government The Constitution
Adaptations
Volocity and forces
Maps
Truils and berodity
Foree and motion
Oceania: geography
World History
Economics
beosystems
Purticle motion and energy
Physical Geography
Cireace
Basic coonomic principles
Classification
Heat and thermal energy States matter
The Americas goography
Ancient Mesopotamin
Supply and demand
Scientifie
Oceans and continents
World religions
Banking and finance
Heredity
Kinetic and potcnttal energy Mixture
Cities
American bistery
Global Studies Society environment
beological interactions
States
Medieval Asia
Plants
Solutions
Chemistry
Supporting argumentis
Writing Strategies
Calegories
Vocabulary
ABC
Verbs
Verb lense
Plant reproduction
Animals
Atoms and molecules
Plysical and chemical change
Sentences, fragments, and run-ons Word usage and nuance Creative techniques
Comprecennsio strategies
Shades of meaning
Capitalization
Formatting
Earth Science
Chemical reactions
Context clues
Punctuation
Fossils
Rocks mincrals
Weather and climate
Astronomy
Engineering practices
Designing experiments
Engineering
Editing and revising Visual elements Opinion wriling
Audience, purpuse, and lone Promouns and antecadents Persuasive strategles
Phrnses and classes
Grammar
Sentences and frugmenns
Rhyming
Fragmentt
Phonolugy
Earth events
Units and Measurement Weather and climate
Figurative Laaguage
Reference
Plate lectonics
Literary devices
Research skills
Figure 4: Domain diversity in SCIENCEQA. Each color corresponds to one subject: natural science, social science, and language science. For visual clarity, only the most frequent classes are shown.
contexts. 66. 1 1% of the questions have at least one type of context information. The image context is in the format of diagrams or natural images, which visualize the critical scenario necessary for question answering or simply illustrate the question for better understanding. Similarly, the textual context can provide either semantically rich information or a simple hint to the question. Therefore, models need to be flexible and general to understand these diverse types of contexts.
Domain diversity. Each SCIENCEQA question belongs to one of the three subjects: natural science, social science, and language science. With each subject, questions are categorized first by the topic (Biology, Physics, Chemistry, etc.), then by the category (Plants, Cells, Animals, etc.), and finally by the specific skill (Classify fruits and vegetables as plant parts, Identify countries of Africa, etc.). SCIENCEQA has total of 26 topics, 127 categories, and 379 skills. The treemap in Figure 4 visualizes the different subjects, topics, and categories and shows that SCIENCEQA questions are very diverse, spanning a wide range of domains.
3.2 Comparisons with Existing Datasets
Table 2 shows comparison of SCIENCEQA and other science problem datasets. As shown in the table, SCIENCEQA is much larger than most other datasets. SCIENCEQA also has the largest set of images, spans across all 12 grades, contains the longest questions, and has the most diverse input sources. As opposed to limiting the subject to only natural science, SCIENCEQA also includes social science and language science, largely adding to the domain diversity of the dataset. Furthermore, most of the questions in SCIENCEQA A are annotated with textual lectures (83.9%) and explanations (90.5%), which reveal the reasoning path the correct answer. To the best of our knowledge, SCIENCEQA is the first large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations.
#Q
#1
AvgQ MaxQ Grades
Science subjects
Contexts
Images
Lecture Explanation
Ge0metry33K [31]
3,002
2,342
10.1
46
6-12
natural (geometry)
image
diagram
x
X
A12D [17]
4,563
4,903
9.8
64
1-6
natural
image
diagram
FOODWEBS [24]
Rd5,000
R=5.00
8
natural (foodweb only)
image
diagram
ARC [5]
7.787
0
20.4
128
3-9
natural
x
X
X
X
TQA [18]
26,260
3,455
9.2
57
6-8
natural
image, text visual
diagram diagram
IconQA [35]
107,439
96,817
8.4
73
Prek-3
math
WorldTree [13]
1,680
0
3-5
natural
x
OpenBooKQA [37]
5,957
10.6
68
1-6
natural
x
QASC [20]
9,980
0
8.0
25
1-9
natural
x
x
SCIENCEQA (ours)
21.208
10,332
12.1
141
1-12
natural, social, language image, text natural, diagram
Table 2: Statistics for SCIENCEQA A and comparisons with existing datasets. #Q: number of questions, #I: number of images, AvgQ: average question length; MaxQ: maximum question length.
5

---

4 Baselines and Chain-of-Thought Models
In this section, we establish baselines and develop two thain-of-thhough models SCIENCEQA.
4.1 Baselines
Heuristic baselines. The first heuristic baseline is random chance: randomly select one from the multiple options. Each trial is completed on the whole test set, and we take three different trials for an average result. The second heuristic baseline is human performance. We post the task Amazon Mechanical Turk and ask workers to answer SCIENCEQA questions. Only workers who obtain a high school or higher degree and pass the qualification examples are qualified for the study. Each worker needs to answer set of 10 test questions, and each question is answered by three different workers. For more details of the human performance study, see Appendix B.2.
Zero-shot and few-shot baselines. We establish the zero-shot baselines on top of UnifiedQA [19] and GPT-3 [4]. The zero-shot setup follows the format of QCM-AA where the input is the concatenation of tokens of the question text (Q), the context text (C), and multiple options (M), while the output is to predict the answer (A) from the option set. We extract the caption from the captioning model based on ViT [7] and GPT-2 [47] for the image as the visual context. In the few-shot setting, we follow the standard prompting [4] where in-context examples from the training are concatenated before the test instance. These io-context examples serve as an instruction for the language model to adjust to the specific task in SCIENCEQA.
Fine-tuning baselines. We first consider the fine-tuning baselines from VQA models [1, 21, 55, 9, 22, 35, 26] proposed in recent years. These VQA baselines take the question, the context, and choices as the textual input, take the image as the visual input, and predict the score distribution over choice candidates via linear classifier. In addition, we build the fine-tuning baseline on top of the large language model UnifiedQA [19]. UnifiedQA takes the textual information as the input and outputs the answer option. Similarly, the image is converted into a caption that provides the visual semantics for the language model.
4.2 Language Models with the Chain of Thought
A chain of thought refers to a coherent flow of sentences that reveals the premises and conclusion of a reasoning problem [54]. chain of thought clearly decomposes a multi-hop reasoning task into intermediate steps instead of solving the task in a black-box way. The chain of thought can be the step-by-step thought process [54] before arriving at the final answer or explanations [41] that come after the answer. The annotated lectures and explanations in SCIENCEQA serve as demonstrations of the chain of thought that mimics the multi-step reasoning steps of human beings. In this paper, we study if large language models can generate reasonable explanations as the chain of thought to reveal the thought process when answering SCIENCEQA questions. Further, we explore how the chain of thought can improve the reasoning ability of language models on SCIENCEQA in both few-shot and fine-tuning learning.
UnifiedQA with the chain of thought. UnifiedQA [19] is state of the art model for multi-option question answering. The original architecture of UnifiedQA takes the question and options as the input and outputs short phrase as the final answer. We make format modification to develop UnifiedQA with the chain of thought (CoT), i.e., UnifiedQA is fine-tuned to generate a long sequence of text which consists of the answer followed by the lecture and explanation.
GPT-3 via (tain-of-thoughhe prompting. Recent research work [4, 38, 34] has shown that GPT-3 [4] can perform various tasks when provided with in-context examples in standard prompt. Take multi-option question answering as an example, the standard prompt [36, 57, 29] builds instructions using in-context examples with components of the question text, options, and the correct answer text. This style of few-shot learning enables the GPT-3 model to answer specific questions without parameter updates. Different from standard prompting, we build GPT-3 via chain-of-thought (CoT) prompting, as shown in Figure 5. To be specific, for each test problem I, we map the prompt instruction I (T+),, Jt into a textual format where (Te), refers to the instruction set of n-shot in-context examples from the training set, while denotes the test instruction. Instead of the way where the explanation comes before the answer [54], we feed the instruction into the encoder-

---

Question: question I
Options: (A) option : 1 (B) option 1o (C) option : 433
Context: context 1eont
Answer: The answer is answer T.. BECAUSE: lecture pect explanation : 1
Question: question: Tryue
Options: (A) option 12 (B) option: 12" (C) option 133 (D) option: : 14
Context: context 1punt
Answer:
Prompt instruction encoding for the test example t in GPT-3 (CoT). The prompt above consists of the instruction { for the 1-shot training example and I, for the test example.
Figure 5:
decoder model GPT-3 to generate the answer a followed by the lecture lect and explanation exp: M (4,,, a, lellet, exp.
5
Experiments
5.1 Experimental Setup
Evaluation metrics. The heuristics and VQA baselines treat SCIENCEQA task as multi-class classification problem with multiple options and are evaluated with the accuracy metrics. UnifiedQA and GPT-3 treat SCIENCEEA as the final prediction to evaluate the question answering accuracy. The generated lectures and explanations evaluated by automatic metrics [44, 28, 49] and human scores by annotators.
text generation problem. So the most similar option is selected
Implementation details. The VQA baselines are trained for a maximum number of 50 epochs with a learning rate of 5. We fine-tune the UnifiedQA for 50k iterations and evaluate every 1k iteration.
The training process is stopped following the early stopping strategy with a patience period of three evaluations. For GPT-3, use the text-davinci- 002 engine, which is the most capable model version suggested in the official documentation. More details can be found Appendix B.I.
5.2 Results for Question Answering
Table 3 demonstrates the empirical results for Science Question Answering.
VQA baselines. We feed the VQA baseline models with the input of QCM format to predict answers A. Out of all the VQA models we benchmarked, VisualBERT [26, 27] performs the best on average (61.87%). Interestingly, Patch-TRM [35] beats VisualBERT in natural science (NAT) and language science (LAN), and it also performs better in higher-grade questions (67.50% v.s. 59.92%). However, in the subject of social science (SOC), VisualBERT outperforms Patch-TRM by large margin (+22.39%). Such drastic changes in performance might imply that current VQA models are not generalized process the challenging questions in SCIENCEQA.
Language models. We evaluate whether large-scale pretraining on text can help language models learn scientific knowledge and thus perform better the SCIENCEQA task. For this purpose, have tried two of the tf-the-rrt pre-trrined language models: UnifiedQA and GPT-3.
(i) UnifiedQA. The results show that without any supervised fine-tuning (zero-shot), UnifiedQA cannot beat any VQA baseline model, while the pretraining does help the model obtain some scientific knowledge to outperform the random baseline. When fine-tuned with the answer labels in SCIENCEQA, UnifiedQABASE reports an accuracy of 70. 12% on average. By further teaching the model to generate the answer along with lecture and explanation, the developed language model with of-hhugh (UnifiedQABAss (CoT)) brings additional improvements of +3.21%% 1% (QCM-AAE and +3.99% (LCM+ALE). These results show that generating the chain of thought along with the answer benefits the reasoning ability of language models.
(ii) GPT-3. The positive effect of pretraining is also proved by the surprisingly good results from GPT-3 in the same zero-shot setting as UnifiedQA. Without any fine-tuning, GPT-3 already reaches almost the best performance we can get. Interestingly. prompting the GPT-3 with two training examples with only answers results in negligible difference. However, we prompt GPT-3 with prompting (QCM~ALE) obtain the state-of-the thette-rttttttt result so far (75.17%).
7

---

Model
Learning
Formal
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Avg
Random chance
M-+
40.28
46.13
29.25
47.45
40.08
33.66
39.35
40.67
39.83
Q only [i]
train set
Q-+A
41.34
27.22
47.00
41.79
35.15
44.60
39.28
40.87
39.85
C, only [1]
train set
C1-AA
41.34
29.25
45.45
42.33
36.09
42.93
39.21
41.07
39.87
Q+M only [1]
train set
QM-+A
52.66
51.86
60.18
55.57
50.37
57.42
52.53
57.88
54.44
Q+Cr+M only [1]
train set
M-y
57.28
49.04
61.36
60.46
52.80
58.82
54.44
60.51
56.61
Q+C++M only [1]
train set
QCyM-+A
58.97
53.77
60.45
62.85
54.49
57.63
56.72
61.04
58.
MCAN [55]
train sel
QCM->
56.08
46.23
58 58.09
59.43
51.17
55 55.40
$1.65
59.72
54.54
Top-Down [1]
train set
QCM-X >A
59.50
54.33
61.82
62,90
54.88
59.79
57.27
62.16
59.02
BAN [21]
train set
QCM-+
60.88
46.57
66.64
62.61
52.60
65.51
56.83
63.94
59.37
DFAF [9]
train set
QCM >A
64.03
48.82
63.55
65.88
54,49
64.11
57.12
67.17
60.72
ViLT [22]
train set
QCM AA
60.48
63.89
60.27
63.20
61.38
57.00
60.72
61.90
61.14
Patch-TRM [35]
train set
QCM-A
65.19
46.79
65.55
66.96
55.28
64.95
58.04
67.50
61.42
VisualBERT [26, 27]
train set
QCM-A
59.33
69.18
61.18
62.71
62.17
58.54
62.96
59.92
61.87
UnifiedQAMMALL [48]
zero-shot
QCM >A
47.78
40,49
46 00
50.24
44,12
44,39
45.56
46.21
45,79
UnifiedQAsAse [48]
zero-shot
QCM-+
50.13
44.54
48.18
53.08
48.09
46.69
47.58
50.03
48.46
UnifieQAAsMALL [48]
train set
QCM-+
53.77
58.04
61.09
52.10
51.51
61.46
58.22
53.59
56.57
UnifiedQAsAse [48]
train set
QCM-A
68.16
69.18
74.91
63.78
61.38
77.84
72.98
65.00
70.12
AnASE (CoT)
set
QCM->AR
70.60
74.02
78.36
65.69
64.80
81.53
75.48
69.48
73.33g 211
UniftedQA BASE (CoT)
sel
QCM-+ALE
71.00
76.04
78.91
66.42
66.53
81.81
77.06
68.82
74. 11,11,,.00 991
GPT-3 [4]
zero-shot
QCM->
75.04
66.59
78.00 .0
74.24
65.74
79.58
76.36
69.87
74.04
GPT-3
2-shot
QCM->A
74.64
69.74
76.00
74.44
67.28
77.42
76.80
68.89
73.97
GPT-3 (CoT)
2-shot
QCM-AE
76.60
65.92
77.55
75.51
66.09
79.58
78.49
67.63
74.61g.647
GPT-3 (CoT)
2-shot
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.171.201
Human
QCM-+
90.23
84.97
87.48
89.60
87.50
88.10
10
91.59
82.42
88.40
Table Evaluation of baselines over different classes in accuracy (%) Model names: Q question, M = multiple options, C= context, Cr = text context, C, = image context, CoT = chain of thought. Format names: A = answer, AE answer with explanation, ALE = answer with lecture and expla- nation. Question classes: NAT natural science, SOC social science, LAN = language science, TXT text context, IMG = image context, NO = no context, G1-6 = grades 1-6, G7-12 grades 7-12. Segments 1: Random chance; Segment 2: Ablation studies on top of Top-Down; Segment 3: VQA baselines; Segment 4: UnifiedQA baselines and UnifiedQA with CoT; Segment 5: GPT-3 baselines and GPT-3 with CoT; Segment 6: Average human performance.
What do these two changes huve in common?\yn bleaching clothes')n (n
Question:
a picce of apple turning brown')n
Options: (A) Both are only physical changes. (B) Both are caused by cooling, (C) Both are chemical chunges. (D) Both are caused by heating. \.n
Answer: The answer C. BECAUSE: physical change a change substance that does not change what the substance is. A physicul chunge only changes the substunce's 's physical properties, such as its size, shape, color. \n A chemical change is change
substance that changes what the substance is. A chemical change changes the substance's chemical properties, such as its chemical structure. An
The changes that you listed are both chemical changes. \n
When you bleach clothes, the chemical structure of the fabric changes. This change makes the fabric less likely to hold color. \n When piece of fruit turns brown, the chemical structure of the fruit changes. This change makes the fruit taste different.
Figure 6: One example of the predicted answer along with the chain of thought from GPT-3 (CoT).
Human performance. Humans outperform all benchmarks consistently across question classes, context types, and grades, e.g., a 20.07% gap for questions with the image context (IMG) between humans and our best performing model. The gap is to be filled by future research on multimodal reasoning for scientific question answering.
5.3 Results for Generated Explanations
One prediction example of GPT-3 (CoT) is visualized in Figure 6. We can see that GPT-3 (CoT) predicts the correct answer and generates a reasonable lecture and ex explanation to mimic the human thought process. We further report automatic metrics (BLEU-1/4 [44], ROUGE-L [44], and (sentence)
8

---

Similarity [49] to evaluate the generated lectures and explanations, as shown in Table 4. The Similarity metric computes the mnsine-similarig of semantic embeddings between two sentences based on the Sentence-BERRe network [49]. The results show that UnifiedQABASI (Co'T) generates the most similar explanations to the given ones. However, it's commonly agreed that automatic evaluation of generated texts only provides a partial view and has to be complemented by human study. By asking annotators to rate the relevance, correctness, and completeness of generated explanations, we find that the explanations generated by GPT-3 (CoT) conform best to human judgment.
Model
Format
BLEU-1
BLEU-4 ROUGE-L Similarity Relevant Correct Complete
Gold
UnitiedQAaass (CoT)
QCM-ALE
0.397
0.370
0.714
0.811
80.4%
76.6%
76.1%
56.9%
GPT-3 (CoT)
QCM-AB
0.234
0.048
0.351
0,561
76.9%
73.0%
70.5%
52.5%
GPT-3 (CoT)
QCM->ALE
0.192
0.052
0.323
0.595
88.5%
78.8%
84.5%
65.2%
Table 4: Automatic metries (BLEU- 1/4, ROUGE-L, Similarity) and human evaluation of generated explanations. Note that a gold explanation refers to one that is relevant, correct, and complete.
5.4 Analysis
Blind studies. Blind studies are conducted on top of the modification the full model, Top-Down [1]. The results achieved in blind studies of Q only and C2 only are close to random chance, showing that the SCIENCEQA A dataset is robust and reliable in distribution. The performance drops in Q+M only, Q+C+++M only, and Q+C++M only indicate that all input components provide critical information for answering SCIENCEQA questions.
Prompt types. We study the effect of prompt types and visualize the comparison in Figure 7 (a). It shows that prompting the GPT-3 model with both lectures and explanations (QCM-ALE) results in the highest accuracy on average and the smallest variance. In contrast, prompting with only explanations (QCM->AB gives the largest variance, resulting less stable model.
80
80
DCM A
OCM-ALE
78
76
Accurscy
Accuracy
70
68
68
66
66
QCM-A
QCM-AL
QOM-AE
QCM-ALE
Prompt type
Number of training examples
(a) Acc. v.s. different prompts with 4-shot examples.
(b) Acc. v.s. different # of training examples.
Figure 7: Accuracy of GPT-3 (CoT) cross different prompt types (a) and # of training examples (b).
Number of in-context examples. In Figure 7 (b), we further investigate how different numbers of training examples encoded in prompts can affect the prediction accuracy. The QCM-ALE prompt type outperforms performs comparably to the QCM-A type with all numbers of examples. And we observe the peak performance of QCM+ALEE with training examples being prompted. After that, the accuracy goes down as more training examples are added the model.
Dynamic sampling. In Table 5, instead of ran- dom sampling, we try dynamically select the in-context examples to prompt with the same class as the test sample. However, slight differ- ences in prediction accuracy are observed when comparing them simple random sampling.
Prompt type Sampling
Acc. (%)
QCM-+ALE Dynamic (same topic)
75.15
QCM-AALE
Dynamic (same category)
74.58
QCM+AALE
Dynamic (same skill)
75.10
Table 5: Dynamic sampling for GPT-3 (CoT).
Upper bound. We search the upper bound of the GPT-3 accuracy by feeding the gold lecture and explanation in the test prompt. As reported in Table 6, QCME* + outperforms the QCM-+ALE baseline by 18.86% and QCMLE** outperforms QCM- ALE by 18.96%, indicating potential improvement direction by generating correct explanations before answering science questions.

---

Prompt type
Sampling
Acc. (%)
Prompt type
Sampling
Acc. (%)
QCML x
Random
73.59
QCM-LA
Random
60.6
QCML* +AB
Random
74.32
QCM -EA
Random
56.0
QCME*+A
Random
94.03
QCM-
Random
55.4
QCMLE*-A
Random 94.1318.961
QCM-ELA
Random
51.5
QCM-ALE
Random
75.17
QCM-ALE
Random
73.6
Table 6: Upper bound of GPT-3 (CoT).
Table 7: Different positions of L/E for GPT-3 (CoT).
Positions of lectures and explanations. We study the performance of GPT-3 (CoT) in terms of different positions of lectures and explanations on 1,000 test examples. The results are shown in Table There could be huge accuracy decreases if GPT-3 (CoT) predicts lectures and explanations before answers. It is mainly because if GPT-3 (CoT) is formulated to generate the long lecture and explanation first, there a greater chance that it will stop generating the prediction early or use up the maximum token limits before obtaining the required answer.
CoT learns with fewer data. To study if the chain of thought helps language models learn more efficiently, we
report the accuracies of UnifiedQA and UnifiedQA (CoT) fine-tuned on different sizes of the training set in Figure 8.
70
Accuracy (96)
%
UnifiedQA (CoT) benefits language models by learning the
coherent reasoning path when answering questions, resulting in similar accuracy with fewer training examples.
60
Model
UnifledIA
Error analysis. GPT-3 via chain-of-thouglee prompting ob- tains promising results but still fails answer a wide range of challenging questions in SCIENCEQA. See examples of failure cases in Appendix B.4. The failure cases can be clas- sified into two types: (a) the model fails to understand the multimodal inputs and lacks domain-speciffe knowledge to arrive at the correct answer; (b) the model generates the wrong chain of thought with irrelevant, incorrect, or incomplete information.
55
(CoT)
10
20
40
60
80
100
Proportion training examoles (%)
Figure 8: UnifiedQA (CoT) learns ef- ficiently with fewer training examples.
Discussion and Conclusion
In this paper, we propose SCIENCEQA, a dataset that features 21, 21,208 multi-option questions with multimodal contexts from the science curriculum. To the best of our knowledge, SCIENCEQA is the first large-scale multimodal science dataset where most questions are annotated with corresponding lectures and explanations. We establish various baselines, including recent VQA models and large language models on SCIENCEQA. We further study language models can generate reasonable explanations and then benefit the reasoning ability. Experiments show that UnifiedQA with the chain of thought can achieve an improvement of 3.99% and few-shot GPT-3 via chain-of mhain-of-thought (CoT) prompting can obtain satisfactory accuracy of 75.17% SCIENCEQA. 65.2% of the generated explanations from GPT-3 (CoT) meet the gold standard by human evaluations.
7 Acknowledgment
We would like to thank the anonymous reviewers for their valuable comments and suggestions. We would also like thank Xiaodan Liang for insightful discussions dataset collection. We thank our colleagues at The Allen Institute of AI (AI2), Jiasen Lu and Jungo Kasai for helpful discussions. The work does relate Liang Qiu's position at Amazon Alexa.
References
[1] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision (CVPR), pages 2425 22233, 2015.
10

---

[3] Jonathan Bragg. Arman Cohan. Kyle Lo. and Iz Beltagy. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learnrrs. Advances in neural information processing systems (NeurlPS), 33:1877-1901, 2020.
[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.088 05457, 2018.
[6] Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. Explaining answers with entailment trees. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In The International Conference on Learning Representations (ICLR), 2021.
[8] Feng Gao, Qing Ping. Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform- carriee-gennrrate Natural language-centric outside-knowlede visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5067-5077, June 2022.
[9] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6639-6648, 2019.
[10] Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang. Question- guided hybrid convolution for visual question answering. In The European Conference on Computer Vision (ECCV), pages 469-44406 485,-885,222 2018
[11] Yash Goyal, Tejas Khot, Douglas Summers Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sommerr--Stay,
[12] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 6700-6709, 2019.
[13] Peter A Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton T Morrison. Worldtre:: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. arXiv preprint arXiv:1802.080 100.00052. 2018
[14] Harsh Jhamtani and Peter Clark. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop guestion-answering arXiv preprint arXiv:2010.03274 2020.
[15] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901-2910, 2017.
[16] Kushal Kafle, Brian Price, Scolt Cohen, and Christopher Kanan. Dvqa: Understanding data visualiza- tions via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5648 000635556,220 2018
[17] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[18] Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Ha- jishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4999-5007, 2017.
[19] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics (EMNLP), pages 1896-1907, 2020.
11

---

[20] Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, and Ashish Sabharwal. Qase: A dataset for question answering via sentence composition. ArXiv, abs/1910. 1473, 2020.
[21] Jin-Hwa Kim, Jachyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems (NeurPPS), pages 1571-1581, 2018.
[22] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Assion-and-languaae transformer without convolution region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 35833-5544 2021.
[23] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma. et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), pages 32-73, 2017.
[24] Jayant Krishnamurthy, Oyvind Tafjord, and Aniruddha Kembhavi. Semantic parsing to probabilistic programs for situated question answering. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages (60--70,220 2016.
[25] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from 2022.
explanations in context? arXiv preprint arXiv:2204.02::555
[26] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang, Visualbert: A simple and performant baseline for vision and language. arXiv preprint K90v:/90..03300 2019.
[27] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kei-Wei Chang. What does bert with vision look at? In Proceedingy of the 58th Annual Meeting of the Association for Computational Linguistics 2020.
(ACL), pages
[28] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarizaaiio branches out, pages 74-81, 2004.
[29] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.00 000004000004...11111 2021.
[30] Jinsen La, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- vision-and-language tasks. In Advances in Neural Information Procexsing Systems (NeurlPS), pages 13-23, 2019.
[31] Pan La,, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu, Inter- gps: Interpretable genmetry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguisties (ACT.), 2021.
[32] Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. R-vqa: learning visual relation facts with semantic attention for visual question answering. In The ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 1880-1889, 2018
[33] Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with modal multiplicatiive feature embedding for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAI), 2018
[34] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209, 14610. 2022.
[35] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurlPS) Track on Datasets and Benchmarks, 2021.
[36] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically or- dered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2 104.0260 2021.
[37] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP),201) 2018.
12

---

[38] Swaroop Mishra, Matthew Finlayson. Pan Lu, Leonard Tang. Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[39] Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. Reframing instructional prompts to gptk's 's language. ACL Findings, 2021.
[40] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. The 59th Annual Meeting of the Association for Computa- tional Linguistics (ACL), 2021.
[41] Sharan Narang, Colin Raffel. Katherine Lee. Adam Roberts. Noah Fiedel, and Karishma Malkan. WL5!! training text-to-text models to explain their predictions. arXiv preprint arXiv:2004. /4546, 2020.
[42] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieher, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models, arXiv preprint arXiv:2112.0011
2021.
[43] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, ct al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022 155, 2022.
[44] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine tramslation. In Proceedings of the 40th ammual meeting of the Association for Computational Linguistics (ACI.), pages 31 1-318, 2002.
[45] Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. In- BoXBART: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112-128 Seattle, United States, July 2022. Association for Computational Linguistics.
[46] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models, Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188)9, 2019.
[48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:1-67, 2020.
[49] Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 11 2019.
[50] Mrinmaya Sachan, Kumar Dubey, and Eric Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from texthooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 773-784, 2017.
[51] Mrinmaya Sachan and Eric Xing. Learning to solve geometry problems from natural language demonstra- tions in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 251-261, 2017.
[52] Shailaja Keyur Sampat, Yezhou Yang, and Chitta Baral. lingusin question answering (vlqa) challenge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings (EMNLP), pages 4606 061666466.2222 2020.
[53] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. The International Conference on Learning Representations (ICLR), 2021.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, arXiv preprint arXiv:2201. /9903, 2022.
[55] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6281-6290, 2019.
13

---

[56] Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference Computer Vision and Puttern Recognition (CVPR), 2016.
[57] Zihao 7hao, Fric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate hefore use: Improving few-shot performance of language models. In International Conference on Machine Learning (ICML.), pages 12697-12706. PMLR, 2021.
[58] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Yes, we did the error analysis in Section 5.4 and discussed the limitations of the work in Appendix B.4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discussed the broader impacts in Appendix B.5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We included 100 data examples and the data visualizer tool the supplemental material. The whole dataset and code will be available at https: / htt////ceaaa enceqa gf thub. io.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 and Appendix B. for experimental details.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We reported the error bars for GPT-3 (CoT) experiments in Figure 7, where each experiment was repeated four times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We discussed compute resources in Appendix B.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We collected the SCIENCEQA dataset from https: /www ix1. com/. The copyright belongs to IXL.
(b) Did you mention the license of the assets? [Yes] SCIENCEQA is under the CC BY-NC- SA 4.0 license and used for non-commercial research purposes.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We included data examples and a visualizer tool in the supplemental material. The dataset will be available at https: // sci enceqa gi thub. io.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The collected data does not contain personally
identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] We included sereenshots of the instructions in Appendix B.2 and B.3.
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] We included the monetary compensation
details in Appendix B.2 and B.3.
15


Reviews:
[{"review_id": "lTNCLQCUTC0", "reviewer": "Reviewer_hjMW", "summary": "This paper provides a novel dataset for science question answering, SQA. Specifically, this paper studies a dataset for evaluating the reasoning/explanation performance of QA models. The authors claimed that existing science question answering datasets either 1) lack annotations for explanations for the answers or 2) has a small data scale/set of topics. SQA consists of 21K examples, a comparably large-scale dataset compared to existing science QA datasets. SQA covers more topics than existing datasets, and SQA is a multi-modal dataset; this dataset provides image context as well as textual context. This paper shows the validity of SQA by showing that the answer prediction performance increases when QA models are fine-tuned on SQA. Also, this paper conducts explanation prediction experiments. In this experiment, QA models predict the annotated explanations/lectures, and humans evaluate the predictions. The experimental results indicate that training QA models on SQA increases the explanation prediction (reasoning) performance of QA models.", "questions": "Q1: This paper provides SQA for better evaluation of the reasoning performance of QA models (lines 29-30). How can we know the reasoning performance of QA models with this dataset? Does predicting the correct explanation guarantee that the model’s prediction (answer) is based on the correct reasoning processes? Although some dataset papers in this field claim similar arguments (evaluating the reasoning performance of QA models can be done by explanation generation), the explanation prediction task is insufficient to argue that SQA enables evaluating QA models’ reasoning performance.\n\nQ2: Line 58-59: How much improvement can we make with other science QA datasets? Please compare SQA with other datasets to show SQA's efficacy. One possible experiment is fine-tuning QA models on the SQA and TQA datasets, then comparing how much improvement the QA models get.\n\nQ3: The image captioning model also affects the models’ performance. How can you conclude that current VQA models are not generalized to process the challenging questions in SQA (lines 229-230)? Would it be possible that the poor performance came from the poor performance of the image captioning model used in this paper?\n\nQ4: In Table 4, the authors reported the semantic similarity scores computed by Sentence-BERT. However, reporting the absolute similarity scores is not a standard way to show semantic similarity. In semantic textual similarity, ranking-based evaluation metrics are usually used. Please use rank-based metrics such as top-k, recall@K, and MRR.\n\n**Minor questions from here**\n\nQ5: What is the reason for choosing social science and language science?\n\nQ6: What is the difference between “Unique” questions and “Total” questions (line 133)?\n\nQ7: In Table 4, what is the reason for using the BLEU-1 score instead of BLEU-4, a more widely used evaluation metric?", "limitations": "The authors have addressed the limitations and potential negative social impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed dataset, SQA, is a large-scale dataset and covers more diverse science topics. Also, SQA consists of more features such as detailed lectures and explanations related to the given question (line 129).\n2. This paper showed that predicting explanation increases the overall SQA performance (Table 3).\n3. This paper is well written and easy to follow.\n4. This paper provides a sufficient analysis of SQA.\n\nWeaknesses\n\n1. This paper claims that SQA is a better dataset for evaluating the reasoning performance of QA models. However, this paper does not compare SQA and other science QA datasets (see Q2 in the “Questions” section for more details).\n2. I have left my concerns in the “Questions” section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j0S21Fs_2QN", "reviewer": "Reviewer_ACnG", "summary": "In this work, the authors propose a new multi-modal multiple-choice question answering dataset for science questions, with explanations. The dataset covers multiple domains, and has questions with text and image contexts.\n\nThe authors evaluate several baselines in different setups: VQA, fine-tuning UnifiedQA, prompting GPT-3 and find that the explanations provided in the dataset lead to improved accuracy on the dataset, both when using in a fine-tuning context as well as when used in prompting GPT-3.\n", "questions": "Q1: what is the evidence that images are actually necessary for answering the questions in VQA experiments? Do you have questions referring to an image without having textual context that contains all the necessary information? Can you include at least one such example in the paper somewhere because I’m not sure the baby example is clear in this regard?\n\nQ1.a: are questions that require an image to be answered included in the evaluation of text-only models and why does that make sense? How do we know the captions capture the necessary information?\n\nQ2: why no QCM → (L)EA?\n\nQ3: questions (i) - (iii) regarding table 3 (see above)\n\nQ4: what is the variance on the reported numbers?\n\nQ5: Are you always using a caption model ? How is the caption added to the context? How is the caption model trained? What is the effect of including generated captions for your text-based experiments? I think the captioning part should be explained in more detail, and evaluated in an ablation. Are these captions provided in the dataset for use by others?\n\nSmall notes:\n* table 1 columns should be explained\n* l. 82: “multi-model”?\n* l. 97: “for examples”?\n* l. 97-99: sentence incomplete\n* l. 219: “will be stopped” → “is stopped”\n* l. 246: “on SQA so far (75.17%) on SQA”\n* Fig 3,a: the lines are a bit too thick, it is a bit difficult to read this figure\n* Fig 3,b: I think this figure is unnecessary because it repeats what was already mentioned in the text, and in table 2, and takes up a lot of space relative to the contained info (unlike Fig. 3,a for example). I would add the % of questions without any context in Table 2 though.\n", "limitations": "QCM→(L)EA not evaluated and not clear why.\n\nCoT not evaluated with VQA.\n\nThe authors includes an impact statement in appendix.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n* (mostly) well-written\n* new dataset appears to fill a few gaps from existing datasets\n* extensive evaluation with VQA models, as well as text-based finetuned and prompt-based models (UnifiedQA and GPT-3)\n\n## Weaknesses:\n* mediocre originality, not clear what really new questions or lessons\n* confusing evaluation presentation and confusing to use\n* some experimental settings are not explored (QCM→(L)EA)\n* a round of proof-reading is needed to fix some mistakes and improve sentence construction\n* I think it needs renaming because SQA already exists: https://paperswithcode.com/dataset/sqa\n\n## Detailed review:\nThe paper is mostly well-written, and presents an interesting dataset for the QA community, and establishes reasonable baselines in various scenarios. Even though the contribution is technically solid (once a few concerns are addressed), there doesn’t appear much new to be learned from this work. It appears to be an interesting dataset for science QA, in particular because it provides explanations. The problem is rather that it’s easy to dismiss this work as showing that chain-of-thought based training works better with some baselines in a similar set-up to what was already explored for different domains. The addition of the visual modality is unfortunately not tied to CoT experiments and appears as another loose dimension of the dataset.\nSome concerns that I had while reading the work were that the way evaluation results are presented is quite confusing, which could lead to confusion in practical use by others.\nAll main results (VQA and text-based) are crammed into Table 3 and there are several issues: (i) the segments in the table are not labeled (ii) when evaluating VQA models, are all the questions used (even the ones without image context, and does this make sense)? (iii) what do the TXT and IMG columns mean? What about questions without either?\nI think the main evaluation needs more explanation and improved presentation.\nThen, some important settings are missing from evaluation, in particular QCM → (L)EA (generating explanation before answer) hasn’t been included. I think there could be a significant difference between QCM→A(L)E and QCM→(L)EA for auto-regressive decoders because the reasoning steps generated before producing the answer can contribute to predicting the answer more correctly (see also Fig. 5 in [48]).\nIn addition, it would have been nice to provide some numbers for VQA with CoT. The currently tested VQA models are using a classifier to choose between the multiple choices, but it seems a baseline can be made by replacing that with a decoder? I think improvements in VQA due to CoT could also be an interesting additional selling point for this work.\n\nConclusion: if the above concerns regarding evaluation presentation and missing evaluation settings are addressed, I think it’s a solid work that presents an interesting dataset for the QA community. However, I wish there were more interesting new questions raised or new lessons learned.\n\n(Note: I tried to formulate all of these concerns in the questions section as well for easier answering)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RCR_IgwBocc", "reviewer": "Reviewer_f7az", "summary": " - The paper presents a Science Question Answering (SQA) dataset, which contains evidence and explanations in multimodalities and covers a diverse category of questions. \n- Authors evaluate standard baselines as well as chain of thoughts (COT) baselines on the established dataset. Empirical result shows that COT can improve the few-shot performance. Due to the introduction of lectures and explanations annotation in SQA, it is also possible to evaluate COT under the supervised finetuning setting. And it could be shown that COT also helps.\n- Human performance is measured as the upper bound of the task.", "questions": "- What is the motivation to concatenate evidence and explanation after the answer instead of before it?\n- What is the insight that few-shot of GPT-3 is better than finetuned UnifiedQA? It is surprising that a few-shot model outperforms a supervised one, so more analysis is expected.\n- For the questions without any context, how were they handled in the evaluation?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a novel dataset that fills in the gap of the existing datasets as shown in table 1. Lectures and explanations annotation are introduced to facilitate future research on COT.\n- Empirical of SOTA models with and without COT is evaluated to set baseline performance of this dataset\n- The paper is clearly written and easy to follow\n\nWeakness:\n\n- See the question section ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JTuc_wW0p4S", "reviewer": "Reviewer_E8dv", "summary": "This paper aims to aid the development of a reliable model that is capable of generating a coherent chain of thought (CoT) when arriving at the correct answer to reveal the multi-step reasoning process.  Since existing science question datasets either lack annotated explanations for the answers or are restricted to the text modality with small data scales and limited topics, the paper presents Science Question Answering (SQA), a large-scale multi-choice dataset that contains multimodal questions with explanations and features rich domain diversity. It further designs language models to learn to generate lectures and explanations as to the chain of thought to mimic the reasoning process. Experimental results show that CoT benefits large language models in both few-shot and finetuning learning. And analysis shows that CoT helps language models learn from fewer data.\n", "questions": "1. Why aren't all examples annotated with lectures and explanations?\n2. Why are CoTs lectures and explanations instead of explanations? Shouldn't the lecture be part of the model's input? Are there any experimental results for QCML->A and QCML->AE?\n3. In table 3, why does GPT perform worse in the 2-shot setting than in the zero-shot setting?\n", "limitations": "The authors have discussed the limitations of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The presentation is good. The paper is easy to read.\n2. The proposed datasets may aid the development of model reasoning in science QA.\n3. Experiments are extensive and enlightening.\n\nWeaknesses:\n1. The novelty of the proposed dataset and baselines are limited.\n2. The evaluation of the generated CoTs (lectures and explanations) still relies heavily on human judgment. Since the automatic metrics used in the paper are less consistent with human judgment, better automatic evaluation methods specifically for CoT are desired.\n3. The correlation between question answering performance and explanation generation performance is not discussed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I6PJn9oScNg", "reviewer": "Reviewer_M8Qt", "summary": "The paper introduces a new dataset ScienceQA which consists of multiple choice questions with knowledge source and explanations. The dataset covers three domains of elementary through high school questions. The authors also conduct experiments using chain-of-thought methods, which prompts GPT3 and unified QA with a concatenation of question context, answer and lecture+explanation.  ", "questions": "Possible improvements directions: \n1. I think the lectures could be retrieved rather than generated, assume that you have a large documents of lecture contents. Using a retrieval based model to retrieve the lectures and use the GPT-3 models to condition on the lecture and generate explanations, and finally the answers could boost the model performance.  \n\n2. Captioning system that converts images to text might be suboptimal, and could also explain why the accuracy for image questions seems to be significantly lower than the text questions. I think you might want to train end-to-end, or find a system that's focused on image description, rather than image captioning. \n\n3. Report standard deviations in the main table. I think there are high variances for in-context learning, but the table doesn't show it.  ", "limitations": "The authors discussed limitations but in a quite narrow and empirical way, would be nice to see broader, higher-level limitations of this class of methods. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The dataset looks promising, given that it covers both image and text modality and various science domains. \n\nWeakness: \n1. I don't understand why the proposed chain-of-thought (QCM-ALE) would be helpful when you generate answer before generating explanations (especially for the few-shot GPT3 examples). I think this might improve the quality of explanations, but given GPT3 is left-to-right autoregressive, the generated answers cannot take advantage of the generated explanations in the right contexts, which seemingly weakens the power of CoT, that looks like  QCM-LEA or QCM-ELA. \n\n2. Should compare to QCM-LEA or QCM-ELA. I think generating answers conditioned on the explanations would probably yield more performance gains, because the answers can condition on the generated explanations. \n\n3. The method is not novel enough. Other than my concerns (1) that the chain-of-thought is not used in its most powerful form: the chain-of-thought idea is not novel and nor is explanation/rationale generation.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept