Title: Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Benchmark:  SCIENCEQA
Summary: This paper introduces SCIENCEQA, a large-scale multimodal benchmark featuring ~21k multiple-choice science questions (grades 1-12) across diverse topics. Questions can involve text, images, or both. Uniquely, each question is annotated with a detailed explanation comprising relevant background knowledge (lecture) and step-by-step reasoning (solution). The authors propose using language models to generate these explanations as Chains-of-Thought (CoT) and demonstrate that this process significantly improves answer accuracy.
Contribution: Key contributions include: (1) Creating SCIENCEQA, a large (~21k), diverse (multiple science topics, grades 1-12), multimodal benchmark for science QA. (2) Providing detailed, structured explanations (Lecture + Solution) for each question. (3) Proposing and evaluating the generation of these explanations as Chains-of-Thought (CoT) using language models. (4) Demonstrating that training models to generate CoT explanations boosts their answer accuracy on the QA task.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal reasoning; Multi-hop reasoning; Explanation generation (Chain-of-Thought); Science question answering.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to answer science questions by integrating information from text and/or images, performing multi-step reasoning that may involve recalling background knowledge (lecture component) and deriving the answer through logical steps (solution component), and explicitly articulating this process as a Chain-of-Thought explanation.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: To provide a rich, large-scale benchmark for evaluating deeper reasoning and interpretability in multimodal science QA, addressing limitations of previous datasets. To investigate Chain-of-Thought generation as a mechanism for improving reasoning.
Task definition: Given a science question with multimodal context (text and/or image), select the correct multiple-choice answer and generate a detailed textual explanation consisting of a relevant lecture (background knowledge) and a step-by-step solution (reasoning process).
Task item definition: An instance includes: the question text, context (text and/or image URL), multiple-choice options (A-E), the correct answer index, grade level, topic, skills tested, and the gold explanation text structured as {Lecture, Solution}.
Task definition detail: Covers grades 1-12. Topics include Natural Science, Social Science, Language Science. Context can be text-only, image-only, or both. Skills cover various scientific practices. Answers are multiple-choice.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Questions sourced from open curriculum websites. Explanations (Lecture + Solution) were written by human annotators with STEM backgrounds recruited via Upwork, following guidelines and multiple verification rounds.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Uses curriculum-style science questions to evaluate reasoning and explanation abilities relevant to science education and understanding.
Dataset size: Total: 21,208 examples. Test set: 4,241 examples.
Dataset size notes: Train: 12,726 examples. Validation: 4,241 examples. Mini-test/val sets also available.
Dataset metadata available: Yes
Dataset metadata detail: Grade, Topic, Skills, Context Type, Question Text, Options, Answer Index, Lecture Text, Solution Text, Image URL (if applicable).
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Response format detail: Models primarily need to select the correct multiple-choice answer. They can also be trained/evaluated on generating the free-form explanation text.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric definition detail: Answer Accuracy (%). Explanation Quality: BLEU-4, ROUGE-L
Metric aggregation: Simple Mean
Metric statistics: Accuracy (%), BLEU-4, ROUGE-L
Metric subscores: Yes
Metric subscores detail: Performance analyzed by input modality, topic, grade level, and question type. Comparison between models trained with vs. without CoT explanation generation.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Large-scale (~21k) and diverse (subjects, grades, modalities). Sourced from real science curricula. Annotations performed by qualified annotators (STEM backgrounds) with multi-round verification. Structured explanations (Lecture+Solution) provide richer signal. Empirical results demonstrate CoT explanations improve reasoning performance.
Taxonomy: Reasoning / 
New taxonomy: Specific Application / Physical Sciences
Procedural notes: Describes the annotation process via Upwork, including annotator qualifications and payment. Details baseline models (UnifiedQA-v2 T5, GPT-3 text-davinci-002) and experimental setups (fine-tuning, few-shot prompting). Describes the Multimodal Chain-of-Thought (MM-CoT) method. Provides dataset statistics and examples.
Other notes: SCIENCEQA's unique contribution is the large-scale combination of multimodality, diverse science topics/grades, and detailed, structured explanations (Lecture + Solution), making it a rich resource for studying complex reasoning and explanation generation. The demonstration of CoT improving performance is a key finding.