Title: SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
Benchmark: SceMQA (Scientific College Entrance Level Multimodal Question Answering)
Summary: This paper introduces SceMQA, a multimodal question answering benchmark focused on science subjects (Math, Physics, Chemistry, Biology) at the college entrance level. It aims to fill the difficulty gap between primary/middle school and college-level benchmarks. SceMQA includes multiple-choice and free-response questions, detailed solution explanations, and knowledge point labels. Evaluation of current MLLMs shows performance around 50-60% accuracy, indicating the benchmark's challenge.
Contribution: Key contributions include: (1) Creating the SceMQA benchmark targeting the underrepresented college entrance difficulty level for multimodal science QA. (2) Providing high-quality annotations, including detailed solution explanations and specific knowledge points for most problems. (3) Incorporating problems with varied questions for the same context to robustly assess reasoning. (4) Benchmarking several SOTA MLLMs and analyzing their performance and error types.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Scientific reasoning; Multimodal question answering capability at the college entrance level.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to perform scientific reasoning across core subjects (Math, Physics, Chemistry, Biology) at a level typical of college entrance examinations, requiring the integration and comprehension of both textual descriptions and visual information (images, diagrams, graphs) to answer questions.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark specifically aims to address the gap in difficulty level between existing primary/middle school and college-level multimodal science datasets. It provides detailed explanations and knowledge points to facilitate finer-grained analysis of model capabilities.
Task definition: Given a scientific problem presented multimodally (text and potentially an image), answer a related question corresponding to college entrance-level difficulty in Math, Physics, Chemistry, or Biology. The answer format is either multiple-choice or free-response.
Task item definition: A problem instance consists of a textual description/question, often accompanied by an essential image (diagram, graph, etc.), potential multiple-choice options, the correct answer, a detailed solution explanation (for >90% of items), and one or more specific knowledge point labels.
Task definition detail: Covers 4 science subjects at the college entrance level. Uses a mix of multiple-choice (4-5 options) and free-response (numerical, yes/no, fill-in-the-blank) formats.
Some instances feature the same context with different questions.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: Problems were collected by annotators from public online materials intended for college entrance exams, respecting licenses. Domain experts reviewed problems for difficulty level and verified annotations (explanations, knowledge points).
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The benchmark uses questions formatted like standardized test items to assess scientific reasoning expected for college admission.
Dataset size: 1,045 problems total. Test set size is not applicable as no standard split is defined; evaluation appears to use the entire set.
Dataset metadata available: Yes
Dataset metadata detail: Subject (Mathematics, Physics, Chemistry, Biology), Problem Format (Multiple Choice, Free Response), Knowledge Point(s), Solution Explanation.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Response format detail: Answers are either a single letter choice (MCQ) or a short, specific free text answer (number, word, yes/no).
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Accuracy based on exact match. Rule-based normalisation or GPT-4 evaluation was used for free-response answers in the paper's experiments.
Metric aggregation: Simple Mean
Metric statistics: Accuracy (%). Kappa score used for error analysis inter-rater reliability.
Metric subscores: Yes
Metric subscores detail: Performance reported per subject (Math, Physics, Chemistry, Biology). Results separated for Multiple Choice vs. Free Response formats. Accuracy distribution across specific knowledge points is analyzed (in appendix). Performance compared across zero-shot, few-shot, and text-only settings.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: No, no link is provided
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Problems sourced from college entrance materials. Difficulty level aligned with high school/pre-college, filling a gap. Annotations (explanations, knowledge points) verified by domain experts. Problem selection required image essentiality. Difficulty confirmed by comparing GPT-4 performance to its performance on primary-level (ScienceQA) and college-level (MMMU) benchmarks. Error analysis conducted by human experts.
Taxonomy: General Science / 
New taxonomy: Specific Application / Physical Sciences
Procedural notes: Data sourced from public online college entrance materials. Adherence to licenses checked. Mathematical expressions converted to LaTeX. Exact-match accuracy used, with GPT-4 as evaluator for free-response questions in experiments. Error analysis involved 2 experts and Kappa score. Detailed prompts planned for release.
Other notes: The benchmark's specific focus on the college entrance difficulty level and its high annotation quality (explanations, knowledge points) are key distinguishing features. The finding that few-shot prompting did not improve, and even slightly hurt performance compared to zero-shot for GPT-4V/Gemini Pro, is unusual and suggests potential negative interference from the text-only examples used in the prompts.