SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark (2024)

Abstract: The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and freeresponse formats, ensuring a comprehensive evaluation of AI models’ abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both opensource and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models.

heading: 1 Introduction
text: In recent years, the evolution of large language models (LLMs) has marked a significant milestone in artificial intelligence. Initially, these models excelled in diverse natural language processing tasks (Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023a,b; OpenAI, 2023; Google, 2023), but their utility has since increasingly expanded, transforming them into incredible agents for various downstream tasks such as reasoning and planning (Li et al., 2023; Wu et al., 2023b; Park et al., 2023; Guo et al.). Notably, LLMs have shown proficiency in tasks that typically pose significant challenges to even highly skilled humans, such as tackling intricate mathematical problems (Lu et al., 2023;
Romera-Paredes et al., 2023) and accelerating scientific discoveries (Birhane et al., 2023). This evolution demonstrates the versatility of LLMs and their potential to revolutionize areas traditionally dominated by human expertise.
Alongside, the rapid development of visionbased LLMs has garnered considerable attention within the AI community, especially with the release of platforms like OpenAI’s GPT4-V (OpenAI, 2023) and Google’s Gemini Ultra (Google, 2023). These models have demonstrated exceptional abilities in tasks requiring advanced reasoning and planning, often surpassing existing benchmarks and approaching human-level performance. This progress has spurred researchers to create more sophisticated and challenging benchmarks for Multimodal LLMs (MLLMs), one of the most representative is the science domain, which is a long-standing focus for humans. For example, the MathVista benchmark (Lu et al., 2023), comprising 6,141 problems, demands a high level of visual understanding and
109
mathematical reasoning. Additionally, the Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark (MMMU) (Yue et al., 2023a) poses college-level multimodal reasoning challenges. Currently, even the most advanced models achieve only about 50% accuracy on these benchmarks. The importance of such benchmarks lies in their role as vital tools for assessing and pushing the boundaries of AI capabilities. By presenting AI models with tasks that mimic complex, real-world scenarios, benchmarks provide a clear measure of progress and highlight areas for future development.
However, in the science domain, a critical observation in multimodal reasoning benchmarks is the disparity in the levels of difficulty. Prior benchmarks like ScienceQA (Lu et al., 2022) primarily focused on elementary and middle-school levels, while MMMU leaps to a college-level challenge. This leaves a significant educational phase in human learning – the high school, or college entrance level – relatively unaddressed. In fact, learning progressively in difficulty levels is not only important for humans, but also can facilitate AI systems including LLMs via curriculum learning (Bengio et al., 2009) and progressive training (Xu et al., 2023; Mitra et al., 2023). Therefore, we fill this gap by introducing a novel benchmark named Science college entrance level Multimodal Question Answering (SceMQA), designed for this critical educational stage, with four key subjects: Mathematics, Physics, Chemistry, and Biology.
Apart from the difficulty level, our benchmark also has a detailed annotation granularity. Firstly, most problems (over 90%) in SceMQA has detailed explanations to solutions except for some straightforward problems. Besides, each problem is associated with a specific knowledge component, facilitating detailed knowledge tracing for models. Moreover, SceMQA uniquely features problems with the same context but different questions. This design is informed by prior research indicating that without diverse question types for each narrative context, models might resort to learning shallow heuristics or patterns rather than developing a deep, semantic understanding (Patel et al., 2021; Yang et al., 2022). This approach ensures a more comprehensive and precise evaluation of reasoning capabilities. In Figure 1, we compare the difficulty level, annotation granularity, and covered modality among existing benchmarks.

heading: 2 Related Work
text: Multimodal Question Answering Multimodal Question Answering (QA) has been a focal area in AI research. The Visual Question Answering (VQA) benchmark (Antol et al., 2015), established in 2015, pioneered free-form, open-ended visual QA, necessitating intricate image comprehension and reasoning. ChartQA (Masry et al., 2022) emphasized complex reasoning about charts, merging visual and logical thought processes. VisIT-Bench (Bitton et al., 2023) tested vision-language models across real-world tasks, ranging from simple recognition to advanced creative generation.
Multimodal LLMs In addition to notable models like GPT4-V and Google Gemini, various opensource Multimodal LLMs (MLLMs) have emerged. MiniGPT-4 (Zhu et al., 2023) improved visionlanguage understanding by syncing a visual encoder with a language LLM. LLaVAR (Zhang et al., 2023b) combined OCR with text-only GPT4 for enhanced visual instruction tuning in textrich image contexts. mPLUG-Owl (Ye et al., 2023) proposed a modular framework for equipping LLMs with multimodal capabilities, focusing on image-text alignment. InstructBLIP (Dai et al., 2023) excelled in vision-language instruction tuning, demonstrating remarkable zero-shot performance in diverse tasks. For a more detailed summary of related studies, please refer to these surveys (Wu et al., 2023a; Yin et al., 2023).
Science Question Answering Various benchmarks have been developed for specific scientific subjects, including MATH (Hendrycks et al., 2021b), MathVista (Lu et al., 2023), chemistry (Guo et al., 2023), etc. More comprehensive science QA benchmarks like ScienceQA (Lu et al., 2022), C-EVAL (Huang et al., 2023), AGIEVAL (Zhong et al., 2023), MMMU (Yue et al., 2023a), and SciBench (Wang et al., 2023b) have recently been introduced, providing a broader scope of assessment.

heading: 3 Our Benchmark SceMQA
text: Our benchmark is designed to bridge a significant gap in existing multimodal benchmarks, which typically span from elementary to college levels, and overlook the crucial high school/college entrance stages. This educational phase is crucial in the human learning process. Although existing benchmarks (Zhong et al., 2023; Zhang et al., 2023a)
incorporate problems at this level, they predominantly feature text-only questions. A comparative analysis of our dataset against existing benchmarks is detailed in Table 1. Although our benchmark appears smaller in total problem count, it focuses specifically on the science domain, offering a substantial average number of problems per subject. Furthermore, it excels in quality, as evidenced by the high proportion of problems accompanied by detailed explanations. The collection and annotation protocol is located in Section A.3. Example problems in our benchmark are shown in the Appendix (Figure 5).
SceMQA has in total 1,045 problems, with an average of 261 problems per subject. Details can be found in Table 2. This set of problems ensures a thorough evaluation across all included subjects.

heading: 4 Experimental Examination of SceMQA
text: In this section, we evaluate the state-of-the-art MLLMs on SceMQA by firstly reporting their answer accuracy across various settings. Additionally, we conduct a detailed error analysis (Section 4.3) and show an accuracy distribution across knowledge categories (Section A.1), which provide significant insights to identify the current MLLMs’ limitations and demonstrate the value of our benchmark in exploring them. We will move those im-
portant experiments to the main body of our paper when we have more space upon paper acceptance.

heading: 4.1 Experimental Settings
text: We choose InstructBLIP (Dai et al., 2023), MiniGPT4 (Zhu et al., 2023) and LLaVa (Liu et al., 2023a) as the open-source MLLM solvers for SceMQA. As for close-sourced models, we focus on three of the most representative MLLMs currently available: Google Bard, Gemini Pro and GPT4-V. Furthermore, we test GPT4-V and Gemini Pro under three distinct settings: zero-shot, fewshot, and text-only. In the zero-shot setting, the models are provided with the problem without any prior examples. The few-shot setting involves giving the models a small number of example problems and solutions to “learn” from, before attempting the new problems. We use hand-crafted textonly problems as examples since it is not flexible to insert multiple images in one API call. The textonly setting is a unique approach under zero-shot where only the textual content of the problem is provided to the model, without any images. All the prompts in our experiments, along with detailed descriptions of each setting, will be available for public view after the paper is accepted.
For the evaluation metric, we have chosen to use exact-match-based accuracy, which is consistent with several prior studies (Lu et al., 2023; Yue et al., 2023a) in this domain. This metric is particularly suitable for our benchmark as both the multiplechoice and free-response problems have definitive, singular correct answers. In the multiple-choice format, this involves selecting the correct option out of the presented choices. For the free-response format, it requires generating an accurate and precise answer, be it a numerical value, a yes/no response, or a specific term for fill-in-the-blank questions. Empirically we use rule-based answer exaction for
multiple choice questions, and GPT4 as evaluators for free response questions.

heading: 4.2 Accuracy for Solving SceMQA
text: The performance of examined MLLMs on SceMQA is presented in Table 3. Foremost, in all evaluated scenarios, the zero-shot GPT4-V consistently outperforms other models. Despite this, the challenge posed by the benchmark remains significant for even the most advanced MLLMs, including GPT4-V and Google Gemini. This parity shows the challenging nature of our benchmark and the necessity for further improving MLLMs’ reasoning capabilities. It can be also observed that the performance of open-sourced models are significantly inferior to close-sourced ones. We have looked into the error cases and found that the both instructionfollowing and reasoning abilities of open-sourced models are not very satisfactory, leaving a huge room for improvement.
Additionally, in the few-shot setting, we noticed an intriguing trend: it underperforms the zero-shot setting. We hypothesize that the few-shot examples, while providing guidance on scientific reasoning, do not enhance the models’ ability to interpret scientific images. This could inadvertently lead the
models to prioritize logical reasoning over critical image interpretation. Also, we can see a significantly lower performance in the text-only setting. This highlights the indispensability of visual information in solving the problems in our benchmark.
Another notable finding is the variation in performance across different subjects. The models perform better in Chemistry and Biology compared to Math and Physics. We infer that this is because Math and Physics often require precise calculations for correct answers, while Chemistry and Biology tend to focus more on conceptual understanding. This pattern suggests that the integration of external computational tools, such as calculators or Python programs, might be beneficial in improving performance on our benchmark, particularly in subjects with extensive calculations like Math and Physics.

heading: 4.3 Error Analysis
text: To delve deeper into the shortcomings of state-ofthe-art MLLMs, we conducted a comprehensive error analysis. We randomly selected 150 instances of errors made by GPT4-V on the SceMQA dataset and enlisted two human experts for a detailed examination. These experts categorized each error into one of six categories: Image Perceptual Errors,

heading: Reasoning Errors, Lack of Knowledge, Rejection to
text: Answer, Annotation Error, and Answer Extraction Error. The inter-rater reliability, assessed using the Kappa agreement score, was found to be greater than 0.5, indicating a moderate level of agreement between the annotators. We then averaged their annotations to determine the proportion of each error type, as depicted in Figure 2. The top-3 error types are shown in Figure 3 and analyzed below:
Reasoning Error The most prevalent error type is categorized under Reasoning Error. It occurs when the model correctly processes image-based information but fails to construct an accurate reasoning chain to arrive at the correct answer. Common mistakes include omitting necessary steps or making incorrect calculations. And we find these errors evenly spread in four subjects in SceMQA, underscoring the need for further development in the reasoning abilities of MLLMs. Drawing on insights from studies on LLMs, approaches such as prompting engineering (Wei et al., 2022) or supervised fine-tuning (Yu et al., 2023; Yue et al., 2023b) might prove beneficial.
Image Perception Error This occurs when the model misinterprets visual information—such as incorrectly reading numbers or coordinates, or failing to differentiate between points in a geometric diagram. This type of error happens more often in the math subject because many math problems require precise diagram or table perception, which suggests that the image perception capabilities of current MLLMs require significant enhancement for precision and interpretation. Incorporation of external tools like OCR, as suggested in studies
like (Liu et al., 2023b), could potentially improve the model’s understanding of visual content.
Lack of Knowledge This type of error arises when the model fails to correctly identify or apply relevant knowledge concepts, such as misusing formulas or misinterpreting theorems. These errors occur more in physics, chemistry and biology, which are indicative of gaps in the model’s learned knowledge base, suggesting that enriching the training datasets of foundation models with diverse and domain-specific knowledge is essential to enhance their expertise in those domains.
Rejection to Answer and Annotation Error Interestingly, a smaller portion of errors were categorized as Rejection to Answer and Annotation Error. Rejection to Answer occurs when the model refuses to provide an answer, possibly due to uncertainty or inability to comprehend the query. Annotation Error, on the other hand, arises from inaccuracies or inconsistencies in the dataset’s annotations, leading to confusion for the model. These categories highlight the importance of robust dataset design and also the need for models to handle ambiguous or complex instructions and questions effectively.
Through this detailed error analysis, we have identified specific patterns and weaknesses of MLLMs’ performance on scientific problems. These findings provide valuable insights and directions for future research aimed at enhancing the capabilities of MLLMs. Addressing these identified issues could lead to significant improvements in the application of MLLMs in educational and research contexts, particularly in the domain of science.

heading: 5 Conclusion
text: In this paper, we introduced SceMQA, a novel multimodal question answering dataset tailored for the college entrance level, including key scientific subjects: mathematics, physics, chemistry, and biology. A standout feature of SceMQA is its high annotation granularity, with over 90% problems accompanied by detailed explanations and associated with specific knowledge points. We conduct extensive experiments including accuracy comparison, error analysis, and category accuracy distribution, employing state-of-the-art MLLMs and highlighting the opportunities and obstacles for multimodal AI models in scientific reasoning.

heading: Limitation
text: Model Comparison Our SceMQA is evaluated on a small number of state-of-the-art MLLMs due to limited computational resources. We plan to evaluate a wider range of models in the future. We will include both open-source models, such as QwenVL (Bai et al., 2023) and CogVLM (Wang et al., 2023a), and closed-source ones like Claude. This comprehensive comparison will provide deeper insights into the capabilities and limitations of those AI models in multimodal scientific reasoning.
Data Scope We will enhance both the depth and breadth of our dataset. In terms of depth, we plan to incorporate more diverse problems within each scientific subject. This will involve adding more complex and varied question types. As for breadth, we aim to extend the range of subjects covered by our dataset beyond the traditional sciences, including more disciplines that are encountered in the human cognitive process.

heading: A Appendix

heading: A.1 Accuracy across Knowledge Points
text: In SceMQA, each problem is associated with a specific knowledge point. The individual accuracy on those knowledge points can be found in Figure 7 and 8. We can observe that the model generally performs better in chemistry and biology than in math and physics. Also, the worst-performed categories of knowledge points are generally related to image understanding (e.g., limits and continuity, optics) or calculation (e.g., one-variable data analysis, integration), which indicate the weaknesses of current MLLMs to some extent.

heading: A.2 Features of SceMQA
text: To evaluate the difficulty of the problems in our benchmark, we utilize GPT-4 to respond to the questions within our dataset, as well as those from both a primary level and a college level benchmark. Figure 4 demonstrates the moderate difficulty level of our benchmark, positioning between the existing benchmark on primary and college levels. The example problems in SceMQA are located in Figure 5, with the following features:
Science Subjects Focusing on the core science subjects such as mathematics, physics, biology, and chemistry, our benchmark aligns with both existing text-only benchmarks, such as SciBench (Wang et al., 2023b), and major human exams like the GaoKao (i.e., Chinese national college entrance

heading: Math Physics Chemistry Biology
text: exam). To effectively address these problems, AI models must demonstrate a robust understanding of images, tables, and diagrams, coupled with deep domain knowledge to recall necessary formulae, theorems, and other elements for advanced reasoning. This presents a suitable challenge for current AI systems, testing their limits in areas typically reserved for advanced human cognition.
Solution Explanation We have meticulously annotated every problem in SceMQA. Almost all solutions (> 90%) are accompanied by detailed, humanverified explanations except for some straightforward solutions, as shown in Figure 5. These explanations are useful for identifying errors in model predictions and could also be instrumental in future supervised fine-tuning (SFT) (Ho et al., 2022; Hsieh et al., 2023) and few-shot prompting methodologies (Wei et al., 2022).
Identified Knowledge Category Additionally, each problem is associated with specific knowledge components within its subject, also shown in Figure 5. The availability of these components aids in building a knowledge state for the evaluated models, facilitating knowledge tracing and under-
standing the depth of the model’s capabilities.
Question Variation Furthermore, our benchmark features a variety of questions based on the same image and context, as shown in Figure 6. Solving such kind of question sets has been demonstrated to be challenging for AI models (Liang and Zhang, 2021), where they usually fail to detect subtle differences among various questions related to the same context (Patel et al., 2021). This onecontext multiple-questions setting can not only test the depth of understanding and reasoning capabilities of these AI models (Patel et al., 2021; Yang et al., 2022) but also have the potential to support advancements in Socratic learning (Shridhar et al., 2022) and interpretable reasoning (Zhang et al., 2021).

heading: A.3 Data Collection Protocol
text: The data for SceMQA was meticulously sourced from publicly available online materials tailored for college entrance level tests in four key subjects: math (including calculus and statistics), biology, physics, and chemistry. In selecting these questions, our team of annotators strictly adhered to the licensing regulations of the source websites, ensuring no copyrighted material was included. This adherence to legal and ethical standards was a priority throughout the data collection process.
For the curation of SceMQA, we specify its intended use to ensure compatibility with the original access conditions. The dataset is designed for academic research and educational technology development. It is not intended for commercial use or outside of research contexts, especially considering that the data is derived from educational resources accessed for research purposes. This specification helps maintain ethical standards and respects the original access conditions of the sourced materials. We also asked annotators to carefully check
whether the data that was collected contained any personal identifier or offensive content and remove them if necessary.
Each problem within our dataset contains one image that is essential for solving the corresponding question, aligning with the multimodal nature of SceMQA. The problems are presented in two formats: multiple-choice and free-response. The multiple-choice questions offer 4 to 5 options, denoted by uppercase letters, a format consistent with other established benchmarks. Following previous studies (Hendrycks et al., 2021a; Lewkowycz et al., 2022), we transform all mathematical expressions into latex codes, making them easy to process for LLMs, as shown in Figure 5 and 6.
The free-response section includes calculationbased problems where answers are numerical values. This format is particularly advantageous for evaluation purposes, as the correctness of modelgenerated answers can be straightforwardly deter-
mined by checking the final numerical value. This approach is in line with other benchmarks like GSM8k, SciBench, and MMMU. Besides calculations, our benchmark diversifies with other freeresponse types like Yes-or-No and fill-in-the-blank questions. These formats not only broaden the range of question types but also maintain ease of evaluation through exact matching. Given these characteristics, accuracy will be the primary metric for assessing performance on our benchmark.
In terms of data features, each problem was thoroughly reviewed by annotators to ensure it aligned with the intended high school and pre-college difficulty level. Moreover, every problem is accompanied by a clear explanation of the answer and is tagged with the main knowledge point from predefined knowledge sets. These annotations and categorizations have been verified by domain experts, ensuring that each problem accurately reflects the intended educational content and difficulty.

heading: Future Work
text: These explanations are useful for identifying errors in model predictions and could also be instrumental in future supervised fine-tuning (SFT) (Ho et al. , 2022; Hsieh et al. , 2023) and few-shot prompting methodologies (Wei et al. , 2022). Identified Knowledge Category Additionally, each problem is associated with specific knowledge components within its subject, also shown in Figure 5. The availability of these components aids in building a knowledge state for the evaluated models, facilitating knowledge tracing and under-
standing the depth of the model’s capabilities. Question Variation Furthermore, our benchmark features a variety of questions based on the same image and context, as shown in Figure 6. Solving such kind of question sets has been demonstrated to be challenging for AI models (Liang and Zhang, 2021), where they usually fail to detect subtle differences among various questions related to the same context (Patel et al. , 2021). This onecontext multiple-questions setting can not only test the depth of understanding and reasoning capabilities of these AI models (Patel et al. , 2021; Yang et al. , 2022) but also have the potential to support advancements in Socratic learning (Shridhar et al. , 2022) and interpretable reasoning (Zhang et al. , 2021)