VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning

## Abstract

Abstract Multi-modal large language models (MLLMs) have demonstrated promising capabilities across various tasks by integrating textual and visual information to achieve visual understanding in complex scenarios. Despite the availability of several benchmarks aims to evaluating MLLMs in tasks from visual question answering to complex problem-solving, most focus predominantly on mathematics or general visual understanding tasks. This reveals a critical gap in current benchmarks, which often overlook the inclusion of other key scientific disciplines such as physics and chemistry. To address this gap, we meticulously construct a comprehensive benchmark, named VisScience, which is utilized to assess the multi-modal scientific reasoning across the three disciplines of mathematics, physics, and chemistry. This benchmark comprises 3,000 questions drawn from K12 education — spanning elementary school through high school — equally distributed across three disciplines, with 1,000 questions per discipline. The questions within VisScience span 21 distinct subjects and are categorized into five difficulty levels, offering a broad spectrum of topics within each discipline. With VisScience, we present a detailed evaluation of the performance of 25 representative MLLMs in scientific reasoning. Experimental results demonstrate that closed-source MLLMs generally outperform open-source models. The best performance observed include a 53.4% accuracy in mathematics by Claude3.5-Sonnet, 38.2% in physics by GPT-4o, and 47.0% in chemistry by Gemini-1.5-Pro. These results underscore the strengths and limitations of MLLMs, suggesting areas for future improvement and highlighting the importance of developing models that can effectively handle the diverse demands of multi-modal scientific reasoning.

## 1 Introduction

Figure: Figure 1: The accuracies of representative MLLMs on VisScience across different subjects and difficulty levels. (Left) The accuracies on different subjects. (Right) The accuracies on various difficulty levels.
Refer to caption: /html/2409.13730/assets/x1.png

Recently, large language models (LLMs) [, , , , , , , , ] have demonstrated remarkable capabilities across a wide range of tasks, including natural language understanding, text generation, and complex problem solving. The success of LLMs facilitates the development of multi-modal large language models (MLLMs) [, , , , , , ], which extends these capabilities by integrating the ability to process and analyze both textual and visual information. Evaluation is a significant component in assessing the ability of these MLLMs across various tasks, which has attracted widespread attention and developed rapidly in recent years. For instance, several benchmark datasets are proposed to evaluate the ability of MLLMs in general visual understanding, including MME [], MMMU [], MMBench [], MMStar [], and SEED-Bench [].

As a primary evaluation domain, mathematical reasoning presents specific challenges, requiring models to handle complex mathematical problems accompanied by visual information. Previous works [, , ] focus on geometric problems, resulting in the emergence of various evaluation datasets such as GeoQA [], Geometry3K [], and UniGeo []. Subsequently, several benchmark datasets [, , ] extend the scope of mathematical reasoning beyond geometry to encompass various branches such as arithmetic, algebraic, statistics, logic, and functions. Notably, MathVista also contains a portion of scientific datasets such as TQA [], SciBench [], and ScienceQA []. However, despite these advancements, there remains some issues:

- •
Existing benchmarks often focus narrowly on specific mathematics, neglecting other crucial scientific disciplines like physics and chemistry.
- •
Existing benchmarks are often collected from limited sources, resulting in a lack of natural difficulty levels and leading to an incomplete evaluation of models’ capabilities.
- •
Current benchmarks are predominantly available in a single language, limiting the evaluation of MLLMs’ multilingual capabilities.

To address the limitations of existing benchmarks and provide a more comprehensive evaluation benchmark, we introduce a more expansive evaluation benchmark, named VisScience, integrating both textual and visual information. This benchmark is designed to assess the performance of MLLMs in multi-modal scientific reasoning tasks across disciplines like physics and chemistry alongside mathematics. To construct this benchmark, we gather a total of 450,000 questions from K12 education and meticulously select 3,000 questions as the final dataset, with each discipline containing 1,000 questions. This benchmark spans a comprehensive range of knowledge points across different chapters, with difficulty levels ranging from 1 to 5, ensuring that models are assessed on both basic and challenging problems.

In order to better understand MLLMs’ performance on more detailed subjects within three disciplines, we categorize VisScience into several subjects across each discipline. Specifically, we divide the mathematical part of VisScience into six subjects such as plane geometry, solid geometry, functions and equations, algebraic operations, probability and statistics, and combinatorial mathematics. For physics, the dataset is categorized as eight subjects, including mechanics, thermodynamics, comprehensive experiments and methods, mechanical motion, vibration and waves, optics, electromagnetism, and modern physics. The chemistry section includes seven topics such as chemical experiments, organic chemistry, material composition, electrochemistry, chemical reactions, inorganic chemistry, and chemical equilibrium. In summary, VisScience contains 21 subjects across the three disciplines of mathematics, physics, and chemistry.

We conduct extensive experiments on VisScience to evaluate the scientific reasoning abilities of 25 representative MLLMs. These models include close-source LLMs, close-source and open-source MLLMs, offering a comprehensive analysis of their performance across various disciplines. As illustrated in Figure [1](#S1.F1), the best performance is observed in close-source MLLMs, with distinct models excelling in different disciplines. In specific, Claude3.5-Sonnet achieves an accuracy of 53.4% in mathematics, GPT-4o reaches a 38.2% accuracy in physics, and Gemini-1.5-Pro records an accuracy of 47.0% in chemistry. Among open-source models, InternVL-1.2-Plus performs best with accuracies of 30.1% in mathematics, 24.8% in physics, and 31.2% in chemistry. Lastly, we systematically analyze the errors made by advanced models like GPT-4o on VisScience, which providing valuable insights into the specific domains where these models excel and where they struggle.

## 2 VisScience Dataset

In this section, we first illustrate the overview of our specially curated VisScience benchmark, designed to assess the capabilities of MLLMs in multi-modal scientific reasoning. Next, we introduce data generation process, which encompasses three core scientific disciplines: mathematics, physics, and chemistry. Lastly, we discuss the difference between our VisScience benchmark and existing benchmarks.

### 2.1 Overview

We introduce the VisScience benchmark, a meticulously curated collection aimed at evaluating the capabilities of multi-modal large language models (MLLMs) in multi-modal scientific reasoning, with a particular focus on bilingual tasks involving both English and Chinese. This dataset incorporates textual and visual contexts as inputs and spans three scientific disciplines, including mathematics, physics, and chemistry. Each discipline comprises 1,000 questions, meticulously gathered from different chapters to ensure comprehensive coverage of topics and concepts. The core statistics of the VisScience benchmark are presented in Table [1](#S2.T1). The distributions of question length in VisScience are provided in Appendix [A.1](#A1.SS1).

In mathematics, the dataset can be divided into six key areas: plane geometry, solid geometry, functions and equations, algebraic operations, probability and statistics, and combinatorial mathematics. The physical component of the VisScience benchmark encompasses eight subjects, including mechanics, thermodynamics, comprehensive experiments and methods, mechanical motion, vibration and waves, optics, electromagnetism, and modern physics. The chemistry section of the VisScience benchmark includes seven topics such as chemical experiments, organic chemistry, substance composition, electrochemistry, chemical reactions, inorganic chemistry, and chemical equilibrium. A detailed introduction of each subjects within the three disciplines is available in Appendix [A.2](#A1.SS2).

The primary objective of the VisScience benchmark is to provide a rigorous and diverse benchmark for assessing the multi-modal scientific reasoning capabilities of MLLMs. This benchmark aims to supplement existing benchmarks that predominantly focus on mathematical reasoning by broadening the scope to include expansive domains such as mathematics, physics, and chemistry. This benchmark aims to supplement existing benchmarks that predominantly focus on mathematical reasoning by broadening the scope to include expansive domains such as mathematics, physics, and chemistry. Through this enhancement, VisScience seeks to provide a more holistic measure of MLLMs’ abilities across a wider spectrum of scientific disciplines.

**Table 1: Key statistics of VisScience.**
| Statistic | Number |
| --- | --- |
| Total questions | 3000 |
| - multiple-choice questions | 2,053 (68.4%) |
| - Free-form questions | 947 (31.6%) |
| Number of categories of math questions | 6 |
| Number of categories of physics questions | 8 |
| Number of categories of chemistry questions | 7 |
| Number of difficulty levels | 5 |
| Unique number of images | 3,000 |
| Unique number of questions | 3,000 |
| Unique number of answers | 1,427 |
| Statistics with Chinese Language |  |
| Maximum question length | 1297 |
| Maximum answer length | 112 |
| Maximum choice number | 5 |
| Average question length | 162.85 |
| Average answer length | 20.93 |
| Statistics with English Language |  |
| Maximum question length | 418 |
| Maximum answer length | 92 |
| Maximum choice number | 5 |
| Average question length | 80.93 |
| Average answer length | 12.30 |

### 2.2 Data Generation

The goal of the VisScience benchmark is to establish a comprehensive, bilingual (Chinese and English) benchmark for evaluating the capabilities of MLLMs in processing and understanding complex, scientifically-oriented tasks across various disciplines. In order to achieve this goal, we present a two-stage data generation pipeline to meticulously construct a benchmark dataset comprising 3,000 questions, evenly distributed with 1,000 questions each in the fields of mathematics, physics, and chemistry. Figure [2](#S2.F2) shows some examples sampled from the VisScience benchmark across three disciplines: mathematics, physics, and chemistry. More cases in VisScience are provided in Appendix [B](#A2).

Data Collection. We gather a total of 450,000 questions from the disciplines of mathematics, physics, and chemistry, each enriched with visual information sourced from K12 education. This collection spans a comprehensive range of knowledge points across different chapters, with the difficulty levels scaled based on education grade. Consequently, we cluster 150,000 questions per discipline and carefully select 1,000 representative questions. These questions exemplify a range of difficulty levels and a variety of subjects, guided by the following principles:

- •
Guaranteeing every knowledge point is included in VisScience benchmark.
- •
Prioritizing the selection of questions from high-frequency knowledge points.
- •
Ensuring a mixture of questions across various difficulty levels.

In the end, the VisScience benchmark is constructed with 3,000 questions, with each of the three disciplines – mathematics, physics, and chemistry – contributing 1,000 questions. This approach ensures that the benchmark comprehensively covers a wide array of topics within each discipline, reflecting the breadth and depth required for a thorough assessment of MLLMs’ capabilities.

Data Annotation. To improve the quality of the VisScience benchmark, we conduct multiple checks using both manual reviews and LLM assessments to confirm the completeness of each question. For textual content, we check for accuracy, coherence and relevance, ensuring that each question aligns with the corresponding scientific discipline and is free of ambiguities. For associated visual content, we rigorously screen out images that are incorrect, unclear, or lacking in detail, retaining only those that are clear and richly informative. To maintain the volume of the VisScience benchmark, we compensate for questions removed due to incomplete information by selecting new questions on identical topics from the original dataset. This approach ensures that the overall number of questions and the breadth of content coverage are consistently maintained. This verification process guarantees that both the textual and visual components of the VisScience benchmark is a reliable and effective tool for evaluating the capabilities of MLLMs in scientific reasoning.

Figure: Figure 2: Examples of the VisScience benchmark comprising three disciplines: mathematics, physics, and chemistry.
Refer to caption: /html/2409.13730/assets/x7.png

### 2.3 Data Analysis

We utilize statistical analysis to assess subject distributions and difficulty levels within the VisScience benchmark. Figure [3](#S2.F3) presents a visual representation of the categorization of question within the VisScience benchmark. This illustration shows the distribution of questions dedicated to each subject area – mathematics, physics, and chemistry – and details the distribution across various difficulty levels, ranging from 1 to 5.

Figure: Figure 3: The distribution of detailed subjects and difficulty levels in the each discipline within the VisScience benchmark. (Left) The distributions of various subjects. (Right) The distributions of difficulty levels.
Refer to caption: /html/2409.13730/assets/x8.png

Subject Distributions. To categorize each discipline into more detailed subjects, we first utilize LLM to segment the overall discipline into specific topics based on knowledge points and terminologies presented in the questions. Subsequently, we conduct a manual review of these categories to confirm its rationality and appropriateness, ensuring that each question is accurately categorized. As shown in Figure [3](#S2.F3), the mathematical part of the VisScience benchmark is divided into six subjects, i.e., plane geometry (43.6%), algebraic operations (15.4%), functions and equations (12%), probability and statistics (11.8%), solid geometry (10.9%), and combinatorial mathematics (6.3%). Furthermore, the distributions for physics and chemistry disciplines are presented in the figure, providing a comprehensive overview of the scope of the VisScience benchmark within these scientific fields.

Difficulty Levels. To classify the questions into distinct difficulty levels, we first utilize LLM for the initial sorting, and then conduct a manual verification. The questions within each discipline are stratified into five difficulty levels ranging from 1 to 5, defined as follows: Basic, Easy, Intermediate, Advanced, and Expert. Figure [3](#S2.F3) shows the distribution of difficulty levels, providing a visual representation of the distribution of questions across different difficulty levels. Each discipline demonstrates a unique profile of topic distribution across the difficulty levels. For instance, in the field of mathematics, plane geometry is classified at the intermediate level, algebraic operations are positioned at the basic level, and functions and equations appears at the highest difficulty level, reflecting their various placement within educational curricula. In physics, mechanics dominates the introductory level, which belongs to a fundamental concept in physics education. Electromagnet is positioned at the highest difficulty level, demanding the application of various advanced knowledge points. In the discipline of chemistry, organic chemistry and chemical equilibrium represent the pinnacle of K12 chemical education, requiring deep conceptual understanding and the ability to apply knowledge to complex scenarios.

### 2.4 Comparison with Other Benchmarks

We compare the VisScience benchmark with 5 existing benchmarks, including MathVista [], Math-Vision [], CMMMU [], ScienceQA [], and SciBench [].

VisScience vs MathVista. MathVista is a comprehensive multi-modal benchmark for mathematical reasoning, comprising data from 28 existing datasets and 3 newly collected datasets. In MathVista, the majority of questions are annotated after collecting images, which results in a certain homogeneity within the data. In contrast, VisScience directly collects its questions from K12 education, featuring an average question length of 80.93 words. Such questions provide more contextual information, which facilitate a more thorough evaluation of the models’ reasoning capabilities. Unlike MathVista that encompasses only seven subjects within mathematics, VisScience offers a far broader scope, including 22 distinct subjects across mathematics, physics, and chemistry. Furthermore, VisScience distinguishes itself by being a bilingual benchmark, including both Chinese and English versions of questions. This feature is particularly advantageous as it assesses MLLMs’ capabilities in scientific reasoning across different languages.

VisScience vs Math-Vision. Math-Vision is a mathematics benchmark derived from 19 competitions, covering 16 topics across 5 levels of difficulty. Different from Math-Vision that collected from competitions, VisScience spans a broader educational spectrum, incorporating a natural gradient of difficulty from elementary school to high school. Furthermore, VisScience extends beyond mathematics to include questions from physics and chemistry, significantly broadening its scope and applicability. While Math-Vision primarily focuses on the unique challenges of competitive environments, VisScience is grounded in real-world educational settings.

VisScience vs CMMMU. CMMMU comprises 12,000 manually collected multi-modal questions from university exams, quizzes, and textbooks, which covers 6 core subjects and 30 specific fields. Similar to VisScience, CMMMU is a bilingual benchmark, offering questions in both Chinese and English. Within this dataset, only 1,601 questions are dedicated to the disciplines of mathematics, physics, and chemistry, accounting for only 13.34% of the total dataset. VisScience features a total of 3,000 questions, significantly outnumbering those in CMMMU dedicated to the same subjects. The questions in CMMMU are set at the university level, characterized by high difficulty, demanding that the model possesses substantial professional domain knowledge and expert-level reasoning abilities. In contrast, VisScience comes from K12 education, with a broader range of difficulty. This range allows VisScience to more comprehensively evaluate MLLMs’ capabilities across different educational stages.

VisScience vs ScienceQA. ScienceQA is a newly developed benchmark featuring approximately 21,000 multimodal multiple-choice questions across a variety of science topics. In the ScienceQA dataset, 30.8% of questions incorporate both image and text contexts, providing a multimodal benchmark to test MLLMs in scientific reasoning. The questions in ScienceQA have an average length of only 12.11 words. In contrast, VisScience also serves as a benchmark for evaluating the scientific reasoning abilities of MLLMs, but it typically features longer and more textually detailed questions. Specifically, the Chinese version of VisScience has an average question length of 162.85 words, providing a more comprehensive and intricate testing ground for evaluating the depth of detailed reasoning in MLLMs. Additionally, VisScience contains mathematical problems, further enriching the benchmark’s scope by testing MLLMs on their mathematical problem solving capabilities alongside their scientific reasoning.

VisScience vs SciBench. SciBench is a benchmark developed to evaluate the reasoning capabilities of LLMs in solving collegiate-level scientific problems within the domains of mathematics, chemistry, and physics. The majority of the data in SciBench focuses on assessing the scientific reasoning of LLMs, it only includes 177 problems that incorporate visual elements to evaluate the performance of MLLMs. In contrast, VisScience is primarily focused on multimodal scientific reasoning, covering similar subjects such as mathematics, chemistry, and physics. VisScience differentiates itself by offering a more comprehensive range of difficulty levels and subjects, making it a broader benchmark for assessing the capabilities of MLLMs in scientific reasoning.

## 3 Experiments

In this section, we conduct experiments to evaluate a variety of MLLMs using the VisScience benchmark. The evaluation encompasses both close-source and open-source models, enabling a comprehensive analysis of their effectiveness in scientific reasoning. Besides, we provide a detailed error analysis of the advanced model GPT-4o.

### 3.1 Experimental Setup

Models. We conduct our evaluation across a diverse array of models, including close-source text-only large language models (LLMs), close-source multi-modal large language models (MLLMs), and open-source MLLMs. This comprehensive assessment covers more than 20 models, which are listed below. The sources of models is reported in Appendix [C.1](#A3.SS1).

- •
Close-source text-only LLMs: ChatGPT [], GPT-4 [], Claude2 [].
- •
Close-source MLLMs: Gemini-1.0-Pro [], Gemini-1.5-Pro [], GPT-4o [], Qwen-VL-Max [], Qwen-VL-Plus [], Claude3.5-Sonnet [], Claude3-Opus [], GLM-4V [], and Step-1V [].
- •
Open-source MLLMs: mPLUG-Owl [], LLaMA-Adapter-V2 [], MiniCPM-Llama3-V2.5 [], LLaVA-1.5 [], DeepSeek-VL [], ShareGPT4V [], SPHINX-Plus [], InternLM-XC2 [], InternVL-1.2-Plus [], InternVL-Chat-V1.5 [], CogVLM [], CogVLM2 [], and GLM-4V-9B [].

Evaluation Details. The evaluation process is conducted through two steps: generation and judgment. During the generation phase, the models are tasked with producing responses based on a set of questions. For zero-shot setting, we directly prompt the models with these questions without any examples. For 2-shot Chain of Thought (CoT) setting, we provide the models with two relevant examples before they are prompted with the questions. For MLLMs, we supply the models with the textual questions and the corresponding image to obtain their responses. During the judgment phase, we utilize GPT-4o to evaluate the models’ responses by comparing them with the standard answers to assess consistency. This phase involves calculating the accuracy across different subjects and levels. The prompts used in two phases is defined in Appendix  [C.2](#A3.SS2).

### 3.2 Experimental Results

Overall Results. Table [2](#S3.T2) demonstrates the performance of several models on VisScience within the version of the Chinese language. Experimental results show that the close-source models achieves best performance on VisScience. Specifically, Claude3.5-Sonnet achieves an accuracy of 53.4% in mathematics, GPT-4o attains a 38.2% accuracy in physics, and Gemini-1.5-Pro accomplishes an accuracy of 47.0% in chemistry. Among open-source models, InternVL-1.2-Plus stands out, demonstrating robust capabilities across various scientific disciplines with accuracies of 30.1% in mathematics, 24.8% in physics, and 31.2% in chemistry. Despite this, there is a notable disparity in performance between close-source and open-source models, with close-source models generally exhibiting superior performance. The performance of InternVL-1.2-Plus, although trailing behind the advanced close-source models such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, showing significant potential for improvement. Notably, the performance in physics underscores unique challenges that necessitate targeted improvements in model training. This discipline often involves the interpretation of conceptual and numerical data, challenging the reasoning and computational abilities of MLLMs. As evidenced in Table [2](#S3.T2), even advanced models like GPT-4o achieve relatively lower accuracies in physics compared to other disciplines. Results on VisScience with the version of the English language are provided in Appendix [D.1](#A4.SS1).

**Table 2: Results on VisScience within the version of the Chinese language across the disciplines of mathematics, physics, and chemistry. For input, $Q$ represents for question, $I$ represents for image. The highest scores among close-source and open-source models are highlighted in red and blue, respectively.**
| Model | LLM | Input | Mathematics | Physics | Chemistry |
| --- | --- | --- | --- | --- | --- |
| Close Source Models (APIs) |  |  |  |  |  |
| Text-only LLMs |  |  |  |  |  |
| Zero-shot ChatGPT | - | Q | 22.4 | 22.7 | 18.6 |
| Zero-shot GPT-4 | - | Q | 25.9 | 30.4 | 33.1 |
| Zero-shot Claude-2 | - | Q | 27.3 | 22.0 | 24.4 |
| Zero-shot Claude3-Opus | - | Q | 29.3 | 30.8 | 32.5 |
| Zero-shot Claude3.5-Sonnet | - | Q | 29.7 | 35.3 | 36.9 |
| Zero-shot GPT-4o | - | Q | 31.1 | 38.0 | 39.6 |
| 2-shot CoT Claude2 | - | Q | 27.8 | 21.7 | 23.9 |
| 2-shot CoT ChatGPT | - | Q | 20.2 | 18.6 | 21.3 |
| 2-shot CoT GPT-4 | - | Q | 32.1 | 31.5 | 32.4 |
| Multi-modal LLMs |  |  |  |  |  |
| Gemini-1.0-Pro | - | Q, I | 26.6 | 23.70 | 27.8 |
| Gemini-1.5-Pro | - | Q, I | 49.4 | 38.1 | 47.0 |
| GPT-4o | - | Q, I | 51.7 | 38.2 | 41.6 |
| GPT-4o-mini | - | Q, I | 42.6 | 29.8 | 28.4 |
| Qwen-VL-Max | - | Q, I | 35.5 | 30.70 | 42.5 |
| Qwen-VL-Plus | - | Q, I | 27.6 | 26.5 | 37.7 |
| Claude3.5-Sonnet | - | Q, I | 53.4 | 38.0 | 43.1 |
| Claude-3 opus | - | Q, I | 34.4 | 31.1 | 34.1 |
| GLM-4V | - | Q, I | 24.2 | 19.2 | 25.0 |
| Step-1V | - | Q, I | 28.1 | 23.5 | 25.0 |
| Open Source Models |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |
| mPLUG-Owl | LLaMA-7B | Q, I | 7.6 | 8.3 | 9.5 |
| LLaMA-Adapter-V2 | LLaMA-7B | Q, I | 9.6 | 10.3 | 10.8 |
| MiniCPM-Llama3-V2.5 | LLaMA3-8B | Q, I | 15.4 | 17.9 | 19.5 |
| LLaVA-1.5 | Vicuna-13B | Q, I | 15.5 | 15.2 | 18.8 |
| LLaVA-1.5 | Vicuna-7B | Q, I | 13.0 | 13.5 | 16.0 |
| DeepSeek-VL | DeepSeek-LLM-7B | Q, I | 8.3 | 16.8 | 21.0 |
| ShareGPT4V | Vicuna-7B | Q, I | 15.7 | 14.0 | 19.0 |
| ShareGPT4V | Vicuna-13B | Q, I | 16.4 | 14.9 | 18.4 |
| SPHINX-Plus | LLaMA2-13B | Q, I | 17.0 | 15.3 | 20.4 |
| InternLM-XC2 | InternLM2-7B | Q, I | 24.9 | 18.3 | 25.6 |
| InternVL-1.2-Plus | Nous-Hermes-2-Yi-34B | Q, I | 30.1 | 24.8 | 31.2 |
| InternVL-Chat-V1.5 | Mixtral 8*7B | Q, I | 26.9 | 20.8 | 23.7 |
| CogVLM | Vicuna-7B | Q, I | 16.7 | 14.5 | 17.0 |
| CogVLM2 | LLaMA-3-8B | Q, I | 23.2 | 14.4 | 21.0 |
| GLM-4V-9B | GLM-4-9B | Q, I | 24.7 | 19.3 | 22.5 |

Results on Mathematics Across Different Subjects. The mathematical part of VisScience encompasses a wide range of subjects, including plane geometry, solid geometry, functions and equations, algebraic operations, probability and statistics, and combinatorial mathematics. Table [3](#S3.T3) reports the comprehensive results across different mathematical subjects. It is evident that models like Claude3.5-Sonnet and GPT-4o in close-source MLLMs excel across multiple subjects, particularly in functions and equations, probability and statistics, and algebraic operations. Conversely, open-source models show a more varied performance with notable strengths in certain areas but generally lower scores compared to close-source models. For instance, InternVL-1.2-Plus and InternVL-Chat-V1.5 perform relatively well in plane geometry, and functions and equations. These detailed performance on different subjects provide valuable insights into the specific strengths and weaknesses of various MLLMs. Additionally, results on physics and chemistry across different subjects are presented in Appendix [D.2](#A4.SS2) and Appendix [D.3](#A4.SS3), respectively. Case studies illustrating correct responses by MLLMs can be found in Appendix [E](#A5).

**Table 3: Results on the mathematical part of VisScience across different subjects. Subjects: PlaneG: plane geometry, SolidG: solid geometry, Fun: functions and equations, Alg: algebraic operations, Stat: probability and statistics, Comb: combinatorial mathematics. The highest scores among close-source and open-source models are highlighted in red and blue, respectively.**
| Model | Mathematics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ALL | PlaneG | SolidG | Fun | Alg | Stat | Comb |  |
| Close Source Models (APIs) |  |  |  |  |  |  |  |
| Text-only LLMs |  |  |  |  |  |  |  |
| Zero-shot ChatGPT | 22.40 | 20.18 | 11.93 | 18.33 | 13.63 | 15.25 | 26.98 |
| Zero-shot GPT-4 | 25.90 | 30.73 | 18.35 | 28.33 | 17.53 | 24.58 | 33.33 |
| Zero-shot Claude-2 | 27.30 | 27.06 | 25.69 | 25.83 | 31.17 | 31.36 | 25.40 |
| Zero-shot Claude3-Opus | 29.30 | 30.28 | 21.10 | 32.50 | 27.27 | 34.75 | 31.75 |
| Zero-shot Claude3.5-Sonnet | 29.70 | 33.94 | 15.60 | 33.33 | 27.27 | 27.12 | 34.92 |
| Zero-shot GPT-4o | 31.10 | 36.24 | 24.77 | 35.83 | 25.32 | 24.58 | 31.75 |
| 2-shot CoT Claude2 | 27.80 | 30.05 | 26.61 | 25.00 | 28.57 | 27.97 | 26.98 |
| 2-shot CoT ChatGPT | 20.20 | 23.17 | 20.18 | 19.17 | 17.53 | 22.88 | 14.29 |
| 2-shot CoT GPT-4 | 32.10 | 37.16 | 31.19 | 28.33 | 22.08 | 30.51 | 38.10 |
| Multi-modal LLMs |  |  |  |  |  |  |  |
| Gemini-1.0-Pro | 26.60 | 24.08 | 22.02 | 23.73 | 35.71 | 29.66 | 34.92 |
| Gemini-1.5-Pro | 49.40 | 48.74 | 33.03 | 47.06 | 61.69 | 55.93 | 52.38 |
| GPT-4o | 51.70 | 48.17 | 44.04 | 57.50 | 68.18 | 56.78 | 41.27 |
| GPT-4o-mini | 42.60 | 41.28 | 29.36 | 44.17 | 54.55 | 44.92 | 38.10 |
| Qwen-VL-Max | 35.50 | 34.86 | 27.52 | 35.83 | 50.00 | 33.05 | 26.98 |
| Qwen-VL-Plus | 27.60 | 27.98 | 18.35 | 29.17 | 31.17 | 35.59 | 20.63 |
| Claude3.5-Sonnet | 53.4 | 50.23 | 35.78 | 57.50 | 74.03 | 63.56 | 39.68 |
| Claude3-Opus | 34.40 | 35.31 | 24.77 | 29.17 | 45.45 | 35.59 | 31.75 |
| GLM-4V | 24.20 | 28.57 | 30.28 | 22.50 | 20.26 | 21.37 | 17.46 |
| Step-1V | 28.10 | 31.68 | 24.71 | 23.15 | 48.85 | 40.57 | 22.64 |
| Open Source Models |  |  |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |  |  |
| mPLUG-Owl | 7.60 | 6.19 | 10.09 | 5.00 | 12.34 | 7.63 | 7.94 |
| LLaMA-Adapter-V2 | 9.60 | 10.78 | 10.09 | 7.50 | 9.09 | 13.56 | 4.76 |
| MiniCPM-Llama3-V2.5 | 15.40 | 23.62 | 19.27 | 15.83 | 26.62 | 26.27 | 15.87 |
| LLaVA-1.5-13B | 15.50 | 15.83 | 15.60 | 12.50 | 18.83 | 14.41 | 14.29 |
| LLaVA-1.5-7B | 13.00 | 12.84 | 12.84 | 15.83 | 14.29 | 11.86 | 11.11 |
| DeepSeek-VL | 8.30 | 13.99 | 8.26 | 10.00 | 11.04 | 10.17 | 7.94 |
| ShareGPT4V-7B | 15.70 | 16.06 | 16.51 | 13.33 | 14.29 | 17.80 | 17.46 |
| ShareGPT4V-13B | 16.40 | 15.60 | 11.93 | 19.17 | 17.53 | 22.03 | 14.29 |
| SPHINX-Plus | 17.00 | 21.79 | 19.27 | 15.83 | 20.13 | 22.88 | 7.94 |
| InternLM-XC2 | 24.90 | 25.92 | 22.02 | 22.50 | 27.92 | 27.97 | 20.63 |
| InternVL-1.2-Plus | 30.10 | 34.40 | 25.69 | 30.00 | 29.87 | 26.27 | 23.81 |
| InternVL-Chat-V1.5 | 26.90 | 28.44 | 25.69 | 23.33 | 29.87 | 24.58 | 26.98 |
| CogVLM | 16.70 | 16.06 | 23.85 | 17.50 | 17.53 | 13.56 | 19.05 |
| CogVLM2 | 23.20 | 21.56 | 22.02 | 29.17 | 22.73 | 26.27 | 20.63 |
| GLM-4V-9B | 14.70 | 25.23 | 20.18 | 19.17 | 27.27 | 33.05 | 19.05 |

### 3.3 Error Analysis

To analyze the causes of errors in model responses, we meticulously review incorrect answers to identify common patterns. We specifically focus on the advanced MLLM, GPT-4o, to illustrate specific instances of errors and their distributions across the disciplines of mathematics, physics, and chemistry. Figure [4](#S3.F4) demonstrates the distributions of these errors, categorizing them into several types such as reasoning error, knowledge error, calculation error, vision recognition error, and question misunderstood error. Notably, across all disciplines, reasoning errors are the most prevalent, indicating a challenge in model’s ability to solve scientific problems that involve visual information. Specifically, reasoning errors account for 56.5% of the total errors in mathematics, 50.1% in physics, and 40.6% in chemistry, respectively. This is followed by knowledge error, which is particularly significant in chemistry, constituting 33.2% of the errors in that discipline. Similarly, knowledge error also represent the second most common error type in physics. However, knowledge error in mathematics is less prevalent, making up only 8.8% of the total errors. This indicates that while the model struggle with conceptual and fundamental principles in chemistry and physics, it demonstrate a better grasp of mathematical concepts. Vision recognition error is another significant type of error, accounting for 18.8% of the errors in mathematics, making it the second most prevalent error type in this discipline. This error category is also significant in physics and chemistry, where it constitutes 17.8% and 15.3% of the errors, respectively. This type of error highlights the challenges faced by the model in processing and understanding visual information. Furthermore, calculation error accounts for a small portion of the errors, especially in chemistry, indicating that the model excels particularly in handling numerical computations. Figure [5](#S3.F5) shows some cases of reasoning error category in the disciplines of mathematics, physics, and chemistry. More detailed examples of these errors can be found in Appendix [F](#A6).

Figure: Figure 4: Error distributions of GPT-4o on VisScience across the disciplines of mathematics, physics, and chemistry.
Refer to caption: /html/2409.13730/assets/x14.png

Figure: Figure 5: Cases of errors from GPT-4o in the disciplines of mathematics, physics, and chemistry.
Refer to caption: /html/2409.13730/assets/x17.png

## 4 Related Works

### 4.1 Multi-modal Reasoning Benchmarks

Recently, the evaluation of multi-modal large language models (MLLMs) [, , , , , , ] in various reasoning tasks has become increasingly crucial. So many benchmark datasets for these tasks span several categories are proposed like MME [], MMMU [], MMBench [], MMStar [], SEED-Bench [], and CMMMU [], which evaluate models’ capabilities to apply logic and inference; mathematical reasoning; scientific reasoning, and agent-based reasoning. These benchmark datasets provide comprehensive measurements of MLLMs’ capabilities in applying specialized knowledge and decision-making in simulated environments. For instance, MMMU covers university-level questions from six domains, which is utilized to assess MLLMs’ advanced perception and reasoning abilities. CMMMU [] evaluates models’ reasoning abilities across various disciplines through bilingual multi-modal questions in Chinese and English. Existing benchmark like ScienceQA [] is a specialized dataset designed to evaluate the capabilities of MLLMs, particularly in the domain of scientific reasoning. It contains over 21,000 questions that span a wide range of scientific disciplines, including physics, chemistry, biology, and earth science. Furthermore, several benchmarks such as MathVista [], MathVerse [], and Math-Vision [] are specially designed to evaluate the mathematical reasoning capabilities of MLLMs. MathVista comprises 6,141 examples drawn from 31 multi-modal datasets, with a commonly used minitest subset containing 1,000 examples specifically designed to evaluate the mathematical reasoning capabilities of MLLMs. MathVerse contains 2,612 high-quality, multi-subject mathematical problems covering key areas like plane geometry, solid geometry, and functions. Each problem is transformed into six distinct versions to provide varying levels of visual information: Text-dominant Version, Text-lite Version, Text-only Version, Vision-intensive Version, Vision-dominant Version, and Vision-only Version. Math-Vision is a carefully curated collection of 3,040 high-quality math problems, sourced from real math competitions, which covers 16 distinct mathematical disciplines and is graded across five levels of difficulty.
While these benchmarks are valuable, they present limitations such as an overemphasis on mathematics and a broad array of topics that often lack depth in science-related questions and exhibit uneven difficulty levels. Our dataset addresses these shortcomings by providing 3,000 scientific reasoning questions across mathematics, physics, and chemistry, which is collected from K12 education. Additionally, it includes bilingual questions in Chinese and English, enriching the knowledge base and offering a more extensive range of difficulty levels to create a more comprehensive evaluation platform.

### 4.2 Multi-modal Large Language Models

Recently, the success of large language models (LLMs) [, , , , , ] has spurred the ongoing development of multi-modal large language models (MLLMs). These MLLMs [, , , , , ] expand upon traditional LLM capabilities by integrating the ability to process and analyze both text and images. For instance, models like miniGPT [] and InstructBLIP [] attempt to utilize a trainable Q-Former or a linear layer to connect a frozen pretrained vision encoder and language model. Subsequently, LLaVA [, ] presents visual instruction tuning, which achieves a end-to-end fine-tuning on a large multi-modal model (LMM) comprising visual encoder and language model. Currently, close-source MLLMs like Gemini [], GPT-4v [], Qwen-VL [], and Claude3 [] demonstrate impressive capabilities in general image understanding and scientific reasoning. Besides, the development of open-source multi-modal large language models (MLLMs) continues to expand, providing an important complement to their closed-source models. These open-source MLLMs, such as mPLUG-Owl [, ], LLaMA-Adapter-V2 [], MiniCPM [], LLaVA-1.5 [], LLaVA-NeXT [], DeepSeek-VL [], ShareGPT4V [], SPHINX [], InternVL [], InternVL 1.5 [], InternLM-XComposer2 [], and CogVLM [], also achieves advance performance, further enriching the landscape of MLLM domain. Here, we utilize our specially curated benchmark to evaluate these MLLMs across tasks in mathematics, physics, and chemistry. This comprehensive evaluation aims to assess their capabilities in image understanding and scientific reasoning.

## 5 Conclusion

In this paper, we introduce a comprehensive benchmark, VisScience, to evaluate the capabilities of MLLMs in multi-modal scientific reasoning. VisScience comprises 3,000 questions across three disciplines of mathematics, physics, and chemistry, spanning 21 subjects and 5 difficulty levels. We conduct evaluations on VisScience with a total of 25 prominent models, including close-source and open-source models. Experimental results demonstrate that close-source MLLMs generally outperform open-source models, showing particularly more better capabilities in complex problem-solving and analytical reasoning. Notable models such as Claude3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro exhibit superior performance across three disciplines. Specifically, Claude3.5-Sonnet achieves an accuracy of 53.4% in mathematics, GPT-4o accomplishes a 38.2% accuracy in physics, and Gemini-1.5-Pro reaches an accuracy of 30.1% in chemistry. Although a gap remains between open-source models and the best-performing closed-source models, the open-source model like InternVL-1.2-Plus exhibits competitive advantages. For instance, InternVL-1.2-Plus outperforms models like Gemini-1.0-Pro across all three disciplines. By providing a comprehensive and challenging set of questions across three scientific disciplines, VisScience ensures a robust assessment of MLLMs’ ability in scientific reasoning.

## Appendix A Dataset Details

### A.1 Question Length Distribution

We provide both Chinese and English versions of the VisScience benchmark. The Chinese version features an average of 162.85 words per question, with the longest question comprising 1,297 words. Answers in this version average 20.93 words, with the longest reaching 112 words. Conversely, the English version shows an average of 80.93 words per question, with the longest question spanning 418 words. Answers here average 12.3 words, with the most detailed answer containing 92 words. Figure [6](#A1.F6) depicts the distribution of word counts, highlighting the diversity and complexity of questions.

Figure: Figure 6: The distributions of word counts per question in the Chinese and English versions of VisScience.
Refer to caption: /html/2409.13730/assets/x18.png

### A.2 Detailed Description of Subjects

VisScience consists of three disciplines: mathematics, physics, and chemistry. The mathematics section includes six subjects: algebraic operations, combinatorial mathematics, functions and equations, probability and statistics, plane geometry, and solid geometry. The physics section is composed of eight subjects: mechanics, optics, modern physics, mechanical motion, electromagnetism, vibrations and waves, comprehensive experiments and methods, and thermodynamics. The chemistry section includes seven subjects: chemical experiments, chemical reactions, inorganic chemistry, organic chemistry, electrochemistry, substance composition, and chemical equilibrium. A more detailed introduction of the above subjects is presented as follows:

#### A.2.1 Mathematics

Algebraic Operations. Algebraic operations include the manipulation of algebraic expressions, such as addition, subtraction, multiplication, and division. They are fundamental for solving algebraic equations and inequalities and are widely applied across various fields of mathematics.

Combinatorial Mathematics. Combinatorial mathematics studies the counting, arrangement, and combination of discrete structures, involving graph theory, number theory, and coding theory. It has significant applications in computer science, optimization, and probability theory.

Functions and Equations. Functions and equations are core parts of mathematics, dealing with relationships between variables and their representations. Functions are mappings between inputs and outputs, while equations are equalities concerning these mappings. Mastering knowledge of functions and equations is fundamental for solving many practical problems and is widely applied in engineering, physics, and economics.

Probability and Statistics. Probability and statistics study the laws of random events and methods of data analysis, including probability distributions, statistical inference, and data analysis techniques. They have broad applications in scientific research, engineering, and economics.

Plane Geometry. Plane geometry studies the shapes and figures in two-dimensional space, including points, lines, angles, and polygons. It is a fundamental part of mathematics education.

Solid Geometry. Solid geometry involves the study of geometric shapes in three-dimensional space, including points, lines, surfaces, and polyhedra. It examines the properties, volumes, and surface areas of these geometric bodies and is foundational for architecture, physics, and engineering.

#### A.2.2 Physics

Mechanics. Mechanics studies the motion of objects and the forces acting upon them, including classical mechanics, quantum mechanics, and relativistic mechanics. It is the foundation of physics and is widely applied in engineering, astronomy, and materials science.

Optics. Optics studies the properties of light and its interactions with matter, including reflection, refraction, interference, and diffraction. Optical technologies have broad applications in imaging, communication, and laser technology.

Modern Physics. Modern physics includes theories developed since the 20th century, such as quantum mechanics, relativity, and particle physics. These theories have expanded our understanding of the fundamental laws of nature.

Mechanical Motion. Mechanical motion studies the movement of objects under the influence of forces, including linear motion, rotational motion, and vibration. Understanding mechanical motion is fundamental for the design and analysis of mechanical systems.

Electromagnetism. Electromagnetism studies the interactions between electric and magnetic fields, including electrostatics, magnetic fields, and electromagnetic waves. It is the basis of modern physics and electrical engineering.

Vibration and Waves. Vibration and waves study vibrating systems and wave phenomena, including sound waves, light waves, and electromagnetic waves. They have broad applications in communication, acoustics, and optical technologies.

Comprehensive Experiments and Methods. Comprehensive experiments and methods involve using various experimental techniques and methods in physics teaching and research. They include designing and conducting experiments to observe and analyze the effects of specific variables on outcomes. Through comprehensive experiments, students can grasp the complexities of scientific research, cultivate scientific reasoning abilities, and understand the meticulousness and uncertainties of experimental work.

Thermodynamics. Thermodynamics studies the processes of energy transformation and transfer, including the laws of thermodynamics, thermodynamic systems, phase transitions, and heat engines. Thermodynamics is a fundamental aspect of both physics and engineering, with broad applications in energy, environmental science, and materials science. By investigating the relationship between internal and external energy of objects, thermodynamics reveals the basic principles of energy conversion and transfer in nature, providing theoretical support for the development of modern industrial technology.

#### A.2.3 Chemistry

Chemical Experiment. Chemical experiments involve studying the properties and changes of substances through experimental methods. Students learn to design experiments, observe chemical reactions, collect and analyze data, and draw conclusions in chemical experiments. Chemical experiments play a crucial role in understanding chemical theories and applying chemical knowledge.

Chemical Reaction. Chemical reactions study the chemical changes between substances, including reaction types, mechanisms, and rates. Understanding chemical reactions is essential for predicting and controlling chemical processes, which have wide applications in pharmaceutical manufacturing, materials science, and environmental engineering.

Inorganic Chemistry. Inorganic chemistry studies the properties and reactions of non-carbon elements and their compounds. It covers a wide range of topics from metals and non-metals to transition metals and coordination compounds and is key to understanding the periodic table of elements and chemical reaction mechanisms.

Organic Chemistry. Organic chemistry studies the structure, properties, and reactions of carbon-containing compounds. It has significant applications in pharmaceutical chemistry, materials science, and biochemistry.

Electrochemistry. Electrochemistry studies the interconversion between electrical and chemical energy, including processes such as batteries, electrolysis, and electroplating. Electrochemistry has important applications in energy storage, corrosion control, and electrochemical sensors.

Substance Composition. Substance composition studies the chemical composition and structure of substances, including the arrangement of molecules, atoms, and ions. It has important applications in chemistry, materials science, and biology.

Chemical Equilibrium. Chemical equilibrium studies the behavior of chemical reactions when they reach a dynamic equilibrium state, including equilibrium constants, Le Chatelier’s principle, and solubility equilibrium. Understanding chemical equilibrium is essential for predicting reaction directions and optimizing chemical processes.

## Appendix B Dataset Case

The VisScience dataset consists of 3,000 carefully selected high-quality questions, evenly distributed across three disciplines: mathematics, physics, and chemistry, with each comprising 1,000 questions. Each discipline within VisScience encompasses several subjects: mathematics includes six subjects, physics contains eight subjects, and chemistry comprises seven subjects. To illustrate the diversity and depth of VisScience, we provide more examples sampled from each discipline. In mathematics, six subjects include algebraic operations, combinatorial mathematics, functions and equations, probability and statistics, plane geometry, and solid geometry are illustrated in Figure [7](#A6.F7) to Figure [12](#A6.F12). Figure [13](#A6.F13) to Figure [20](#A6.F20) demonstrate eight subjects within the physics section of VisScience, comprising mechanics, optics, modern physics, mechanical motion, electromagnetism, vibrations and waves, comprehensive experiments and methods, and thermodynamics. The chemistry section includes seven subjects: chemical experiments, chemical reactions, inorganic chemistry, organic chemistry, electrochemistry, substance composition, and chemical equilibrium, which are illustrated in Figure [21](#A6.F21) to Figure [27](#A6.F27).

## Appendix C Evaluation Details

### C.1 The Sources of Models

In Table [4](#A3.T4), we present the sources of the models tested on VisScience.

**Table 4: The source of the models used in the evaluation.**
| Model | Input | LLM Size | Source |  |
| --- | --- | --- | --- | --- |
| Closed Source Models |  |  |  |  |
| Text-only LLMs |  |  |  |  |
| ChatGPT | Q | - | [gpt-3.5-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo) |  |
| GPT-4 | Q | - | [gpt-4](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4) |  |
| Claude-2 | Q | - | [claude-2](https://www.anthropic.com/api) |  |
| Multi-modal LLMs |  |  |  |  |
| Gemini-1.0-Pro | Q, I | - | [gemini-pro](https://ai.google.dev/) |  |
| Gemini-1.5-Pro | Q, I | - | [gemini-1.5-pro](https://ai.google.dev/) |  |
| GPT-4o | Q, I | - | [gpt-4o](https://platform.openai.com/docs/models/gpt-4o) |  |
| Claude3-Opus | Q, I | - | [claude-3-opus-20240229](https://www.anthropic.com/api) |  |
| Claude3.5-Sonnet | Q, I | - | [claude-3-5-sonnet-2024620](https://www.anthropic.com/api) |  |
| Qwen-VL-Plus | Q, I | - | [qwen-vl-plus](https://help.aliyun.com/zh/dashscope/developer-reference/vl-plus-quick-start) |  |
| Qwen-VL-Max | Q, I | - | [qwen-vl-max](https://help.aliyun.com/zh/dashscope/developer-reference/vl-plus-quick-start) |  |
| GLM-4V | Q, I | - | [glm-4v](https://open.bigmodel.cn/dev/api#glm-4v) |  |
| Step-1V | Q, I | - | [step-1v](https://platform.stepfun.com/docs/llm/vision) |  |
| Open Source Models |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |
| mPLUG-Owl | Q, I | 7B | [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl) |  |
| DeepSeek-VL | Q, I | 7B | [deepseek-vl-7b-base](https://github.com/deepseek-ai/DeepSeek-VL) |  |
| LLaMA-Adapter-V2 | Q, I | 7B | [LLaMA-Adapter V2](https://github.com/ml-lab/LLaMA-Adapter-2) |  |
| LLaVA-1.5 | Q, I | 7B | [LLaVA-v1.5-7B](https://github.com/haotian-liu/LLaVA) |  |
| LLaVA-1.5 | Q, I | 13B | [LLaVA-v1.5-13B](https://github.com/haotian-liu/LLaVA) |  |
| ShareGPT-4V | Q, I | 7B | [ShareGPT4V-7B](https://huggingface.co/Lin-Chen/ShareGPT4V-7B) |  |
| ShareGPT-4V | Q, I | 13B | [ShareGPT4V-13B](https://huggingface.co/Lin-Chen/ShareGPT4V-13B) |  |
| GLM-4v-9B | Q, I | 7B | [GLM-4v-9B](https://huggingface.co/THUDM/glm-4v-9b) |  |
| SPHINX-Plus | Q, I | 13B | [SPHINX-Plus](https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/main/SPHINX/README.md) |  |
| InternVL-Chat-V1.5 | Q, I | 20B | [InternVL 1.5](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5) |  |
| InternVL-1.2-Plus | Q, I | 34B | [InternVL-Chat-V1-2-Plus](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2-Plus) |  |
| InternLM-XC2 | Q, I | 7B | [InternLM-XComposer2-VL-7B](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b) |  |
| CogVLM | Q, I | 17B | [CogVLM-17B](https://huggingface.co/THUDM/cogvlm-chat-hf) |  |
| CogVLM2 | Q, I | 19B | [cogvlm2-llama3-chat-19B](https://huggingface.co/THUDM/cogvlm-chat-hf) |  |
| MiniCPM-Llama3-V-2_5 | Q, I | 19B | [MiniCPM-Llama3-V 2.5](https://huggingface.co/THUDM/cogvlm-chat-hf) |  |

### C.2 Prompts

We introduce the prompts used to guide models in generating responses in Chain-of-Thought (CoT) settings and judging the LLMs’ answers. The specific prompts can be found in Table [5](#A3.T5).

**Table 5: Prompts for response generation and answer judgment.**
| Task | Prompt |
| --- | --- |
| Response Generation | You are an exceptionally talented mathematics (physics/chemistry) instructor. Kindly furnish an elaborate, step-by-step solution to the question. |
| Answer Judgment | You are a highly skilled mathematics (physics/chemistry) teacher. I will provide you with a mathematics (physics/chemistry) problem, along with its ground answer and the model response from the model. Please determine whether the ground answer and the model response are consistent. Note that you do not need to judge the correctness of either answer, only whether they are consistent. If it is a multiple-choice question, both answers must choose the exact same option to be considered consistent. If it is a calculation problem, the relative error between the model response and the ground answer must be less than 0.05 to be considered consistent. If the problem has multiple sub-questions, each sub-question’s answer must be identical for consistency. If you find them consistent, please add [Consistent] at the end of your response. If you find them inconsistent, please add [Inconsistent] at the end of your response. |

## Appendix D More Experimental Results

### D.1 Results on VisScience in English Version

Table [6](#A4.T6) reports a comprehensive comparison of various models on the VisScience benchmark in the English version. The benchmark evaluates performance across three disciplines: mathematics, physics, and chemistry. Among close-source models, GPT-4o demonstrates the highest performance across two disciplines, achieving an accuracy of 53.6% in mathematics and 42.7% in physics. However, Claude3.5-Sonnet surpasses GPT-4o in chemistry with a higher accuracy of 43.6%. Open-source models generally show lower performance compared to close-source counterparts. Notably, InternVL-1.2-Plus displays competitive performance, reaching up to 26.0% in mathematics, 23.6% in physics, and 27.8% in chemistry. The English version of VisScience is designed to facilitate the evaluation of MLLMs that specialize in English, assessing their capabilities in scientific reasoning.

**Table 6: Results on VisScience within the version of the English language across the disciplines of mathematics, physics, and chemistry. The highest scores among close-source and open-source models are highlighted in red and blue, respectively.**
| Model | LLM | Input | Mathematics | Physics | Chemistry |
| --- | --- | --- | --- | --- | --- |
| Close Source Models (APIs) |  |  |  |  |  |
| Text-only LLMs |  |  |  |  |  |
| Zero-shot ChatGPT | - | Q | 17.4 | 20.7 | 25.2 |
| Zero-shot GPT-4 | - | Q | 29.9 | 37.7 | 38.7 |
| Zero-shot Claude-2 | - | Q | 24.6 | 22.7 | 25.6 |
| Zero-shot Claude3-Opus | - | Q | 21.7 | 15.8 | 29.4 |
| Zero-shot Claude3.5-Sonnet | - | Q | 27.2 | 35.7 | 35.2 |
| Zero-shot GPT-4o | - | Q | 35.2 | 40.3 | 42.5 |
| 2-shot CoT Claude2 | - | Q | 25.7 | 21.9 | 24.1 |
| 2-shot CoT ChatGPT | - | Q | 24.4 | 20.1 | 22.1 |
| 2-shot CoT GPT-4 | - | Q | 36.5 | 39.0 | 38.1 |
| Multi-modal LLMs |  |  |  |  |  |
| Gemini-1.0-Pro | - | Q, I | 26.4 | 39.1 | 27.9 |
| Gemini-1.5-Pro | - | Q, I | 47.8 | 35.1 | 39.1 |
| GPT-4o | - | Q, I | 53.6 | 42.7 | 43.3 |
| GPT-4o-mini | - | Q, I | 43.2 | 33.7 | 34.9 |
| Qwen-VL-Max | - | Q, I | 30.7 | 26.4 | 36.3 |
| Qwen-VL-Plus | - | Q, I | 21.9 | 20.9 | 29.7 |
| Claude3.5-Sonnet | - | Q, I | 50.8 | 36.6 | 43.6 |
| Claude3-Opus | - | Q, I | 34.4 | 29.4 | 34.7 |
| GLM-4V | - | Q, I | 23.1 | 18.5 | 23.4 |
| Step-1V | 7B | Q, I | 32.0 | 19.5 | 27.6 |
| Open Source Models |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |
| mPLUG-Owl | LLaMA-7B | Q, I | 7.4 | 12.3 | 12.3 |
| LLaMA-Adapter-V2 | LLaMA-7B | Q, I | 12.6 | 11.4 | 16.2 |
| MiniCPM-Llama3-V2.5 | LLaMA3-8B | Q, I | 24.4 | 20.6 | 24.4 |
| LLaVA-1.5 | Vicuna-13B | Q, I | 15.0 | 17.4 | 21.1 |
| LLaVA-1.5 | Vicuna-7B | Q, I | 17.4 | 16.6 | 18.9 |
| DeepSeek-VL | DeepSeek-LLM-7B | Q, I | 16.0 | 16.9 | 17.8 |
| ShareGPT-4V | Vicuna-7B | Q, I | 14.7 | 17.7 | 21.3 |
| ShareGPT-4V | Vicuna-13B | Q, I | 14.5 | 16.0 | 20.2 |
| SPHINX-Plus | LLaMA2-13B | Q, I | 17.9 | 15.7 | 22.4 |
| InternLM-XC2 | InternLM2-7B | Q, I | 20.7 | 20.5 | 25.0 |
| InternVL-1.2-Plus | Nous-Hermes-2-Yi-34B | Q, I | 26.0 | 23.6 | 27.8 |
| InternVL-Chat-V1.5 | Mixtral 8*7B | Q, I | 24.9 | 23.0 | 25.9 |
| CogVLM | Vicuna-7B | Q, I | 18.5 | 15.9 | 23.1 |
| CogVLM2 | LLaMA-3-8B | Q, I | 24.2 | 16.6 | 24.9 |
| GLM-4V-9B | GLM-4-9B | Q, I | 24.7 | 19.2 | 23.9 |

### D.2 Results on Physics Across Different Subjects

Table [7](#A4.T7) presents a detailed analysis of various models on VisScience across different subjects within the physics section, which includes mechanics, electromagnetism, thermodynamics, comprehensive experiments and methods, optics, vibration and waves, modern physics, and mechanical motion. The table highlights that while GPT-4o exhibits the top performance on the entire physics discipline, the best performance in individual subjects varies notably. For instances, Claude3.5-Sonnet excels specifically in modern physics with an accuracy of 66.67%, significantly surpassing other close-source models in this area. This variation in performance by subject underscores the specialized capabilities of different models. Moreover, this detailed analysis provides more insights, emphasizing the need for targeted improvements to achieve balanced performance across all physics subjects.

**Table 7: Results on the physics part of VisScience across different subjects. Subjects: Mech: mechanics, Ele: electromagnetism, Threm: thermodynamics, Comp: comprehensive experiments and methods, Opt: optics, Vib & Waves: vibration and waves, Mod Phys: modern physics, Mech Motion: mechanical motion. The highest scores among close-source and open-source models are highlighted in red and blue, respectively.**
| Model | Physics |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ALL | Mech | Ele | Therm | Comp | Opt | Vib & Waves | Mod Phys | Mech Motion |  |
| Close Source Models (APIs) |  |  |  |  |  |  |  |  |  |
| Text-only LLMs |  |  |  |  |  |  |  |  |  |
| Zero-shot ChatGPT | 22.70 | 22.08 | 19.94 | 23.53 | 4.62 | 40.98 | 29.79 | 19.05 | 23.33 |
| Zero-shot GPT-4 | 30.40 | 34.26 | 30.21 | 33.33 | 15.38 | 40.98 | 34.04 | 42.86 | 20.00 |
| Zero-shot Claude-2 | 22.00 | 24.62 | 23.56 | 25.49 | 12.31 | 27.87 | 21.28 | 28.57 | 23.33 |
| Zero-shot Claude3-Opus | 30.80 | 34.26 | 32.02 | 33.33 | 10.77 | 39.34 | 31.91 | 42.86 | 10.00 |
| Zero-shot Claude3.5-Sonnet | 35.30 | 40.36 | 35.95 | 35.29 | 15.38 | 40.98 | 34.04 | 47.62 | 26.67 |
| Zero-shot GPT-4o | 38.00 | 43.91 | 38.67 | 45.10 | 9.23 | 49.18 | 38.30 | 52.38 | 23.33 |
| 2-shot CoT Claude2 | 21.70 | 24.87 | 22.96 | 25.49 | 10.77 | 18.03 | 23.40 | 28.57 | 10.00 |
| 2-shot CoT ChatGPT | 18.60 | 20.30 | 20.54 | 13.73 | 12.31 | 22.95 | 23.40 | 23.81 | 13.33 |
| 2-shot CoT GPT-4 | 31.50 | 35.03 | 32.02 | 37.25 | 12.31 | 44.26 | 29.79 | 47.62 | 23.33 |
| Multi-modal LLMs |  |  |  |  |  |  |  |  |  |
| Gemini-1.0-Pro | 23.70 | 26.97 | 23.03 | 17.65 | 6.15 | 31.15 | 34.04 | 19.05 | 10.00 |
| Gemini-1.5-Pro | 38.10 | 46.56 | 33.74 | 47.06 | 20.00 | 45.00 | 34.04 | 52.38 | 43.33 |
| GPT-4o | 38.20 | 41.37 | 39.27 | 56.86 | 23.08 | 42.62 | 36.17 | 42.86 | 43.33 |
| GPT-4o-mini | 29.80 | 31.73 | 30.51 | 29.41 | 10.70 | 36.07 | 19.15 | 47.62 | 30.00 |
| Qwen-VL-Max | 30.70 | 36.13 | 26.59 | 39.22 | 9.23 | 34.43 | 31.91 | 28.57 | 30.00 |
| Qwen-VL-Plus | 26.50 | 31.04 | 24.77 | 33.33 | 6.15 | 36.07 | 36.17 | 23.81 | 16.67 |
| Claude3.5-Sonnet | 38.00 | 41.62 | 36.56 | 43.14 | 13.85 | 44.26 | 38.30 | 66.67 | 30.00 |
| Claude3-Opus | 31.10 | 33.25 | 29.91 | 39.22 | 12.31 | 45.90 | 34.04 | 61.90 | 23.33 |
| GLM-4V | 19.20 | 23.16 | 17.82 | 15.69 | 12.31 | 25.00 | 17.02 | 19.05 | 23.33 |
| Step-1V | 23.50 | 21.55 | 24.35 | 28.57 | 7.84 | 12.82 | 25.00 | 31.25 | 39.13 |
| Open Source Models |  |  |  |  |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |  |  |  |  |
| mPLUG-Owl | 8.30 | 11.93 | 8.46 | 1.96 | 4.62 | 8.20 | 10.64 | 4.76 | 10.00 |
| LLaMA-Adapter-V2 | 10.30 | 10.41 | 10.88 | 8.00 | 4.84 | 13.11 | 25.53 | 14.29 | 3.33 |
| MiniCPM-Llama3-V2.5 | 17.90 | 21.57 | 19.64 | 15.69 | 6.15 | 26.23 | 19.15 | 9.52 | 23.33 |
| LLaVA-1.5-13B | 15.20 | 17.26 | 14.80 | 7.84 | 7.69 | 21.31 | 17.02 | 9.52 | 16.67 |
| LLaVA-1.5-7B | 13.50 | 15.28 | 15.12 | 11.76 | 3.12 | 15.25 | 15.56 | 5.26 | 17.24 |
| DeepSeek-VL | 16.80 | 18.77 | 19.33 | 13.73 | 7.69 | 16.67 | 13.04 | 19.05 | 3.45 |
| ShareGPT4V-7B | 14.00 | 13.71 | 15.41 | 9.80 | 3.08 | 19.67 | 19.15 | 28.57 | 6.67 |
| ShareGPT4V-13B | 14.90 | 15.23 | 16.92 | 9.80 | 6.15 | 14.75 | 19.15 | 19.05 | 16.67 |
| SPHINX-Plus | 15.30 | 16.50 | 18.43 | 17.65 | 4.62 | 11.48 | 12.77 | 19.05 | 13.33 |
| InternLM-XC2 | 18.30 | 20.81 | 17.82 | 13.73 | 10.77 | 26.23 | 21.28 | 14.29 | 6.67 |
| InternVL-1.2-Plus | 24.80 | 29.69 | 22.94 | 29.41 | 12.31 | 31.67 | 25.53 | 35.00 | 10.00 |
| InternVL-Chat-V1.5 | 20.80 | 23.97 | 20.87 | 23.53 | 9.23 | 25.42 | 17.02 | 14.29 | 17.24 |
| CogVLM | 14.50 | 18.02 | 13.29 | 7.84 | 6.15 | 14.75 | 19.15 | 19.05 | 6.67 |
| CogVLM2 | 14.40 | 16.75 | 16.00 | 12.00 | 6.15 | 13.11 | 19.15 | 4.76 | 10.00 |
| GLM-4V-9B | 19.30 | 21.78 | 21.12 | 24.00 | 4.62 | 25.42 | 15.91 | 15.00 | 13.33 |

### D.3 Results on Chemistry Across Different Subjects

Table [8](#A4.T8) presents a nuanced view of the performance of various models across different subjects within the chemistry discipline of the VisScience benchmark. The chemistry discipline includes chemical experiment, chemical reaction, inorganic chemistry, electrochemistry, organic chemistry, chemical equilibrium, and substance composition. Notably, Gemini-1.5-Pro stands out among close-source models, excelling across the entire chemistry discipline. It demonstrates particular prowess in organic chemistry and substance composition, achieving impressive accuracies of 57.02% and 61.16%, respectively. Additionally, Qwen-VL-Max leads in chemical experiment and inorganic chemistry, achieving the highest accuracies of 46.28% and 51.94%, respectively. Open-source models demonstrate a range of performances, with InternVL-1.2-Plus leading this group. It achieves the highest open-source accuracy in nearly all subjects. This comprehensive review of model performances within the chemistry section of the VisScience benchmark highlights the need to enhance MLLMs’ capabilities in scientific domains, ensuring models are both accurate and adaptable across various disciplines.

**Table 8: Results on the chemistry part of VisScience across different subjects. Subjects: Chem Exp: chemical experiment, Chem React: chemical reaction, Inorg Chem: inorganic chemistry, Electrochem: Electrochemistry, Org Chem: organic chemistry, Chem Equil: chemical equilibrium, and Sub Comp: substance composition. The highest scores among close-source and open-source models are highlighted in red and blue, respectively.**
| Model | Chemistry |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ALL | Chem Exp | Chem React | Inorg Chem | Electrochem | Org Chem | Chem Equil | Sub Comp |  |
| Close Source Models (APIs) |  |  |  |  |  |  |  |  |
| Text-only LLMs |  |  |  |  |  |  |  |  |
| Zero-shot ChatGPT | 18.60 | 26.35 | 23.86 | 23.26 | 23.75 | 35.43 | 24.64 | 30.89 |
| Zero-shot GPT-4 | 33.10 | 40.54 | 30.68 | 38.76 | 32.50 | 36.22 | 30.43 | 31.71 |
| Zero-shot Claude-2 | 24.40 | 24.32 | 26.36 | 31.71 | 23.86 | 20.29 | 0.30 | 25.98 |
| Zero-shot Claude3-Opus | 32.50 | 37.16 | 30.68 | 31.78 | 31.25 | 36.22 | 30.43 | 39.84 |
| Zero-shot Claude3.5-Sonnet | 36.90 | 34.80 | 36.93 | 39.53 | 46.25 | 45.67 | 23.19 | 47.15 |
| Zero-shot GPT-4o | 39.60 | 42.57 | 40.34 | 44.96 | 35.00 | 41.73 | 26.09 | 54.47 |
| 2-shot CoT Claude2 | 23.90 | 23.99 | 26.70 | 22.48 | 30.00 | 26.77 | 27.54 | 27.64 |
| 2-shot CoT ChatGPT | 21.30 | 19.93 | 23.30 | 20.93 | 22.50 | 22.83 | 26.09 | 30.08 |
| 2-shot CoT GPT-4 | 32.40 | 29.05 | 32.39 | 32.56 | 32.50 | 42.52 | 28.99 | 53.66 |
| Multi-modal LLMs |  |  |  |  |  |  |  |  |
| Gemini-1.0-Pro | 27.80 | 24.03 | 26.70 | 26.36 | 31.25 | 35.54 | 31.82 | 37.19 |
| Gemini-1.5-Pro | 47.00 | 43.46 | 47.43 | 51.59 | 50.00 | 57.02 | 35.29 | 61.16 |
| GPT-4o | 41.60 | 43.58 | 46.02 | 38.76 | 46.25 | 43.31 | 43.48 | 50.41 |
| GPT-4o-mini | 28.40 | 22.30 | 27.27 | 27.13 | 30.00 | 34.65 | 20.29 | 42.09 |
| Qwen-VL-Max | 42.50 | 46.28 | 41.48 | 51.94 | 35.00 | 41.73 | 36.23 | 53.66 |
| Qwen-VL-Plus | 37.70 | 33.78 | 40.34 | 44.19 | 41.25 | 48.03 | 33.33 | 41.80 |
| Claude3.5-Sonnet | 43.10 | 40.54 | 41.48 | 42.64 | 50.00 | 42.52 | 33.33 | 59.35 |
| Claude3-Opus | 34.10 | 35.47 | 30.11 | 31.78 | 31.25 | 40.16 | 33.33 | 51.22 |
| GLM-4V | 25.00 | 23.65 | 25.86 | 21.71 | 28.75 | 27.78 | 31.88 | 32.52 |
| Step-1V | 25.00 | 32.51 | 27.48 | 25.26 | 25.45 | 17.72 | 13.33 | 21.95 |
| Open Source Models |  |  |  |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |  |  |  |
| mPLUG-Owl | 9.50 | 7.77 | 11.36 | 7.75 | 12.50 | 12.60 | 13.04 | 9.76 |
| LLaMA-Adapter-V2 | 10.80 | 7.77 | 13.64 | 8.53 | 12.66 | 12.80 | 17.65 | 17.07 |
| MiniCPM-Llama3-V2.5 | 19.50 | 20.96 | 26.29 | 26.61 | 18.18 | 24.00 | 28.79 | 30.83 |
| LLaVA-1.5-13B | 18.80 | 15.54 | 16.48 | 24.03 | 20.00 | 22.05 | 23.19 | 19.51 |
| LLaVA-1.5-7B | 16.00 | 13.49 | 17.14 | 19.20 | 16.25 | 20.49 | 26.09 | 10.74 |
| DeepSeek-VL | 21.00 | 18.84 | 20.57 | 20.16 | 21.25 | 23.62 | 36.76 | 20.66 |
| ShareGPT4V-7B | 19.00 | 13.85 | 19.32 | 26.36 | 18.75 | 23.62 | 28.99 | 15.45 |
| ShareGPT4V-13B | 18.40 | 13.51 | 21.02 | 19.38 | 23.75 | 22.83 | 13.04 | 19.51 |
| SPHINX-Plus | 20.40 | 20.27 | 21.02 | 24.03 | 22.50 | 22.83 | 27.54 | 21.95 |
| InternLM-XC2 | 25.60 | 22.64 | 27.27 | 26.36 | 21.25 | 33.86 | 26.09 | 24.39 |
| InternVL-1.2-Plus | 31.20 | 22.29 | 31.82 | 33.58 | 31.46 | 39.57 | 32.47 | 38.84 |
| InternVL-Chat-V1.5 | 23.70 | 20.07 | 25.00 | 25.20 | 22.37 | 28.80 | 25.00 | 28.46 |
| CogVLM | 17.00 | 15.54 | 20.45 | 10.85 | 16.25 | 22.05 | 20.29 | 17.07 |
| CogVLM2 | 21.00 | 13.10 | 21.39 | 25.78 | 20.51 | 31.45 | 22.73 | 30.17 |
| GLM-4V-9B | 22.50 | 21.00 | 25.44 | 26.23 | 23.08 | 26.83 | 17.39 | 25.83 |

## Appendix E Case Study

The VisScience dataset includes three disciplines: mathematics, physics, and chemistry. The mathematical section comprises 6 subjects, the chemistry section contains 7 subjects, and the physics section includes 8 subjects, culminating in a total of 21 distinct subjects across the VisScience benchmark. Here, we present one question from each subject, along with its standard answer and the correct response provided by GPT-4o. Figure [28](#A6.F28) to Figure [30](#A6.F30) demonstrate cases of the mathematical part of VisScience. Figure [31](#A6.F31) to Figure [34](#A6.F34) illustrate sampled questions from the physics section of the VisScience benchmark. Each figure provides insight into the diverse range of topics covered, showing GPT-4o’s capabilities to handle complex physical principles and calculations. Figure [35](#A6.F35) to Figure [38](#A6.F38) display examples from the chemistry section. These examples not only demonstrate the diversity of the VisScience benchmark within chemistry but also illustrate how effectively GPT-4o can generate accurate responses across different scientific subjects.

## Appendix F Error Case

We conduct rigorous tests on a series of open-source and close-source models on VisScience and perform a detailed analysis of the models’ responses. These errors in the models’ answers can be classified into five categories: reasoning error, vision recognition error, knowledge error, calculation error, and question misunderstood error. We present examples of these five error types across the disciplines of mathematics, physics, and chemistry, with a specific focus on errors made by GPT-4o. Additionally, we demonstrate error examples from other representative close-source models such as GLM-4V, Qwen-VL-max, and Claude 3.5, as well as open-source models like LLAVA-1.5 , GLM-4V-9B and InternVL-Chat-1.5. Notably, it should be noted that the types of errors made by these models in response to the same questions can differ from those made by GPT-4o. This analysis helps to underline the varied challenges faced by different models in processing complex scientific questions, providing insight into their respective strengths and limitations. Figure [39](#A6.F39) to Figure [48](#A6.F48) demonstrate cases of errors from representative models in the mathematical part of VisScience. Figure [49](#A6.F49) to Figure [61](#A6.F61) show the incorrect answers in the physics section. Figure [62](#A6.F62) to Figure [70](#A6.F70) demonstrate the errors in the chemistry section.

Figure: Figure 7: Cases of algebraic operations in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/math_Algebraic_Operations.jpg

Figure: Figure 8: Cases of combinatorial mathematics in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/math_Combinatorial_Mathematics.jpg

Figure: Figure 9: Cases of functions and equations in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/math_Functions_and_Equations.jpg

Figure: Figure 10: Cases of probability and statistics in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/x20.png

Figure: Figure 11: Cases of plane geometry in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/math_Plane_Geometry.jpg

Figure: Figure 12: Cases of solid geometry in mathematical part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/math_Solid_Geometry.jpg

Figure: Figure 13: Cases of mechanics in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x21.png

Figure: Figure 14: Cases of optics in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/physics_Optics.jpg

Figure: Figure 15: Cases of modern physics in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/physics_Modern_Physics.jpg

Figure: Figure 16: Cases of mechanical motion in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x22.png

Figure: Figure 17: Cases of electromagnetism in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x23.png

Figure: Figure 18: Cases of vibration and waves in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x24.png

Figure: Figure 19: Cases of comprehensive experiments and methods in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x25.png

Figure: Figure 20: Cases of thermodynamics in physics part of VisScience.
Refer to caption: /html/2409.13730/assets/x26.png

Figure: Figure 21: Cases of chemical experiment in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Chemical_Experiment.jpg

Figure: Figure 22: Cases of chemical reaction in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Chemical_Reaction.jpg

Figure: Figure 23: Cases of inorganic chemistry in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Inorganic_Chemistry.jpg

Figure: Figure 24: Cases of organic chemistry in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Organic_Chemistry.jpg

Figure: Figure 25: Cases of electrochemistry in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Electrochemistry.jpg

Figure: Figure 26: Cases of substance composition in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Substance_Composition.jpg

Figure: Figure 27: Cases of chemical equilibrium in chemistry part of VisScience.
Refer to caption: /html/2409.13730/assets/figs/chemistry_Chemical_Equilibrium.jpg

Figure: Figure 28: A question, the standard answer, and GPT-4o’s correct response from the mathematical part in VisScience.
Refer to caption: /html/2409.13730/assets/x27.png

Figure: Figure 29: A question, the standard answer, and GPT-4o’s correct response from the mathematical part in VisScience.
Refer to caption: /html/2409.13730/assets/x28.png

Figure: Figure 30: A question, the standard answer, and GPT-4o’s correct response from the mathematical part in VisScience.
Refer to caption: /html/2409.13730/assets/x29.png

Figure: Figure 31: A question, the standard answer, and GPT-4o’s correct response from the physics part in VisScience.
Refer to caption: /html/2409.13730/assets/x30.png

Figure: Figure 32: A question, the standard answer, and GPT-4o’s correct response from the physics part in VisScience.
Refer to caption: /html/2409.13730/assets/x31.png

Figure: Figure 33: A question, the standard answer, and GPT-4o’s correct response from the physics part in VisScience.
Refer to caption: /html/2409.13730/assets/x32.png

Figure: Figure 34: A question, the standard answer, and GPT-4o’s correct response from the physics part in VisScience.
Refer to caption: /html/2409.13730/assets/x33.png

Figure: Figure 35: A question, the standard answer, and GPT-4o’s correct response from the chemistry part in VisScience.
Refer to caption: /html/2409.13730/assets/x34.png

Figure: Figure 36: A question, the standard answer, and GPT-4o’s correct response from the chemistry part in VisScience.
Refer to caption: /html/2409.13730/assets/x35.png

Figure: Figure 37: A question, the standard answer, and GPT-4o’s correct response from the chemistry part in VisScience.
Refer to caption: /html/2409.13730/assets/x36.png

Figure: Figure 38: A question, the standard answer, and GPT-4o’s correct response from the chemistry part in VisScience.
Refer to caption: /html/2409.13730/assets/x37.png

Figure: Figure 39: Cases of errors in the mathematical of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x38.png

Figure: Figure 40: Cases of errors in the mathematical of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x39.png

Figure: Figure 41: Cases of errors in the mathematical of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x40.png

Figure: Figure 42: Cases of errors in the mathematical of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x41.png

Figure: Figure 43: Cases of errors in the mathematical of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x42.png

Figure: Figure 44: Cases of errors in the mathematical of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x43.png

Figure: Figure 45: Cases of errors in the mathematical of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x44.png

Figure: Figure 46: Cases of errors in the mathematical of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x45.png

Figure: Figure 47: Cases of errors in the mathematical of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x46.png

Figure: Figure 48: Cases of errors in the mathematical of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x47.png

Figure: Figure 49: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x48.png

Figure: Figure 50: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x49.png

Figure: Figure 51: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x50.png

Figure: Figure 52: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x51.png

Figure: Figure 53: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x52.png

Figure: Figure 54: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x53.png

Figure: Figure 55: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x54.png

Figure: Figure 56: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x55.png

Figure: Figure 57: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x56.png

Figure: Figure 58: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x57.png

Figure: Figure 59: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x58.png

Figure: Figure 60: Cases of errors in the physics part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x59.png

Figure: Figure 61: Cases of errors in the physics part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x60.png

Figure: Figure 62: Cases of errors in the chemistry part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x61.png

Figure: Figure 63: Cases of errors in the chemistry part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x62.png

Figure: Figure 64: Cases of errors in the chemistry part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x63.png

Figure: Figure 65: Cases of errors in the chemistry part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x64.png

Figure: Figure 66: Cases of errors in the chemistry part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x65.png

Figure: Figure 67: Cases of errors in the chemistry part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x66.png

Figure: Figure 68: Cases of errors in the chemistry part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x67.png

Figure: Figure 69: Cases of errors in the chemistry part of VisScience for several classic close-source models.
Refer to caption: /html/2409.13730/assets/x68.png

Figure: Figure 70: Cases of errors in the chemistry part of VisScience for several classic open-source models.
Refer to caption: /html/2409.13730/assets/x69.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - In the experimental section of this paper, a text-only setting was adopted, but it is not explicitly stated whether the problems used are the same as those in the multimodal setting. If the two are indeed consistent, for questions that rely on graphical information, posing them through text alone may not constitute a complete question. If this design is intended to validate the effectiveness of graphical information, it is recommended that the authors provide clearer explanations in the text to elucidate the purpose of the experimental design and the interpretation of the results.
- The paper categorizes the inference results of the model but does not provide detailed definitions and classification criteria for each type of error. To enhance the transparency and reproducibility of the paper, it is recommended that the authors clearly define each type of error in the text and describe the specific methods used for error classification.
- The source of the data is not clearly specified in the paper. It is suggested that the authors provide additional explanations regarding the data sources in the paper.
- In Fig. 26, Fig. 32, and Fig. 33, the titles indicate that they display cases of standard answers and correct GPT-4o answers. However, it can be observed that in some cases, the answers provided by the model are actually incorrect. It is recommended that the authors carefully review all figures and cases in the paper to ensure that the displayed results match the annotations and to avoid such errors, thereby improving the accuracy and reliability of the paper.
soundness: 2
strengths: - The paper introduces VisScience, a multimodal scientific reasoning benchmark covering mathematics, physics, and chemistry, with bilingual (Chinese and English) support. This benchmark addresses the shortcomings of existing benchmarks in multidisciplinary and multilingual evaluations. The dataset has undergone rigorous screening and verification by both LLMs and human experts to ensure its quality. Comprehensive evaluations were conducted on 25 closed-source and open-source MLLMs, revealing the strengths and limitations of different models in scientific reasoning.
summary: The paper introduces the VisScience benchmark, which is designed to evaluate the performance of multimodal large language models (MLLMs) in scientific reasoning tasks. This benchmark covers three major subjects—mathematics, physics, and chemistry，consists of 3,000 questions, spanning both English and Chinese language tasks. It is further divided into 21 sub-disciplines and five difficulty levels. Through experiments, the authors tested 25 MLLMs, and the results showed that closed-source models performed better across all subjects.
weaknesses: - VisScience includes bilingual tasks in Chinese and English; however, in the case studies presented in the appendix, it is observed that the text within images corresponding to English questions might be in Chinese. This bilingual mixing situation could lead to different reasoning capabilities of the models when processing information in different languages, especially during the integration of visual and linguistic information. Therefore, it is suggested that the authors further investigate and address this issue in future research to ensure that the models can perform stably in true multilingual environments.
- The paper mentions that the data underwent multiple checks using manual review and LLM assessment, along with some screening principles. However, the details of this process are not elaborated upon in the paper. This process is crucial for ensuring the quality of the dataset and the accuracy of model evaluation. It is recommended that the authors supplement the paper with a detailed description of this process.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses Section above.
soundness: 2
strengths: There are several noticeable strengths in this paper:
* This paper presents a well-justified motivation for creating a benchmark focusing on K12 math, physics, and chemistry knowledge.
* This submission conducts extensive experiments ranging from open-source and closed-source LLM and MLLMs.
* The manuscript is commendable for its clarity and structured writing style, which greatly facilitates reader comprehension.
* Additionally, the inclusion of clear and illustrative figures and tables is a notable strength, as it significantly aids in conveying the main claims of the paper to the audience.
summary: This submission introduces VisScience, an extensive benchmark for evaluating K12 educational multi-modal scientific reasoning across mathematics, physics, and chemistry using large language models.​ The authors highlight the strengths and limitations of current MLLMs and suggest areas for future improvement.
weaknesses: This submission creates a benchmark, which is a notable contribution without a doubt. However, there are some unignorable weaknesses that, if addressed, could greatly enhance the clarity and usability of the benchmark.

---

**Weakness #1: Ambiguity in Dataset Annotation Process**

The dataset annotation process described in Section 2.2 is ambiguous, with multiple steps left unclear. Specific questions regarding this process include:

(1-1) Source of Translations: How were the English and Chinese versions of each question obtained? Which version served as the source, and how were questions and answers translated—via large language models (LLMs) or human annotators?
  
(1-2) Quality Control and Verification: What steps were taken to ensure quality and correctness in the annotations? The authors mention “we check xxx” and “we screen out xxx” (Lines 202-211), but it is unclear who "we" refers to. Was this verification conducted solely by the authors or by additional human annotators? Additionally, was the check single-pass, or did multiple judges perform repeated rounds of verification?

(1-3) Presence of Bilingual Images: The K12 questions listed in the Appendix indicate that many images contain Chinese characters. Are English versions of these images also available?

(1-4) Question-Image Pairing: In Table 1, the authors report 3k questions and 3k images. Does each question have an associated image, or are there cases where a single question is linked to multiple images?

---

**Weakness #2: Lack of Clarity in Experimental Setup and Analysis**

Certain aspects of the experimental setup are unclear, and additional analysis is needed for some experimental results. Specific questions include:

(2-1) Input for English-specialized MLLMs: In Lines 1052-1053, the authors state that “The English version of VisScience is designed to facilitate the evaluation of MLLMs that specialize in English, assessing their capabilities in scientific reasoning.” Were English versions of the images provided as inputs to these MLLMs during evaluation?

(2-2) Unexpected Performance of LLMs vs. MLLMs: In Table 2 and Table 3, text-only LLMs that accept text questions but no image inputs sometimes outperform MLLMs with access to both text questions and images. Why might this discrepancy occur? 

(2-3) Questionable Necessity of Image Inputs: A followup question of (2-2) -- the accuracies on some text-only LLMs are higher than random guess (25% for multiple choices with 4-choice), is it because the image inputs are not necessary, or is it because the LLMs have already been posted to these questions during training that they don't even need to look at the paired image input to answer?


---


**Weakness #3: Benchmark Score Evaluation Methodology**
  
The evaluation method for benchmark scores is unclear, particularly in Section 3.1, Lines 316-317, where the authors mention GPT-4o as the judge. Questions related to score evaluation include:

(3-1) Handling Multiple Correct Answers: For example, in the first example in Figure 11 on page 30, a multiple-choice question has multiple correct answers (ground truth C & D). If a model selects only C, is this considered correct, partially correct, or incorrect? How is accuracy calculated in such cases?

(3-2) Accuracy for Multi-part Free-form Questions: In Figure 19 on page 38, a free-form question contains multiple sub-questions. How is accuracy determined here? Do all sub-questions carry equal weight, and how many must a model answer correctly for GPT-4o to deem the entire question correct?

(3-3) Weighting Free-form vs. Multiple-choice Questions: Free-form questions often contain multiple sub-questions. When reporting overall accuracy, do these free-form questions carry the same weight as single-answer multiple-choice questions? For instance, if a free-form question has five sub-questions, how is the overall score calculated?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weaknesses section. The two points mentioned will impact the practical contributions of this paper, and I hope the authors can provide a detailed explanation. Moreover, VisScience includes the disciplines of physics, chemistry, and mathematics. Mathematics is already widely recognized as an essential foundational capability. However, is the significance of physics and chemistry limited to specific application scenarios for the model? I also hope the authors can provide a more detailed interpretation of the motivation and significance of these two disciplines from the perspective of model evaluation.
soundness: 2
strengths: 1. The paper presents the VisScience benchmark, encompassing multiple disciplines to provide a more comprehensive evaluation guide for evaluating MLLMs.
2. The paper conducts extensive experiments and performs error analysis across different disciplines.
summary: This paper proposes VisScience, a benchmark for evaluating MLLMs, covering multiple disciplines such as mathematics, physics, and chemistry, with a total of 3000 questions. The authors aim to distinguish VisScience from existing benchmarks through "difficulty level", "bilingual" and ultimately present a more comprehensive set of experimental results.
weaknesses: 1. **Bilingual Dimension:** As a benchmark for evaluating MLLMs, the language of the text within images is highly significant. The authors present two languages in VisScience. However, through the Appendix, it can be observed that in English questions, most images still contain Chinese text.
2. **Difficulty Level Classification:** The authors describe the use of LLM for classifying the difficulty levels, while VisScience is a multimodal benchamrk. I have some concerns about using text alone for classification without considering the images. For instance, in a math problem asking about an area, the complexity presented by the image can vary greatly, which cannot be detected by simply using LLM for classification.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: Will the authors be making the VisScience benchmark and the associated evaluation tools publicly available? If so, what are the plans for supporting the community in using these resources?

The paper categorizes errors into several types, but does not provide specific recommendations for model improvement. Could the authors provide more detailed insights or suggestions on how to address the most common error types observed?
soundness: 3
strengths: The paper presents the VisScience benchmark, which is an original contribution to the field of multi-modal large language models (MLLMs). It addresses a significant gap in the evaluation of MLLMs by extending the scope beyond mathematics to include physics and chemistry, covering a comprehensive range of scientific disciplines. The benchmark's design, integrating questions from K12 education and classifying them into detailed subjects and difficulty levels, is a creative approach that provides a more nuanced assessment of MLLM capabilities.

The quality of the VisScience benchmark is evident in its thorough construction and rigorous evaluation process. The dataset is meticulously curated, with 3,000 high-quality questions selected to represent a breadth of knowledge points and difficulty levels. The authors' commitment to ensuring the benchmark's reliability through multiple checks and annotations reflects a high standard of quality in dataset curation.
summary: The paper introduces a comprehensive benchmark designed to assess the capabilities of multi-modal large language models (MLLMs) across various scientific disciplines, including mathematics, physics, and chemistry. The benchmark, named VisScience, comprises 3,000 questions drawn from K12 education, evenly distributed across the three disciplines, and covers 21 distinct subjects categorized into five difficulty levels. The authors conducted extensive experiments with 25 representative MLLMs, revealing that closed-source MLLMs generally outperform open-source models in scientific reasoning tasks. Notable performances include Claude3.5-Sonnet with 53.4% accuracy in mathematics, GPT-4o with 38.2% in physics, and Gemini-1.5-Pro with 47.0% in chemistry. The paper highlights the strengths and limitations of MLLMs and underscores the importance of developing models that can effectively handle multi-modal scientific reasoning.
weaknesses: While the paper provides a general categorization of errors, a more in-depth analysis of the types of errors made by MLLMs across different subjects could offer more targeted insights for model improvement. Suggestions for specific model enhancements or training techniques based on error analysis would be valuable additions.