OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024)

Abstract: Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at https: //github.com/OpenBMB/OlympiadBench

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks such as text generation (Zhao et al., 2023), code generation (Zan et al., 2023) and mathematical reasoning (Lu et al., 2023; Zhou et al., 2023), garnering significant attention from both academia and industry (Wei et al., 2022; Zhao et al., 2023; Bubeck et al., 2023). The most powerful models
*Corresponding authors: Xu Han and Zhiyuan Liu.
such as GPT-4 (OpenAI, 2023a) and Gemini Ultra (Team, 2023) have even surpassed oridinary human level on a wide variety of benchmarks such as MMLU (Hendrycks et al., 2020), MMMU (Yue et al., 2023), and even surpassing human expert in many area. These results show a promising future that LLMs can serve as proficient assistants for human scientists (Nguyen, 2023; Qiu et al., 2023). Among the array of expert-level skills exhibited by LLMs, scientific reasoning consistently emerges as one of the most brilliant, showcasing some of the most distinguished intellectual properties that experts possess. Therefore, this paper primarily focuses on mathematical and physical reasoning.
In recent years, several benchmarks related to mathematics have been proposed, such as the
3828
dataset GSM8K (Cobbe et al., 2021) as well as the dataset MATH (Hendrycks et al., 2021). However, these benchmarks, are primarily developed before the advent of highly capable LLMs, and now lack sufficient challenge for the latest models. For instance, GPT-4 with prompting techniques(Zhou et al., 2023) has achieved a 97.0% success rate on GSM8K and 84.3% on MATH. The rapid evolution of LLMs may soon lead to saturated results on these benchmarks. Concurrently, LLMs are not yet fully equipped to assist mathematicians in solving complex problems (Collins et al., 2023; Zhang et al., 2023), nor are they capable of performing expert-level mathematical reasoning independently. This discrepancy underscores the need for more challenging datasets to benchmark future advancements of LLMs in this domain. Similarly, physics presents comparable challenges for AI to those found in mathematics. Nevertheless, existing benchmarks related to physics (Lu et al., 2022; Arora et al., 2023; Wang et al., 2024) are characterized by their relatively low difficulty and limited scope. There is also a significant lack of a rigorous and challenging benchmark in physics.
In addition to the issue regarding the benchmark difficulty, it is important to note that these benchmarks predominantly focus on text. This presents a significant limitation, as a wide range of scientific reasoning contexts require multimodal reasoning abilities. For example, grasping geometry reasoning in mathematics or understanding experiments designs in physics are scenarios where multimodal reasoning capabilities are crucial. Notably, various large multimodal models (LMMs) have been developed (Team, 2023; Liu et al., 2023) and demonstrate proficiency on a variety of tasks (Lu et al., 2022; Yue et al., 2023; Zhang et al., 2024b; Lu et al., 2024), offering the potential for multimodal scientific reasoning. Nevertheless, there is still a lack of sufficient benchmarks to prove whether these LMMs are capable of handling scientific problems. Consequently, a challenging multimodal benchmark is essential for advancing scientific reasoning tasks(Zhang et al., 2024a; Lu et al., 2023).
To address the aforementioned inadequacies, we introduce OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark. This collection comprises 8,476 math and physics problems sourced from International Olympiads, Chinese Olympiads, and the most challenging segments of the Chinese College Entrance Exam (GaoKao). We download PDF data from official
websites and utilize Mathpix1 for OCR parsing. We meticulously inspect, clean, and revise the data, and further adopt LLMs for deduplication. Finally, we annotate the data with crucial information such as answer types and subfields, yielding a dataset that is clean, accurate, and detailed. As shown in Figure 1, OlympiadBench features numerous distinct characteristics such as difficulty, free-form generation, expert-level solution annotation, detailed labeling of difficulty, wide-coverage of modality and language, etc. These features are summarized more clearly from Table 1.
We conduct an evaluation of current state-of-theart LLMs and LMMs on the OlympiadBench. The best-performing model, GPT-4V, is a multimodal version of GPT-4 developed by OpenAI that can understand images. Despite its advanced capabilities, GPT-4V achieves a score of only 17.97% on OlympiadBench, with individual scores of 21.70% in mathematics and 10.74% in physics.
Importantly, the experiment results show that LMMs still struggle in computational error, incorrect reasoning or induction. For the process involved in the correct responses, the process occasionally includes hallucinated reasoning, or choosing a more complex solution when a simpler solution exists. All these results highlight the substantial challenge OlympiadBench presents to contemporary large models and point the direction of future efforts.
OlympiadBench is inspired by the significant advances made by DeepMind AlphaGeometry (Trinh et al., 2024), which nearly matches the proficiency of International Mathematical Olympiad (IMO) gold medalists in geometry proofs. It is clear that OlympiadBench, along with other challenging datasets like the AI-MO challenge2, will witness and benchmark the swift progress towards expertlevel AI assistants for solving scientific problems.

heading: 2 Related Work
text: This section gives an overview of the existing datasets in solving mathematics and physics problems as well as multimodal datasets.
Mathematics Benchmarks. Solving mathematics problems and proving theorems in natural languages has been a key research focus in machine learning and natural language processing since the 1960s (Bobrow et al., 1964). Previous bench-
1https://mathpix.com/ 2https://aimoprize.com/
marks (Koncel-Kedziorski et al., 2016; Wang et al., 2017; Ling et al., 2017; Amini et al., 2019; Cobbe et al., 2021; Wei et al., 2023) focus predominantly on math word problems (WMPs) which involve four basic arithmetic operations with single or multiple operation steps (Lu et al., 2023). Typically, the GSM8K (Cobbe et al., 2021) dataset targets elementary-level questions within 8 steps of basic arithmetic operations. However, these problems are typically text-only (Lu et al., 2023) and of lower difficulty, with reasoning limited to a few computations. As the complexity of the problems rises, some works (Hendrycks et al., 2021; Frieder et al., 2023; Arora et al., 2023) introduce competitionlevel problems integrating mathematical logic and background knowledge. Yet, these challenging datasets are increasingly being surmounted (Zhou et al., 2023). Theorem proving is a problem to demonstrate the truth of a mathematical claim (a theorem) through a sequence of logical arguments (a proof) (Lu et al., 2023). Earlier efforts mainly focused on translating natural language proofs into formal representations, facing significant expertise and labor challenges (Zheng et al., 2022; Welleck et al., 2021). The emergence of LLMs has facilitated notable advancements in the domain of natural language proof (Jiang et al., 2023). OlympiadBench presents mathematical reasoning and theoretical proofs all in natural language with detailed solution annotations.
Physics Benchmarks. Physics questions in SciQ (Welbl et al., 2017), ScienceQA (Lu et al., 2022) and E-EVAL (Hou et al., 2024) are mainly elementary and high school level multiple-choice questions, lacking complex reasoning and computational tasks. In MMLU-STEM (Hendrycks et al., 2020) and C-Eval-STEM (Huang et al., 2023), physics questions also adopt a multiple-choice format. JEEBench (Arora et al., 2023) extends this format to include multistep reasoning with physics knowledge, yet it is limited in scope and purely text-only. TheoremQA (Chen et al., 2023) is the first theorem-driven question-answering dataset. Curated by domain experts, it contains 800 highquality questions that encompass 350 theorems from Mathematics, Physics, Electrical Engineering and Computer Science (EE&CS), and Finance. SciEval (Sun et al., 2023) consists of a total of about 18,000 challenging scientific questions, spanning three important basic science fields: chemistry, physics and biology. SciBench (Wang et al., 2024) and OCWCourses (Lewkowycz et al., 2022) of-
fer college-level physics questions in free-response formats, where SciBench contains multimodal information. In contrast, OlympiadBench escalates in difficulty, diversifies in question types, and surpasses in volume, setting a new benchmark for complexity and variety in the domain.
Multimodal Benchmarks. For assessing multimodal capability, works such as Geometry3K (Lu et al., 2021), GeoQA (Chen et al., 2021), GeoQA+ (Cao and Xiao, 2022), and UniGeo (Chen et al., 2022) have employed multimodal information for tackling geometric problems, integrating natural language descriptions with diagrams. ScienceQA (Lu et al., 2022), MMMU (Yue et al., 2023), CMMMU (Zhang et al., 2024b) and CMMU (He et al., 2024) are multimodal, multidiscipline evaluation sets, encompassing a broad range of subjects. MathVista (Lu et al., 2024) integrates 28 existing and 3 newly constructed multimodal datasets involving mathematics, aiming to establish a benchmark that encapsulates challenges from a variety of mathematical and visual tasks. However, it does not concentrate on delving into the complexity of mathematics problems.
In summary, we introduce a new benchmark to address these gaps. Table 1 presents a comparison between OlympiadBench and several related benchmarks, highlighting the significant advantages of OlympiadBench across all aspects.

heading: 3 The OlympiadBench Dataset
text: To evaluate the reasoning abilities of LLMs and LMMs in mathematics and physics problems, we have created OlympiadBench, a bilingual and multimodal scientific benchmark at the competition level. This section provides a detailed account of the construction process of OlympiadBench. Summarized statistics of the dataset is shown in Table 2, and more detailed statistics are in Appendix A.2.

heading: 3.1 Design Principle
text: The motivation behind OlympiadBench is to establish a benchmark that represents the pinnacle of human intellectual achievement, thereby encouraging researchers of large models to push the boundaries of mathematical and physical reasoning capabilities. We focus on curating challenges that epitomize the highest level of competition worldwide. Specifically, OlympiadBench includes:
1. Inclusion of Olympiad-Level Problems. We collect mathematics and physics problems
from the International Olympiad competitions, which cater to the most outstanding high school students in a region. These problems are open-ended, differing from traditional multiple-choice or fill-in-the-blank formats. This selection aims to more accurately reflect the complexity of advanced scientific reasoning, providing insight into the actual reasoning process of the models.
2. Provision of Detailed Solutions. Given the advanced difficulty of these problems, which may exceed the comprehension of individuals
without a specialized background in mathematics, each problem is accompanied by expertly crafted solutions that detail the reasoning steps involved. This approach can not only reduces the difficulty of annotation and evaluation but also enhances the accuracy of the solutions provided. Furthermore, detailed expertlevel solutions are valuable for research in model reasoning.
3. Incorporation of Visuals. Recognizing the crucial role of visual information in conveying complex ideas, OlympiadBench incorporates problems that require understanding images, identifying spatial relationships, and other advanced reasoning tasks. This inclusion aims to assess the model’s capabilities in interpreting visual data as part of its reasoning process.
4. Minimization of Data Leakage Risks. To minimize the risk of data leakage, we have sourced problems from official Olympiad competitions, converting them from their original PDF files provided by official websites to the markdown format required. This strategy is aimed at reducing the likelihood of the data being inadvertently incorporated into the pre-training corpora of models.
Through these carefully designed criteria, OlympiadBench aims to not only challenge but also significantly advance the capabilities of
models in mathematical and physical reasoning.

heading: 3.2 Data Processing
text: The data processing pipeline is structured into three distinct phases: data collection, format conversion & deduplication, and classification labeling.
Data Collection. OlympiadBench is meticulously compiled from three primary sources: Global Mathematics and Physics Olympiad Problems, Regional and National Chinese Math Competitions, and Gaokao Mock Questions for Mathematics and Physics 3. Each chosen for its distinct advantages in creating a robust and comprehensive benchmark for evaluating LLMs and LMMs in mathematical and scientific reasoning. Their challenges progressively increase in difficulty, not only distinguishing the reasoning capabilities of models of various sizes but also offering guidance on scaling laws (Kaplan et al., 2020) for specialized models in these domains.
Format Conversion and Deduplication. After collecting all PDF files, we utilize the Mathpix tool for OCR recognition and convert them into markdown format. However, no conversion process is flawless, necessitating manual verification by our team members between the original PDF files and the converted Markdown texts. The Markdown texts are further structured into a format akin to "Problem—Solution—Answer", employing its markup language for text organization. Subsequently, we leverage a specialized small-scale language model 4 trained on mathematical symbol corpora for vectorizing the data and performing deduplication based on cosine similarity.
Classification Labeling. We note that both mathematics and physics problems predominantly comprise two types of questions: the open-ended problems and the theorem proving problems. We also note that the dataset, enriched by both Olympiad and national examination questions, covers a broad spectrum of subfields, as illustrated in Figure 2. Therefore, we manually annotate each question with topic and problem type annotations.

heading: 3.3 Data Characteristics
text: In contrast to previous benchmarks, OlympiadBench unveils two unique characteristics within its dataset: the incorporation of Progressive Prob-
3Clean and correct college entrance exams as well as mock questions from Wisdom Way AI Lab.
4https://huggingface.co/Laurie/ Bloom1b7-deepspeed-chat-Chinese-math
lems in Physics and the categorization of answers to most open-ended questions into a limited number of types.
Progressive Problems in Physics. In physics competitions such as the International Physics Olympiad (IPhO), problems are often structured around a common material or scenario, with subsequent questions potentially relying on the answers or information from previous questions. One example is given in Figure 10 and Figure 11. This design characteristic is commonly referred to as "progressive problems". By linking a series of questions together, progressive problems require participants to apply their knowledge and skills comprehensively to gradually solve more complex issues. This type of question design aims to test students’ depth of understanding, application capabilities, and innovative thinking, rather than just basic knowledge. To better utilize this feature, we compile the material, questions, and their answers into the ’context’ field for each set of progressive problems.
Answer Type Classification. Whether in mathematics or physics, the answers to problem requiring
definitive response can largely be categorized into the following types: numeric, expression, equation, interval, and tuple. Simple examples of these can be seen in Table 3.

heading: 3.4 Automatic Scoring Pipeline
text: We design an automated scoring pipeline (see Algorithm 1) to evaluate model-generated answers across complex fields like mathematics and physics, where answers vary from numbers to equations. This method simplifies answers into two categories: numeric values, handled through floating-point operations, and symbolic expressions, requiring symbolic computation.
For equations, we ensure all terms are on one side before dividing to check for mathematical equivalence. Intervals and tuples are compared by extracting and evaluating each element. Numeric answers are verified against a small tolerance of error, defaulting to 1e-8 but adjustable for physics problems to allow for a specific error margin. For expressions, we use the SymPy5 library to confirm if the subtraction of two expressions approaches zero, indicating correctness.

heading: 4 Experiments

heading: 4.1 Settings
text: We conduct evaluations of open-source and closedsourced LMMs that have been selected with consideration of their comprehensive capabilities on OlympiadBench. At the same time, we have selected LLMs with strong mathematical and logical abilities for evaluation on text-only questions.
As no accurate automatic evaluation method for theorem proving exists, we run full experiment on the automatic-scoring-available open-ended problems with answer type included in the Table 3, which is discussed in this section. We do manual sampling check of GPT-4V for theorem proving problems with analysis reported at Section 5.1.

heading: 4.1.1 Prompts
text: We evaluate the models in a zero-shot setting. Due to the high difficulty of the OlympiadBench questions, there should be considerable randomness in the results when using small batch data as the validation set, so we directly use a specific prompt template for all models instead of conducting promptengineering for each model respectively. The
5https://www.sympy.org
prompt template for English and Chinese openended questions is shown in the figure 3. To ensure the most complete extraction of the model’s final results, we explicitly prescribe the types and formats of the answers in the prompt to promote the accuracy of the machine’s automatic scoring. In order to test the native mathematical and physical abilities of the models, the prompts used in the test do not introduce knowledge points and other extra information contained in the dataset, but this information can be applied in subsequent research. Note that deepseek-math-7B-RL (Shao et al., 2024) requires the addition of a specific chain-of-thought prompt at the end of the input, which we adhered to during the evaluation.

heading: 4.1.2 Evaluation Workflow
text: We first apply each model to generate answers for questions in OlympiadBench using prompts formed by prompt template, with open-source models running on NVIDIA A800 GPUs. Then, we run the automatic scoring pipeline to judge the correctness of the answers as described in subsection 3.4. Finally, we calculate the micro-average accuracy as the comparing metric. The code of the whole workflow is provided in the supplementary material.

heading: 4.2 Baselines
text: In our study, we evaluate the performance of current leading bilingual large multimodal models (LMMs), as well as bilingual large language models (LLMs) that has strong mathematical and reasoning abilities. We take both open- and closedsource models into consideration, using either the largest and latest released checkpoints or the bestperforming official APIs available.
For LMMs, we select GPT-4V(GPT-4Vision) (OpenAI, 2023b), Gemini-ProVision (Team, 2023), Qwen-VL-Max (Bai et al., 2023) for closed-source models, while YiVL-34B (01-ai, 2024) and LLaVA-NeXT-34B (Liu et al., 2024) for open-source models. For models that demand compulsory image input, we take their LMM counterpart (corresponding text-model API or base LLM) for evaluation. Specifically, for LLaVA-NeXT-34B, we use its base LLM, Nous-Hermes-2-Yi-34B (NousResearch, 2023). The text model corresponding to Yi-VL-34B is Yi-34B-Chat (01-ai, 2023). Similarly, for the Gemini-Pro-Vision, we utilize the Gemini-Pro API interface. To examine the impact of replacing LMM with base LLM for processing text-only
data, we subsequently compare the performance differences between GPT-4V and GPT-4 6 on text-only questions in OlympiadBench.
For LLMs, we select DeepSeekMath-7BRL (Shao et al., 2024) as the primary baseline for text-only questions, and report the results of the selected LMMs (or their LLM counterparts) on the text-only questions for comparison, and additionally evaluate GPT-4 as described above.

heading: 4.3 Main Results
text: The overall experiment result is shown in table 4. Based on the results, our key findings can be summarized as the following:
OlympiadBench is more challenging than existing benchmarks, which provides new perspective to compare LMMs. As shown in table 7, the most advanced model only achieves an average accuracy of 17.97% on OlympiadBench, which is much lower than that of existing benchmarks. Moreover, the gap between the models has been widened, thereby becoming more significant, which helps people to compare the differences in capabilities between different models more accurately.
There still exists a huge difference between the most powerful closed-source models and open-source models, but a large model size is needed. The average accuracy of GPT-4V is more than 5 times larger than the best-performing open-source model (Yi-VL-34B). But Gemini-ProVision, being closed-source models of the secondtier size, is much less compatible on complicated tasks such as OlympiadBench, for it achieves an average accuracy that is only slightly higher than open-source model.
6The version of GPT-4 is "0125-preview" and GPT-4V is "1106-vision-preview".
The challenge lies more on question-withimages, Physics and none-English text. The model performance on text-only questions is significantly above average, showing the challenging spirit of multi-modal questions. Meanwhile, Physics questions, especially Physics questions with images, are more challenging than math questions, as they require knowledge of the laws of Physics as well as other world knowledge besides mathematical abilities such as calculation and reasoning. Moreover, LMMs with a focus on bilingual image-text training data, such as Qwen-VL-Max and Yi-VL-34B, perform better on Chinese questions then English questions.
Open source LLMs is catching at fast speed in the area of maths and physics. Although with a relatively small size, DeepSeekMath-7BRL outperforms or is on par with Gemini-ProVision and Qwen-VL-Max on the text-only part of OlympiadBench, especially in Math problems, showing promising future of open-source model of pre-training and fine-tuning on fine-grained mathematical and reasoning data.
Multi-modal training slightly hurts performance on text-only math and physics tasks, but may also bring some improvement. The text-only version GPT-4 performs slightly better on all datasets of OlympiadBench, except for the En_COMP dataset. We hypothesis that the improvement in the En_COMP dataset shows an enhancement of long-context text reasoning capabilities, which is discussed in Appendix B.2.

heading: 5 Analysis
text: In this section, we conduct analysis on the GPT4V’s answers of specific open-ended questions that have been sampled, as well as giving preliminary
examination of theorem proving questions.

heading: 5.1 Examination of Theorem Proving Questions
text: For GPT-4V, we do manual sampling check to evaluate the mathematical theorem proving questions. In the questions drawn according to the knowledge point distribution, GPT-4V only answers 6 out of 81 questions correctly in Math-Zh_COMP, all of which are relatively simple and classic conclusions (e.g. AM-GM inequality), or involved only simple computational derivations, and was basically unable to complete the proof within the token limitation in Math-En_COMP, indicating that existing models still cannot effectively solve lengthy reasoning and proofs, which is consistent with the conclusions in existing papers (Trinh et al., 2024).
In solving proof problems, GPT-4V exposes several important issues, including: inability to fully utilize image information (figure 9 as an example); tending to make mistakes in simplifying and transforming algebraic expressions; proposing simple, basic incorrect conclusions;struggling with classification discussions, etc. Detailed examples can be found in the Appendix C.

heading: 5.2 Mistake Analysis of GPT-4V
text: We manually sample and check 97 maths (55 for English and 42 for Chinese) and 67 physics Olympiad-level open-ended problems that GPT-
4V fails, and analyze the type of mistakes, the overall results are shown in figure 4. In maths problems, the typical errors of GPT-4V include: insufficient classification discussion, especially in combinatorial problems; poor performance in problems requiring large calculations (e.g. conic curve problems), manifests as a lack of logic in the calculation process, resulting in the model being unable to provide a reasonable answer. However, we also found that GPT-4V has strong abilities in solving quadratic equations and derivative problems. In physics problems, GPT-4V tends to fall in conceptual confusion, or introduce unnecessary variables or concepts, but its capability to simplify and transform algebraic expressions is stronger than in purely mathematical situations, with nearly no numerical calculation errors.

heading: 6 Discussion and Future Work
text: Here, we discuss the automated evaluation of theorem problems and the future research directions in advanced disciplines.
Evaluation of Theorem’s Proof Problems. The inability to automatically evaluate theorem’s proof problems remains a significant challenge today. Currently, mainstream methods for the automatic evaluation of proofs require formalization, necessitating domain expertise and background knowledge. Exploring how to automatically evaluate natural language proofs represents an important research direction. Our dataset includes expert-level comprehensive annotations in a fusion of natural language and LaTeX formats, making it a highquality resource for research in natural language proof problems and fostering further development.
Expansion of Disciplines. Mathematics, much like physics, serves as a litmus test for artificial intelligence, requiring a strong foundational knowledge, rigorous high-level computations, and precise logical reasoning. Currently, large models still face significant challenges in mathematics and physics, which are critical hurdles that must be overcome for the development of AGI. In our future work, we will integrate additional fields such as geography, biology, and chemistry to provide a more innovative and comprehensive evaluation of a model’s reasoning capabilities.

heading: 7 Conclusion
text: We create OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark to assess the capabilities of large models in mathematics and physics reasoning. Each problem is detailed with expert-level annotations for step-by-step reasoning. In our benchmarking, we provide a detailed analysis of model performance, pinpointing prevalent error types and potential areas for enhancement. This significant and challenging effort fills a notable void, and we intend to open-source the benchmark to advance AGI and scientific reasoning research. Future efforts will focus on gathering more challenging questions and broadening the range of subjects to further develop rigorous scientific benchmarks.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (No.2022ZD0116312), National Natural Science Foundation of China (No.
62236004), Quan Cheng Laboratory (Grant No. QCLZD202301) and Institute Guo Qiang at Tsinghua University.

heading: Ethical Considerations
text: In this paper, we introduce OlympiadBench, a highly challenging bilingual, multimodal scientific benchmark aimed at evaluating the mathematical and physical reasoning of large models now and AGI in the future. The paper outlines the dataset construction, including data gathering, OCR processing, cleansing, deduplication, and detailed annotation. OlympiadBench’s data, derived exclusively from official sources, substantially reduces the likelihood of pre-training data leakage. We offer precise annotations for each problem and have implemented an exhaustive evaluation script for more accurate model performance assessment. Additionally, being bilingual and providing expert-level reasoning annotations for every question, OlympiadBench serves as a crucial resource for propelling AGI’s prowess in scientific reasoning. Committed to environmental sustainability, we intend to release the dataset and accompanying scripts publicly to cut down on unnecessary carbon footprint. In experiments, we comply with all licenses for models and data.

heading: Limitations
text: In pursuit of understanding the logical reasoning abilities of LLMs and LMMs within the multimodal domains of mathematics and physics, we develop OlympiadBench, a challenging bilingual multimodal scientific benchmark. Despite filling a notable void, this work acknowledges inherent limitations. First, in the OlympiadBench, some questions feature answers that require categorical discussions or textual descriptions, such as proofs, which currently cannot be assessed using regular expressions or tools like SymPy at the code level and necessitate manual review. However, this data holds significant research value. Secondly, the automated scoring system we propose cannot perform specific analysis based on the particulars of each question. It makes logical judgments solely based on the two symbols or numerical expressions inputted, without integrating any special constraints that may exist within the actual problem context. What’s more, the development of datasets for multimodal scientific reasoning requires extensive manual effort in gathering and annotating data,
which constrains the diversity and difficulty of multimodal scientific challenges. As a result, this hampers AI’s capacity to learn from and address more intricate scenarios.

heading: A Dataset Details

heading: A.1 Data Sources
text: Our data sources can be split into the following three parts:
1. Global Mathematics and Physics Olympiad Problems. The Mathematics and Physics Olympiad problems are globally recognized for their complexity and quality. These problems often require multiple methods of solution and the ability to integrate sub-disciplines from within the broader fields of mathematics and physics. The participants in these competitions represent some of the most proficient individuals worldwide in logical reasoning within mathematics and physics. This not only sets a high standard for problem-solving but also fosters a diverse set of analytical skills that are crucial for the advancement of large models.
2. Regional and National Chinese Mathematics Competitions. In addition to maintaining a high level of difficulty, regional competitions and the CMO introduce elements specific to the Chinese context. This inclusion is instrumental in furthering the development and research of Chinese-oriented and multilingual large models. By encompassing a wide array of mathematics and physics problems, these competitions provide a unique opportunity to develop models that are adaptable and proficient across different mathematical queries, enhancing their versatility and effectiveness.
3. Gaokao Mock Questions for Mathematics and Physics. Given that the resolution of Olympiad-level problems typically necessitates models with substantial parameter sizes, we also incorporate Gaokao simulation problems to evaluate smaller models’ capabilities in answering free-form mathematics and physics questions.
The integration of data from Gaokao simulation problems, regional and national competitions, to the global Olympiads constructs a smooth difficulty transition curve. This methodology not only distinguishes the mathematical and physical problemsolving capabilities of different models but also provides guidance on the scaling laws for models specialized in these domains.

heading: A.2 Data Curation Process
text: Our initial step involves a comprehensive survey of well-known Olympiad competitions, and the list of which is accessible through the AoPS community platform 7. We cataloged these competitions based on several criteria: difficulty level, volume of questions, availability of materials in public domains, language, discipline, and coverage years. Following the design principles outlined in Section 3.1, we meticulously select specific contests and years that not only adhere to our dataset design criteria but also try to span the widest possible range of years (Table 5).
In the format conversion phase, we also manually annotated the subfield of each question in maths or physics, with their distribution detailed in Table 9.

heading: A.3 Example of Progressive Problem in Physics
text: Figures 10 and 11 present a sequential challenge from the International Physics Olympiad (IPhO) 2021, illustrating the intricacies of progressive problem-solving in a competitive context. This particular problem set exemplifies a common trait in advanced physics competitions: the dependency of many questions on the solutions and materials of preceding ones. These dependencies are sometimes explicit, but most are implicit, weaving a complex web of interconnected knowledge and reasoning.
An explicit instance of this dependency can be observed in problem C.2, where the prompt directly requires the use of the symbol β defined in B.1 for the calculation of an unknown quantity. This re-
7https://artofproblemsolving.com/community/c13
quirement not only tests the participants’ ability to understand and apply physical concepts but also assesses their skill in navigating through and linking various parts of a problem set. Such explicit instructions are crucial for guiding participants through the logical progression of the problems, yet the majority of dependencies remain implicit, demanding a deeper level of comprehension and integration of the material.
This structure of problem-solving reflects a realistic scientific inquiry, where discoveries and solutions often rely on previously established knowledge. The explicit mention of β in C.2 as derived from B.1 is emblematic of this educational approach, aiming to foster a holistic understanding and the ability to build upon existing information to solve complex problems. It underscores the importance of thorough comprehension of earlier sections for successful problem-solving in later sections, simulating real-world scientific challenges where new solutions are often predicated on a foundation of established knowledge.

heading: B Evaluation Details

heading: B.1 Details of the Evaluated Models
text: B.1.1 LMMs We have selected current mainstream LMMs that have performed the best on past scientific multimodal datasets for evaluation.
The closed-source models include: GPT4V (OpenAI, 2023b), developed by OpenAI, which is currently the most powerful multimodal model. Gemini (Team, 2023) is the LMM series developed by Google Deepmind, with Gemini-UltraVision being purported to have surpassed GPT-4V
on datasets like MMMU. However the unavailability of Google’s API for Gemini Ultra, we test the accessible Gemini-Pro-Vision as an alternative. Qwen-VL-Max (Bai et al., 2023), developed by Alibaba, is the largest LMM, and stands on par with GPT-4V and Gemini-Ultra in multi-modal tasks. Due to the large proportion of Chinese data used in its training, Qwen-VL-Max has a certain advantage in Chinese language ability.
The open-source models include: Yi-VL34B (01-ai, 2024) is the first open-source 34B multi-modal model that has demonstrated satisfying performance on several latest datasets. With Chinese text-image pairs included in the training process, Yi-VL-34B offers adequate multilingual support. LLaVA-NeXT-34B (Liu et al., 2024) claims to be the strongest open-source LMM, with enhancements in reasoning, OCR, and world knowledge. Despite being trained exclusively with English multi-modal data, it demonstrates an emergent zero-shot Chinese multi-modal capability on Chinese benchmarks.
It should be noted that an image must be passed for Gemini-Pro-Vision, LLaVA-NeXT, and Yi-VL during inference. Therefore, for the text-only questions in OlympiadBench dataset, we use the corresponding text-model api (for closed-source models), or their base LLM (for open-source models). To examine the impact of replacing LMM with base LLM for processing text-only data, we subsequently compare the performance differences between GPT-4V and GPT-4 on text-only questions in OlympiadBench.
B.1.2 LLMs The field of LLM starts early in scientific areas such as mathematics and physics, with models specifically trained occurring. We select DeepSeekMath7B-RL (Shao et al., 2024) as the primary baseline for text-only questions. DeepSeekMath-7BRL is pre-trained on 120B math-related data and enhanced chain-of-thought (CoT) reasoning capabilities using reinforcement learning, in the result scoring close to GPT-4 and Gemini-Ultra on the MATH (Hendrycks et al., 2021) dataset. We report the results of the selected LMMs (or their LLM counterparts) on the text-only questions for comparison, and additionally evaluate GPT-4 in order to compare with GPT-4V 8.

heading: B.2 Detailed Experiment Result
text: The comparison of the performance of mainstream closed-ended models on different datasets are shown in Table 7.

heading: Benchmark GPT-4(V) Qwen Gemini
text: To further discuss the performance difference between GPT-4 and GPT-4V on the PhysicsEn_COMP, we split the En_COMP dataset into two sub-datasets, with normal-PhO being normal
8The version of GPT-4 is "0125-preview" and GPT-4V is "1106-vision-preview".
PhO questions, and long-PhO being PhO questions that show in a relational series, therefore having long context. As shown in table 8, GPT-4 keeps performing slightly better on normal-PhO, but lags much behind on long-PhO, which may indicate improvement of long-context text reasoning capabilities after multimodal training.

heading: B.3 Unavailable Responses for Closed-Source Models
text: As described in table 4, the response for some problems are not available, the main causes are as follows:
1. Exceeding input limit: Some of the context of the problems are too long, which exceed the input token limitation for the API. This case mainly occurs in Physics-En_COMP that contains long-context problems of over 6,000 tokens.
2. Inappropriate response: Some problems trigger inappropriate response, which are banned by the API to return.
3. No response: Some problems continuously get no or empty response from the API.
4. Request timed out: Some problems continuously fail to get a response.
We removed the problems with unavailable response when calculating the accuracy.

heading: C Additional Analysis and Examples

heading: C.1 Performance analysis of GPT-4V
text: We analyzed GPT-4V’s performance (accuracy on open-ended problems) on different knowledge points based on the knowledge point labels in OlympiadBench, the results can be found at Figure 5.
For Math problems, GPT-4V has poor performance in geometry, with the lowest scoring knowledge points being almost exclusively geometryrelated. This may show the need of improving the
ability of understanding and imaging plane or 3d situations. GPT-4V also performs poorly on knowledge points that are more computationally intensive such as conic curves; and struggles to give a complete and comprehensive classification discussion, therefore prone to making mistakes on combinatorial problems. However, GPT-4V is stronger in knowledge related to derivatives and complex numbers.
As for Physics problems, none of the knowledge points surpass an accuracy of 16%, and GPT-4V struggles more in thermodynamics and mechanics.
C.2 Detailed Description of the Error Types in GPT-4V’s solving or proving process
The error types are as follows:
1. Question Misunderstanding: GPT-4V sometimes misunderstands the intention or settings of the question.
2. Value Calculation Error: GPT-4V make simple calculation mistakes sometimes, such as outputting b2 + 7 = b+7 2 , these mistakes ap-
pears more in Chinese and Math contents.
3. Expression Calculation Error: Similar to value calculation error, but happens when transforming between two expressions.
4. Logical Reasoning / Induction Error / Conceptual Confusion: GPT-4V sometimes makes false reasoning or induction, as well as encounters conceptual confusion (see Figure 7 for example).
5. Introducing Unnecessary variables or concepts: GPT-4V sometimes suddenly introduce variables or try to use concepts that have no contribution to solving the problem, which not only makes the output longer, but also may confuse GPT-4V itself and leads to incorrect output.
6. Conclusion Hallucination: GPT-4V sometimes hallucinates for a conclusion that is not reached in former output, or hallucinates a theorem that does not really exist (for example, when solving geometric proving problem, GPT-4V always mention "The Power Theorem", which does not exist, and all the proof thereafter will lost their logic).
7. Unfinished Answering: GPT-4V sometimes says the question have confliction in settings
(which is not true), or degenerates after some tokens.
8. Insufficient Classification Discussions: When doing classification discussion, GPT-4V may miss some possible situation, or have overlapped discussion (see Figure 6 for example).
9. Incorrect Judging: Sometimes GPT-4V gives the right answer, but is judged as incorrect due to the limitation of the automated scoring system: One important problem is that many problems, especially Physics problems, accept answers that fall in a specific range due to rounding up, rather than a fixed numerical answer, so a precision is needed for automatically calculating the range, which was not given in many cases. Using relative precision (such as a percentage of 1%), may leads to accepting answers with unacceptable error when the origin answer is big, so we manually decide the precision for OlympiadBench problems if there is no official precision given, which may leads to incorrect judging. For example, in a Physics question (Physics-En_COMP#995), the precision is manually set as 1e1, with the official answer being √ 2 × 3970 ≈ 5614.43. GPT-
4V calculates the answer as √ 2 × 3970 ≈ 1.414 × 3970 ≈ 5615.88, which falls out of the acceptable range, but is actually correct. Another problem is the current automated scoring system can’t judge the equality between expressions such as a √ b and √ a2b with a ≥ 0. These flaws call for future work such as automatically deciding required precision of the answer, and automatically judging equality between expressions.

heading: C.3 Other Observation for Open-Ended Problems
text: 1. Given a simple solution, GPT-4V may choose a more complex method to solve the problem (see Figure 8).
2. Models may give correct answers with a false process. Mainly observed for problems with a simple answer, such as the variables takes 0 as the answer.
3. GPT-4V may success in giving correct overall idea, but fail in calculation (such as solving quadratic equations with extra negative signs), which leads to a wrong answer.
4. GPT-4V may not fully utilize the information from the image (see Figure 9).

heading: D Automatic Scoring Pipeline
text: The pipeline workflow is shown in Algorithm 1.
Algorithm 1: Auto Scoring Judge Input: GroundTruth, ModelOutput; Output: Boolean value indicating match;
Preprocess GroundTruth and ModelOutput; if GroundTruth equals ModelOutput then
return True; else
if GroundTruth and ModelOutput are intervals or tuples then if GroundTruth equals ModelOutput as intervals or tuples then
return True; else
return False; end
else if GroundTruth equals ModelOutput numerically then
return True; else
if Both GroundTruth and ModelOutput contain "=" then return EquationEqual(GroundTruth, ModelOutput); else return ExpressionEqual(GroundTruth, ModelOutput);
end end
end end
Theory
Q2-1 English (Official)
Electrostatic lens (10 points) Consider a uniformly charged metallic ring of radius 𝑅 and total charge 𝑞. The ring is a hollow toroid of thickness 2𝑎 ≪ 𝑅. This thickness can be neglected in parts A, B, C, and E. The 𝑥𝑦 plane coincides with the plane of the ring, while the 𝑧-axis is perpendicular to it, as shown in Figure 1. In parts A and B you might need to use the formula (Taylor expansion)
(1 + 𝑥)𝜀 ≈ 1 + 𝜀𝑥 + 12𝜀(𝜀 − 1)𝑥 2, when |𝑥| ≪ 1.
Theory
Q2-2 English (Official)

heading: Future Work
text: These flaws call for future work such as automatically deciding required precision of the answer, and automatically judging equality between expressions