UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Authors: Xin Xu
Qiyun Xu
Tong Xiao
Tianhao Chen
Yuchen Yan
Jiaxin ZHANG
Shizhe Diao
Can Yang
Yang Wang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
benchmark and dataset
physics reasoning
Primary area: deep_learning->large_language_models
TL;DR: We introduce UGPhysics dataset along with MARJ answer assessment pipeline to advance AI for physics reasoning.

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs’ abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and diverse benchmark specifically designed to evaluate **U**nder**G**raduate-level **Physics** (**UGPhysics**) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese across 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (**MARJ**) pipeline specifically tailored for assessing physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the need for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning. Codes and data are available at \href{https://github.com/YangLabHKUST/UGPhysics}{https://github.com/YangLabHKUST/UGPhysics}.

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Xin Xu
Qiyun Xu 2
Tong Xiao
Tianhao Chen
Yuchen Yan 4 Jiaxin Zhang'
Shizhe Diao
Can Yang'
Yang Wang'
Abstract
Mechanics Thermody nasmies
Classical Mochanics; Theommtral Mochanics; Relativity: Themmody nemiiiss Stotistical Mechanics
Large language models (LLMs) have demon- strated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate--evel physics, underscoring the need for comprehen- sive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-Ievel physies problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoming skills, all screened for data leakage. Additionally, we develop a Mosia-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physies problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAl-ol-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements Al for physics reasoning. Codes and data are available how.ai///oooomaaaaaaaaaauu.
Uephissics
Electromagmetsm
Classical Flectromagnatimm Flaztrodynamics: Ceometrical Opties Wave Optics
Modera Pbysics
Mechunics, lomic Physics Sulid Physics Semiconductor Physics
Question: A non-mlativist a particle of mass in maves one dimansi on within potentaal VGx) 60x1. The particle is in bound stats, Find the value of such that the probebiity the particle being the region [+] xo is 1/2.
Solution: Bound
0),
m5(x) vGx) p(()
Discontinuity Condition: Using lim, J. dx on both sides yielde
('(0t) d'(0 2mm
vC00 (2)
Wavelunction From equation considering the finitrress
go:: (k Be Ai
Ac x
inr
Substauting into (2): 57 Therefore: d(x) = !).
Probabiiity P(ki < xx) = 21A1,, 1,"" exp (-2mu () de = 2|Al e: exp 2ma ())
N Normazzation; 1 ['_ [GG da -2lA]2 Aime
Condition: exp (-2ma
(2ma (t) 1/s, Solution for xo ln
A nnswr: In
Domain: Modern Physics
Subject: Qumtuum Mechanies
Reasoning Skill: Malt. D dlon
Figure 1. An overall illustration of UGPhysics. The top part repre- sents the hierarchical physics domains and subjects. The bottom part showcases one concrete example.
1. Introduction
"Physics is the foundation of all the natural sci- ences."
-Mxx Planck
Physics forms the foundation for natural sciences (Planck, 1949; Hawking, 1988; Giancoli, 2000), and physics prob- lem solving constitutes a significant aspect of reasoning for artificial intelligence (AI) (Bakhtin et al., 2019; Ding et al., 2023; Pang et al., 2024; Jaiswal et al., 2024a). Efforts to solve physics problems with machines date back to the mid-to-late 20th century (Larkin et al., 1980; Mendelson & Zelinski. 1984; Klahr & Waterman, 1986). After large language models (LLMs) have revolutionized the natural language processing community, significant attention has been paid to solving complex mathematical reasoning prob- lems, which span several areas, such as creating challenging benchmarks (Tang et al., 2024; Gao et al., 2024; Xu et al., 2025), exploring advanced prompting techniques (Wei et al.,
'Equal contribution 'The Hong Kong University of Science and Technology "Tsinghua University "The Universify of Science and Technology China 'Zhejiang University 'NVIDIA Research. Correspondence to: Can Yang <macyang@ust.ht ust.hk>..
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
2022; Wang al., 2022), applying supervised fine-tuning (SFT) (Tong al., 2024; Xu et al., 2024c), and leveraging continued pretraining strategies (Lewkowycz et al., 2022; Azerbayev et al., 2023). In contrast, despite its comparable, or even greater challenges for Al reasoning (He et al., 2024; Pang et al., 2024), physies has not yet garnered the same level of attention as mathematics.
tributes are outlined more clearly in Figure 1 and Table 1. To address the challenges of answer assessment brought by unique features of physics problems (e.g., physical con- stants, equivalent quantities defined in problem descriptions, examples given in Table 4), we develop Model-Assistant Rule-based Judgment (MARJ) (see Section 3.3) that com- bines the high calculation precision of rule-based judgment methods with the flexibility of model-based approaches. Hu- man evaluation has shown MARJ's reliability answer judgment of physics problems (see Section 5.2).
Early investigations into solving physics problems were of- ten studied alongside other scientific domains (Lewkowyez et al., 2022; Lu et al., 2022; Wang et al., 2023a). However, physics, an ancient and well-cstablished discipline, has its unique characteristics and deserves separate treatment for AL. Unlike mathematics, which predominantly relies on log- ical reasoning, physics problems often require additional domain-specific knowledge (e.g., laws and principles) for resolution (Bakhtin et al., 2019; Pang et al., 2024). More- over, physics problem-solving typically involves multiple applications of physical laws or formulas, making physics reasoning even more demanding than math reasoning. Thus, evaluating the capabilities of LLMs in physics reasoning of significant importance. Existing physics benchmarks (Zhong et al., 2023; Zhang et al., 2023; Arora et al., 2023; Ma et al., 2024; Ding et al., 2023; Jaiswal et al., 2024b) primarily target middle or high school physics and typically are multiple-choice questions, which are relatively easy for current LLMs to handle. Although some recent benchmarks have begun to explore competition-level (He et al., 2024; Huang et al., 2024b) or college-level (Hendrycks et al., 2020; Huang et al., 2024a; Li et al., 2023) physics, they remain limited in either scope or size of undergraduate-level physics (see Table 1), which encompasses broad range of topics and is widely used in educational assessments for humans (McDermott & Redish, 1999; Heller et al., 1992; Redish, 2003). These underscore the demand for a comprehensive benchmark specifically designed to evaluate the physics reasoning abilities of LLMs at the undergraduate level.
We perform comprehensive evaluation of 31 advanced LLMs, incorporating proprietary models, peneroa-purpose open-source models (e.g. Qwen-2.5-Inss Innssrructt (Yang et al., 2024a)), specialized math L.LMs (e.g. NuminaMath (Beech- ing et al., 2024)), and even ol oikee LLMs (e.g. DeepSeek- R1 11-Llama-70B (DeepSeek-Al et al., 2025)). The inclusion of math L.MSs aims to assess the extent to which training on specialized math corpus contributes to physics reason- ing. Despite LLMs' strong math reasoning abilities, the best overall accuracy achieved in UGPhysics is 49.8% by OpenAl-ol-ming (OpenAI, 2024b). These results highlight the challenges that UGPhysics pose to current LLMs in terms of physics problem-solving underscoring the signif- icance for future research with emphasis on physics as well. To summarize our key findings:
1. UGPhysics is a challenging dataset for LLMs in physics problem-solving with OpenAl-ol -mini achieving the high- est overall accuracy of 49.8%.
2. Unlike math problem-solving, math-specialized LLMss yield only minor improvements over their general-purpose counterparts in UGPhysics, suggesting the compulsion for more high-quality physics corpora.
3. Oi-like LLMs suggest a promising direction for future ad- vancements in physics reasoning. Among them, DeepSeek- I--istill-Llama--75 achieves the second-highest overall accuracy UGPhysics, though there remains a perfor- mance gap compared to top-tier closed-source LLMs.
In this paper, we introduce UGPhysics, large-scale and comprehensive benchmark tailored for evaluating the physics problem-solving abilities of LLMS across multiple UnderGraduate-level Physies (UGPhysies) disciplines, as shown in Figure 1. We carefully collect, format, split, and fil- ter undergraduate-level physics problems (see Section 3.2), creating benchmark comprising 5,520 distinct problems in three main domains, 13 core subjects, and 59 key topics, classified into six atomic answer types and one compound answer type. We further translate these problems into En- glish to enable bilingual evaluation, resulting in 1 11,040 problems in total. To better delineate the skills needed to solve various physics problems, we categorize the problems into four distinet physics reasoning skills correlated with the requisite solution capacities (see Appendix A.3). We also conduct data leakage detection some mainstream LLMs to validate the quality of our UGPhysics. These at-
4. Unlike abstract math reasoning, math derivation in the context of physics requires additional knowledge and in- volves practical meanings, where LLMs currently fall short.
5. Error analysis reveals that, unlike math reasoning, the primary types of errors made by OpenAl-ol-min are flawed reasoning, knowledge deficiency, and wrong application.
2. Related Work
Physics Benchmarks. The growing interest in LLM evalua- tion has led to the creation of benchmarks across various do- mains, such as mathematics (Cobbe et al., 2021; Hendrycks et al., 2021; Tang et al., 2024; Liu et al., 2024; Gao et al., 2024; Xu et al., 2025), visual reasoning (Chen et al., 2021;
2

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table Comparison of various henchmarks. For "Level", 1: Middle School, 2: High School, 3: College Entrance Examination, 4: Competition, 5: Undergraduate or above. "#Test" shows the number of textual test examples in physics, while "#UG" refers to the number of textual physics test examples of at least the undergraduate level. "#Subjects" specifies the number of physics subjects, and "." means that the dataset does not divide undergraduuate-levll physics into more fine-grained subjects. "# Ans. Types" is the number of answer types in physics. "Language": "EN" for English and "ZH" for Chinese. "Eval." describes the evaluation methods and "Leak, Det." states whether data leakage detection is performed, which can alleviate potential test set contamination.
Dataset
Level
Test
# UG
# Subjects
Ans. Types
Language
Eval.
Leak. Det.
MMLU
2,5
548
118
EN
Rule
X
AGlEval
3
200
ZH
Rule
C-Eval
1,2, 5
601
200
ZH
Rule
X
GAOKAO
3
111
0
2
ZH
Rule & Model
X
JEEBench
3
123
O
EN
Rulc
X
CMMLU
2,5
423
147
ZH
Rule
ThcoremQA
5
131
131
EN
Rule
SciEval
1,657
3
EN
Rule
X
PhysQA
1
1,770
0
5
EN
Rule
X
GPQA
5
227
227
EN
Rule
X
OlympiadBench
4
376
EN ZH
Rule
OlympicArena
4
796
0
EN & ZH
Rule & Model
PhysicsQA
2
370
0
Rule
X
UGPhysics
5
11,040
11,040
13
7
EN & ZH
Rule & Model
Cao & Xiao, 2022; He et al., 2024; Huang et al., 2024b; Lu et al., 2023), embodied AI (physical reasoning) (Bisk et al., 2019; Bakhtin et al., 2019; Wang et al., 2023b), dynamic benchmarks to mitigate test set contamination (Srivastava et al., 2024; Zhang et al., 2024; Qian et al., 2024; White et al., 2024), and many others (Chen al., 2023; Xu et al., 2024b; Lambert al., 2024; Mudur al., 2024; Xu et al., 2024c; Zheng et al., 2024). Physics is an ancient yet dynamic dis- cipline and researchers have increasingly turned to bench- marks to assess LLMS in physics reasoning. While high school-level benchmarks (Welbl et al., 2017; Lu et al., 2022; Hou et al., 2024) are valuable, they lack complex reasoning or computational challenges (He et al., 2024). College en- trance exam-based benchmarks (Zhong et al., 2023; Zhang et al., 2023; Arora et al., 2023) present more advanced problems, but they often lack fine-graines subject catego- rization. Benchmarks like MMLU-STEM (Hendrycks et al., 2020), C-Eval-STEM (Huang et al., 2024a), and CMMLU- STEM (Li et al., 2023) include some college-level physics questions, yet they are predominantly multiple-choice ques- tions. Advanced benchmarks such GPQA (Rein et al., 2023), OlympiadBench (He et al., 2024), and OlympicArena (Huang et al., 2024b) provide challenging physics problems but are limited in size and breadth, often incorporating other scientific domains. Specialized physics benchmarks like PhysQA (Ding et al., 2023) and PhysicsQA (Jaiswal et al., 2024b) remain relatively simple. In contrast, our proposed UGPhysics encompasses a broader range of undergraduate- level physies subjects, offering diverse answer types, and providing significantly larger number of test examples.
lenging task. Researchers have primarily relied on two approaches: rule-based methods, often combined with elab- orate answer-cleaning codes (Hendrycks et al., 2021; He et al., 2024; Jaiswal et al., 2024b), and model-based meth- ods that employ LLMs as evaluators (Gao et al., 2024). While rule- based methods are efficient, they struggle with handling complex answers (Gao al., 2024). On the other hand, model-based methods offer more flexibility but often fall short in accurately assessing numerical values, possi- bly due to the current limitations of LLMs in performing precise calculations (Chen et al., 2022; Xu et al., 2025). This issue is particularly pronounced in physics, where cus- tomized relative error requirements for different problems are required. As evidenced by 12% judgment error rate for physics problems in OlympiadBench (He et al., 2024), evaluating model-generated answers for physics problems presents an even greater challenge due to frequent appear- ance of physical constants and equivalent quantities (see Table 4). To address the challenge of answer assessment of physics problems, we propose MARJ, a two-stage evalua- tion framework that integrates both the precise calculation of rule-based judgment with the flexibility of model- assessment (details in Section 3.3). Human evaluation has confirmed the reliability of our MARJ scoring framework Additionally, several works (Didolkar et al., 2024; Huang et al., 2024b) utilize model-based evaluation to obtain addi- tional metrics for assessing effectiveness.
LLMs for Reasoning. Significant efforts have been de- voted to leveraging LLMs for solving reasoning problems, particularly in mathematics. Beyond evaluation (Liu et al., 2024; Tang et al., 2024; Gao et al., 2024), researchers have explored various approaches, including advanced prompting
Answer Judgment. Evaluating model-generated answers to complex mathematical problems has long been a chal-
3

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 2. Benchmark Statistics
Table 3. Examples of different answer types.
Statistic
Number
Type
Abbrev.
Example
Total Problems
5520
Numerical Value
NV
2.51 10
Number of Language
x2
Expression
EX
v 2//pow
Total Domains
3
Equation
EQ
J
lpp 0
Total Subjects
13
Interval
IN
0o, E E/cB]
Total Topies
59
True/False
TF
Yes
Total Answer Types
7
Multiple Choice
MC
A w/c, k. Ao
Total Difficulty Level
4
Compound
Average Problem Tokens
82.4
Average Solution Tokens
318.5
Average Number of Answers 1.34
techniques (Wei et al., 2022: Wang et al., 2022), supervised fine-tuning (SFT) (Tong et al., 2024; Xu et al., 2024c), and continued pretraining strategies (Lewkowycz et al., 2022: Azerbayev et al., 2023). To assess the impact of math-related training on physies reasoning of LLMs, we also evaluate several math-specific LLMs in UGPhysics. More recently, there has been much work dedicated specifically to physics reasoning (Pang et al., 2024; Jaiswal et al., 2024a), yet there is still a lack of a specialized physics corpus for LLMs to continually pretrain or further SFT. Our findings underscore the necessity for further research in this area.
processing & filtering, and problem annotation.
Data Collection & Cleaning. The UGPhysics is sourced from several undergraduate--evel physics exercise books ("The Great Compendium of Physics Problems" from the University of Science and Technology of China, seven books in total). The corresponding PDF files are converted to LaTeX format using the Mathpix tool for optical charac- ter recognition. Both the original PDFs and the converted LaTeX files are manually reviewed and corrected by our team. The LaTeX files are then structured into a "Prob- lem ---ouootooonnsseeee Deduplication carried out based on model embeddings to eliminate potential repeated or similar problems. Cur- rently, problems containing images are excluded to focus on text-only reasoning of UGPhysics.
format using various markups.
3. The UGPhysics Benchmark
3.1. UGPhysics and MARJ Overview
We introduce UGPhysics, a large and comprehensive undergraduatt- physics benchmark specifically de- signed to thoroughly evaluate the physics problem-solving ability of LLMs. UGPhysics is large in size, including 5,520 physies problems presented bilingually for better evaluation. covers three domains: Mechanics & Thermodynamics, Electromagnetism, and, Modern Physics, encompassing 13 core subjects and 59 different topics in undergraduate-lever physics (details are in Appendix A.4). Similar to He et al. (2024); Huang et al. (2024b); Xu et al. (2025), each problem is structured with 7 answer types to facilitate answer judg- ment, including six atomic answer types and one compound type that is a list of atomic ones. To provide a more granular analysis of LLMs" physics reasoning ability, we categorize each test example into four distinct physics reasoning capa- bilities, which could possibly show which skill sets certain families of models succeed or fail on. Detailed statistics of UGPhysics are shown in Table 2. Additionally, data leakage detection on several LLMs is conducted to identify potential data contamination (see Section 5.4).
Problem Processing & Filtering. In physics, some prob- lems are progressive, where subsequent questions may de- pend on the answers or information from previous ones. Unlike He et al. (2024); Huang et al. (2024b), we split these progressive problems into independent new problems, in- corporating all relevant information in each new problem. Additionally, we exclude problems that lack definitive an- swers for assessing correctness, such as estimation, proof, and explanation problems. Several examples are provided in Appendix 1. All problems are initially in Chinese and then translated into English facilitate bilingual evaluation.
Problem Annotation. He et al. (2024); Huang et al. (2024b); Xu et al. (2025) suggest that classifying answer types can facilitate the evaluation pipeline. In our UG- Physics, we categorize answers into seven types: six atomic answer types and one compound type, which consists of a list of atomic answers separated by commas. One concrete example for each atomic answer type is presented in Table 3. To emphasize the focus on physics reasoning, we label each test example with one of four distinct physics reasoning skills: Knowledge Recall, Laws Application, Math Deriva- tion, and Practical Application ("Others" for the remaining). We use GPT-40 as the annotator for the categorization of skill sets. Further details are provided in Appendix A.3.
3.2. UGPhysics Creation
Our UGPhysics creation process can mainly be divided into three distinct phases: data collection & cleaning, data
4

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
notation and only the base of the scientific notation is consid- ered, allowing for relative error of up le-2 to account for unit differences or rounding. EX and EQ are normal- ized by removing all physical constants. IN are judged by comparing the two endpoints, treating them as either NV or EX. For the model-based judgment stage, we require the evaluator model to pay attention to physical constants as well as equivalent quantities in the problem description. The few-shot judging prompt is long and will be released in code repository. From the analysis in Section 5.2, the MARJ pipeline offers a reliable answer assessment.
Table 4. Examples of Challenges for Answer Judgment, wh stands for ground-truth answers and "Model Ans." is the model- generated answers.
GT
Model Ans.
Comments
0.055 8
55 ms
unit conversion intermediate precision & unit
3000 bv ET
3.02 10
Xo h
Xo RT
E F1
lv = E2 Ei inclusion of physical constants
3.3. MARJ Evaluation Framework
Evaluating model-generated answers for physics problems presents great challenge, as evidenced by a 12% judgment error rate for physics problems in OlympiadBench (He et al., 2024). Illustrative examples are provided in Table 4. Several reasons are given as follows:
4. Experiments
4.1. Experimental Setup
Evaluated LLMs. Our evaluation covers 31 leading LLMs, including closed-source commercial LLMs, open-source general-purpose LLMs, ol-like LLMs, and specialized math LLMs. Based on our UGPhysics, we provide a thorough evaluation of the physics reasoning capabilities of current LLMs. The evaluated LLMs are listed below:
The occurrence of physical constants poses challenges to the calculation precision, which could be exacerbated in multi-step reasoning, where intermediate values accumulate errors. Additionally, unit conversions or providing final answers in different units can further complicate the evaluation.
Precision Issues:
For proprietary LLMs, we select OpenAl-ol-min (Ope- nAI, 2024b), GPT4o (OpenAI, 2024a), and GPT4o-mini (OpenAI, 2024a).
Physies problems often define equivalent quantities in problem descriptions, leading to multiple correct ways express the final answer. These expressions may not always be mathematically equivalent, making it difficult to apply rule-based evaluation. Addition- ally, it is conventional to omit certain physical constants in the final answer, further complicating the answer judgment.
2. Equivalent Quantities:
For open-source general- purepol LLMs, evaluated the LLaMA-3.1-Insttue series (8B, 70B) (Dubey et al., 2024),
Qwen2.5-Instruet (7B,
72B)(Yang et al., 2024a). Yi-1.5-Chat (6B, 9B, 34B) (01-AI et al., 2024), tralAI, 2024), Mistral-Nemm--Instuut-222 (Mistral, 2024b), (Mistral, 2024c), Mistral- (Mistral, 2024a), DeepSeck-MOE- 16B-Chat (Dai et al., 2024), and DeepSeek-V22 V2-Lite-Chat (DeepSeek-Al, 2024).
(Mis-
These highlight the unreliability of relying solely on tradi- tional rule-based or model-based methods for evaluating answers to physics problems. To address these evaluation challenges, we propose Model-Assistant Rule-based Judg- ment (MARJ) pipeline, which combines the efficiency of customized rule-based methods for simple answers like nu- merical values with the flcxibility of model-based methods to handle more complex cases (see Algorithm 1).
Large-Instruct-e
We also incorporate specialized math LLMs to assess the extent to which continued training and SFT on math-related content can enhance physics reasoning: DeepScekMath- 7B (-RL,, -Instruct) (Shao al., 2024), Qwen2.5-Math (7B, 72B)(Yang et al., 2024b), Mathstral-7B (Mistral, 2023), NuminaMath-785 BB-thh--BBCaa CoT (Beeching et al., 2024), and OpenMath2- (8B, 70B) (Toshniwal et al., 2024).
The MARJ involves a two-stage evaluation process. In the first stage, rule-based judgment system is employed, fol- lowed by a second stage where GPT-4o is used to assess cases flagged as "False" by the rule-based method. The entire pipeline is described in detail in Appendix B.4. In the rule-based matching stage, multiple answers to given problem are evaluated individually. If any one answer devi- ates from the ground-truth, the result is marked as "False". For the model-based judgment stage, all answers are as- sessed collectively in a single evaluation prompt. For the rule-based matching stage, different types of answers are handled separately. TF and MC are judged after transform- ing the monel-generated answers to the same format of ground-tren -round-truth. For NV, answers are converted into scientific
For ol-like LLMs, we cover QwQ-32B-Preview (QwQ- Team, 2024), Skywork-ol--peen-Llame 3-B (Skywork, 2024), and DeepSeck-RI (DeepSeek-AI al., 2025) dis- tilled series DeeepSee-RR--issiillLLLlaaa 70B; -Qwen-BB, -Qwen-32B).
-Llama-
We provide the details of these LLMs in Appendix B.
Evaluation Setting. Following He et al. (2024); Huang et al. (2024b), all our experiments use zero-shot prompts, tailored to different answer types for better answer extrac-
5

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
tion and rule-based matching. Detailed prompts are given in Appendix B.2. We use vL.LM' to speed up the evalua- tion process. To maintain consistency in evaluations and facilitate reproduction, we set the maximum output length to 4,096 tokens and employ a greedy decoding strategy with the temperature O. For LLMs with a maximum output length of less than 4,096 tokens during SFT, such as NuminaMath- CoT-7B, we adjust the maximum output length to align with their specific SFT configurations. More Details of the evaluation setting are given in Appendix B.3.
large margin (around or over 10%) (Liu al., 2024; Xu et al., 2025). This suggests that continued pre-training and further supervised fine-tuning on mathematical corpora yield only marginal gains in physics problem-solving highlighting the need for future efforts to incorporate physics-specific content during training.
O1-like LLMs yield surprisingly strong results. DS-RI- Llama-70B achieves the second-highed overall accuracy UGPhysics. Notably, QwQ-32B attains competitive accuracy of 37.3%, closely approaching Qwen2.5 Mathh- 72B-Instruct (9)5%) despite its significantly smaller model size. Furthermore, QwQ-32B outperforms DeepSeek-R1- Distill-Llama-70F on problems in ZH.
4.2. Main Results
The main results shown in Table 5 and more detailed results are given in Appendix C. From Table 5, we have the following observations.
5. Analysis
Our UGPhysics presents a significant challenge for cur- rent LLMs. The highest overall accuracy, 49.78%, is
5.1. Fine-grained Analysis
achieved by OpenAllol-minte followed by DeepSeek-RI Distill-Llama- 70B with 40.17%. Notably, 15 out of 31 evaluated LLMs score below 20%, and only two models surpass the 40% overall accuracy. In contrast, OpenAl-ol- mini achieves over 90% accuracy on MATH (Hendrycks et al., 2021), over 60% on olympic and undergraduate math problems (Gao et al., 2024; Xu et al., 2025). Although cur- rent L.MSs have powerful math reasoning abilities, they still struggle with complex physics reasoning.
In this section, we conduct an in-depth analysis using 8 strong LLMs and delay the complete results to Appendix C.
LLMs show varying performance across different sub- jects, although the disparity is relatively small. As shown in Figure 2a, the average overall accuracy of eight strong LLMs reveals that they perform particularly well in Semi- conductor Physics (31.0%) and Atomic Physies (26.7%). In contrast, their performance is slightly lower in Theoret- ical Mechanics (16.5%). Additionally, LLMs show minor performance variation across six out of 13 subjects, with accuracies hovering around 20%. In comparison, LLMs' performance can vary from 10% to 70% across different math topics (Liu et al., 2024).
Open-source LLMs are catching up with closed-source LLMs, but a performance disparity remains. Five open- source LLMS achieve overall accuracy comparable to or even exceeding GPT-4o. However, the best performing open-source LLM, DeepSeck-R1-De RI-Distill-Llama by around 10% behind OpenAl-ol -miii. Furthermore, GPT- 4o-mini still outperforms many open-source LLMs, even surpassing some ol-like LLMs.
lags
LLMs exhibit varying levels of physics reasoning skills. As shown in Figure 2b, the selected 8 LLMs display similar performance trends across different physics reasoning skills, They perform well on Knowledge Recall tasks but struggle with Math Derivation problems. This suggests that recalling physics concepts is relatively simple for LLMs, whereas performing complex math derivations in a physics context (usually require physics knowledge and practical meanings) is more challenging. Notably, OpenAI-ol-ming outperforms the other models across all four distinct physics reasoning skills (as well as "Others").
Model performance improves with increasing parameter size within the same model family. As model size grows from 7B to 72B, Qwen2.5-Mathh --nttt---ntttt exhibits an ap- proximate 17% increase in overall accuracy, with consistent improvements across different domains and languages. A similar trend is observed in ol-like LLMs, where DeepSeek- E1-Distill-Llame sees even more pronounced perfor- mance gain by 27% when scaling from 8B to 70B.
LLMs exhibit varying performance across different lan- guages when solving physics problems. From Table 5, some LLMs demonstrate only minor discrepancies in per- formance between English (EN) and Chinese (ZH), such as OpenAll-ol-miii, 5wen2.5-Man 1-t.555--at--tttttee Instruct, and QwQ-32B, However, other LLMs exhibit a significant performance gap between ZH and EN, such as Yi-1.5-Chat and LLaMA-3.1. For further illustration, Figure 3 presents the performance of subset of L. in both languages, with the models sorted by the difference in accuracy between EN and ZH. It
Math-specialized LLMs outperform their general- purpose counterparts, but the improvement is less pronounced than in mathematics. Qwen2.5-Math-7B- Instruct achieves only a 0.14% higher average accuracy than Owen2.5-7B--nsttene while Owenn255-Math-72BB-me Instructt outperforms Qwen2.5-72B- Insstruct by 3.3%. In con- trast, math-specific LLMS usually outperform their general- purpose counterparts in solving mathematical problems by
6

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table Main Results UGPhysics figures are in %). Models are classified into four different categories according to their purpose and origin. The best results within each column are bolded and the best results of LLMs within a similar group are underlined. "Mec. and "Elec." represents Electromagnetism.
Ther." stands for Mechanics & Thermodynamics, Mec. and Ther.
Elec.
Modern Physies
Overall
Average
Models
EN
EN
EN
Z.H
Closed- -ource LLMs
OpenAl-ol-mini-
48.47
49.08
43.58
43.15
54.06
52.75
49.96
49.60
49.78
GPT--4--2024-08-66
36.97
36.84
36.40
34.58
42.80
40.63
39.29
38.01
38.66
GPT-44-mini-2224-077-1
27.81
26.07
24.84
22.38
30.72
29.28
28.51
26.78
27.64
Open-source Chat LLMs
Y1-1.3-6B-Chat
10.99
77288
1T.99
7.82
13.66
9.29
12.26
8.21
10.24
Owen2.5-7B-Instrug
23.89
20.23
24.09
17.99
25.57
21.77
24.62
20.49
22.55
-LaMA3.1-8B-Inst.
12.64
7.67
14.35
9.85
16.80
13.00
14.66
10.25
12.45
Simisttal-SB-Insttuc--22
13.95
10.81
15.52
8.99
19.20
12.17
16.39
11.07
13.73
Yi-1.5-9B-Chat
16.00
11.73
15.85
13.28
19.94
15.40
17.61
13.51
15.56
Mistral-Nemo--nstruee
14.08
11.64
14.78
11.99
18.41
16.27
16.00
13.62
14.81
DeepSeck-MOE-Il 16t-Chat
3.75
3.18
4.93
4.82
7.16
4.49
5.36
4.00
4.68
DeepSeek--22-Lite-Ch5
6.50
4.93
6.75
5.78
9.47
7.59
7.77
6.18
6.97
Mistral-Smaall--nstuue---00
22.71
21.27
22.70
18.42
29.97
23.21
25.72
21.59
23.66
Yi-1.5-34B-Chat
18.79
13.38
18.63
12.74
23.17
17.84
20.58
15.13
17.85
T.
27.90
24.89
26.98
22.81
32.98
27.23
29.86
25.51
27.68
I.I.MA3.3-70B-Imm t
33.61
25.81
33.30
24.84
39.18
26.83
35.87
26.07
30.97
Qween255-7BB-nnstmuet
35.96
35.70
33.19
34.05
37.13
38.22
35.98
36.47
36.22
Mistral-Lagge-Inssuu-22244
38.06
36.70
38.33
34.90
44.20
40.36
40.65
37.92
39.28
Specialized
Mathematical L.MMs
MeepSeck-MathhTBBBBe
13.25
T2.34
16.49
13.17
19_07
T5.62
16.21
13.84
15.03
DeepSeck-Math-78
15.17
11,68
15.20
12.74
18.54
15.40
16.58
13.41
14.99
NuninaMath-7B-CoT
13.60
15.52
14.45
15.52
18.06
18.50
15.60
16.76
16.18
7B-vstt
14.82
12.47
17.77
19.94
17.06
14.80
16.12
OpenMath2-LLama--.11e
8.63
6,28
10.17
7.39
11.95
9.55
10.27
7.83
9.05
Qwen2.5 -atth- Maath--B-Inntruee
23.84
21.05
22.38
18.09
26.53
21.34
24.71
20.67
22.69
OpenMath22 -aa----L-aaa3.111227 3.B-70B
20.31
18.70
22.16
18.63
25.44
21.51
22.75
19.86
21.30
Qwen2.5-Matte Mathh- IIstruct
39.54
39.84
35.87
38.44
41.19
39.44
39.60
39.44
39.52
o1 ol-like LLMs
DeepSeek-RI RI--sstill Qwen- BB
29.25
20.62
27.41
17.45
29.80
20.68
29.17
20.11
24.64
Skywork-ol -Open- Open--pama-a- Llama-3.1 3--8BB
13.47
9.55
14.45
8.67
15.36
10.12
14.42
9.64
12.03
DeepSeek-R1-n RI-Distill-Llamaal
16.35
7.15
17.13
6.53
20.90
9.08
18.37
7.84
13.11
QwQ-32B- Prevviww
36.84
38.01
35.65
31.58
38.61
38.92
37.37
37.30
37.34
DeepSeek-RI -----tt--l---------- Siiill--wen-328
35.22
28.51
33.40
23.34
38.09
28.80
36.11
27.75
31.93
-I-Disiill Llama-70B
43.24
34.26
42.93
28.80
49.91
36.78
45,96
34.38
40.17
Newly-added
Results
Phi-4
34.T3
32,48
36.51
29.87
40.45
35.86
377.66
"3.44
35.30
DeepSeck-RI
55.49
56.67
54.50
48.39
59.90
57.29
57.16
55.53
56.34
is evident that LLaMA-3.3-708 -nstrruutt and DeepSeek-R1- I. Llama-70B show substantial discrepancy between 7HH and EN, while Diven22.-72B-Inntteee and DWQ-32B-Preview exhibit negligible differences. This discrepancy is reason- able. as LLaMA models have limited Chinese corpus for pretraining and fine-tuning (Dubey et al., 2024), whereas Qwen LLMs are trained much larger Chinese corpus (Yang et al., 2024a;b).
substantiate MARJ evaluation method, conduct hu- man evaluation to determine its alignment with human judg- ment on a randomly selected subset of 100 test examples. Specifically, we initially annotate whether each solution adheres to the ground-trune answer for its corresponding problem, establishing these annotations the gold standard. Subsequently, we compare the evaluations generated by rule-model combination with the gold standard. We find that our MARJ evaluation achieves an accuracy of 98% % when compared to human annotations, underscoring the reliability of our evaluation methods and outcomes. Furthermore, our evaluation approach efficient in assessing correctness for examples whose answers can be easily verified by Sympy, while also demonstrating resilience in handling complex answers that suitable for rule-based judgments.
5.2. Reliability of Evaluation
Despite several studies utilizing LLMS to evaluate correct- ness across all test examples (Gao et al., 2024) specific subsets (Zhang et al., 2023), the capability of our MARJ to reliably evaluate physics problems remains inconclusive. To

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Semiconductor Physics
11.0%
KR
Atomic Physics
26.7%
Therodyynamics
Mechanics
22.7%
Solid-State Physics
22.5%
HommaA-al-minn--2224-000
Classical Electommannetim
22.3%
OT
DeepSeek-R1-
Statistical Mechanics
Qwea2.5
Classical Mechanics
Relatisity
20.7%
RwwQ-32-PPree
32B-Prasinn
Wave Optics
Eleetrodynamics
Knowledge
Geometrical Optics
Appliiation
Math Derivation
Theeretical Mechanics
16.5%
Pructical Applicationn
MD
PA
(a) Accuracy Across Subjects
(b) Accuracy Across Reasoning Skills
Figure 2. The distribution of overall accuracy across subjects, and physics reasoning skills. (a) The overall accuracy of different subjects averaged across strong LLMs listed in Figure (b). Each bar consists of several segments with colors indicating their corresponding reasoning skills. (b) The overall accuracy of reasoning skills, averaged across all subjects. Only 8 strong LLMs are included for brevity, "KR": Knowledge Recall; "LA": Laws Application; "MD": Math Derivation; "PA": Practical Application; "OT": Others.
0.5
zb
0.4
31.0%
flawed reasoning knowledge deficiency wrong application computation misunderstanding
0.2
25.0%
0.1
2.0%
exceeding max output instruction following others
Owp-228-Peeiie
Uerret Mast 33
mnted
Mittra Larpe
Lanaa
16.0%
7.0%
8.0%
Figure 3. Performance in different languages, sorted by the differ- ence ZH.
Figure 4. Distribution of Error Types of OpenAI-ol-man
5.3. Error Analysis
age within different LLMs. Concretely, combined each problem with its solution in the dataset and randomly chose K positions for extracting 5-grams. A sample is considered contaminated if the 5-grams predicted by the model match the actual 5-grams from the dataset. The results for a subset of LLMS are presented Table 6. evident that most models exhibit some degree of data leakage. Among them, Dwwen25--MATH-72B--me
To gain deeper insights into the performance of LLMs, we select 100 incorrect answers generated by OpenAl-ol- mini and have these errors annotated by human evaluators to determine failure reasons. As illustrated in Figure 4, primary error types are flawed reasoning, knowledge deficiency, and incorrect application, which contrast with those mathematics, where caleulation is one of the ma- jor sources of errors (Chen et al., 2022; Xu et al., 2025). This suggests that reasoning and math derivation in physics, which require additional knowledge and involve real-world meanings, are more challenging than the abstract reasoning in mathematics. Several cases are provided in Appendix D.
shows the highest level of leakage, accurately predicting 5 grams in 78 samples. Ad- ditionally, we report on the contaminated samples that are subsequently answered correctly by the tested models. The numbers of both "Contaminated" and "Contaminated & Correct" samples are extremely low, suggesting that data leakage has minimal impact on UGPhysics.
5.4. About Data Leakage
5.5. Compare Existing Benchmarks
We perform data leakage detection to alleviate the poten- tial data contamination UGPhysics. Following Xu et al. (2024a), we utilize n-gram accuracy to detect any data leak-
Accuracy Compared to Existing Benchmarks To con- textualize our UGPhysies's difficulty, we evaluate GPT-4o
8

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
the limit to 16,384. Results in Table 9 show marginal per- formance gains, with small proportion of generations still exceeding the extended limit. These findings highlight the need to address token redundancy in ol-like LLMs during reasoning, as noted by Chen et al. (2024).
Table The proportion (in %) of data leakage detection for: (a)
The proportion of contaminated examples. (b) The proportion of contaminated and correct examples. "Prop." stands for proportion.
Model
(a) Prop.
(b) Prop.
DeepSeck-Matee Math--B-RL
0.0%
0.0%
LLaMA3.1-8 1-88-188-111 Instructt
0,53%
0.06%
LLaMA3.3 Instruct
0.65%
0.29%
Table 8. The Clip Ratio (in %) for Different ol-like LLMss
Breen2.5-Mtth-7B--mmse
0.65%
0.36%
Models
8192
16382
0.75%
0.68%
QwQ-32BB-Peeiee
0.71%
0.65%
DeeesSee--R--Disiil-------e
44,40
38.90
DeepSeek-RR RI RI-Distill-Qwent
0.0%
0.0%
DeepSeek-RR --ieek-RR----iiiei --Dittil -wwssill--wen-3BBBB
38.55
34,47
DeepSeek-RR RI- ELl-Disiil-QQee--22B
0.0%
0.0%
DeepSeek- RI-ik--R--Dtilll Disttill-Llaa--88 Llama-8B
52.37
43,80
DeepSeek-RI DeepSeck-R1-Distil-LLaee Ri-----iill---aam--000
19.16
12.25
B00Q-32B-Previce
19.01
8.54
on established physics reasoning datasets (Table 7).
01-mini-2024.09--2
2.01
GPT-4o achieves 38.67% accuracy, weaker than its 53.60% on GPQA (Rein et al., 2023), highlighting UGPhysics's greater challenge. We also include MATH (Hendrycks et al., 2021) for reference.
Table Performance (in %) of Different Token Budget. Models
8192
16382
DeepSeek-FR BeepSeek-RI-Diiade Diistill-Qeen711
24.64
24.86
Tokens Spent on UGPhysics We further analyze the to- ken usage required to solve problems in our benchmark. DeepSeck-RI (DeepSeck-Al et al., 2025), when used with- out maximum token constraints (via API), requires an aver- age of 5,555 tokens to solve problems in our benchmark. To assess the impact of token restrictions, set maximum output token limit of 8, 8,192 for DeepSeek-RI -iistilttlllQQe -een- 32B in our experimental setup. Under this constraint, the model requires 3,079 tokens on the MATH (Hendrycks et al., 2021) benchmark and 4,081 tokens on our UGPhysics. These results highlight the computational cost of solving problems our benchmark relative other tasks.
DeepSeck-R1 R1-ock-----iitiil Diistill--Qee--255
31.93
32.21
DeepSeck-R1-Diieg DDistill-Llamm-88
13.11
14,51
DeepSeek-RI -a---RDDiiil--aaaaaae
40.17
41.77
SwQ-322-Previse
37.34
38.90
6. Conclusion
In this study, we propose UGPhysics, a comprehensive undergraduate-level physics benchmark, and the MARJ an- swer scoring framework evaluate LLMs' capabilities in solving physics problems. Through our extensive experi- ments, we find that although current LLMs excel in mathe- matical reasoning, there remains considerable potential for improvement in their performance on physics problems. We believe our dataset and codes will be valuable for the future development Al with strong physics reasoning abilities.
Table 7. Performance on various benchmarks.
Dataset
Performance (%)
MMLU (college physics)
68.60
MMLU (high school physics)
72.80
MMLU (conceptual physics)
92.30
Impact Statement
MMLU- -pro
75,06
OlympicArena
55.92
This paper introduces undergraduate- physics bench- mark aimed at advancing Al capabilities in physics problem- solving. Future directions include incorporating problems with images to enable multi-modal evaluation or more lan- guage to facilitate multi multi-linngall assessment.
GPQA
53,60
MATH
76.60
UGPhysics
38.67
5.6. About the Clip Ratio
A Acknowledgments
The maximum generation token limit was set to 8,192 in our evaluation. Open-source ol-like LLMs typically con-
This work partially supported by grant from the Re- search Grants Council of the Hong Kong Special Adminis- trative Region, China (Project Reference Number: AoE/E- 601/24-N), We would like to thank all the reviewers for their valuable suggestions improve our work.
sume more tokens than OpenAl's o1-mini when solving UG- Physics problems. With 8, 192 token limit, only 2% of ol- mini's ' generations exceed this limit, whereas significantly higher proportions of other open- open-source ol-like LLMs fail to terminate within the limit (Table 8). To assess whether in- creasing the token limit improves performance, we extended

---

UGPhysics:
A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
References
that much for 2+ 3=? the overthinking of ol-like Ilms. arXiv preprint arXiv:2412.211 2024.
01-AI, Young, A., Chen, B., Li, C., Huang, C., Zhang, G.,
Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue. S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., NIu, X., Nie, P. Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by O1. 2024.
Cobbe, K.,, Kosaraju, V., Bavarian, M.. Chen, M.. Jun, H.,
Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. ArXiv preprint, abs/21 10.1111........ 2021. URL https: / //arxiv... www.iii.rg/ab////o /2tabbss21000 2110. 68.
Arora, D., Singh, H. G., et al. Have iims advanced enough? challenging problem solving benchmark for large lan- guage models. arXiv preprint 2...:2305. 2023.
Dai, D., Deng. C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K. Huang, P., Luo, F., Ruan, C.,, Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models. CoRR, abs/2401 06066, 2024. URL httpt::tt///aaee /ttp:///aaxvv//////oo abs 0ats/2001.0000
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q.. Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for math- ematics. ArXiv preprint, abs/2310. 10631,,,0006,,, 2023. URL https: aariiv.vvvv http//axivv.gg.////////////o abs///ss///// 221023100.00 10631.
DeepSeek-AL. Deepseck-v2: A strong, economical, and efficient mixture- -f-eeeerts language model, 2024.
Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., and Girshick, R. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F. Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji. D., Li, E., Lin, F., Dai, F., Luo, F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu H., Li, H., Guo, J., Li, J.,, Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L.., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K. Gao, K., Guan, K., Huang, K. Yu, K., Wang. L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L.,, Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S. Wang, S., Yu, S., Zhou, S., Pan. S., Li, S. S., Zhou, S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang. T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang. W., Xiao, W. L., An, W., Liu, X., Wang. X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y.. Shi, Y., Xiong, Y., He. Y., Piao. Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z.. Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma, Z.,, Yan, Z., Wu, Z.,, Gu, 7.,, 7hu, 7., Liu, Z., Li, Z. Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z. and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcemett learning, 2025. URL https: /ttp:///axiv.vv. /tt/a//www...///////....... 12948.
Beeching, E., Huang, S. C., Jiang, A., Li, J., Lipkin, B., Qina, Z., Rasul, K., Shen, Z., Soletskyi, R., and Tunstall, L. Numinamath 7b cot, https: / / /uuggi face.e. MM-MMO//umiin
7B- 2024.
Bisk, Y., Zellers, R., Le Bras, R., Gao, J., and Choi, Y. Rea- soning about physical commonsense in natural language, 2019.
Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Con- ference on Computational Linguistics, pp. 1511-1520, 2022.
Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P., and Lin, L. Geoqa: geometric question answering bench- mark towards multimodal numerical reasoning. arXiv preprint ar 115::2005..411., 2021.
Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint, abs/2211.12558 2022. URL https: /ttp:i//////vv.e/// .org/ 12588.
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., and Xia, T. Theoremqa: theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889-7901 2023.
Chen, X., Xu, J., Liang. T., He. Z., Pang, J., Yu, D., Song, L., Liu, Q. Zhou, M., Zhang, Z.,, et al. Do not think
10

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Didolkar, A., Goyal, A., Ke, N. R., Guo, S., Valko, M., Lillicrap, T., Jimenez Rezende, D., Bengio, Y., Mozer, M. C., and Arora, S. Metacognitive capabilities of Ilms: An exploration in mathematical problem solving. Ad- vances in Neural Information Processing Systems, 37: 19783-19812, 2024.
multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024a.
Huang, Z.. Wang. Z., Xia, Li, X., Zou, H., Xu, R., Fan, R.-Z., Ye, L.,, Chern, E.,, Ye, Y., et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXiv:2406. 12753, 2024b.
Ding, J., Cen, Y., and Wei, X. Using large language model to solve and explain physics word problems approaching human level. arXiv preprint arXiv:2309.08182. 2023.
Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmad- hikari, A., Marathe, A., and Shah, R. R. Improving physics reasoning in large language models using mixture of refinement agents. arXiv preprint arXiv:2412.0082), 2024a.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Gao, B., Song, F., Yang. Z., Cai, Z., Miao, Y., Dong, Q.. Li, L., Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmad- hikari, A., Marathe, A., and Shah, R. R. Improving
physics reasoning in large language models using mixture of refinement agents. arXiv preprint arXiv:2412.0082 2024b.
Giancoli, D. C. Physics for Scientists and Engineers. Pren- tice Hall, 2000.
Klahr, P. and Waterman, D. A. Applications of Artificial Intelligence to Physics. AAAI Press, 1986.
Hawking, S. A Brief History of Time: From the Big Bang to Black Holes. Bantam Books, 1988.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13797, 2024.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang. Y., Zhang, Y., et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. arXiv preprint ar Arv:22240.. /4008, 2024.
Larkin, J. H., McDermott, J., Simon, D. P., and Simon, H. A. Expert and novice performance solving physies problems. Science, 208(4450) 1980.
Heller, P., Keith, R., and Anderson, S. Teaching problem solving through cooperative grouping. part I: Group ver- sus individual problem solving. American Journal of Physicx, 60(7):627-636, 1992.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman- Collo, T., al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mcasuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.0333 2020.
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y.. Duan, N., and Baldwin, T. cmmlu: Measuring mas- sive multitask language understanding in chinese. arXiv preprint arXiv:2306. 09212,992222.2 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Mcasuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.03874, 2021. URL https: //axxiv. org/ 2tabs/2100 03874.
Liu, H., Zheng, Z.,, Qiao, Y., Duan, H., Fei, Z., Zhou, F,, Zhang. W., Zhang, S., Lin, D., and Chen, K. Math- bench: Evaluating the theory and application proficiency of Ilms with hierarchical mathematics benchmark. arXiv preprint 240i /2209, 2024.
Hou, J., Ao, C., WU, H., Kong, X., Zheng, Z., Tang, D., Li, C., Hu, X., Xu, R., Ni, S., et al. E-eval: compre- hensive chinese k- 4-12 education evaluation benchmark for large language models. arXiv preprint 15X 2024.
Lu, P., Mishra, S., Xia, T., Qiu, I.., Chang, K.WW., Zhu, S.-C., Tafford, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521 2022.
Huang, Y., Bai, Y., Zhu, Z., Zhang, J.. Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Fu, Y., et al. C-eval: A multi-level

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Lu, P. Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Qian, K., Wan, S., Tang, C., Wang, Y., Zhang, X., Chen, M., and Yu, Z. Varbench: Robust language model bench- marking through dynamic variable perturbation. arXiv preprint arXiv:2406. 176811 2024.
QwQ-Team.
Qwq-32b-previee.
Ma, Y., Gou, Z.., Hao, J., Xu, R., Wang, S., Pan, L.., Yang, Y., Cao, Y., Sun, A., Awadalla, H., et al. Sciagent: Gool-augmented language models for scientific reasoning. arXiv preprint arXiv:2402.11455 2024.
2024.
Redish, E. F. Teaching Physics with the Physics Suite. Wiley, 2003.
McDermott, L. C. and Redish, E. F. Resource letter physics education research. American Journal of Physics, 67(9):755-767, 1999.
Rein, D., Hou, B. L., Stickland, A. C., Petty. J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-prood proole-proofoee q&a benchmark. arXiv preprint urXiv:2311 2022, 2023.
Mendelson, E. and Zelinski, D. E. The Physics Problem Solver. 1984.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song. J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.033000, 2024. URL
Mistral. Mathstral. Accessed: 2024-09-23.
mathstral/, 2022
Mistral. Mistral large =ttgs./mmisrralaalleewssmmaaaa large-2407/, 2024a.
https: /ttp///awwvvorr/s////aa/2.// 03300.
Skywork. Skywork-ol model card, 2024. URL https://hugginggaaee cokkkworrk/ Skywork-ol-Open-
Mistral. The future of ai: Trends and predictions. https: h/misttralaa/////mmm......aaaaat ai/.ews/ /mi stral-trennt 2024b.
Accessed: September 29, 2024.
Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 2024.
Mistral.
misrraaimmistallssallnnnmeeeeeee
httpes//hogghaageee...///ttttttt
2024c.
MistralAI.
Ministral model card, 2024. URL
Tang, Z., Zhang, X., Wan, B., and Wei, Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.0288. 2024.
http:////ggggggacccco/o/ Minist:al-8B-Inst:ene
Mudur, N., Cui, H., Venugopalan, S., Raccuglia, P., Bren- ner, M., and Norgaard, C. Feabench: Evaluating lan- guage models real world physics reasoning ability. In NeurlPS 2024 Workshop on Open- Open-World Agents, 2024.
Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. Dart- math: Difficulty-awan rejection tuning for mathemati- cal problem-solving. arXiv preprint arXiv:2407. 13690, 2024.
OpenAL. Hello gpt-4o. attps::/opents com/ hatps//helooggpt-440 2024a.
Toshniwal, S., Du W. Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-22 Accelerating ai for math with massive open-source in- struction data. arXiv preprint arXiv:2410.01560. 2024.
OpenAL. 1lms.
Learning to reason with index/ -earning-to-rees000 watho-wwith--e 11ms/, 2024b. Accessed: 2024-09-23.
https:://peet com/
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, Self-consistency im- proves chain of thought reasoning language models. ArXiv preprint, abs/2203.11171, 2022. URL https: //arxiiv org abs httpa/bb//22222/
Pang, X., Hong, R., Zhou, Z., Lv, F., Yang, X., Liang, Z.,, Han, B., and Zhang, C. Physics reasoner: Knowledge- augmented reasoning for solving physics problems with large language models. arXiv preprint 13Xiv:2412.13791, 2024.
Wang. X., Hu, 7., Lu, P., Zhu, Y., Zhang. J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models. arXiv preprint arXiv:2307.1066 2023a.
Planck, M. Scientific Autobiography and Other Papers. 1949.
12

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Wang, Y. R., Duan, J., Fox, D., and Srinivasa, S. Newton: Are large language models capable of physical reasoning? arXiv preprint arXiv:2310.0700 2023b.
Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. Processbench: Identifying process emrors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Lc, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022.
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang. Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centrie benchmark for evaluating foundation mod- els. arXiv preprint arXiv:2304.06366 2023.
Welbl, J. Liu, N. F,, and Gardner, M. Crowdsoure- ing multiple choice science questions. arXiv preprint arXiv: (ttv::777.00099 2017.
White, C., Dooley, S., Roherts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: challenging, contamination-fre Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Xu, R., Wang, Z., Fan, R.-Z., and Liu, Benchmark- ing benchmark leakage in large language models. arXiv preprint ar arXiv:2404.1 0et.:4444.8884,, 2024a.
Xu, X., Diao, S., Yang, C., and Wang, Y. Can we verify step by step for incorrect answer detection? arXiv preprint arXiv:2402. 10528, 2024b.
Xu, X., Xiao, T., Chao, Z., Huang, Z.,, Yang. C.,, and Wang. Y. Can Ilms solve longer math word problems better? arXiv preprint arXiv:2405. 14804, 2024c.
Xu, X., Zhang, J., Chen, T., Chao, 7.,, Hu, J., and Yang, C. Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models. arXiv preprint arXiv:2501.13766, 2025.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.12122 2024b.
Zhang, H., Da, Lee, D., Robinson, V. Wu, C., Song, W., Zhao, T.. Raja, P., Slack, D., Lyu, Q., al. A careful amination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332. 2024.
Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint A.V:22305.12474 2023.
13

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
A. Detailed Statistics of UGPhysics
A.1. Problem Filtering
During problem processing, we find some physics problems do not have definite answers for evaluation. Two examples are given Table 10. We exclude these problems for automatic evaluation in our UGPhysics.
A.2. Answer Types
By carefully reviewing a large collection problems and referring to various past benchmarks (He et al., 2024; Huang et al., 2024b; Xu al., 2025), we classify all answers to seven categories in total: six atomic and one compound type. Each compound type is composed of list of atomic ones. These types are designed encompass wide range of problems. Detailed definitions for each answer type can be found in Table 11, Similar He et al. (2024), the compound answer type characterized by the attribute is issmultti wwwunnipppeeeee answess.
A.3. Physics Reasoning Skills Annotation
The definitions of different physics skills are as follows:
Knowledge Recall: Refers to the recall and understanding of basic physics concepts, formulas, and definitions. This category assesses the ability accurately remember key physics points, such as the units, definitions, and fundamental properties of physical quantities, without requiring deep reasoning complex calculations.
Laws Application: Involves understanding and applying important physical laws. Students are expected to correctly describe the content, conditions, and scope of these laws and determine their applicability to specific scenarios. Examples include Newton's 's laws, the laws of thermodynamics, and Coulomb's law.
Math Derivation: Focuses the mathematical derivation and logical proof of physics formulas or principles. This requires students to use known laws or principles and apply rigorous mathematical reasoning to derive new relationships or theorems. Examples include deriving the momentum conservation equation from Newton's 's second law or deriving thermodynamic relations from the ideal gas law.
Practical Application: Emphasizes the practical use of physics knowledge, laws, and derivations solve real-world problems. This includes analyzing scenarios, building physical models, and using calculations and logical reasoning arrive at solutions. Examples include calculating the final velocity of using the work-energy theorem analyzing the motion of charged particle using electric field equations.
The prompt used for classification is given in Table 12. Note that there are problems that fail to be classified into certain types of skill, we annotate them as "others".
A.4. Distribution of Problems
Our UGPhysics include key domains and 13 subjects in undergraduate-level physics. The detailed topics across different subjects and the corresponding number of examples are listed in Table 13. There are 59 topics in total. We illustrate the words of topics in Figure 5. Furthermore, the distribution information of our benchmark on different subjects and reasoning skills is presented in Table 14.
Estimation Problem Table 10. Estimation and Explanation Problem Example.
Provide the best estimate the cosmic ray flux at sea level.
Explanation Problem
Water is polar molecule. Discuss the effect of electrode polarization its dielectric constant at high and low frequencies.
14

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
B. Detailed Experimental Setup
B.1. Evaluated LLMs
Our evaluation encompasses a range LLMs, including both proprictary commercial models and publicly accessible models, For open-source LLMs, we cover ceneral-purpone LLMs, ol-like LLMs, and specialized math LLMs.
Closed-source L.LMs listed follows:
ol-preview (OpenAI, 2024b): An early preview of OpenAl's ol model, designed reason about hard problems using broad general knowledge about the world. We used ol www-ppeeeeeweee 00-224-9-122 for our evaluation.
GPT-4o (OpenAl, 2024a): GPT-4o is multimodal LLM, and has the same high intelligence as GPT-4 Turbo but is much more efficient. For evaluation,
this specific version: GPT-40-2024- -
GPT-4o-mini: GPT-4o-mini is even more efficient and cheaper than GPT-4o with the cost of minor performance drop. We use GoT--o-TT-- ii 2204-000222------ for our experiments.
The following open-source general-purpose LLMs evaluated UGPhysics: LLaMA-3.I-Instrum (Dubey al., 2024): LLaMA-3.1 models are the most capable of the LLaMA families of writing this paper. We used instruction finetuned 8B and 70B versions of the model. These models are licensed under Llama 3.1 Community License.
Ministral-SB-IInstrut---11 (MistralAI, 2024): mrl License
Mistral-Nemo-Insttrre440777 (Mistral, 2024b): Apache 2.0
Mistral-umalll--sstuut--29999 (Mistral, 2024c); MRL License.
MistralLLagge-nnstrue2247777 (Mistral, 2024a): MRL License.
Qwen2.5-Instruct (Yang et al., 2024a): Qwen2.5 series are developed with dedication to math and coding. We used 7B and 72B models. 7B models are licensed under Apache 2.0, while 72B models are under Qwen License.
Yi-1.5-Chat (01-AI et al., 2024): Yi-1.5 delivers stronger performance in coding, math, reasoning, and instruction- following capability compared to its predecessor. We used 6B, 9B, 34B variants. Yi-1.5 series are licensed under Apache 2.0.
-2-pSeeek-V-Litee--e (DeepSeek-AI, 2024): model under Model License code under MIT License.
deepsek-moo--1b-chaat (Dai al., 2024): Model under Model License, code under MIT License.
For ol-like LLMs, the following:
QWQQ-3B-Preeicee Previer (QwQ-Team, 2024): HWWQ-3BB-rreiiee developed by the Qwen Team, focused on advancing AI reasoning capabilities. under Apache 2.0 License.
Table Descriptions answer types included in evaluation prompts. only include English version for simplicity. English Answer Type Description
Answer Type
NV
numerical value without units
EX
an expression
EQ
equation
INT
range interval
TF
either True False
MC
one option of multiple choice question
15

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
is LLM that incorporates ol-like slow thinking and reasoning capabilities. It is developed by the Skywork team.
Slyworkol--ppo--Lama-...111 (Skywork, 2024): -kywork-ol-Open-JJanaze
DeepSeek-RI Distilled serires (DeepSeck-AL al., 2025): These LLMs are distilled from DeepSeck's 's first-generation reasoning models DeepSeek-RI Model under Model License, code under MIT License. In our evaluation, DeepSeek-R1 -ainee-RR--issillld-aaaaas -Llama-70B; -Qwen-7B, -Qwen-32B.
We also experiment with the following specialized math LLMs our study:
Deth (Shao et al., 2024): DeepSeekMath is initialized with DeepSeck-Coder-v1..5 7B and continues pre-training math-related tokens. We tested both DeepSeekMath-7B-RR and DeepSeekMath-7B-nnstue variants. Models are under Model License while code is under MIT License.
Qwen2.5-Math (Yang al., 2024b): Qwen2.5-Math is series of specialized math language models built upon the Qwen2.5 LLMs. We evaluated 7B and 72B variants. They are under the same license as the Qwen2.5- Instrucct series.
Mathstral-7B (Mistral, 2023): Mathstral stands on the shoulders of Mistral 7B and specializes in STEM subjects. This model is published under Apache 2.0.
Numinamath-7B-Co7 (Beeching al., 2024): This model is finetuned from DeepSeekMath-7B -B-baas with two stages of supervised fine-tuning to solve math problems using chain of thought (CoT). It is licensed under Apache 2.0.
OpenMath2-LLama-33 (Toshniwal et al., 2024): These are specialized math LLMs that have undergone SFT on 2.3M augmented GSM-8K and MATH training examples. These models are trained by Nvidia and are licensed under Llama 3.1 Community License.
B.2. Evaluation Prompts
The prompts employed in our experiments are presented in Table 15, with detailed explanations of response types available in Table 11. For simplicity, we here only show prompts used by English problems. The prompts for Chinese problems are quite similar and will be released in our code repository. Based on He et al. (2024); Huang et al. (2024b), these prompts are tailored for diverse subjects and response types to improve evaluation efficiency. should be noted that for chat models, we will adhere to their official chat template.
B.3. Evaluation Parameters
To maintain consistency in evaluations and facilitate reproduction, we set the maximum output length to 4,096 tokens and employ a greedy decoding strategy with the temperature For LLMs with maximum output length of less than 4, 4,096 tokens during SFT, such as NuminaMath-CoTT-75 we adjust the maximum output length align with their specific SFT configurations. For OpenAI-ol olmmmni, we set it to 8,192 tokens, as this model often requires higher token count for reasoning tasks. Exceeding the maximum output length can result in no output being returned. Similarly, we also set the maximum output length ol lliee LLMs to be 8, 192 tokens. The temperature for the OpenAI aoo--ooiin commnini model is restricted to value of due to new regulations from OpenAl. Setting the temperature to other values would result in an error from the API.
B.4. MARJ Details
The whole pipeline of MARJ is given in Algorithm 1. For the first stage, different types of answers (TF, MC, NV, EX, EQ, IN) are handled as follows:
For TF and MC, answers are judged after transforming model-generated answers to the same format as the golden answers.
For NV, answers are converted into scientific notation. Only the bases of the scientific notation are considered, allowing for relative error up to to account for unit differences or rounding.
For EX and EQ, answers are normalized by removing all physical constants. The physical constants are listed in Table 16.
16

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
For IN, answers are judged by comparing the endpoints, treating them either NV or EX based on the context.
In the second stage, GPT-4o is employed to evaluate answers that were flagged as "False" by the sule-based system. We manually design judging prompts based on the prompt provided by Gao et al. (2024). As this few-shot prompt long, we will release it our code repository.
C. More Results
Results across different subjects are given in Table 17, 18 and 19. Results across different physies reasoning skills are provided in Table 20.
D. Error Analysis
We perform error analysis in Section 5.3, and here showcase several examples of various error types in Table 21, 22, and 23.
17

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 12, The prompt for annotating physics reasoning skills, where problem, solution, and answer represent the components of the
problem to be annotated.
Problem: {problem}
Solution:
{solution}
Answer:
{answer)
Classification Categories:
Refers the recall and understanding of basic physics concepts, formulas, and definitions. This category assesses the ability to accurately remember key physics points, such as the units, definitions, and fundamental properties of physical quantities, without requiring deep reasoning complex calculations.
Knowledge Recall:
Laws Application: Involves understanding and applying important physical laws. Students expected correctly describe the content, conditions, and scope these laws and determine their applicability to specific scenarios. Examples include Newton's 's laws, the laws of thermodynamics, and Coulomb's 's law.
3. Math Derivation: Focuses on the mathematical derivation and logical proof of physics formulas or principles. This requires students to use known laws principles and apply rigorous mathematical reasoning to derive new relationships theorems. Examples include deriving the momentum conservation cquation from Newton's second law deriving thermodynamic relations from the ideal gas law, Practical Application: Emphasizes practical use of physics knowledge, laws, and derivations to solve real-world problems. This includes analyzing scenarios, building physical models, and using calculations and logical reasoning to arrive at solutions. Examples include calculating the final velocity of car using the work-energy theorem or analyzing the motion of charged particle using electric field equations.
Instructions for Classification: Please classify the above problem by selecting the most appropriate category that best represents the type of thinking and approach required to address the physics problem. Consider the complexity, the need for creativity, and the depth of knowledge required. You should conclude your response with So, the problem can be categorized ANSWER,, where ANSWER should be one of the indexes in 2, 3,4. 4.
Rigid Body
y
Special Re
RelativityRelativvee
Quantuin
exprrime ntal
Electrostatio Flac
Probl ens Orhi ta
Methodd
Optics
Forms
Body!
Dynamics Dynamics
Particl
ellomentun Probl lmms
ems
Elestrom
onagnetic
HaveS
Physics
Particle o Small Vierati
Part
Probless
Al abate
Thermodyannt
State
Dynamles
Medhantes
e
Electrons
S
Bo Emmann Statistin
sistiic
Hamil tonson
Dirac Statistics
Nuclear
Formulation
PhysiCs
Physics
Finite
System
Physics Experj
Quantum Optics
Fluld Mechanics
Atomic C
Particle
Dynanics Rigid
Rigid
Physic
iss
Dirac
Ensemble Theory
Dynanics Vibrations
tu
Statistiyy Ranell.....
DDyaamii i CS Problems
Central Force
Physics
Nuclear
Ferni
Molecul maronn
Angular Momen
tum
ApprotInTIO Prebiems
Physic
Phy
Problams Central
Second
LOW
Nae
Scattering
Probleme
ems
Ma XWelell
Bolt
zmann
hotarti Linear
Dynamics
Adi
ablabatic
Approxination
Particl
timetic Theary
Beams
Spin
Angular
Lagrangian
formulation
Basic Principles
Figure 5. Word Cloud of Topics in UGPhysics
18

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Subject
Topics
Examples
Classical Mechanics
Purticle Dynamics
644
Classical
Vihrations
Classical Mechanies
Central Force Motion
156
Classical Mechanics
Bigid-Bddy Dynamics
360
Classical Mechanies
Fluid Mechanics
154
Theoretical Mechanics
Lagrangian Formulation of Mechanies
202
Theoretical Mechanics
Small Vibrations Finite System
Theoretical Mechanies
Hamiltonian Formulation of Mechanics
254
Relativity
General Relativity
Relativity
Special Relativity
Relativity
Relativistic Cosmology
10
Ibermodynaniis
Phase Transition and Equilibrium
Thermodynamics
Thernodynamic State and First Law Thermodynamies
146
Thermodynamies
Second Law of Theroodnaaiiss and Entropy
218
Thermodynamics
Themmdynnamcc Functions Equilibriom Conditions
Thermodynanics
Non-equilibriue l'hermodynanies
78
Statisticll Mechanics
Ensemble Theory
Statistical Mechanics
Maxwell-Bolune Bolizmann Statisties
294
Statistical Mechanics
Distribution Function and Statistical Entropy
Statistccal Mechanics
Pemi--irac Statistics
228
Statistical Mechanies
Base Einstein Statisties
140
Statistccl Mechanics
Theory
Classical Elestromagnetiin
Mugstalies
296
Classical Electromagnetism
Flectrostatics
368
Classical Electromagnetism
Cireuit Analysis
166
Electrodynamies
Propagation and Radiation of Electromagnetic Waves
194
Electrodynamics
Relativistic Flectrodynamics
Electrodynamics
Interaction of Electromagnetic Fields with Matter
Geometrical Optics
Imaging Light
Geometrical Optics
Reflection and Refraction Light
Wave Ophiss
Diffractio
192
Wave Optics
Interference
110
Wave Optics
Polarization
302
Quantum Mechamics
Linear Dynamics Problems
332
Quantum Mechanics
Principles Quantum Mechanics
Quantum Mechanics
Central Force and Scattering Problems
192
Quantum Mochanies
Orbital and Spin Angular Momentum Problems
Quantum Mechanics
Motion of Charged Particles in Electric and Magnetic Fields
60
Quanium Mechanics
Adiabatic Approximation Problems
Quantum Mechanies
Scattering Problems
Quanium Mechanics
Variational Methods and Perturbation Theory Information Physics
120
Quantum Mechanics
Quantum Mechanies
Few-body Problems
104
Quantum Mochanies
Others
Quantum Mechanics
Quantum Optics
246
Atomic Physics
Atomic and Molecular Physics
622
Physics
Particle Physics
Atomic Physics
Nuclear Physics
400
Atomic Physies
Experimental Methods Particle Beams
Solid-Stute Physics
Crystal Structure
Solid-State Physics
Lattice Vibrations and Mechanical Propertics of Solids
Solid-Stul Physics
Crystal Defects and
Solid State Physics
Binding in Solids
54
Solid-State Physics
Transport Properties Electrons Holes
Solid-State Physics
Motion of Electrons in Electromagnetic Fields Energy Bands Solids
Solid-State Physics
Semiconductor Physics
Magnetic, Optical, Superconnucttngg Properties
Semicondnctor Physics
Transport Properties
80
Semiconductor Physics
Others Statistical Distribution of Electrons and Holes in Space
Semiconductor Physics
Table 13. Topics of each subject and corresponding number of examples included in UGPhysics.
19

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Knowledge Recall
Laws Application
Math Derivation
Practical Application
Others
All
Classical Mechanies
774
172
1,672
Thcoretical Mechanics
30
124
448
16
638
Relativity
188
154
18
414
Thermodynamics
36
388
246
46
28
744
Statistical Mechanics
112
322
590
38
58
1,120
Classical Electromagnetism
60
338
314
44
24
780
Electrodynamics
28
142
156
38
368
Geometrical Optics
8
48
26
30
116
Wave Optics
52
236
130
160
26
604
Quanturn Mechamics
226
584
1,052
100
76
2,038
Atomic Physies
422
282
282
1,830
Solid State Physics
36
158
Semiconductor Physies
110
Grand Total
1.168
3,914
4,440
1,106
412
11,040
Table 14. Statistics of UGPhysics across different subjects and physics reasoning skills.
Table 15. Evaluation prompts for English problems with single answers or multiple answers. {problem} is the specific problem to evaluate. (subject) denotes the subject this problem belongs all subjects are given Figure 1. [answer.yype.desriiee description) are specified Table 11.
Evaluation Prompt for Single Answer
The following is an open-ended problem from (subject) undergradduate-Ievd Physics.
The answer of The problem should be fanswer type description).
Please calculate the answer according to the given requirements and the information provided. Please LaTeX format represent the variables and formulas used the solution process and results.
Please end your solution with "So the final answer is answer (unit)." and give the result explicitly, note that the unit of the answers should not be included in D
{problem}
Evaluation Prompt for Multiple Answers
The following is open-ended problem from (subject) of undergraduuat-level Physics. The question has multiple answers, with the answers order being (answer type description). Please calculate the answer according to the given requirements and the information provided. Please use LaTeX format to represent the variables and formulas used in the solution process and results.
Please end your solution with "So the final
is multiple answers connected with commas and give the result explicitly, note the the answers should not be included in +
{problem}
20

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Algorithm 1 MARJ Pipeline
Input: Problem P. Solution S, Golden Answer List GT, Model Solution s, Model Answer List A. Initialize Correctness = Palse
if Len(A) not equals (en then
return False end if for gt, in GT, do flag False
if gt equals a then
flag True; continue end if
if P is a T/F or MC question then Transform gt, standard forms: gt', a' if gl' equals a' then
flag True; continue end if end if
if gt is expression or equation then
Normalizing a, gf to a', gt' by removing physical constants
if gt equals as equation or gt' equals a' as equation then flag True; continue
else if gt equals expression gt' equals a' expression then
flag True; continue
end if end if
if gl is Numeric Value then Transform a, gl into scientific notation: a Abase X 10 gl = glaas x 10
if [boase gtiusel/ /gtbaee < then
flag True; continue end if end if'
if gt is interval then
let c, d be endpoints of gt; e, be endpoints of a if c equals e as NV EX and equals f as NV or EX then
flag True; continue
end if end if end for
if flag equals True then
return True
else
return ModelJudge P, S. S.GT,,,A). end if
21

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 16. Physical Constants MARJ.
Physical Quantity
Symbol
Speed of light in vacuum
Newtonian constant of gravitation
G NA
Avogadro constant
Universal gas constant
R
Boltzmann constant (macroscopic)
R/NA
Molar volume of ideal gas
Vm
Elementary charge (proton charge)
Electron mass
me
Electron charge-to to-mass ratio
e/m
Proton mass
mp
Neutron mass
ma
Vacuum permittivity (electric constant)
Eo
Vacuum permeability (magnetic constant)
Ho
Electron magnetic moment
He
Proton magnetic moment
Pp
Bohr radius
d0
Bohr magneton
PB
Nuclear magneton
Planck constant
h h a
Planck constant
Fine-structure constant
Rydberg constant
R
Compton wavelength
h
Proton-electron mass ratio
Boltzmann constant
k
22

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 17. Results subjects of Mechanics & Thermodynamics UGPhysies (all figures in %). Models classified into four different categories according to their purpose and origin. The best results within each column are bolded and the best results of LLMs within similar group underlined.
Class. Mech.
Theor. Mech.
Relativity
Ther.ynn
Stat. Mech.
Models
EN
EN
EN
ZH
ZH
EN
Closed-source LLMs
OpeniAA--o--miin-0224-0000 GPT-44-2024400-06
47.01
47.61
40.13
35.74
61.35
36.04
34.30
36,45
46.79
31,43
37.68
36.84
31.03
24.45
36.71
42,03
38.71
39.78
38.21
40.00
GPT-4o-mini-2244007--
29.19
27.27
20.69
20.06
26.09
22.22
29.84
28.49
29.11
27.50
Open-sounce
Chat I.LMs
Y1-1.3-6B-Char
12.08
9.09
6.27
4.08
10.63
5.80
14.78
9.41
9.64
3.34
Dwenn.5--B-Instruee
26.67
23.68
20.06
15.36
20.77
14.01
28.76
25.27
19.82
16.79
LLaMA3.1-BB-nstte
12.68
11.29
4.39
9.66
5.31
16.13
8.87
12.14
Simistral-8B-Instruet--23
14.00
12.56
11.91
8,46
9.66
8.70
17.47
12.63
14.29
9.11
Yi-1.5-9B-CCL
17.58
12.80
10.66
8.46
14.01
10.14
20.16
15.32
14.64
10.18
Mistral-Nemo--nnsttutt-44
13.88
10.29
12.23
11.29
12.56
8.70
15.86
13.98
14.82
13.39
DeepSeek-Mon M-E--MMME----BB--
3.71
3.45
2.19
2.90
3.86
5.11
3.93
DeepSeck-V244 V2-LLte-Chhe
6.46
5.38
5.64
3.13
4.35
3.38
8.60
7.53
6.43
4.11
Mistral-Sman Sonall-Instruct-24100
22.61
19.74
20.69
15.67
15.94
16.91
22.85
23.12
26.43
27.14
Vi-1.5-34B-Chr
20.33
14.83
13.48
7.52
22.71
11.59
21.77
17.47
16.07
12.50
-
28.23
25.48
24.45
27.05
21.26
31.72
28.23
27.14
26.43
LLaMA3.3-700 7-0----trt7
32.66
26.56
28.21
16.61
32.37
22.71
41.67
29.84
33.21
28.39
Awve2.5-72BB Instruct
38.52
34.33
36.05
25.71
38.16
33,82
36.29
41.13
31.07
40.54
Mistral- Large-Lnsttuut--24444 Instruct-2407
36.72
34.33
35.74
28.53
38.65
37.68
42.74
41.13
38.04
41.61
Specialized
Mathematical LLMs
DeepSeeK-Math- Math-
14.47
12.92
1[.60
9.72
11.11
9.18
12.90
13.71
13.39
13.21
15.55
12.68
13.79
8.46
10.63
9.18
18.55
12.37
14.82
12.50
NuminaMath-7BBCool
14.23
18.42
13.17
11.91
8.21
10.14
14.78
16.13
14.11
14.82
-
15.79
13.04
12.23
10.34
13.04
7.73
14.52
12.63
15.71
14.46
Open -aa-aaahh--llaaa3111111 -BBB
10.05
6.94
5.64
4.08
4.35
4.83
11.02
7.53
8.21
6.25
Qwen2. 1ean22.5-Mathh7B--Insttee
26.08
25.72
21.00
19.12
23.19
16.43
26.61
20.70
20.54
17.14
OpenaMath2-Llama-33-17000
21.05
20.10
15.99
16.61
16.43
14.98
23.39
20.70
21.07
17.86
Oween2.5MMahh772B-nnnnnn
42.22
39.83
36.36
35,42
43.96
42.51
37.10
38.98
37.32
41.96
ol-like
LLMs
33.13
23,56
25.03
17.24
29.95
28.02
30.38
T8.0T
2488
T7.14
Skywork-ol- Saaii--ol--Opp----lma------- 1
Ri--eck--R-Diiili-------- 1-8B
17.46
13.88
10.03
5.96
7.25
5.31
16.40
9.14
9.82
6.96
17.70
7.30
9.40
4.39
21.26
9.66
17.20
8.33
15.89
6.79
QwQ-32B-Prees --Be-rew
37.44
39,95
33.54
29.78
43.96
43,96
40.05
42.47
33.04
34.64
DeepSeek-RL. Beneck-R---iing RI--ittill Siiill--een-aae 32B
37.44
29.07
28.53
22.57
42.03
40.10
35.48
31.18
33.04
25.00
DeepSeek-RA R1 -Lama-llm000
44.26
33.97
38.24
30.09
50.24
37.68
45.70
40.86
40.36
31.43
23

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 18. Results subjects of Electromagnatism on UGPhysics (all figures in %). Models elassified into four different categories according to their purpose and origin. The hest results within each column are bolded and the best results of LLMS within similar group underlined.
Class. Elec.
Dl
Geo. Optics
Wave Optics
Models
ZH
EN
ZH
EN
EN
ZH
Closed-source LLMss
OpennAl--i1-miii--224-----------
45.38
48.21
41.85
40.76
27.59
31.03
45.36
40.40
G00-40-0024-08-06
38.46
40.77
32,07
35.87
20.69
25.86
39.40
27.48
GPT-4o-mini-222407--B8
25.13
26.41
21.20
17.93
18.97
20.69
27.81
20.20
Open-sounce
Chat L.LMs
Y1-1.3-6B-Chär
12.56
8.2T
78.70
8.70
88.62
12.07
13.9T
3.96
Qwen2.5-7B-Instruut
26.15
20.00
17.39
18.48
20.69
15.52
26.16
15.56
LLaMA33J-B-Insttue
12.05
9.74
10.87
15.52
16.89
9.60
Ministral-BD-nnstmue2441
14.87
6.67
11.96
11.96
8.62
12.07
19.87
9.60
Yi-1.5-9B-CL
17.18
15.38
16,30
11.41
17.24
15.52
13.58
11.26
15.38
12.82
16.85
11.41
13.79
10.34
12.91
11.59
PrepSeek-MOE-1633
3.85
4.89
4.35
5.17
1.72
6.29
6.95
DeepSeck- 12-Lite-Chat
7.69
4.87
6,52
7.07
6,90
3.45
5.63
6.62
Mistral-Small Snnaallnnttrutt2200 2409
25.13
19.74
17.93
20.65
17.24
15.52
23.51
15.89
Yi-1.5-34B ---taB-Chht
20.00
12.05
14.13
14.13
15.52
15.52
20.20
12.25
LLaMA3.1 --7B-IInstrue
24.62
26.09
23.37
10.34
28.81
21.19
3LaMA3.3-70B-Inss..
33.85
29.49
30.98
23.91
20.69
17.24
36.42
20.86
Qwen2.5-72B-Inss
35.64
37.18
31.52
33.70
18.97
27.59
33.77
31.46
Mistral-Large Losttal-Lrge--nstuutt--222 Instruct-2407
43.33
40.00
33.15
34.78
24.14
29.31
37.75
29.47
Specialized Mathematical
LLMs
DeepSeek-Maile 7B-th-BB-ue IB-nnstrcct
17.69
14.87
13.04
11.41
17.24
10.34
16.89
12.58
DeepSeek-Mathe Math-aB-RR
17.44
14.62
17.93
10.87
13.79
12.07
10.93
11.59
NuminaMath-7B-CCS
15.90
15.13
15.22
15.22
10.34
22.41
12.91
14.90
16.41
13.08
18.48
19.02
15.52
15.52
19.54
14.90
OpenMath2-LLamma-.11
10.00
6.15
8.70
7.61
13.79
8.62
10.60
8.61
Owenn2.5-Math--BBnnstnuc
24.62
22.56
19.57
13.59
15.52
13.79
22.52
15.89
OpenMath2-Llama-311-000
21.54
21.28
23.91
15.76
17.24
25.86
22.85
15.56
Qween2.5-Math-72B-nnsee
40.26
40.77
28.26
38.59
29.31
22.41
36.09
38.41
ol-like
LLMs
DeepSččeKRRI-Diae --DbseE-RR---DiillS-Ceeee----
32.05
20.51
18.48
13.59
75.52
10.34
29.14
T7.2
Skywork-ol-Opee (amen-L-lama33.11
11.54
8.97
14.13
9.78
20.69
6.90
17.22
7.95
DeepSeek-RI- D--stiill--lameeee
18.46
6.92
12.50
7.07
13.79
6.90
18.87
5.63
QwQ-32B --Q-32B-Preview
35.38
35.13
31.52
36.96
20.69
22.41
41.39
25.50
DeepSeek-RR R.
37.44
27.44
22.28
21.74
15.52
17.24
38.41
20.20
DeepSeek RI Distill-Llama-i
46.41
31.54
34.78
28.80
25.86
29.31
46.69
25.17
24

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 19. Results subjects of Modern Physics UGPhysics (all figures in %). Models classified into four different categories according to their purpose and origin. The hest results within each column are bolded and the best results of LLMS within similar group underlined.
Quan. Mech.
Atomic Phy.
S.S.. Phy.
Semi. Phy.
Models
EN
ZH
EN
ZH
EN
ZH
EN
ZH
Closed-source LLMs
49.95
30.54
38.58
34,86
43.60
45.93
63.98
60.75
G00-40-002400
41.71
39.06
44.48
39.23
36.05
41.28
46.77
58.94
GPT-4o-mini-222407--BB
27.28
28.66
33,99
28.63
27.91
27.33
36.02
37.63
Open-sounce Chat I.LMs
Yf-1.3-6B-Chär
11.48
7.56
15.19
9.84
'9.0
'8.72"
22.04
T6.67
Qwen2.5-7B-instruet
23.06
20.22
27.10
21.75
22.09
21.51
34.95
30.65
LLaMA3.J-B-Instmne
13.05
10.79
19.89
14.75
10.47
27.96
18.28
Ministaa-SDD-nnstmtt-411
15.90
10.21
21.86
12.68
15.70
11.63
27.42
20.97
Yi-1.5-9B-CH
16.58
14.82
22.84
14.43
14.53
14.53
29.03
24.19
16.68
14.33
20.44
17.38
15.12
13.37
20.97
24.19
PrepSeek-MOE-1633
4.22
9.73
6.40
6.99
5.91
DeepSeck- 12-Lite-Chat
6.67
6.48
13.01
9.07
2.91
4.07
13.44
9.68
Mistral-Small Snnaal-nnttruut2222 2409
27.67
23.06
31.91
22.40
25.58
21.51
37.10
29.57
Yi-1.5-34B 3--3B--hhat
18.84
18.06
26.56
17.27
22.09
11.05
31.18
25.81
LLaMA3.1 --7B-IInstrue
31.11
27.38
33.22
29.65
24.42
45.16
34.95
3LaMA3.3-70B-Inss..
34.54
26.30
43.83
25.36
36.63
27.33
44.09
36.56
Qwen2.5-72B-Inss
34.25
37.98
40.00
37.81
33.14
34.30
42.47
45.16
Mistral-Large Losttal-Lrge--nstuutt--222 Instruct-2407
42.39
41.1
47.10
38.69
34.30
36.05
48.92
45.16
Specialized Mathematical
LLMs
DeepSeek-Maile 1B-th-BB-ua IB-nnstruct
15.51
13.15
23.17
16.39
15.70
17.44
21.51
23.66
DeepSeek-Math- Math- Ma---
16.98
12.76
20.44
17.81
15.12
12.21
20.97
20.97
NuminaMath-7B-CCC
17.27
16.19
18.91
20.22
12.21
14.53
23.66
26.34
16.88
16.00
20.77
17.81
23.26
12.21
29.57
23.66
OpenMath22LLama--.111
9.91
8.34
13.33
10.16
11.05
8.14
17.20
14.52
Owenn2.5-Math--BBnnstnuc
24.44
24.14
28.74
18.58
19.77
16.28
33.33
24.19
OpenMath2-Llama-311-000
21.10
18.25
28.52
22.62
28.49
21.51
31.18
33.87
Qween2.5-Math-72BBnnsee
39.35
41.22
43.61
37.60
31.98
34.30
47.85
43.55
ol-like
LLMs
DeepSčekKRRIDDSae --Dii
27.67
20.00
32,33
20.33
79.77
12:21
38.17
28.45
Skywork-ol-Opee 1appe---lmma-3....
12.95
9.52
17.70
10.38
11.63
7.56
20.43
14.52
DeepSeek-RII L-1-Disiill-Llama
16.39
9.13
25.57
9.07
11.05
6.98
31.72
10.75
QwQ-32B N-Q-32B-Preview
33.56
37.88
45.36
40.87
27.33
28.49
43.55
44.62
DeepSeek-RR R. Distiill-wen-322
32.78
26.10
44.70
31.48
26.16
21.51
45.70
37.10
DeepSeek RI
46.71
35.82
53.66
35.63
40.70
36.05
57.53
48.39
25

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Table 20. Results different skill UGPhysics (all figures in %). Models classified into four different categories according to their purpose and origin. The best results within each column are bolded and the best results of LLMss within a similar group underlined.
Know. Recall
Laws App.
Math Deri.
Prac. App.
Others
Models
EN
EN
EN
ZH
ZH
EN
ZH
Closed-source LLMs
OpeniAT--11-mii---22440000
69.18
6...77
33.60
31.46
42.16
45.05
37.92
45.75
30.49
30.97
GPT-44-2024400-06
63.36
55.25
42.46
38.94
31.40
33.71
34.54
32.72
38.83
40.69
GPT-4o-mini-224-077-B
53.77
48.29
30.81
28.00
20.72
22.48
24.59
17.00
29.61
26.70
Open-sounce
Chat L.MSs
Y1-1.3-6B-Char
26.37
21.23
14.46
7.97
6.7T
5..8
10.67
44.4"
T5.53
9.7T
Dwenn.5--B-Innsruet
48.29
40.24
26.78
21.72
17.48
15.68
19.71
13.92
27.18
22.33
LLaMA3.1-BB-nstte
34.93
25.51
15.99
8.02
6.44
14.10
17.48
12.62
Simistral-8B-Instrutt--22
37.67
26.37
16.91
10.42
10.41
8.65
14.83
5.61
19.90
14.56
Yi-1.5-9B-CCe
36.13
29.62
20.49
13.64
10.95
9.95
14.29
9.40
18.45
16.02
Mistral-Nemo--nnsttutt-44
35.62
32.71
15.94
12.62
11.58
10.50
12.84
8.50
16.99
16.50
DeepSeek-Mon Bareeee---E---------
16B-hht
13.18
5.52
4.45
3.06
2.43
2.71
7.77
DeepSeck-V244 V2-LLte-Chhe
22.26
17.98
8.94
6,34
3.78
3.74
4.16
3.25
8.25
5.34
Mistral-Sman Somall-nnstruct-24100
50.17
41.78
25.45
20.75
20.54
18.74
19.53
12.48
31.55
27.67
Vi-1.5-34B-Chr Chat
42.81
33.56
23.91
15.59
12.43
10.81
17.36
11.93
22.33
13.59
-
52.23
42.47
30.76
24.83
23.87
23.06
27.12
18.81
29.61
28.16
LLaMA3.3-700 7-0----tt-7
59.25
40,92
37.71
24.48
28.47
24.37
34.54
20,25
35.44
33.01
Awve2.5-72BB Instruct
56.68
56.34
38.02
37.61
29.01
32.39
33.27
26.76
40.29
39.32
Mistral- Large- Large-Lnstruut--24000 Instruct-2407
64.73
58.73
42.51
38.17
33,56
34.86
37.43
27.12
39.81
38.35
Specialized Mathematical
LLMs
DeepSeeK-Math-2 Math-
37.33
32,36
16.96
14.05
10.50
10.27
13.02
6.69
19.42
16,99
DeepSeek-Mathh
38.36
30.14
16.35
12.62
12.39
10.27
9.58
10.67
20.87
14.56
NuminaMath-7BB-C
32.36
33.90
14.51
18.14
12.57
11.85
12.12
11.03
20.39
23.30
-athstral-7B-xo.1
41.10
28.94
16.66
14.41
12.70
12.43
13.56
9.76
19.42
17.48
Open 3al-aatth---laaa311111111 -BBB
21.92
18.66
11.24
8.07
6.71
4.95
7.78
7.23
13.11
7.28
Swen2.5-Math--71
45.21
35.10
26.26
20.34
18.87
18.78
20.61
11.39
25.73
28.16
OpenaMath2-Llama-33-17000
47.60
37.67
22.79
21.67
16.67
14.10
19.89
17.54
25.24
20.39
Owenn255MMahh772---nnnnn.
61.30
56.51
41.44
38.83
33.69
38.15
33.82
28.57
39.81
39.81
ol-like
LLMs
DeepSedf-R13SSaallloeeeee-
47.26
34.59
35.40
20.93
22.03
76.53
26.40
T5.73
31.55
21,36
31.16
21.58
16.04
9.04
9.01
7.61
12.12
6.51
16.02
11.65
DeepSeck-RI-N RI-Distill-Llama
39.55
19.86
21.15
8.28
10.68
4.91
16.46
4.52
19.90
10.19
QwQ-32B-Prees B--rrreee
58.05
57.19
41.29
39.04
28.15
31.85
36.53
31.46
43.20
38.83
DeepSeek-RL. -i-kki-R-DDsiill Seeill-Qwen--e Qwen-32B 32B
56.85
45.89
39.29
30.10
27.61
22.07
37.25
22.24
35.44
30.10
DeepSeek-RA R1 -Lama-llm00
64.38
46.06
49.41
36.02
38.56
31.35
43.04
28.39
48.54
34.47
26

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Problem: A crystal has atoms and possesses Frenkel defects. The numbers of normal positions and interstitial positions are N and N'' respectively. The energy when an atom is in normal and interstitial positions is é and E2, with O. Derive the function relationship of the most probable number of interstitial atoms as function temperature (assuming n, N, N' 1).
Eu E1
ol-mini Completion: To determine the most probable number of interstitial
atoms n as function of temperature
consider the statistical mechanics of
Frenkel defects in a crystal.
**Multiplicity ($2)*:: The number of ways to arrange interstitial atoms among N normal sites and N' interstitial sites is given by:
N
N
[should be Ω Taking the natural logarithm and using Stirling's approximation for large NN No:
N
N'
hΩ = In
In
2. Energy (U)*s; The total energy associated with interstitial atoms is:
Ne1 + n(s2 - E1) = Nei + nw
Here, = E2 E1.
**Free Energy Maximization**: To find the most probable maximize the entropy subject the energy constraint:
d dn
[nS2
U
=0
KT
E:oN: multiplier method, where N1 - n, N22 = nl This lcads to:
[should ò - Σ J... B 3
0]. applying Lagrange
In
NN'
w
0
n²
KT
Solving for n:
In (In(NN') 2 KT
Exponentiating both sides gives:
NNN' e (W/(2TT)
So the final answer is Ground-Truthh Answer:
VNN' w/(2kT)
W KT
exp
Model Output:
NNE sir
Table 21. An example of Statistical Mechanics with flawed reasoning error.
27

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Problem: An clectromagneti eddy current brake consists a disc with electrical conductivity and thickness f, which rotates around an axis through its center an angular velocity w. There is magnetic field B with coverage of a² perpendicular to the disc. The area a² located at distance from the axis. Determine the magnitude of the eddy current generated under this configuration.
ol-mini Completion: To determine the magnitude the eddy current gener- ated in the electromagnetic eddy current brake, we follow these steps:
Induced Electric Field: The disc rotates with angular velocity in uniform magnetic field B perpendicular its surface. The tangential velocity at distance from the center = wrl The induced electric field E due to this motion is given by:
BF
Current Density: The current density J is related to the electric field by Ohm's 's law:
== owBrr
Total Eddy Current: To find the total current 1, integrate the current density over the area where magnetic field present. Considering the thickness I of the disc and integrating from the center (r = 0) to the extent of the magnetic field coverage (r
a):
1= 10 J. dA = 1-" TowBr (2mrl) dr
J.dA=
Ishould be / Tw Bao
ant
Simplifying the integral:
I = Bt S dr Bt
a
2x ow Bta
3
However, considering the standard configuration and simplifying constants for the practical scenario, magnitude of the eddy current can expressed as:
TtwBtaa
So the final answer Blw 2
(A).
Ground-Truth Answer: raBart
Model Output:rowBlu*
Table 22, An example of Classical Electromagnetism with wrong application error.
28

---

UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Calculate the effective mass a hole mi, It is known that static diclectric constant of InP is 9.6, E, 1.400 eV, Ee = 1.386eV, and holes in InP are much lighter than electrons (m < m,).
Problem:
ol-mini Completion:
To calculate the effective mass of hole mi in InP, we use the effective Rydberg formula adapted for semiconductors. The effective Rydberg energy R! is given by:
pe
R
2(4
where: - ya is the reduced mass, which for mb m simplifies j mi- is the elementary charge. Eg is the vacuum permittivity. e the static dielectric constant. h is the reduced Planck's 's constant.
Assuming the exciton binding energy Es in Inl is approximately 0.01 eV, we solve for the reduced mass jc:
2R" (4
e4
Given that R' E. 0.01 eV, and substituting the known values:
0.01
9.6
2 mi
& 0.07 mu
13.6
So the final answer is 0.07 Correct knowledge should look like this:
For hydrogen- like cases, the distance of these absorption lines from the funda- mental absorption threshold is:
m e
En = Eg
where m,, is the reduced mass of the electron-hole pair. Since mi & mg, it follows that:
m
RO mi
me + m
For the ground state, n = 1, and Fi = Fec = eV thus:
8E2E8h?(E, Ec)
mi
ea
8 9.6 x (8.854 x 10 12)2 x (6.626 x 10 31 x 0.014 x 1.6 x 10-19
(1.6 10 10-19)a
8.67 x 10 36
8.67 X 10
kg
~ 9.52 10 mo
9.1 x 10 31
Ground-Truth Answer: 9.52 x 10
Model Output: 0.07
Table 23. An example of Semiconductor Physics with knowledge deficiency error.
29


Reviews:
[{"review_id": "OPjKHKcTkX", "reviewer": "Reviewer_zXs9", "summary": "This paper introduces UGPhysics, a large‐scale, bilingual benchmark specifically designed for evaluating undergraduate-level physics reasoning with large language models. UGPhysics comprises 5,520 distinct physics problems (11,040 when including both English and Chinese versions) spanning 13 subjects and 59 topics. In addition to the dataset, the paper proposes a novel evaluation framework called Model-Assistant Rule-based Judgment (MARJ) that combines rule‐based precision with model‐based flexibility to assess complex, multi-step physics problem solutions. Extensive experiments across 31 LLMs reveal that even state-of-the-art models, such as OpenAI-o1-mini, achieve only about 50% accuracy, underscoring the challenges posed by physics reasoning compared to math-focused tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zXs9,\n\nThank you for your constructive feedback! We will reply to your questions one by one as follows:\n\n> Prior works on mathematical reasoning (such as [1]) evaluation of LLMs use evaluation techniques similar to the MARJ evaluation (i.e. using a combination of rule based + LLM-as-a-judge) method described in the paper (without explicitly describing the procedure). Regardless, I believe that stating the use of, and describing the procedure explicitly is a valuable contribution.\n\nThank you for pointing out this relevant paper and acknowledging our contribution.  \n\nAfter reading [1], they employ a model-based evaluation to get additional metrics from three angles, which differs slightly from our setting.\n\nWe will **include the following sentence in the \"Related Work\" section** (in \"Answer Judgment\") for discussion:\n\n\"Additionally, several works [1, 2] utilize model-based evaluation to obtain additional metrics for assessing effectiveness.\"\n\n\n> TheoremQA also contains some physics questions but has not been discussed in the paper.\n\nThank you for pointing this out. We will **add the following line to Table 1**: \n\n| Dataset     | Level | # Test | # UG | Subjects | # Ans. Types | Language | Eval. | Leak. Det. |\n|-------------|-------|--------|------|----------|--------------|----------|-------|------------|\n| TheoremQA   | 5     | 131    | 131  | –        | 5            | EN       | Rule  | No         |\n\n> Including a comparison of performances of models with some standard physics reasoning benchmarks (such as MMLU Physics subset, PhysicsQA, etc. to that on UGPhysics would help give the reader a better idea of the overall difficulty of the benchmark as compared to existing benchmarks.\n\nThank you for your suggestion! We will **add this comparison for the GPT-4o model as follows** (we also include MATH for reference) and will include a figure to illustrate this table in our manuscript as well:\n\n| Dataset          | Performance       |\n|------------------|-------------------|\n| Ours             | 38.67%           |\n| MMLU   (college physics)          | 68.6%   |\n|       MMLU  (high school physics)          | 72.8%  |\n|      MMLU  ((conceptual physics) )          | 92.3%  |\n| MMLU-pro         | 75.06%           |\n| OlympicArena     | 55.92%           |\n| GPQA             | 53.6%            |\n| MATH             | 76.6%            |\n\n\n\n\n> The authors mention that the initial questions are in Chinese and are then translated to English. How is this translation done? If it is done using LLMs / some other machine translation methods, are any measures undertaken in order to ensure a high quality of the translations?\n\nFollowing [3, 4], we leverage LLMs (specifically GPT-4o-2024-08-06) for translation. As demonstrated in [3] (using GPT-4 for translation) and [4] (using GPT-3.5-turbo), the quality of translation produced by LLMs is high. Since we utilize a significantly more powerful model, the translation quality is expected to be even higher.\n\nFurthermore, during the initial stages of translation, we manually reviewed several examples (typically 5-20) for each subject (particularly checking whether it can handle physics-specialized terminology). This process confirmed that GPT-4o excels at translating them.\n\n\nThank you once again for your valuable suggestions to improve the quality of our work. If you have any further questions or feedback, please do not hesitate to reach out to us.\n\nSincerely,\n\nAuthors\n\n[1] Didolkar et al., 2024; Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving \n\n[2] Huang et al., 2024; Olympicarena: Benchmarking Multi-Discipline Cognitive Reasoning for Superintelligent AI.  \n\n[3] Liu et al., 2024; Mathbench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. \n\n[4] Tang et al., 2024; Mathscale: Scaling Instruction Tuning for Mathematical Reasoning.", "claims_and_evidence": "The paper provides sufficient empirical evidence of the claims made. The main claim of the paper that physics reasoning abilities of LLMs have not received sufficient attention and as a result LLMs struggle on the task has been well substantiated by the observation that the best performance on the proposed benchmark is 49.8% on the proposed benchmark while several math reasoning benchmarks have been saturated.", "methods_and_evaluation_criteria": "The methodological description and as described are generally speaking quite sound. Prior works on mathematical reasoning (such as [1]) evaluation of LLMs use evaluation techniques similar to the MARJ evaluation (i.e. using a combination of rule based + LLM-as-a-judge) method described in the paper (without explicitly describing the procedure). Regardless, I believe that stating the use of, and describing the procedure explicitly is a valuable contribution. \n\n[1] Didolkar et al., 2024;  Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving", "theoretical_claims": "The paper does not make any theoretical claims", "experimental_designs_or_analyses": "The authors provide an elaborate set of experiments and discussion of their results on the benchmarks. The observation that Math specialised LLMs do not necessarily perform better on Physics as compared to their general counterparts is interesting, showing that finetuning on specific maths problems does not necessarily lead to an improvement in general reasoning capabilities. At the same time, the fact that o1 like models which are post-trained predominantly on Math / Code rasoning data, perform the best suggests that RL based post training can lead to improvements in general reasoning capabilities of models.", "relation_to_broader_scientific_literature": "This paper falls within the vast literature on LLM evaluation - specifically evaluating Physics reasoning capabilities of LLMs. While there exist multiple evalution benchmarks for phsics, most of them are either simple for existing LLMs, do not require elaborate CoTs, are limited in size or do not cover a wide range of topics. This value of this work stems from its elaborate subject categorization, support for two languages, difficulty level, larger size and an elaborate evaluation pipeline.", "essential_references_not_discussed": "TheoremQA [2] also contains some physics questions but has not been discussed in the paper. \n\n[2] Chen et al., 2024; TheoremQA: A Theorem Driven Question-Answering Dataset", "other_strengths_and_weaknesses": "All strengths and weaknesses of the paper have been discussed in other sections.", "other_comments_or_suggestions": "Including a comparison of performances of models with some standard physics reasoning benchmarks (such as MMLU Physics subset, PhysicsQA, etc. to that on UGPhysics would help give the reader a better idea of the overall difficulty of the benchmark as compared to existing benchmarks.", "questions_for_authors": "The authors mention that the initial questions are in Chinese and are then translated to English. How is this translation done? If it is done using LLMs / some other machine translation methods, are any measures undertaken in order to ensure a high quality of the translations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VuNAirCqt6", "reviewer": "Reviewer_3YX8", "summary": "This paper proposes a new benchmark that targets underground-level physics prompts. The prompts are mined from physics textbooks via a rigorous processing pipeline. The two stage eval protocol is designed for this benchmark, in which a rule based metric used followed by using llm (gpt4o) to double check those marked as wrong. The authors compare many LLMs on this benchmarks and the best one's score is less than 50 out of 100, so it could be a good benchmark for reasoning models for some time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3YX8,\n\nThank you for your helpful comments! We will reply to your questions one by one as follows:\n\n> It would be great if the authors can provide an analysis how robust the MARJ eval method is. e.g., how often it makes wrong judgement? In what scenarios LLMs can't correctly compare the given solution with gold?\n\nThank you for your suggestion! We **have conducted such an analysis in Section 5.2**: “We find that our MARJ evaluation achieves an accuracy of 98% when compared to human annotations.” During our manual inspection, we observed that our MARJ sometimes still fails to correctly evaluate answers that are equivalent in physics but require several steps of conversion. \nFor instance, consider the ground-truth answer $RT/\\mu$ and the model-generated answer $p/\\rho$. While both are physically equivalent ($p/\\rho= pV/m = nRT/m = RT/\\mu$, using the formula $PV = nRT$ and the definition $μ = m/n$), our MARJ fails to recognize the equivalence due to the need for multi-step conversion.\n\n> It would be great if the authors can run some stats on the complexity / difficulty of this benchmark. e.g., for o1-like reasoning models, how many tokens do them need to solve the problem on average?\n\nThank you for your comment. In fact, we have analyzed the difficulty of UGPhysics through “physics reasoning skills” in Section 5.1 (Figure 2b). As suggested, we will also **add the stats of the tokens** DeepSeek-Distill-Qwen-32B used to solve the problem on average: \n\n| Dataset    | Avg. Tokens  |\n|-------------|--------|\n| UGPhysics   | 4081   |    \n| MATH        | 3079   | \n\nIn this table, we also include the average tokens DeepSeek-Distill-Qwen-32B spent to solve MATH [1] for reference. \n\nIn addition, the average number of tokens that DeepSeek-R1 spent to solve problems in UGPhysics is 5555.\n\n> I'm curious how does the most frontier model e.g., o3-mini performs on this benchmark, since this basically measures the lifecycle of this benchmakr. (I understand there is lots of overhead to run this, especially if the authors are from academia, so I totally understand if the authors don't give this in rebuttal.)\n\nThank you for your valuable question and understanding! There is indeed a lot of overhead to evaluate o3-mini. We will **add the results of DeepSeek-R1**, whose performance is catching up with o3-mini high(90.8% vs. 86.9% on MMLU). (DeepSeek-R1 is much cheaper)\n\n\n  |Mec. and Ther. (EN) | Mec. and Ther. (ZH) | Elec. (EN) | Elec. (ZH) | Modern Physics (EN) | Modern Physics (ZH) | Overall (EN) | Overall (ZH) | Average  |\n---------------------|---------------------|------------|------------|----------------------|---------------------|--------------|--------------|----------|\n 0.5549              | 0.5667              | 0.5450     | 0.4839     | 0.5990               | 0.5729              | 0.5716       | 0.5553       | 0.5634   |\n\nFrom the table, the overall accuracy is 56.34%, which is higher than o1-mini as expected and there is still much room for improvement.\n\n\n> Regrading \"The UGPhysics is sourced from several undergraduate-level physics exercise books.\" What exercise books are used as data source?\n\nThank you for your question! There is a risk to reveal the institute of several authors and we give the links to these books at a later stage (if possible). \n\n> Why using math-specialized LLM for this physics benchmark?\n\nAs we mentioned in L105-108: “The inclusion of math LLMs aims to assess the extent to which training on specialized math corpus contributes to physics reasoning.”  From experiments, we find that “math-specialized LLMs yield only minor improvements over their general-purpose counterparts in UGPhysics, suggesting the compulsion for more high-quality physics corpora.” (L119 –L122). \nWe believe the Reviewer zXs9 gives an interesting discussion about this in the section of \"Experimental Designs Or Analyses\" in his/her review.\n\nWe would like to thank you once again for your useful suggestions to improve the quality of our manuscript. Feel free to discuss if you have any further questions or comments.\n\nSincerely,\n\nAuthors\n\n[1] Hendrycks et al., 2021; Measuring Mathematical Problem Solving with the MATH Dataset.", "claims_and_evidence": "This is a new benchmark paper, so i have less concern on this.", "methods_and_evaluation_criteria": "- The method of creating this benchmark is reasonable. All questions are grounded on physics textbooks and the extracted latex format goes through manual check, so the quality should be good.\n- Eval for STEM questions is not easy since the gold answer could be freeform (unlike math is more formalized), so the proposed two stage eval protocol is a reasonable approach, though i'd hope there is a \"STEM sympy\" someday.", "theoretical_claims": "No theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiments are mainly comparing LLMs in the proposed benchmark, which i don't find clear concern.", "relation_to_broader_scientific_literature": "- This paper mainly targets pushing the frontier of LLM reasoning models.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Weakness:\n - It would be great if the authors can provide an analysis how robust the MARJ eval method is. e.g., how often it makes wrong judgement? In what scenarios LLMs can't correctly compare the given solution with gold?\n-  It would be great if the authors can run some stats on the complexity / difficulty of this benchmark. e.g., for o1-like reasoning models, how many tokens do them need to solve the problem on average?\n- I'm curious how does the most frontier model e.g., o3-mini performs on this benchmark, since this basically measures the lifecycle of this benchmakr. (I understand there is lots of overhead to run this, especially if the authors are from academia, so I totally understand if the authors don't give this in rebuttal.)", "other_comments_or_suggestions": "Please check my previous section.", "questions_for_authors": "- Regrading \"The UGPhysics is sourced from several undergraduate-level physics exercise books.\" What exercise books are used as data source? \n- Why using math-specialized LLM for this physics benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zl42MR7sqJ", "reviewer": "Reviewer_SPQT", "summary": "The paper introduces a comprehensive bilingual benchmark UGPhysics for evaluating undergraduate physics reasoning, featuring 5520 questions across 13 subjects. The benchmark also comes with a proposed evaluation pipeline that combines rule-based and model-based methods for improved accuracy. Notably, the study finds that even top-performing LLMs achieve less than 50% accuracy on this proposed benchmark, highlighting a critical need for improvement in LLM capabilities for physics reasoning.\n\n\n## update after rebuttal\nI have read the author response. I'll keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SPQT,\n\nThank you for your valuable suggestions! We will reply to your questions one by one as follows:\n\n> However, one concern I have is that in 5.2 Reliability of Evaluation, only 100 random test examples are being examined and it's not clear what are the answer types of those questions. Do they cover all the answer types, or only some of the seven answer types?\n\nThank you for your question! Previous studies [1, 2, 3] have shown that using similar or even smaller sample sizes (100 [1], 80 [2], and 50 [3]) is sufficient for human evaluation tasks, even for more subjective tasks such as text summarization [3].\nRegarding the types of answers, after examining 100 randomly selected test examples, we observed that all seven answer types are represented, except for True/False (TF). We believe it is OK because the evaluation of the TF question is relatively straightforward.\n\n> In line 372-382: \"LLMs show varying performance across different subjects, although the disparity is relatively small...\" I don't know where the numbers mentioned are coming from (e.g., 27.0%, 22.9%, 13.8% etc), I don't see those numbers from Figure 2 (a), could you clarify this?\n\nThank you for pointing this out! We apologize that we mistakenly put the wrong numbers after the update of Figure 2(a). We will **correct these numbers accordingly**: \n\"As shown in Figure 2a, the average overall accuracy of eight strong LLMs reveals that they perform particularly well in Semiconductor Physics (31.0%) and Atomic Physics (26.7%). \nIn contrast, their performance is slightly lower in Theoretical Mechanics (16.5%). Additionally, LLMs show minor performance variation across six out of 13 subjects, with accuracies hovering around 20%” \n\n> For o1 like models, I would like to know how many percentages of the inference generations didn't end due to length limit (i.e., 8192 as mentioned in the Appendix).\n\nThank you for your insightful question! In our experiments, we found that most cases of o1-mini generation will end within 8192 tokens. From the analysis in Section 5.3, the error incurred by the length limit is around 5% in all failure cases (approximately 2.5% = 5% * 50% in total). After checking the other open-source o1-like LLMs, we find the percentage is much higher than o1-mini.\n\nWe will **add the following table of this percentage as follows** (in %):\n\n| Models                        | 8192  | 16382  |\n|-------------------------------|-------|--------|\n| DeepSeek-R1-Distill-Qwen-32B  | 38.55 | 34.47  |\n| DeepSeek-R1-Distill-Qwen-7B   | 44.40 | 38.90  |\n| DeepSeek-R1-Distill-Llama-70B | 19.16 | 12.25  |\n| DeepSeek-R1-Distill-Llama-8B  | 52.37 | 43.80  |\n| o1-mini-2024-09-12            | 2.01  |     -   |\n| QwQ-32B-Preview               | 19.01 | 8.54   |\n\nWe believe this is also a gap between open-source o1-like LLMs and OpenAI o1 series. We will **add a paragraph in Section 5.1 to discuss this as follows**:\n\n\"Open-source o1-like LLMs typically consume more tokens compared to OpenAI's o1-mini when solving problems in UGPhysics. When the maximum length of generation is set to 8192 tokens, only around 2% of OpenAI o1-mini’s generations exceed this length limit. In contrast, a significantly higher proportion of inference generations for other open-source o1-like LLMs fail to terminate within the specified limit, as shown in the previous table. To assess whether increasing the maximum generation length improves the performance of these o1-like LLMs, we conducted additional experiments by extending the token limit to 16384. The results, presented in the following table, demonstrate that doubling the maximal generation tokens only slightly improves the performance of o1-like LLMs. Additionally, we report the proportion of cases where the generation did not terminate due to the extended length limit of 16384 tokens. These findings suggest addressing the redundancy in token consumption of o1-like LLMs [4] during reasoning remains an important direction for further research.\"\n\n| Models /Acc (in %)                       | 8192  | 16384 |\n|-------------------------------|-------|-------|\n| DeepSeek-R1-Distill-Qwen-7B   | 24.64 | 24.86 |\n| DeepSeek-R1-Distill-Llama-8B  | 13.11 | 14.51 |\n| QwQ-32B-Preview               | 37.34 | 38.90 |\n| DeepSeek-R1-Distill-Qwen-32B  | 31.93 | 32.21 |\n| DeepSeek-R1-Distill-Llama-70B | 40.17 | 41.77 |\n\n\n\nThank you once again for your insightful comments to improve the quality of our work. Feel free to discuss if you have any further questions or comments.\n\nSincerely,\n\nAuthors\n\n[1] Gao et al., 2024; Omni-Math: A Universal Olympiad-Level Mathematics Benchmark for Large Language Models.\n\n[2] Shaib et al., 2024; How Much Annotation is Needed to Compare Summarization Models.\n\n[3] Zheng et al., 2023; Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.\n\n[4] Chen et al., 2024; Do Not Think That Much for 2+3=? On the Overthinking of o1-like LLMs.", "claims_and_evidence": "The proposed benchmark is well-curated and manually reviewed.", "methods_and_evaluation_criteria": "The proposed evaluation pipeline MARJ makes sense to me. However, one concern I have is that in 5.2 Reliability of Evaluation, only 100 random test examples are being examined and it's not clear what are the answer types of those questions. Do they cover all the answer types, or only some of the seven answer types?", "theoretical_claims": "The paper does not have theoretical claims.", "experimental_designs_or_analyses": "The experiments that examine 31 leading LLMs' performance on UGPhysics are well executed.", "relation_to_broader_scientific_literature": "While mathematical reasoning has numerous benchmarks, AI for physics remains underexplored, lacking challenging evaluations and diverse question types. This paper addresses this gap by proposing a comprehensive physics benchmark that surpasses previous ones in size, difficulty, and subject coverage.", "essential_references_not_discussed": "The paper covers related work well.", "other_strengths_and_weaknesses": "Please see other sections.", "other_comments_or_suggestions": "It would be nice to also have a table that lists the information of how many questions are there for each answer type.", "questions_for_authors": "- In line 372-382: \"LLMs show varying performance across different subjects, although the disparity is relatively small...\" I don't know where the numbers mentioned are coming from (e.g., 27.0%, 22.9%, 13.8% etc), I don't see those numbers from Figure 2 (a), could you clarify this?\n- For o1 like models, I would like to know how many percentages of the inference generations didn't end due to length limit (i.e., 8192 as mentioned in the Appendix).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CJCHfPAigQ", "reviewer": "Reviewer_wqeU", "summary": "The paper proposes a new benchmark for reasoning in physics by LLMs, they used 31 LLMs to evaluate the performance on the proposed benchmark, then they have introduced a new method MARJ to evaluate the outputs on these 31 LLM on the benchmark. Overall they show that OpenAI-01-mini gives the best performance on this benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear wqeU,\n\nThank you for your time and effort to review our work! We will reply to your questions one by one as follows:\n\n>  the human evaluation of the proposed method was only checked with 100 examples, which I feel is a very small sample, apart from this the model used for evaluation is Open AI-4o which in turn is the same family of models that shows the best performance.\n\nPrevious studies [1, 2, 3] have shown that using similar or even smaller sample sizes (100 [1], 80 [2], and 50 [3]) is sufficient for human evaluation tasks, even for more subjective tasks such as text summarization [3]. The second-best LLMs  (QWQ and DS-Distill) are not within the same family of GPT-4o. Additionally, OpenAI o1-mini is a Long CoT LLM, which is quite different from GPT-4o. [1] has also shown that LLMs-as-judge is a valid approach even if the LLMs are within the same family. \n\n> The authors have considered 31 leading models, but the choice of models is not clear, as in the closed source LLM only one family of models are considered, there are other families of closed source models which I feel should have been considered for better evaluation. Also models of Phi family which are specifically trained on text book data are not considered. The selection of models according to the task should be considered even if it is a lesser number of models.\n\nThank you for your suggestion to include Phi. We will **add the results of Phi-4 as follows**:\n\n| Mec. and Ther. (EN) | Mec. and Ther. (ZH) | Elec. (EN) | Elec. (ZH) | Modern Physics (EN) | Modern Physics (ZH) | Overall (EN) | Overall (ZH) | Average  |\n|---------------------|---------------------|------------|------------|----------------------|---------------------|--------------|--------------|----------|\n| 0.3413              | 0.3248              | 0.3651     | 0.2987     | 0.4045               | 0.3586              | 0.3716       | 0.3344       | 0.3530   |\n\n\nFrom the results, Phi-4 is a very strong fast-thinking LLM. \n\nWe have listed all the details of the chosen LLMs in Appendix B.1. We have covered OpenAI, Qwen, Llama, DeepSeek, Mistral, Skywork, Yi, Numina, and OpenMath2, which we believe are very diverse. It is quite expensive to cover more closed-source LLMs, especially Claude-series, and we could not afford to do so for more closed-source LLMs. \n\n\n> The paper writing has a good amount of redundancy, specifically the section 3.1, which was mentioned multiple times and also about the evaluation method.\n\nThank you for your suggestion, we will **change the name of Section 3.1** to “UGPhysics and MARJ Overview”. \n\n> In line 87 they say rigorous data leakage, but the methods to check data leakage were not that rigorous. More details on checking the data leakage should be provided, like the number of times each problem was run and then checked if in every case the model didn’t give the output as in the question then you could say there is no data leakage.\n\nThank you for pointing this out. We will **change our wording about \"rigorous\"**. This data leakage detection [4] is widely adopted [1, 5], and is believed to be in some sense useful.  All our settings align with [5]. Although this method is not perfect, we believe that **conducting such detection is a merit rather than a shortage**. \n\n> In line 176, there is no reference to the books used for data creation.\n\nThank you for your question! There is a risk to reveal the institute of several authors and we give the links to these books at a later stage (if possible).\n\n> In the appendix, more examples and one whole example of whole work from textbook to translating to getting output from LLM and then evaluation of it with ground truth would be better.\n\nThank you for your comments. We will consider adding more examples to the appendix and add a whole example as well.\n\n\nThank you again for your effort and suggestions. We hope our rebuttal has addressed your concerns. Feel free to discuss if you have any further questions or comments.\n\nSincerely,\n\nAuthors\n\n[1] Gao et al., 2024; Omni-Math: A Universal Olympiad-Level Mathematics Benchmark for Large Language Models.\n\n[2] Shaib et al., 2024; How Much Annotation is Needed to Compare Summarization Models.\n\n[3] Zheng et al., 2023; Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.\n\n[4] Xu et al., 2024; Benchmarking Benchmark Leakage in Large Language Models.\n\n[5] Huang et al., 2024; OlympicArena: Benchmarking Multi-Discipline Cognitive Reasoning for Superintelligent AI.", "claims_and_evidence": "**Claim:** Introduction of UGPhysics, benchmark to evaluate the Physics reasoning with LLMs, \n\n**Evidence:** Though they do introduce a benchmark, it’s not totally about reasoning, it’s also about problem solving abilities.\n\n**Claim:** Introduction of MARJ for evaluating the outputs given by different LLMs on the benchmark.\n\n**Evidence:** They certainly do introduce MARJ, and give details of the method. In section 3.3.\n\n**Claim:** Open Ai-o1 Achieves the best score on this benchmark:\n\n**Evidence:** Table 5 shows the results of all the models and OpenAI-01 scores the highest.", "methods_and_evaluation_criteria": "The evaluation method used doesn’t really support this work, the human evaluation of the proposed method was only checked with 100 examples, which I feel is a very small sample, apart from this the model used for evaluation is Open AI-4o which in turn is the same family of models that shows the best performance.", "theoretical_claims": "There aren’t any theoretical claims", "experimental_designs_or_analyses": "The authors have considered 31 leading models, but the choice of models is not clear, as in the closed source LLM only one family of models are considered, there are other families of closed source models which I feel should have been considered for better evaluation. Also models of Phi family which are specifically trained on text book data are not considered. The selection of models according to the task should be considered even if it is a lesser number of models.", "relation_to_broader_scientific_literature": "If mentioned weaknesses are corrected, the benchmark can be of use for fine tuning and working with LLMs for science.", "essential_references_not_discussed": "The related work is well discussed, they mention the existing work related to Physics, existing benchmarks related to Physics and also reasoning.", "other_strengths_and_weaknesses": "**Strengths**:\n\nNew benchmark for evaluating how well LLMs can work on Physics problems\n\nEvaluating the performance of 31 different models on the proposed benchmark\n\n**Weaknesses**:\n\nThe Evaluation method is not robust enough, as mentioned above in Methods And Evaluation Criteria*\n\nThe paper writing has a good amount of redundancy, specifically the section 3.1, which was mentioned multiple times and also about the evaluation method.\n\nUntil line 81 there is not mention of language of the data and then there is a mention of translating into English\n\nIn line 87 they say rigorous data leakage, but the methods to check data leakage were not that rigorous. More details on checking the data leakage should be provided, like the number of times each problem was run and then checked if in every case the model didn’t give the output as in the question then you could say there is no data leakage. \n\nIn line 176, there is no reference to the books used for data creation.\n\nIn the appendix, more examples and one whole example of whole work from textbook to translating to getting output from LLM and then evaluation of it with ground truth would be better.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces UGPhysics, a large-scale benchmark of 5,520 undergraduate-level physics problems in English and Chinese, designed to comprehensively evaluate LLMs across 13 subjects and multiple reasoning skills. Using a novel Model-Assistant Rule-based Judgment (MARJ) pipeline, the authors assess 31 LLMs and find that even top models achieve only 49.8% accuracy, highlighting the need for improved physics reasoning capabilities in LLMs.

Overall the benchmark seems useful and helpful, and it addresses a timely problem. All reviewers are positive and the concerns of the less positive reviews seem to be overall answered in the discussion.
Decision: Accept (poster)