CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics
Authors: Weida Wang
Dongchen Huang
Jiatong LI
Tengchao Yang
Ziyang Zheng
Chuyi Peng
Di Zhang
Dong Han
Benteng Chen
Binzhao Luo
Zhiyu Liu
kunling liu
Zhiyuan Gao
Shiqigeng
Wei Ma
Jiaming Su
Xin Li
Shuchen Pu
Yuhan Shui
Qianjia Cheng
Zhihao Dou
Dongfei Cui
Changyong He
Jin Zeng
Zeke Xie
Mao Su
Dongzhan Zhou
Yuqiang Li
Wanli Ouyang
Yunqi Cai
Xi Dai
Shufei Zhang
LEI BAI
Jinguang Cheng
Zhong Fang
Hongming Weng
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Benchmark
Condensed Matter Physics
LLM Evaluation
AI for Physics
Primary area: datasets and benchmarks

Abstract: We introduce CMPhysBench, designed to assess the proficiency of Large Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark. CMPhysBench is composed of more than 520 graduate-level meticulously curated questions covering both representative subfields and foundational theoretical frameworks of condensed matter physics, such as magnetism, superconductivity, strongly correlated systems, etc. To ensure a deep understanding of the problem-solving process,we focus exclusively on calculation problems, requiring LLMs to independently generate comprehensive solutions. Meanwhile, leveraging tree-based representations of expressions, we introduce the Scalable Expression Edit Distance (SEED) score, which provides fine-grained (non-binary) partial credit and yields a more accurate assessment of similarity between prediction and ground-truth. Our results show that even the best models, Grok-4, reach only 36 average SEED score and 29% accuracy on CMPhysBench, underscoring a significant capability gap, especially for this practical and frontier domain relative to traditional physics.

Published as a conference paper ICLR 2026
CMPHYSBENCH: A BENCHMARK FOR EVALUATING LARGE LANGUAGE MODELS IN CONDENSED MATTER PHYSICS
Weida Wang Dongchen Huang2. Jiatong Li Tengchao Yangl. Ziyang Zheng Chuyi Peng*1 Di Zhang'. Dong Han', Benteng Chen'. Binzhao Luo", Zhiyu Liu Kunling Liu
Zhiyuan Gao Shiqi Geng', Wei Ma Jiaming Su Xin Li Shuchen Pul, Yuhan Shui!,
Qianjia Cheng', Zhihao Dou', Dongfei Cui Changyong He .in Zeng'. Zeke Xie*, Mao Su', Dongzhan Zhou', Yuqiang Li,, Wanli Ouyang', Yunqi Cai
Xi Dai Shufei Zhang"i, Lei Bai" Jinguang Cheng"), Zhong Fang" Hongming Weng 'Saanhaai Artificial Intelligence Laboratory
Beijing National Laboratory for Condensed Matter Physics and Institute of Physics.
Chinese Academy of Sciences
Condensed Matter Physics Data Center, Chinese Academy of Sciences
*Fudan University 'Tongji University 'Hong Kong Polytechnic University 'Hong Kong University of Science and Technology
HHong Kong University of Science and Technology (Guangzhou) { (wangweida, da, zhangshufei, baileil @pp lab. org. cn, (huangdongchen, caiyq, zfang, hmweng iph...h.... aophy.cc.. cn, daix@ust.hr
ABSTRACT
We introduce CMPhysBench, designed to assess the proficiency of Large Language Models (LLMs) in Condensed Matter Physics, as novel Benchmark. CMPhysBench is composed of more than 520 graduate-level meticulously curated questions covering both represen- tative subfields and foundational theoretical frameworks of condensed matter physics, such as magnetism, superconductivity. strongly correlated systems, etc. To ensure a deep understanding of the problem-solving process, we focus exclusively on calculation prob- lems, requiring LLMs to independently generate comprehensive solutions. Meanwhile, leveraging tree-based representations of expressions, we introduce the Scalable Expression Edit Distance (SEED) score, which provides fine-grained (non-binary) partial credit and yields a more accurate assessment of similarity between prediction and ground-truth, Our results show that even the best models, Grok-4, reach only 36 average SEED score and 29% accuracy on CMPhysBench, underscoring a significant capability gap, especially for this practical and frontier domain relative to traditional physics. The code and dataset are publicly available http://ghhitaaocm//////sseeeeien
1 INTRODUCTION
Recent advances in large language models (LLMs) have revolutionized natural language processing, demon- strating exceptional capabilities in understanding and generation tasks (Brown et al., 2020 Devlin et al.) 2019), particularly in commonsense and mathematical reasoning, often enhanced by reinforcement learning techniques (Guo et al. 20255 Kojima et al. (2022). Leveraging these strengths, LLMs have achieved impressive results in Olympiad-level mathematics (Zhang al. 2025a), complex programming (il-Kishky et al. 2025). and even scientific discovery (Bai et al. 2025) Yang et al. 2023 Wang et al., (2025), fueling expectations for their applicability in physics. As a field grounded in uncovering the fundamental laws of nature, physics

---

Published as a conference paper ICLR 2026
Example Problem
Consider the Anderson s-d exchange model with Hamiltonian
E Rao as Ma + > Par +
U
Σ Yea ( doCoo)
Answer Type: Equation
H =
E FRollKo +
where
Final Answer:
Exe Eg apgh, Ego Exg oright
Here, M===
=(<and ((2ind)) (n) is the Bohr magneton, with Landé factor of go 91 = for both
(a FaodOko = Vxo
electrons and impurities. This the non-dogenerate orbital Anderson s-d mixing model.
Derive equation of motion for the s-d exchange model concerning the mixed Green's
Topic:
function < Cko ldž >.
Strongly Correlated Systems
Hint: Let Axe symbolize < Cealdy and ba symbolize & d d 2
Scalable Expression Edit isstacce
Expression Eitt istance
Ground Truth: (a) - aas V Da
Model Response I: (al Eau ar + Aho Venhal
Model Response 2:
Ground Truth:
EED Score
Bag Teahe
100
Model Response
EED Score
(co Egg Clen V Vao
100
Kg
Model Response
EED Score 100
V Veabo
Kid
ito
Aecuracy
F
SEED Score
E
SEED Score
SEED Score 0
Ground Truth
Resporse
Response 2
100
52
ACC Score
ACC Score
ACC Score
Figure 1: Example problem from CMPhysBench comparing three metrics for model performance evaluation: Expression Edit Distance (EED) (Quu et al. 2025b), Accuracy (Acc) (He et al., 2024) and the proposed Scalable Expression Edit Distance (SHED). Scores for three different responses to the problem shown, where SEED excels at both accuracy and fine-grained evaluation. The detailed interpretation of symbols is shown in Appendix H
imposes uniquely rigorous demands LLMs, requiring not only advanced reasoning and mathematical precision but also deep conceptual understanding of physical principles, concepts and approximations making an ideal testbed for evaluating whether LLMs truly comprehend the structure of the real world.
Previous benchmark efforts, such as SciQ (Welbl et al., 2017) and ScienceQA (Saikh et al. 2022). have played an important role in facilitating the evaluation of LLMs physics-related questions. However, these benchmarks primarily focus on high school-level content, which may not adequately test the complexity of reasoning the degree of mathematical rigor required for evaluating advanced understanding in physics. More recent benchmarks, including PHYBench (Qiu et al. 2025b) and UGPhysics (Xu et al., (2025a), have made meaningful progress by incorporating undergraduate-
problems. Nonetheless, these benchmarks remain limited in depth, as they often underrepresent the most critical and frontier areas of contemporary physics research. Considering the inherent conceptual and mathematical complexity of physics, broader and more rigorous benchmarks are essential for assessing whether LLMs can support real-world scientific tasks and facilitate cross-discipe crossdisccplinary integration.
In this work, we focus on Condensed Matter Physics (CMP), which becomes the mainstream of current physical research and investigates the physical properties and microscopic structures of condensed phases of matter, namely solids and liquids (Marder) 2010). As central area of modern physics, condensed matter has become a driving force behind many recent theoretical and experimental advances, contributing to our understanding of phenomena such superconductivity. topological states, and quantum phase transitions. This field integrates concepts from quantum mechanics (Messiah) (0014). statistical physics (Wannier [9877, ), solid-state physics (Grosso & Parravicini 2013), and many-body theory (Inkson 2012), posing significant challenges due to its complexity, inter-disciplinarity data-scarcity, and demand for precise mathematical formulation evaluation.
2

---

Published as a conference paper ICLR 2026
To address these challenges and test the performance of LLMs in modern physical science, we present CMPhysBench, novel benchmark specifically designed to evaluate the problem-solving abilities of LLMs in CMP. It comprises 520 questions, manually authored by Ph.D. students and postdoctoral researchers based on standard graduate textbooks spanning key CMP subfields, with difficulty levels ranging from undergraduate to advanced graduate coursework. Unlike multiple-choice benchmarks (Saikh et al. 2022; Yue al. 2025) that are ignorant of intermediate reasoning and procedural correctness, IMMPhysBench emphasizes open- ended calculation problems, requiring models to produce complete solutions that reflect both conceptual understanding and computational precision. Furthermore, quantify the differences between mathematical responses and handle various answer types, we propose the Scalable Expression Edit Distance (SEED) metric shown in Figure [
The SEED metric is inspired by Expression Edit Distance (EED) (Qiu al. 2025b) and offers a more robust and interpretable performance measure than exact string matching (He et al. 2024
To summarize, our contribution lies in the following aspects:
Graduate-level CMP benchmark with open-ended calculation. We release CMPhysBench, a 520-question benchmark manually authored by P). students and postdoctoral researchers based on standard graduate textbooks, spanning core subfields and emphasizing open-ended calculation tasks that require complete, step-by-step solutions across five answer types.
SEED: fne-grained, accurate evaluation metric. We propose the Scalable Expression Edit Distance (SEED), which maps diverse answer types to ASTs and computes tree-edit distance with unit conversion, encentifc-notation parsing, and rounding within tolerance, yielding non-binary partial credit and interpretable error localization.
Comprehensive empirical study and diagnosis. We evaluate 18 proprietary and open-source LLMs on CMPhysBench, finding consistently low performance and pronounced variability across models, and providing quantitative analyses that illuminate failure modes and opportunities for improving domain-specific reasoning in CMP. Our experimental also results reveal notable performance gap between mathematical reasoning and physical reasoning in CMP.
2 CMPHYSBENCH
The whole CMPhysBench benchmark consists of metric.
three parts: dataset overview, data curation and evaluation
2.1 OVERVIEW
As shown in T Appendix B CMPhysBench covers 520 carefully curated questions with difficulty spanning from introductory undergraduate exercises to advanced graduate-level challenges from CMP. CMPhysBench comprises six representative topics of CMP, structured as follows. Firstly, ensure domain representativennsss we include four core topics: Magnetism, Superconductivity, Strongly Correlated Systems, and Semiconductors. Secondly, to holistically evaluate LLMS beyond narrow domain expertise, we extend the benchmark with two additional dimensions of CMP. One of the additional categories is Theoretical Foundations, which encompasses, crystallography, plasmonics, phase transitions, and condensed matter field theory. The other is Others, which further includes quantum mechanics, statistical physics, electrodynamics and quantum field theory. This hierarchical categorization allows simultaneous assessment of domain-specific knowledge and general physical reasoning capabilities.
At the same time, following the settings OlympiadBench (He et al. 2024). we also categorize these questions based on different answer types. Specifically, there are five answer types in CMPhysBench, including tuple, equation, numeric, expression, and interval. The categorization of the questions performed by human experts to ensure its correctness. Details of the data categorization and distribution are listed in
3

---

Published as conference paper ICLR 2026
(b)
(c)
Foundations (130)
(110)
21.2%
0,8%
Answer Type
Example
6.0%
Tuple
(x,y,) =
(0 1.1)
21.2%
CMPhys Bench
8.55
A Answer Type
Expression
zm0"
65.6%
Problem Topic
Interval
(0.1)
19
Superconductivity
10.6%
Numeric
16
Strengly Systems (15)
Tuple
Equation
> +" =1
Expresion
Equatiin
Semicondaciors
Interval
(115)
Figure Overview of the CMPhysBench datasct and answer types. (a) Distribution of problem topics across various condensed matter physics domains in CMPhysBench. (b) Distribution of answer types across the dataset, highlighting the prevalence of numeric answers. (c) A table displaying examples of each answer type.
Figure a and (b), and our benchmark contains topics across various fields in condensed-matee physics, and the problems can be divided into five types: Tuple, Numeric, Expression, Equation and Interval, and the examples of them are shown in Figure. (c).
2.2
DATA CURATION
Collection
Structuring
Quality Control
Annotation
DO
A
Condensed Matter
Markdown Parsing
Question& Spliting
Auto Fixing
Expert Review
Problem Label Tagging
Physies Textbooks
Figure
The data curation pipeline of CMPhysBench.
We initially collect course materials and exercise problems from 17 textbooks with difficulty spanning from introductory undergraduate exercises to advanced graduate-level challenges. We mainly choose classical textbooks in CMP like An Introduction to Quantum Field Theory [Peskin][018). Classical Field Theory (Soper 2008), Condensed Matter Field Theory (3rd edition) (Altland & Simons, 2010), Introduction to Many--odyy Physics (Coleman) 2015), Statistical Physics (Landau Lifshitz 1980) etc. As shown in Figure 3 the data curation pipeline consists of four stages ensure the quality and usability of the benchmark.
Collection. Firstly, the collected textbook materials are first converted from PDF to Markdown format, followed by transformation into structured, machine-readan readable text formats. Specifically, convert the PDF documents of textbooks into Markdown format via MathPix
https://mathpix..t
4

---

Published as a conference paper ICLR 2026
Structuring. Subsequendly, we carefully modify the selected the problems relevant calculation tasks and adapted them to standardized iicculation-uuestinn format suitable for benchmarking. Specifically, we propose only calculation problems.
Quality Control, Expert Review and Annotation. Finally, each adapted question is manually checked by Ph.D. students and postdoctoral researchers specialized Condensed Matler Physics. During this review process, incomprehensible ambiguous questions are removed and detailed answers and solutions were carefully verified, ensuring that retained data could be clearly interpreted and evaluated. In addition, all questions are further classified based on the type of answer they require, demonstrated by Figure 2 (c).
2.3 EVALUATION METRIC: SCALABLE EXPRESSION EDIT DISTANCE (SEED)
To provide robust and fine-grained evaluation, we follow the core EED pipeline. We first extract the mathematical expression from the model output and canonicalize it to standard LaTeX. we then convert it to a SymPy object via latex2s syppyeetttteee ended, normalize terms positive canonical form, and apply s simplify () to stabilize and accelerate subsequent comparison.
Illustration of SEED Calculation
Expression
Equation
Ground Truth:
Model Response
Model Response Vx
GT: exp;
exp, =exp2
(o0 Ean Mair
V Veale
(ax Ea Maa + Oea = V Vunb,
Pred: exp,
Pred: exp, =xxp,
SEEb(ex,, exp)
SEED SEED(spp.epp..
expp))
Tuple
Interval
ALO
b
GT: (a. b) (exp,, exp.)
GT: (exp., Pred: [esp,, expy)
Pred: (b. (expa. exps)
SEED (SEED(exp..oom exp +SEED(exp, exp
SEED (SEED(o"ap, ctopp) SEED(="op, d'ou,JV2
Edit Distance
Edit Distance
Total Nodes 13
Scientific Parsing
Rownding
SEED
SEED 100 0
Numeric
Cunversion
Tolerance
Figure 4: SEED calculation process for different answer types, including edit-distance examples and rules for expressions, equations, tuples, intervals, and numeric answers. For detailed explanation of the SEED scoring function, see the Appendix
While EED struggles with noisy LaTeX and varied answer types, SEED standardizes them and provides fine-grained, physics-aware evaluation. We extend the evaluation in three directions. First, answer-type support and unification (as shown in right side of Figure 4 (1) Expressions are directly parsed into abstract syntax trees (ASTs). (2) Equations are standardized by moving all terms to one side. (3) Tuples are evaluated component-wise by positional matching, and the SEED scores are averaged. (4) Intervals incorporate boundary openness through symbolic representations. (5) Numeric answers are evaluated with attention to unit conversion, scientific notation parsing, and rounding within relative tolerance. Second, expanded symbolic coverage: we add native handling of matrices/vectors and inequalities (<, S. >, 2), which we canonicalize as f((-) # O (with # E [<,,,>,>) )) while preserving semantics under operations that flip inequality direction. Third, robust LaTeX preprocessing: we strip wrappers such as \boxed [, remove \left and |right, normalize implicit multiplication (e.g., 2x, ab), unify Unicode symbols (c.g.. the minus sign), standardize function aliases and font commands ( (} {mathhal (mathbb }), discard extrancous
https: :/wp:///www..//// http://wwwwwppp...////p// .wg/
5

---

Published as a conference paper ICLR 2026
natural-language boilerplate (c.g., "Final Answer:") and auto-balance parentheses and fractions. These improvements enable SEED to build ASTs reliably from noisy LLM outputs and, via tree-edit distance, deliver non-binan -inarrr partial credit together with interpretable error localization.
Its type-agnostic AST design and pluggable, physics-aware normalization allow easy extension to new answer types and domain rules, enabling application across CMP and other STEM tasks while maintaining unified, fine-grained evaluation.
3 EXPERIMENTS
3.1 MODELS
2024a);; ol (OpenAl 2024b: o3 (OpenAl 2025b); 03-mini (OpenAI 2025a); 04-mini (OpenAl )025b)), Google (Gemini 2.5 Pro, Gemini 2.0 Flash
We group models by provider families: OpenAl (GPT-40 OpenAA
Thinking (Team et al.J (023)), Anthropic (Claude 3.7 Sonnet; Claude 3.7 Sonnet Thinking (Anthropic 1-rra-....-70-----sss--
)025 xA/ (Grok Beta 2025), Grok 4), Meta/LLLama
Llama-3.3-7008 Instruct Grattafioni et al. 2024)), Alibaba/Qwen (Qwen3-32B (Team, 2025a); QWQ-32B (Team 2025b)), and DeepSeek (DeepSeek- V3 (Deepseek] 2024): DeepSeek-RI and its distilled variants RI-Distill-Laam
70B, RI-Distill Owen-32B (Guo et al., (025)). This family- -bssed taxonomy spans both proprietary and open-source ecosystems as well as general-purpose and Long-CoT reasoning models, enabling controlled comparisons on CMPhysBench.
3.2 EXPERIMENT SETUP
For proprictary LLMs, we utilize API services to query these models. Mcanwhile, for DeepSeek-v3 and DeepSeek-R1, due their requirement on huge GPU memory, we also adopt API services for the query. In contrast, for the remaining open-source general and reasoning L.LMs, we adopt vlim for parallel acceleration.
3.3 MAIN RESULTS
As shown in Figure 5 CMPhysBench is challenging across the board. A small lead cluster, Grok 4, o3, and Gemini 2.5 Pro, achieves scores ranging from 30 on the SEED score scale, with expert-labeled accuracies between 23% and 29% (e.g., Grok achieves 36.0 SEED score and 28.9% accuracy). This cluster clearly separates from the mid pack. Most remaining systems lie in middle band (approximately 23-28 SEED score; 16-20% accuracy), while instruction- 14-15% accuracy), and distilled/smaller variants are the weakest (15-17 SEED score; 10-12% accuracy).
open-source baselines fall lower (20-22 SEED;
However, an interesting phenomenon suggests that reasoning LLMs may not perform better than general LLMs on these challenging -omainn problems in condensed matter physics, because the problems require domain-specifie knowledge and become highly difficult, making it easy for reasoning models to make mistakes during the reasoning process, which then will propagate to the final answer. In this case, the LLMs think, the more likely they could make mistake. We also observe many near-miss solutions (e.g., unit handling, constants, boundary conditions): expert-labeled accuracy is strict and stays low, whereas SEED systematically yields higher values (typically +5-9 points) by explicitly crediting partial correctness. Collectively, these patterns provide a more comprehensive understanding of prevailing limitations of LLMs and underscore the necessity of physics-aware training and evaluation protocols.
hhttps:/ddccs.w
6

---

Published as conference paper ICLR 2026
45
SEED
40
Accuracy
36,14
14.31
50.79
30
28.86
25.63
26.28
25,77
Score
25
23.11
24.14
24.35
19.61
20.49
20.56
19.81
16.72
16.73
17.69
18.08
18.27
15
15.39
13.85
14.23 14.23
15.19
10.58
10
Deppterk &t Disull Diatll-proe-aaa Proe---oo-- 321
OwQ-328
Llama-3 3 1-708- hormer
Cladk 3.7 Somet Thinking
Oward 1281
Liame-3 3- 3-208- Insmeet
Camini 2.0 Flesh Thinking
GPT
Clasde 1.7 Sonner
Clok Beta
DeeySaek-12 1/3
01-mini
o4-miai
DrspSickk
Gemini 2.5
Grok
Figure Model performance on CMPhysBench. For each model, we report the SEED score along with the
expert-labeled accuracy.
4 DISCUSSION
4.1
ERROR ANALYSIS
0x0-338
Coucept Miusee
Unit Dimensional Exrors
System Wrong,
Eau)tte Vecb,
(w - Ei Pizr Vardyy
(s Ngre
Elam--1--1B-tara burrac
Confined healinal anal ieiccaizoa seare lumes the Haniioona
Introboool atime facur that basaks. dimsasiuual comiitenss,
The zurpu ioooppec Inmarre
Clode Seerd Chibing
Matheratical or Logical Eimors
Notational Incensidencyy
Mode
Grak220
Col
Groand Truth
(s Eas V.B mul anibiogtts
Eug Joky Vayla
DeppledW15
in
Tisk: Misundersianding
Redundant Irvolevant
Approsimation
Geeed
"We aced to calculate or di Jof 20...
which Ilyon'' is equatiin
"Grees''s 's functions are importent deseriibing
plasics
ignere
2.30
Stntecced for the woong Croen's
Mosnnerrooo roungt and
Stection
afsehiigg speaife, eountion notiee.
Gass peneral bockgreend inskral
Applicd an Invalid apporciration
benured plasko.
Krrer Count
Figure 6: Analysis of error types across models. Left: Error count breakdown by type for each model on CMPhysBench. Right: Representative examples for each error type, where the background color corresponds to the error category in the left plot. Blue text highlights the specific error location, and the reason provided below each example.
7

---

Published as a conference paper at ICLR 2026
LLMs can make many types of mistakes. To investigate model failure patterns on CMPhysBench, we conduct a detailed error analysis by passing incorrect predictions to GPT-44 and prompting it to infer the underlying reasons. To ensure the reliability of this automated approach, inspired by recent work like x xVerify, we validated it against a set of 300 diverse question-responss pairs manually annotated by domain experts. Our method achieved a 98% agreement rate with human consensus, giving us high confidence its ability to serve as scalable proxy for expert evaluation. This allows us to categorize error types in consistent manner. Notably, Grok 4 is excluded from this analysis as it does not generate intermediate reasoning chains, making it difficult to assess its internal logic. Based on an initial classification by domain experts, errors grouped into eight categories, as detailed in Figure 6
As shown in Figure Table in the appendix, the following two errors account for significant proportion: Concept and Model Misuse and Mathematical or Logical Errors. Concept and Model Misuse are the most dominant error type, and account for over 40-50% of all normalized errors in models such as GPT-4o (66.5%), Claude 3.7 Sonnet Thinking (51.6%), and DeepSeek-V V3 (56.3%). This indicates that many models, even higghperforming ones, struggle with the correct application of fomain-sseciifn physical principles. Another major category is Mathematical or Logical Errors, typically contributing 20-30% of total errors. For instance, o4-mini and o3 exhibit logical mistake rates of 31.0% and 29.4%, respectively, despite having relatively good task-following ability. These issues range from incorrect algebraic manipulation to invalid approximations and reveal persistent gaps in symbolic reasoning.
Task Misunderstanding is more prominent in instruction -utned models like Qwen3-322 (24.2%) and QwQ 32B (27.0%), which often fail to interpret specific constraints. In contrast, more advanced models such as Gemini 2.5 Pro and o3 demonstrate better prompt adherence, with lower task misunderstanding rates (e.g., Gemini 2.5 Pro: 7.5%), suggesting that superior reasoning techniques improve problem comprehension. While other error types like Unit Errors remain rare (<2%), the overall analysis underscores the need for improved scientific alignment and symbolic precision. This diagnostic analysis provides a direct roadmap for mitigating these failures, with specific improvement directions detailed in Appendix G Furthermore, it allows us to form concrete hypotheses about future, comainn-speciff models. We hypothesize that fine-tumin fine-tuning will reduce knowledge-based "Concept and Model Misuse" errors, while core "Mathematical or Logical Errors" may persist. A key goal of CMPhysBench is to provide a platform to test such hypotheses.
4.2 ANALYSIS OF DIFFERENT PROBLEM TOPICS
As shown in Figure a and Table the appendix, performance varies markedly across topics and model families. Grok 4 leads most categories, achieving the highest scores in Magnetism (35.30), Superconductivity (43.42), and Theory (41.21). Mcanwhile, o3 demonstrates strong all-around performance, placing first in Others (46.42) and second in Superconductivity (35.77), Strongly Correlated Systems (37.34), and Semiconductors (27.80). Topic-specific peaks also emerge: DeepSeek-Rt attains the best score Strongly Correlated Systems (62. Gemini 2.5 Pro leads in Semiconductors (29. 18) and is competitive in Theory (40.50), and DeepSeek-V. ranks second in Magnetism (25.75). Notably, even top models display pronounced asymmetries; for example, Grok 4 is strong in Superconductiiity and Theory but weaker in Strongly Correlated Systems, indicating that strengths do not transfer uniformly across CMP subfields.
These patterns highlight the importance of domain-speeift reasoning
generic mathematical skill. Although instruction- tuned open-source basclines generally trail proprictary models, some exhibit localized strengths. For instance, Qwen3-32B performs relatively well in Theory with score of 35.47 but remains weak in Magnetism (8.47), underscoring its uneven competence across topics. This domsin spread suggests the need for subfield-aware training.
8

---

Published as conference paper ICLR 2026
SCS
SC
1.00
6,90
0.51
0.56
0.41
0.36
00
0.38
Semi
Mag
0.51
607
0.56
0.59
0.45
1.00
0.33
0.36
0:41
0.42
0.49
0.33
1.00
0.36
605
Theary
0.36
0.18
0.58
0.36
0.56
1.00
Decpock-R1
Claske Screet
XWwdy
Figure 7: Comparison of model performance and metric correlations. (a) Radar chart of model performance across six domains. Abbreviations: Mag = Magnetism, SC Superconductivityy SCS = Strongly Correlated Systems, Semi Semiconductors, Theory Theoretical Foundations, Others Others. (b) Spearman correlation between human expert ratings and automatic evaluation metrics.
4.3 COMPARISON WITH DIFFERENT METRICS
To systematically assess the reliability and alignment of various evaluation metrics, we compare SEED against four widely used alternatives: Expression Edit Distance (EED) (Qiu et al., 20255),, GPT-4o-based judgment (OpenAl 2024a)), Verify-9B-C (Chen et al., 2025), OlympiadBench-rule based metric (He et al. 2024) and human labels. Human experts have labeled answers as strictly correct (1) or incorrect (0). We then converted the SEED score into a corresponding binary value, where only perfect score (SEED 100) considered correct (1), Spearman correlation coefficients between these metrics and human expert ratings are shown in Figure b).. SEED exhibits the highest correlation with human experts (p = 0.90), demonstrating superior agreement with expert expertjudgment judgment. This performance stems from SEED's design a discrete, structure-aware metric that supports partial credit and accommodates a wide range of symbolic answer types commonly found in CMP, such as equations, intervals, and tuples. Unlike binary accuracy metrics, SEED distinguishes near-miss cases from completely incorrect outputs, providing a more nuanced assessment of symbolic reasoning. Furthermore, SEED is designed for polynomial expression similarity evaluation which is very common in graduate-level CMP.
contrast, EED, though fast and interpretable, struggles with generalization beyond simple expressions. fails to handle complex structures like equations with symbolic manipulations or multi-component answers. GPT-4o and Veriffy while more flexible in language understanding, are less reliable for evaluating highly structured mathematical expressions. Their performance (p 0.56 and 0.51, respectively) suggests limita- tions symbolic alignment, particularly for multi-step derivations and dense expressions common CMP problems. Specifically, these two evaluation methods do not explicitly consider equivalent transformation of expression, making not be the most suitable metric in CMP. OlympiadBench Rule supports multiple answer types, but its mule-based approach is overly simplistic and often fails to account for meaningful structural mathematical equivalence, resulting in the lowest correlation (p- 0.41).
To summarize, these findings indicate that SEED provides fine-grained partial correctness credit and robustness, alongside wide applicability and interpretability, making it a stronger metric for domain-specific scientific reasoning.
9

---

Published as a conference paper ICLR 2026
5 RELATED WORK
Due to the rapid development of LLMs and their potential in scientific research, there is growing trend toward evaluating their performance on scientific problems. For example, benchmarks such as SciQ (Welbl
ct al. 2017). ScienceQA (Saikh et al. 2022). ARC (Clark et al. 2018). OpenBookQA (Mihaylov et al. 2018)
Pub (Jin et al. 2019). SciBench Wang et al. 2024), SciEval (Sun et al. 2023). and E-Eval (Hou
et al. 2024 provide platforms for testing LLMs general scientific questions across multiple disciplines. Normally, lly these benchmarks cover a broad spectrum of topics but often cap difficulty at K-12 or introductory
college levels and favor multiple-choice formats, which increasingly lag behind frontier models and limit exploration of deeper scientific reasoning, especially in physics. In contrast, emerging benchmarks like
UGPhysics (Xu al., 2025a), GPQA (Rein et al. 2024). SuperGPQA (Du et 2025), PHYSICS (Zheng (Qiu et al. 2025b), and PhysReason (Zhang et al. (025b raise the bar by introducing undergraduate- to graduate-level problems, step- or expression- -wwrsi grading, and physics-specific evaluation pipelines, which impose stricter requirements on domain knowledge, reasoning, and moobeem-oolving. However, most of these still emphasize broad coverage rather than depth within a specific research direction; they do not thoroughly examine sustained knowledge acquisition and structured derivations in narrowly defined subfields. summary, while existing work has substantially advanced the evaluation of LLMs' physies problem-solee solving abilities, there remains notable gap for benchmarks that probe rigorous, subfield-sper -ppecffc physics tasks with fine- ane-grained, structure- scoring.
al. 2025). SciCode (Tian et al. 2024).
6 CONCLUSION
In this work, we have introduce CMPhysBench, a novel benchmark tailored to evaluate the proficiency of LLMs in the domain of Condensed Matter Physics. Comprising 520 carefully selected questions based on authoritative texthooks, CMPhysBench encompasses a wide range of representative topics such as magnetism, superconductivity, strongly correlated systems, semiconductorss, etc. To ensure accurate evaluation, propose the Scalable Expression Edit Distance (SEED) score to measure the similarity between various mathematical expressions. Our findings reveal a significant performance gap, with LLMs excelling in general mathematical tasks yet falling short in the specialized context of Condensed Matter Physics, which further underscores the necessity to enhance the effectiveness of LLMs this domain. Further, we believe Nomain-specifte dataset is crucial in promoting the performance of LLM in the future.
10

---

Published as conference paper ICLR 2026
ACKNOWLEDGEMENT
We thank the anonymous reviewers and area chair for their helpful comments. This work was supported by Shanghai Artificial Intelligence Laboratory, Shanghai Municipal Science and Technology Major Project (Grant No. 2025SHZDZX025D066 New Generation Artificial Intelligence- National Science and Technology Major Project (Grant No. 20252D0121802) Intern-Diss Discoveryy and the Postdoctoral Fellowship Program of CPSF (Grant No. GZC20232943).
ETHICS STATEMENT
We study the performance of LLMs in condensed matter physics, and this work does not involve explicit ethic issues.
REPRODICIBILITY STATEMENT
Our SEED score calculation code and CMPhysBench benchmark questions are placed in the supplementary ensure the reproducibility of the article.
REFERENCES
X AI. Grok 3 beta the age of reasoning agents, 2025. httpp:://s.aai/as//gooos
Alexander Altland and Ben D Simons. Condensed matter field theory. Cambridge university press, 2010.
Anthropic. Claude 3.7 sonnet and claude code, 2025.
http:://www.nthhrooiccccoo
claude- - -- sonnet
Daman Arora, Himanshu Singh, and Mausam. Have LLMs advanced enough? a challenging problem solving benchmark for large language models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7527-7543, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.18633/1/2233 www.022.mmmlpmmi.i....m
http://:://.lllaaooooooooorr//// emnlpp weillp-mminn. maiin.c 468
Lei Bai. Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, et al. Intern-sl: A scientific multimodal foundation model. arXiv preprint arXiv:2508. 15763, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901. 2020.
Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504. 10481, 2025.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or Ilms as the judge? study judgement biases. arXiv preprint 24t::2402..1669)) 2024.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disen- tangling computation from reasoning for numerical reasoning tasks. arXiv preprint 1. 2022.
11

---

Published as a conference paper ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18X3v 05057, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168. 2021.
Piers Coleman. Introduction to many-body physics. Cambridge University Press, 2015.
Deepseek. Deepseek-v3 technical report. arXiv preprint arXiv:24/2. /9437, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North Americun chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp.
2019.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprint wrXi::2502.14739 2025.
Ahmed EI-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint wrXiv:2502.06807 2025.
Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models university-level physics problem solving, 2025. URL https
oraxxiv.org/aate hbs////b//aa/ 2503. 21821
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd models. arXiv preprint arXiv:2407.21788, 2024.
Giuseppe Grosso and Giuseppe Pastori Parravicini. Solid state physics. Academic press, 2013,
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey Ilm-as-a-judge. arXiv preprint arXiv:2411.1559. 2024.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.1774,, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepscek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang. Linjie Li, Zhengyuan Yang. Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501 0rX1v:2501.05444, 05444, 2025.
12

---

Published as conference paper ICLR 2026
Chaoqun Hc, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yu- jie Huang, Yuxiang Zhang, et al. Olympiadhench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp.
2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, et al. E-eval: comprchensive chinese k- education evaluation benchmark for large language models. arXiv preprint urXiv:2401.1597, 2024.
John C Inkson. Many-body theory of solids: an introduction. Springer Science & Business Media, 2012.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprini arXiv:1909.00 04966 2019.
Drishya Karki, Michiel Kamphuis, and Angelecia Frey. Easymath: A O-shot math benchmark for slms. arXiv preprint arXiv:2505. 14852, 2025.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
L.D. Landau and E.M. Lifshitz. Statistical Physics. Elsevier Science, 1980. ISBN 9780750633727.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Jingyu Liu, Jiaen Lin, and Yong Liu. How much can rag help the reasoning of Ilm? arXiv preprint arXiv:2410.02338, 2024.
Michael Marder. Condensed matter physics. John Wiley & Sons, 2010.
Albert Messiah. Quantum mechanics. Courier Corporation, 2014.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 1xXiv:8809002780 2018.
OpenAl. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024a.
OpenAl. Openai ol system card. arXiv preprint arXiv:2412. 2024b.
OpenAl. Openai o3- mini: Pushing the frontier of cost-effective reasoning, 2025a. attps://openai
com/ httpinidxx/ppp// matndee//oenaiio3-minnn
OpenAL. Introducing openai o3 and o4-mini, 2025b.
https::/openai..cm//
introducing-o3--nee
Michael E Peskin. An Introduction quantum field theory. CRC press, 2018.
13

---

Published as a conference paper ICLR 2026
Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang. Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Weike Wang. Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming xing Luo, Muhan Zhang, and Hua Xing Zhu. Phybench: Holistic evaluation of physical perception and reasoning in large language models, 2025a. URL 2504. 1 6077
Shi Qiu, Shaoyang Guo, Zhuo-Yarg Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, al. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprini 2rXiv:2504.16074 2025b.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3): 289-301, 2022.
Davison E Soper. Classical field theory. Courier Dover Publications, 2008
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Sci- eval: A Iulti-level large language model evaluation benchmark for scientific research. arXiv preprint arXiv:2308. 13149, 2023.
Gemini Team, Rohan Anil. Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 11805 2023.
Qwen Team. Qwen3: Think deeper, act faster, 2025a.
hwtp://w..gchub..oo/oe.oo qwen3/
Qwen Team. 32b: Embracing the power of reinforcement learning, 2025b. httpss:/cwent
github. io/blelloooooo hoabbl////g-3330 32b/
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Min Zhu, Kilian Adriano Lieret, Yanxin Lu, Genglin Liu, Yufeng Du Tianhua Tao, Ofir Press, Jamie Callan, E. A. Huerta, and Hao Peng. Scicode: A research coding benchmark curated by scientists. arXiv, abs/2407.1316s 2024. URL https://arxiv.e
abs 2207..111611
Weida Wang, Benteng Chen, Di Zhang, Wanhao Liu, Shuchen Pu, Ben Gao, Jin Zeng. Lei Bai, Wanli Ouyang, Xiaoyong Wei, et al. Chem-r: Learning to reason as a chemist. arXiv preprint arXiv:2510. 16880, 2025.
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scißench: Evaluating College-Level Scientific Problem- Solving Abilities Large Language Models. In Proceedings of the Forty-First International Conference on Machine Learning, 2024.
14

---

Published as a conference paper ICLR 2026
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self- self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume long papers), pp.
13484-13508, 2023.
Gregory H Wannier. Statistical physics. Courier Corporation, 1987.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01633 2021.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1007.06200 2017.
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint 200::20020000344 2025a.
Yinggan Xu, Yue Liu, Zhiqiang Gao, Changnan Peng, and Di Luo. Physense: Principle-based physies reasoning benchmarking for large language models. arXiv preprint arXiv:2505.24823, 2025b.
Xiao-Wen Yang. Jie-Jing Shao, Lan-Zhe Guo, Be-Wen Zhang, Zhi Zhou, Lin-Han Jia, Wang-Zho Dai, and Yu-Feng Li. seuro-symbolic artificial intelligence: Towards improving the reasoning abilities of large language models. arXiv preprint arXiv:2508. 13678, 2025.
Zonglin Yang. Xinya Du, Junxian Li, Jie Zheng. Soujanya Poria, and Erik Cambria. Large language models for automated open- open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.0272) 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Xiang Yue, Tianyu Zheng. Yuansheng Ni, Yubo Wang, Kai Zhang, Shenghang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings ACL, 2025.
Boning Zhang, Chengxi Li, and Kai Fan. Mario eval: Evaluate your math Ilm with your math llm-a mathematical dataset evaluation toolkit. arXiv preprint arXiv:2404. 13925, 2024.
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che. Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. llaaabberr:: Pairwise optimization for ol-like olympiad-leve mathematical reasoning. NAACI, 2025a.
Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics- based reasoning. arXiv preprint arXi::2502.1205: 2025b.
Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, et al. Scaling physical reasoning with the physics dataset, arXiv preprint arXiv:2506.00022 2025.
15

---

Published as a conference paper ICLR 2026
A OVERVIEW OF THE APPENDIX
Section contains details about the composition of CMPhysBench, the data curation process, and comparisons with existing benchmarks, highlighting its uniqueness and advantages in the domain of condensed matter physics.
Section C introduces the SEED evaluation metric and compares it against EED, EM, and GPT-44-based scoring, demonstrating SEED's scalability and improved alignment with human judgment in symbolic reasoning tasks. It also contains metries for evaluating complex reasoning related work.
Section Dutlines the experimental settings, including prompt design, tested models, and implementation details used for both answer generation and error analysis.
Section E presents an in-depth analysis of model performance on CMPhysBench, including breakdowns by error type and topics, as well representative case studies of physics problems and model predictions.
Section discloses our use of Large Language Models in the preparation of this manuscript.
Section discusses future research directions suggested by our error analysis. This includes domain -spccffc fine-tuning to address conceptual misuse, neuro- veuro-symbolic methods for mathematical errors, and instruction tuning for system limitations, in addition to leveraging the SEED metric for advanced training paradigms.
Section H provides a detailed explanation of the notations used in Figure covering fundamental operators, creation and annihilation operators, energy parameters, and the Green's function, to aid readers unfamiliar with quantum many-body physics.
B CMPHYSBENCH DETAILS
B.1 COMPOSITION OF CMPHYSBENCH
In this study, we categorize the benchmark question set into six major domains: Magnetism, Superconductiv- ity, Strongly Correlated Systems, Semiconductors, Theoretical Foundations and General Concepts, as shown in Figure 2 Each domain encompasses key theoretical frameworks and representative problems appro- priate for graduate- level physics education, reflecting a progressive trajectory from foundational understanding advanced modeling.
Theoretical Foundations encompass a wide range of topics from quantum field theory (e.g., Klein-Gordon fields, Dirac fields, path integrals, spontaneous symmetry breaking) to statistical physics (e.g., Gibbs distribution, fluctuation theory). Given their central role in supporting advanced topics and their broad applicabilityy, this domain also includes 110 questions, aiming reinforce systematic understanding of modern theoretical physics.
Magnetism and Semiconductors are each represented by 115 questions. These domains focus on phe- nomena such as spin dynamics, magnetic interactions, charge transport, band theory, and device-level tehavior-topics of both fundamental and applied significance in condensed matter physics and materials science. The higher question volume reflects the practical complexity and frequency of these systems in real-worid physical problems, encouraging students to develop robust modeling and analytical skills. Supereonductivity includes topics such as the macroscopic Ginzburg-Laame Landau theory, microscopic BCS theory, and related experimental phenomena. Although conceptually challenging, the theory is relatively self-contained and often revolves around paradigmatic problems. Thus, moderate number of questions (55) is sufficient to assess students' depth of understanding through carefully selected, representative examples.
Strongly Correlated Systems cover advanced topics such as quantum many-body fluctuations, the Hubbard model, and Mott transitions. As one of the most intellectually demanding and
areas in
16

---

Published as conference paper ICLR 2026
theoretical physics, is included as an extension module with 15 high-level questions. These problems are designed to challenge students with strong theoretical backgrounds and facilittte further exploration of frontier topics.
Others cover fundamental problems and computational techniques in quantum mechanics, including harmonic oscillators, perturbation theory, and spin systems. As these topics span multiple subfields and serve as essential tools across the curriculum, relatively large number of questions (110) are assigned this domain to ensure comprehensive training in basic wroblem-solvingg skills and physical intuition.
Generally, the distribution of questions reflects both the structural organization of knowledge in graduate-leva physics and deliberate balance between representativeness, theoretical depth, computational rigor, and pedagogical utility. The design seeks to ensure both breadth and depth, enabling the benchmark to serve as a comprehensive tool for assessing general competence while also identifying advanced reasoning capabilities.
Furthermore, following the settings in Olympiadßench (He al. 2024). we also categorize these questions based the answer types. Specifically, there are five answer types in CMPhysBench, including tuple, equation, numeric, expression, and interval, whose distributions are illustrated in Figure) The categorization of the questions is performed by human experts to ensure its correctness.
B.2 COMPARISON WITH OTHER BENCHMARKS
Table 1: Comparison of our benchmark with existing datasets. For Level: COMP Competition level, CEE University Entrance Exam, KI-K12 Primary and Secondary School. For Question Type: OE = Open-ended, MC = Multiple-choice.
Benchmark
Size
Level
Question Type
Scoring Type
JEEBench Arora et al.
2023)
123
CEE Graduate
OE, MC
Binary
GPQA (Rein et 2024)
227
OE
Binary
SciQ (Welbl et al.
2017)
13,679
K4-K8
OE, MC
Binary
SciEval (Sun et
2023
1,657
OE, MC
Binary
SciBench Wang et al. Z024)
295
University
OE
Binary
ScienceQA (Sarkh et al. (0222)
617
KI-K12
MC
Binary
MMMU Yue et al. 2024)
443
University
OE, MC
Binary
MMMU- Pro Yue et al. (225
3,460
University
MC
Binary
OlympiadBench (He et al. 2024
2,334
COMP
OE
Binary
EMMA (Hao et al. 2025
156
MC
Binary
PHYSICS (Feng et al. 2025
1,297
University
OE
Binary
SciCode (Tian et al. 2024)
338
University
OE
Binary
PhySense (Xu al. 2025b)
380
University-Gradun
OE, MC
Binary
PHYBench (Quu et al.
2025a)
500
K10-COMP
OE Detailed
CMPhysBench
520
Graduate
OE
Detailed
Table[proviees a comparison between CMPhysBench and a range existing scientific and physics-related benchmarks. While earlier benchmarks such as PHYSICS, PHYBench, and SciBench have advanced the development Al systems capable of handling Nomain-specifit problems, CMPhysBench distinguishes itself through its graduate-level difficulty, richer answer representations, and more robust evaluation protocol.
Unlike PHYBench, where open-ended (OE) questions are limited to symbolic expressions and evaluated powerful and extensible metric
using EED (Expression Edit Distance). CMPhysBench introduces named SEED (Scalable Expression Edit Distance). This allows for nuanced grading and flexible equivalence matching beyond symbolic forms.
17

---

Published as a conference paper ICLR 2026
Key distinctions of CMPhysBench include:
Answers are not restricted to expressions or numerics; they also include tuples, intervals, and equation systems, reflecting the diversity physical reasoning and solution strategies required in real-world scientific practice.
Advanced Answer Types:
Graduate- Scope: Questions are curated from advanced textbooks and course materials in theoretical and condensed matter physics, ensuring alignment with the cognitive demands of graduate education and early-stage research, rather than standard undergraduate or competition-level problems.
Semantic Evaluation Flexibility: The SEED metric enables fineggriieed evaluation that supports partial credit, symbolic and numeric equivalence, and structural matching-offen more meaningful feedback on models' reasoning capabilities.
In contrast, many prior benchmarks (e.g., PHYSICS, MMMU, ScienceQA) focus multiple-choice formats or expression-only open-ended questions at the high school or early undergraduate level, and often rely on binary correctness. CMPhysBench, by contrast, aims to bridge the gap between academic problem-sol and scientific reasoning, providing a more rigorous, diverse, and research-oriented benchmark for evaluating LLMsin physics and beyond.
C EVALUATION METRIC
C.I SCALABLE EXPRESSION EDIT DISTANCE
Feature
Original EED
Our SEED Method
Supported Structures
Simple Expressions Basic (symbols/functions) Minimal
Expressions, Equations, Tuples, Intervals Extended (Matrices, Derivatives, Inequalities) Extensive Standardization and Disambiguation Enhanced Parsing Robustness
Parse Tree Nodes
Preprocessing Robustness
Limited
Table 2: Comparison of SEED and original EED.
In this part, we briefly introduce the differences and advantages of our proposed Scalable Expression Edit Distance (SEED) compared with the original Expression Edit Distance (EED). The term "scalable" refers our method's 's capability of extending to more complex and varied answer types, including intervals, tuples, and equations, beyond the simple mathematical expressions handled by EED, Key differences and advantages are listed as follows.
Enhanced Expression Parsing:
SEED supports parsing and scoring of complex LaTeX structures including matrices, derivative expressions (e.g., d)) logical relations (=, <, >), and various special formatting cases, significantly extending EED's capabilities.
2. Extended Node Types in Parse Trees:
Beyond basic numeric, constant, and symbolic nodes, SEED introduces dedicated nodes for matrices, inequalities, derivatives, and logical operators, ensuring richer semantic representations.
3. Advanced Preprocessing and Standardization:
SEED standardizes special fonts (e.g., |mathscr (L]), derivative notations, exponent formats, vector notations, fraction formats, and removes problematic LaTeX commands (e.g., \text(). significantly reducing parsing ambiguities and errors.
Support for Varied Answer Types: Expressions:
Handled similarly to EED, with improved robustness and accuracy.
18

---

Published as a conference paper ICLR 2026
Equations: SEED extracts both sides of equations separately and then combines them into a unified form (typically by subtraction) for scoring. This approach allows direct handling of equation-type answers, addressing EED's inability to process equations effectively.
Tuples: Answers structured as tuples (e.g., (a, b,)) (,) are transformed into key-value pairs, allowing structured and accurate component-wiss evaluation.
Intervals: Interval expressions (e.g.. (a. b)) are transformed into evaluable mathematical forms, includ- ing explicit handling of open and closed boundaries, facilitate robust scoring.
5. Robust Symbol and Format Handling:
Enhanced recognition logic prevents parsing errors from similar LaTeX commands (e.g., distinguishing
\left from \_e), and uniformly standardizes ambiguous formatting and special characters.
Beyond these structural improvements, SEED provides a fine- anne-grined. non-binary score by quantifying the similarity between the predicted and ground-truth expression trees. The score is calculated based on the relative edit distance, r, between the ground-truth tree (Tgr) and the generated tree (Tgen). This scoring function is adapted from the methodology used in PHYBench (Qiu al. 2025b) and defined as follows:
100,
if (exact match),
r Distance gen) Siz)(Tgg)
score
60 100r, 0,
0 << r < 0.6,
r > 0.6.
Here, Distance(-,-) is the tree-edit distance and Size(-) is the number of nodes in the tree. This function assigns a full score of 100 for a perfect match, linearly scales the score down from baseline 60 to award partial credit for answers with minor errors, and assigns score of 0 for expressions that are significantly incorrect (r 2 0.6).
C.2 RELATED WORK: METRICS FOR EVALUATING COMPLEX REASONING
The evaluation of complex reasoning in artificial intelligence, critical aspect of measuring progress in the field, has evolved significantly beyond simple accuracy metrics. As models become more sophisticated, so too must the methods we use to assess their capabilities, moving towards more nuanced and comprehensive techniques. Evaluation methods for complex reasoning broadly fall into four families. (1) Outcome-based scoring. Many benchmarks judge only the final answer via exact match (EM), e.g., GSMSK (Cobbe et al. 2021) and MATH (Hendrycks et al. 2021), sometimes with minor normalization, which is simple but brittle to equivalent forms and formatting noise. To reduce false negatives, several pipelines (Lewkowycz et al. 2022 Hendryeks et al.) 2 augment EM with CAS-based checks using SymPy to test symbolic/numeric equivalence (and lightweight tolerances), as popularized by Minerva and now embedded in common evaluators. Recent math (Karki al. (025) suites further combine exact, numerical, and symbolic equivalence in a single grader. (2) Fine-grained structure-aware similarity. Instead of only the final token string, expression- level metrics compare the structure of predicted and reference solutions. PHYBench's Expression Edit Distance (Qiu et al. (2025b) computes tree-edit distances over SymPy expression trees and converts them to a fine-gras ane-grained score, capturing "almost-correct" derivations that EM misses. (3) Judge- and verifier-based evaluation. LLM-as-a-Judge. (Gu et al.) 2024 Chen et al.) 2024) offers flexible rubric-style grading but is susceptible to systematic biases (e.g.. position/verbositt)) motivating protocols and debiasing improve reliability. In contrast, lightweight answer verifiers target objective tasks by extracting the final answer from long chains and checking equivalence across formats; recent models such as Verify (Chen al., 2025) report strong accuracy across math/short-ansee answer settings. Toolkits like MARIO-Eve (Zhang al. (2024) unify CAS checks with optional LLM judging to improve robustness across datasets. Overall, recent trends move from brittle EM toward type-aware, fine-grained structure- and process- aware evaluation, often blending CAS equivalence, expression-level distances, and calibrated judges/verifiers to better match expert judgments complex reasoning.
19

---

Published as conference paper ICLR 2026
D EXPERIMENTAL DETAILS
D.I PROMPTS FOR RESPONSE GENERATION
This prompt is designed to assess model's ability to perform symbolic, step-by-step reasoning in advanced physics. The model must use only the symbols provided, avoiding any external assumptions, and present the final result in clear LaTeX (boxed } format. This ensures precision, interpretability. and alignment with expert-level problem-solving.
Prompts for Response Generation
You are condensed matter physics expert. Please read the following question and provide step-by-step solution using only the given symbols. Do not introduce any symbols that are provided in the problem statement. Your final answer must be presented as readable LaTeX formula, enclosed [booxedf] environment.
D.2 PROMPTS FOR ERROR ANALYSIS
This prompt instructs GPT-4o, acting a physics expert, systematically evaluate model-generated answers by checking correctness, categorizing errors (e.g., conceptual, mathematical, dimensional) and providing concise reasoning. Responses structured in JSON format, enabling precise and efficient error analysis and scoring.
20

---

Published as conference paper ICLR 2026
Prompts for Error Analysis
You are condensed matter physics expert. task to evaluate model-generated answer to a physics question.
Please perform the following:
1. Determine whether the model's answer correct.
2. If incorrect, identify which of the following error categories (a-h) the falls into (multiple selections allowed):
a) Concept and Model Misuse: Misuse or misapplication of core physical principles, laws, or models (c.g.. using Newtonian mechanics in relativistic regimes).
b) Task Misunderstanding: Misunderstanding of what the question is asking (e.g., solving for the wrong quantity, ignoring critical constraints).
<) Mathematical or Logical Errors: Incorrect mathematical manipulations, derivations, or reasoning steps (e.g., algebraic mistakes, sign errors, invalid inferences).
d) Notational Inconsistency: Incorrect, inconsistent, ambiguous of symbols notation (e.g.. mixing variables, wrong subscripis, undefined terms).
e) Unit or Dimensional Errors: Violations dimensional consistency or incorrect unit conversions (e.g.. adding quantities of different dimensions).
I) Approximation Misuse: Applying approximations or assumptions that are unjustified in the given context (e.g.. small-angle approximation where angle is large).
g) System Limitations: Errors clearly stemming from generation failures, hallucinations, or limitations of the Al system (e.g., nonsensical steps, abrupt output truncation).
h) Redundant or Irrelevant Content: Inclusion of content that is redundant, off-topic, or distracts from the solution (e.g., repeating known facts copying question text unnccessarily). Respond JSON format as follows:
("is_cormi cs_rorte::" "true" "false", "error.types" ["a", "c", ...], "explanation": "Your reasoning 1-2 sentences"
Question: {question}
Ground Truth: {ground truth)
Model Response: responss))
21

---

Published as conference paper ICLR 2026
D.3 MODELS AND SETTINGS
We evaluate diverse set of proprietary and open-source large language models, summarized in Table For OpenAl (GPT-4o, ol, o3, 04-mini) and Authropic (Claude series) models etc, we use their official APIs. Google Gemini and xAI Grok models are also accessed via respective APIs. For open-source models such Qwen, DeepSeek, and LLaMA variants, we employ the vLIM inference engine for efficient batched decoding. In cases where vLLM is not supported (e.g., vision- languae models), fall back the HuggingFace I trans formers library for direct model loading.
Model
Param
Src
URL
QwQ-32R
temperature 0.6
local checkpoint local checkpoint
https:/ /hogg ingane .co/ /Owen OwMO- 325
temperature
htLpst//hug.ngg.et 007 Geegseex 6_7
DeepSeek S1-Distill Cwen 32B
Qwen3-32B
temperature 0.6
local checkpoint
nEtp5:77 huggangaace co/ Qwenm
DeepSeek-RI Rll-Seck-Rl-issilll Liama-70B
temperature 0.6
local checkpoint
BeapSeck-R htt.:/accc..o/eeceeeeeee-ee
DEtps://huggingee .tw.eecco//ooooo
amm- 708
1lama-311770B-nnsee
temperature 0.6
local checkpoint
https:/ Inglace 007
ama/
Llams 3.1 705 Instruct
Llama-3,3 3,3-70B-Instruu
lemperature = 0.6
local checkpoint
hEtps:/ wugginggaccc..... cw/mett- 1amaa
Llama- 708-700110000 Instruct
Clande-3-7-Sonnet
claude-3-7 5-0-l------onne--ateeee
nttpSt7/wate
.omm
Claude 3-7 Sonmet thinking
claude 3-7. sumnet thinking
httes:/ //0. anthoopic....cccccccomcccc com
OpenAl ol 03-mini
https://plan Form. opena
ol
hLLpst//plal Torm
03-mini
REtF5:// Form. openas.......................e.. com
u3
od-minn
o4-mini deepscek-rl
htt /ttp:/ppttttttt Form. opens .com
DeepScek-R1
nLLP5:// www.uugggggggg... coo.eeeeeee---
DeepSeek 21
DeepScek-V3
deepseek-v3
https://hugg httess//huggnnggane Ingface httpp//....///eppeeeaa1///
DeepSeek V3
Gemini-2.0-laan 200-iiah--hinnkin
gemini-2. 2.0-Ilash-thhikiine D-Ilaa--hhinking-eap gemini 2.5 pro preview 03.25
dev!
Gemini 2.5.
wwoglo.cor
Grok-3-Beta
grok-3-heta grok-4
httpst//x.at1
Grok-4
itLps://x.a17
Table 3: The sources of models used in the experiments and the hyperparameters configuration. "." stadds for default parameters.
22

---

Published as conference paper ICLR 2026
E EXPERIMENT RESULTS
E.I ERROR TYPES COUNTS
Table 4: Error types counts, Abbreviations: CM = Concept and Model Misuse, ML = Mathematical or Logical Errors, UD Unit Dimensional Errors, TM Task Misunderstanding. SL System Limitations, NI = Notational Inconsistency, RI = Redundant or Irrelevant Content, AM = Approximation Misuse.
Model
CM
ML
UD
TM
SL
NI
RI
AM
QwQ-32B
202
64
SeepSeck-RI-Disill--Nee
281
91
3
34
43
Llama-3.1 1ama-3.1170BB Iist-nnttrutt
314
102
2
21
2
Qwen3-32B
233
72
0
64
73
Claude 3.7 Sonnet Thinking
131
104
21
hama-333-770BImme
317
97
20
GPT-4o
305
114
4
12
O
Gemini 2.0 Flash Thinking
269
118
2
24
Claude 3.7 Sonnet
257
150
Grok Bela
263
DeepSeek-V3
272
130
3
11
0l
266
114
7
19
10
03-mini
250
117
5
26
04-mini
235
128
DeepSeck-Ro R1
247
124
20
10
Gemini 2.5 Pro o3
215
132
4
30
191
151
5
20
12
23

---

Published as conference paper ICLR 2026
E.2 MODEL PERFORMANCE DIFFERENT DOMAINS
Table Model performance across condensed matter physics domains (normalized scores, two decimal places). Abbreviations: All SEED of all problems, Mag Magnetism, SC Superconductivitt, SCS Strongly Correlated Systems, Semi = Semiconductors, Theory Theoretical Foundations, Others = Others.
Blue
= highest, Purple second highest in each column.
Model
All
Mag
SC
SCS
Semi
Theory
Others
QwQ-32B
15.39
8.93
8.75
26.29
14.97
22.23
17.56
DeepSeck-R1- Diricc-RR---Dsslll-----------
16.72
8.41
12.65
20.12
12.30
24.01
24.31
Mama-3.1-70B-Inse
19.61
8.56
29.63
19.05
27.24
27.92
Qwen3-32B
20.49
8.47
15.65
17.25
16.30
35.47
25.32
Claude 3.7 Somnet Thinking
20.56
10.68
22.38
24.53
13.65
33.44
23.77
1lmma-33-770-Innstn
20.87
10.19
13.08
24.25
17.68
30.10
29.58
GPT-4o
21.74
19.04
18.90
29.03
11.58
28.95
28.42
Gemini 2.0 Flash Thinking
23.11
13.85
13.47
11.15
26.66
29.82
28.85
Claude 3.7 Sonnet
24.14
22.55
19.13
34.93
13.61
30.05
31.93
Grok Beta
24.35
17.74
26.34
26.74
16.26
34.37
28.39
DeepSeek-V3
25.04
25.75
29.67
9.25
15.30
27.73
31.62
25.63
23.75
26.02
28.42
12.72
32.39
33.78
03-mini
26.28
19.51
27.67
19.08
14.40
35.63
36.72
04-mini
26.85
17.50
27.63
22.32
18.33
38.13
34.49
DeepSeek-R1
28.86
20.49
28.88
42.16
22.50
37.10
34.18
Gemini 2.5 Pro
30.79
19.65
25.86
26.67
29.18
40.50
37.44
34.31
23.68
35.77
37.34
27.80
38.99
46.42
Grok
36.04
35.30
43.42
20.49
21.01
41.21
45.79
E.3
CMPHYSBENCH PROBLEM AND RESPONSES DEMO
51: Interacting Fields and Feynman Diagrams, Decay of scalar particle
Problem: Decay of scalar particle. Consider the following Lagrangian, involving two real scalar fields 4 and Φ:
C -
M -
1 (0, m pdrs
2
The last term is interaction that allows & partiele to decay into provided that M 2m. Assuming that this condition met, calculate the lifetime the lowest order μ.. Final Answer:
ST 4
4m2
M
Gemini-2.5-pro Answer:
The lifetime of a particle is the inverse of its total decay rate r. The decay process is & - & 6. We will calculate I' the lowest order in the coupling constant Step I: Determine the matrix element M.
The Lagrangian density is given by:
= (8, 2)² 1 M2 2 (0,
m p000
The interaction responsible for the decay is Lint - This trilinear interaction. The Feynman rule for the vertex connecting one particle two & particles is derived from this term. Since
24

---

Published as a conference paper ICLR 2026
two o fields are identical, there is symmetry factor of 2! associated with permutations of these fields if they were externally labelled, or equivalently, one considers the derivative the interaction term with respect to the fields. For interaction term of the form Rabe Qa 2 te where N is the number NI
of identical fields of a certain type, the vertex factor is -ígube. Our interaction -u This can be written as 2/: do² Thus, the vertex factor is -i(pp). The matrix element M for the decay & + Φ + therefore 2μ (by convention, the -i often factored out). So, |M (2p)² 44p..
Write down the formula for the decay rate r. For a particle of mass M decaying from rest frame into final state particles with momenta pt and P2. the decay rate is given by:
Step 2:
d'p2 p2 (2
r
PL
(2T)
(P- P1 P1P)))
2M
(2T) (2T))2F1
Here, P = (M, 0) is the four-momentum of the initial 4 particle.
=EE, pi) and (E2, p2) are
the four-momenta of the two outgoing & particles. ES = / Mr 2 + m² and F2 1 + m Since two final state & particles are identical, statistical factor i Substituting [M|² 4y² and 1/2:
d'pt
d' p2 (2T)32E2
T= 2M (4n2))
(2 -p1 p1-- -p2)
(2T)
r n²
M ʃ
d'p
d'p2
(0M - -E - E2) - p2)
(2T)
(2m)²2Ez
Step 3: Evaluate the phase space integral. Let 7 be the phase space integral:
d'p1 (2
d'p2 (2T)* - Ei - E2) (5r + +12)
-=
(2m)822
Integrate over Pae using the 513) (pi p2), which sets pa = -pi. This implies lpal = lpil. I.t py
lpil-
Then Ei Es E,
P3 + m
d' (2T) 8(M -2E,) = (2m)²
d' 4E7
1 (2E)
8(M 2EE)
Convert d'p1 to spherical coordinates: d'p1 p3dp The angular integration gives 4m the integrand is spherically symmetric.
4m
pp 8(M 2B,) 4E;
( pÿPPs 4m
1
1
8(M -2E,)
(2
E3
The delta function 6(M 2F,z) constrains F, M/2. Since E, 12 m², have M/2
V pJ m², which gives M p m So, p = 22/4-m² For p. to be real, M > m²
i... M > 2m, which is given. The magnitude of the final momentum is /M2/4 /4 m
VMM= 4m² To evaluate the integral over dp we use the property J (()))([(==))d = E, g(mz)
where f(x)) 0. Here Py. g(pr) P3. and ((1) M 2E, M 2 p2 + m²
25

---

Published as conference paper ICLR 2026
(((r)) -2 2 2 N. Bj"
At Es
M/22 ps
po 1 4m (f)
2pa M72
4pu
dpp Po/
the integral is (po))
Pg
M
4pg
M
Substituting this into the expression for 1:
(M/22)3
M
M.
Po
VM 4m
VM2 4m
4T
M
4T
2
8m
This is a standard result for the two- noobody phase space factor.
Step 4: Combine results get r.
Substitute the value back into the expression for
12
M2
4m
11 V
M
4m
8T
8 M
Step 5: Calculate the lifetime
The lifetime is the inverse of the decay rate I:
M
/2 4m
The condition M > 21m ensures that the term under the square root is positive, so the decay is kinematically allowed and the lifetime real and positive. The calculation is to the lowest order in μ., which means considering only the tree-level diagram, resulting
x μ²²
Final check of units: In natural units (h c 1), mass M, m and coupling μ have units of energy
[E]. The Lagrangian density L has units [E] The kinetic terms (0,.b)² imply & has units [B]. The interaction term po has units [u] [ [e]. So [u] must be [E]. The decay rate I has units of [Æ]]
[
(inverse time). Our result for T:
mm]
[ VEE
[E"
[F]]. This is correct. The
lifetime has units of [ (time). Our result for
pap
[
This is correct.
[n] [mm]
[EP[AT]
The final expression lifetime r
8 ST
1² R2 1m
SEED score:
100
o3 Answer:
Tree-level (lowest order in the coupling
mu)
Vertex rule
From the interaction term -p tb0 the (constant) Feynman amplitude for - 0ó is -ips so that
|MP 2
2. Two-body phase-space factor For a particle mass M decaying at rest into two identical particles of mass m, the phase space gives
/
1
4m
JPS2
8T
2
26

---

Published as conference paper ICLR 2026
Because the two final particles are identical, an extra factor 1/2. is required.
3. Decay probability per unit time
(probability per unit time
f
dPSs-
122
V
4m
2M
32t.M
M
4. Lifetime The lifetime is the inverse of the above quantity, hence
32x
lifetime
4m
V
M
which is valid provided M >> 2m.
SEED score:
77.5
GPT-4o Comments:
The model answer contains mathematical error in calculating the decay rate: shows factor of 32 instead of the correct factor of present the reference answer. The error likely arises from incorrect consideration of symmetry or phase space factors.
Expert Comments:
The analysis is correct. The model wrongly calculate the symmetry factor, and this dates back to the problem. The error type can be logical error concept misunderstanding. In another word, the model know partially about the physical concept.
Error Type: "c": Mathematical Logical Errors
27

---

Published as conference paper ICLR 2026
DeepSeek-RI Answer:
The lifetime of the & particle decaying into two & particles is calculated using the decay rate I, which is determined by the interaction term in the Lagrangian. The decay rate to lowest order in is given by:
VM2 4m
32x
The lifetime is the inverse of the decay rate:
32
12 4m
Final Answer:
32TM12
122 / 4m
SEED score:
77.5
Comments:
The model's answer is mathematically incorrect. When the ground truth expression is simplified,
8T
becomes /22 4m
which differs from the model's answer by a factor the numerical coefficient (32 in the model vs. 8 in the ground truth).
Expert Comments:
The analysis is correct. The model wrongly calculate the symmetry factor, and this dates back to the problem. The error type be logical error. In another word, the model know partially about the physical concept because it calculate the right target but give a wrong answer.
Error Type:
Mathematical Logical Errors
DeepSeek- V3 Answer:
To calculate the lifetime of the & particle decaying into two particles, we'll follow these steps:
Identify the Interaction Term:
The interaction term in the Lagrangian is -etoo which allows the decay d -> dń.
2. Compute the Decay Amplitude: The Feynman rule for the vertex Qá -ip (since the interaction term -udeos the vertex factor -ij).
3. Phase Space Calculation:
The decay rate I' for & dó in the frame of & is given by:
1
d'L
P d'ps (2 -
==
2M
(2m))2E: (2T)32B2
where | is the squared matrix element for the decay.
Matrix Element Squared:
For the decay + + ΦΦ. the matrix element M -ip. Thus, |M = 1
28

---

Published as conference paper ICLR 2026
Phase Space Integral: The phase space integral for two-body decay with identical particles (symmetry factor 1/2 gives:
(
I
p'idpp 5(M Ei F2)6 (p: + p2)
2M
(2T)
Integrating p2 and using p1 p2. have:
T= 1²²
/
d'p E7
8(M - 2Ei).
16TM
Using F1 = m² pil and
P1 = 4T|p1 ?dlpil the delta function enforces:
2E = M
|pil
VM2
M2
4m
2
Evaluating the integral:
1²
P1
4m
16T M
4F
M/2
Since Ei M/2 and |pı| = 1VM² 4m we get:
1ra
7
4m
r
16
M
M
Final Decay Rate:
Simplifying, the decay rate is:
n
4m
16
M
Lifetime Calculation:
The lifetime of the particle is the inverse of the decay rate:
16
The final result for the lifetime of the & partiele is:
16nM
1m
SEED score:
55
29

---

Published as conference paper ICLR 2026
Comments:
The model's answer for the lifetime T twice the correct value. The correct expression r Sm
) The model's 's answer has numerator of 16 instead of 8TM. This discrepancy by factor of could arise from conceptual misunderstanding (e.g., incorrect matrix element |M| symmeery factor) or a mathematticl error in phase space integration.
Am² )-1/2,
Expert Comments:
The analysis is partially correct, and model knows partially about the concept wrongly calculate the integral.
Error Type:
Concept and Model Misuse, Mathematical or Logical Errors
Grok-4-0709 Answer:
8 M
1ra 12VM2 4m
SEED score:
100
228: Movement of electrons in crystal in electric and magnetic fields
Problem: atoms form a two-dimensional square lattice, with each atom contributing one electron to form a two-dimensional free electron gas. The electron energy expression is
k/
E(k)
Derive the formula for the density of states of two-dimensional free gas. Final Answer:
(
m
DeepSeek-R1 Answer:
To derive the density states (DOS) formula for a two-dimensional fiee electron gas, we first consider an annulus k-space corresponding to iso-energy contours. The energy expression is:
12 ( + k2))
E
2m
2m
where k= /k3 + k.. In k-space, the iso-energy contour a circle of radius k. The area of the annulus corresponding to energies from E to E + dE (i.c., between radii and l + dk) is 2 dk. The number of allowed k-states per unit area of k-space is (L/(2r))? where L is the linear dimension of the system (and A I!2 is the real-space area). The number of states dN within the k-space annulus 2 dk, per unit real-space area, is:
1: dk
2 dk
12
2x
2T
30

---

Published as conference paper ICLR 2026
From E = (2m), differentiation yiclds dk gag dE. Substituting this into the expression for the number states per unit real-space area, we ohtain the density of states g(F)
dN
7.dE
1
dk
o(B)
dE
2m
k
dk
d.
VIL
2x
The final result the density states is:
g(E)
2x
SEED score:
49
Comments:
The model's answer for the 2D density states, g m
incorrect because is missing the spin
hn degeneracy factor of 2. rather. it has an extra factor of 1/2. The correct density of states for a 2D free
electron gas, including spin, is g xh²
Expert Comments:
The analysis correct. The model do not understand concept spin and leads to ignorance of spin degeneracy factor
Error Type:
Concept and Model Misuse
DeepSeek-Ve v3 Answer:
To derive the density of states g( (FF for two-dimensional free electron gas, follow the steps below:
Energy dispersion relation:
The energy of an electron is given by
1.2 1:2 + R2))
A²K"
B(K)
2m
2m
where à: = 12
2. Number of states in wavevector space:
two-dimensional le-sace, area per state (2m/L)2, where the length the lattice. For a system composed of atoms, the total k-space area is determined by a circle with radius from 0 to kr (kr is the Fermi wavevector). The number of states is
L
2
Number of states
2T
because the area k-space is m
31

---

Published as conference paper ICLR 2026
Relation between total number of electrons and energy:
Each state can accommodate electrons (spin up and down), so the total number of electrons N satisfies:
L
N = 2
kkp
2T
From this, we find Fermi wavevector ko as
N
kp L
or kp 22n, where n. N/L2² is the arcal electron density.
4. Definition of density of states:
The density of states (15) the number of states per unit energy interval (including spin degeneracy). i.e.,
dN
g dE
From steps 2 and we obtain the relation between number of and energy
L
N(E) = 2-
Tk²
2
Substituting x²
67 get
1m
mT,2
(EE
n/?
5. Differentiate obtain the density states: Differentiating (E) gives (E):
dN
mT. x/?
(E))
dE
This the total density of states of the system. The density of states per unit area is:
2(b) m
g
L
Th
6. Final result:
The density of states for a two-dimensional free electron gas is constant, independent of energy E:
g(E)=
171
xh²
SEED score:
100
Grok-4-0709 Answer:
The density of states per unit area without magnetic field is g (E).
In the presence of magnetic field B, the energy levels condense into Landau levels with energy spacing AE
32

---

Published as conference paper ICLR 2026
The number of states per unit area in an energy interval A without magnetic field g (E) These states condense into one Landau level, so the degeneracy per unit area is D = gl AE
D - ((E)
SEED score:
Comments: The model's answer incorrectly applies the concept of degeneracy, as it relies the density of states and energy interval, without addressing how the Landau level degeneracy determined by the magnetic field. correct expression for Landau level degeneracy depends magnetic flux through lattice area, yielding the given formula: = eB =
Expert Comments:
The analysis is correct. The model do understand both the problem and the concept, especially the Landau level because this problem do not contain magnetic field, leading to wrong answer.
Error Type:
Concept and Model Misuse
116: Electrostatics of Dielectrics, Thermodynamic relations for dielectrics an electric field
Problem: Determine the height by which the liquid surface inside a vertical parallel-plat capacitor rises. Final Answer:
h
E²
8mP
gemini-2.0-flaan Semini--.0-flas--tiiee tiiiiiii Answer:
Here's step-by-step solution to determine the height h by which the liquid rises inside the capacitor, using only the provided symbols:
Energy considerations:
The free energy change, F. must account for the gravitational potential energy increase due to the liquid rising and the reduction in electrostatic energy due the introduction of the dielectrie.
2. Gravitational Potential Energy:
The increase in gravitational potential energy unit area of the capacitor plates is pgh((1/2) pigh?
Electrostatic Energy:
The electric field is E without the liquid. When the liquid of dielectric constant e present, it displaces air (dielectric constant= 1). The energy density reduction is proportional to (s 1)E². Integrating this energy density reduction over the volume the liquid occupics, get the reduction in electrostatic energy per unit area of the capacitor plates { E) h. However, where E the electric displacement. Furthermore, because the field held constant here using voltage source, we have 2 (s 1)e,
express this { Eh
4. Free Energy Minimization:
The system will adjust h to minimize the free energy This means the change in free energy due to small change must be zero, ie., 0h, 0. Therefore, we are balancing the gain in gravitational potential energy with the loss of clectrostatic energy.
of
33

---

Published as conference paper ICLR 2026
Balancing Energies:
We set the change in gravitational potential energy equal to the "negative" R of the change in electrostatic energy (per unit area) when the liquid rises to height /::
pggh {s 1) Jeo E h However, 6o is not in our permitted variables so have to remove it. Since V/d, where is the separation between plates and V is the voltage. The capacitance C e //d. So represents relative permittivity which is
E. is constant, and the parallel plate capacitor
expressed
50 = :.- Then we can get pggh2 2 1) h, which becomes 1/ pihh
) - 1) E
6. Solving for h: We solve for h by dividing both sides by h/2, have:
pgh (5 - 1) ' E²
(e E²
h
Pgs
7. Final Expression:
h (e E)E²
PgE
SEED
36.5
Comments:
The model's derivation is fundamentally incorrect it starts with dimensionally inconsistent and physi- cally flawed expressions for both gravitational potential energy and electrostatic energy. also demon- strates severe notational confusion, using the symbol "e" for both relative absolute permittivity, which culminates in nonsensical final equation.
Expert Comments:
The analysis is correct, and the units required in the problem misunderstood by the model.
Error Type:
Concept and Model Misuse
Gemini-2.5-pp 2.5-pro Answer:
is step-by-step derivation to determine the height
The liquid rises a height h where the total free energy of the system, F, is at minimum. This equilibrium condition is found by setting the derivative of the total free energy with respect to the height zero. The total free energy is sum of the electrical free energy and the gravitational potential energy. Step 1: Gravitational Potential Energy
As a volume of liquid rises into the capacitor, its gravitational potential energy increases. Let's consider the cross-sectional
of the capacitor gap be A.a. The volume of the raised liquid V Acah. The mass of this liquid m pV pAc.h. The center mass of this raised liquid column height h/2. The increase in gravitational potential energy, Ug, is therefore: U, mg h)g Acsh
Step 2:
Electrical Free Energy
The problem specifies that F is the relevant free energy, which is minimized when the potential across the plates is held constant. When the liquid with dielectric constant rises, it displaces the air (with Eair 1). This changes the electrical energy stored the capacitor.
34

---

Published as conference paper ICLR 2026
The problem's use of the symbol æ and the absence of the permittivity of free space eo indicates that Gaussian units are appropriate. In these units, the energy density an electric field in a linear diclectric is
8T F=²
E². The energy density in the liquid-filles region The energy density the air-filled region uaor 8x
Uisquid s- E²
When the liquid rises by height h, volume = Aash previously filled with air is now filled with the liquid. The change in the stored electrical energy, eUele in this volume is: AUete (Uisquid Mair) V
( E²
8m E Acch = -=1 Acah
For system constant potential, the change in the relevant free energy, A Feelec is the negative of the change in the stored energy: AFdle = -Udee s-1
2 Acah 8
Step 3: Total Free Energy and Minimization
The total change the free energy of the system as function of h the sum of the gravitational and electrical contributions: (h) = Uy + Afatt Ag To find the equilibrium height, minimize this free energy by taking its derivative with respect to h and setting it zero:
6-1 E² Acck 8r
d(AF)
(EpgAcsh? e= E² Acsh) pg e=1 E² Acs 0
dh
Step 4: Solve for The cross-sectional area Acs is a common factor and cancels out, pgh (s-1)
8
Solving for the height h: h (s-1)ø²
8mp9
This final expression for uses only the symbols provided in the problem statement.
1)
h
8m
SEED
100
DeepSeek-Ve V3 Answer:
To determine the height by which the liquid surface rises inside vertical parallel-plas capacitor, we analyze the balance between the gravitational potential energy and the electrostatic energy.
1. Gravitational Potential Energy: The work done against gravity raise the liquid to height h is given by the gravitational potential energy:
Uya
2 pghi
(The factor arises because the center mass the raised liquid column is height $.)
2. Electrostatic Energy:
When the liquid rises, it displaces air between the capacitor plates. The change in electrostatic energy due to presence of the dielectric liquid
1 50 - 1)F²
Ude:
Here, so is the permittivity of free space, and the relative permittivity (dielectric constant) of the liquid. The negative sign indicates reduction energy when the dielectric liquid is introduced.
3. Equilibrium Condition: The system reaches equilibrium when the total free energy Ugan + Ustes is minimized, Taking the derivative with respect to h and setting it zero:
dF dh pgh 2 colE 1)E²
35

---

Published as a conference paper ICLR 2026
Solving for h:
EolE 1}
2pg
Thus, the height by which the liquid surface rises is:
Eu(E 1)E²
2pg
SEED score:
36.5
Comments:
The model fundamentally misidentifies the energy involved. It calls specific energy "clectrostatic energy." when term actually represents total energy change of the entire system, including the work done by the battery. This is critical distinction in physics. After making that initial conceptual mistake, the model performs all the subsequent math the calculus and algebra-perreeee the recipe's instructions (the math) correctly. but used the wrong ingredient (the physics concept).
It followed
Expert Comments:
The model misunderstand the physical concept and the units required by the problem.
Error Type:
Concept and Model Misuse, Mathematical or Logical Errors
DecpSeek-RI R1 Answer:
To determine the height & by which the liquid surface rises inside the vertical parallel-plate capacitor determined by minimizing the free energy F of the system, which minimized when the potential between the plates is constant. The relevant symbols provided: h. F.
and E.
The free energy includes the Gibbs free energy of the capacitor and the gravitational potential energy of the liquid. Under constant voltage, the Gibbs free energy for the capacitor is - - CV2 where C is the capacitance and V is the voltage. The gravitational potential energy is pgwdh? where w the width of the plates and d is the plate separation. However, and d cancel out in the minimization process.
The capacitance C' depends the height of the liquid. The electric field E is constant and uniform between the plates due to the constant voltage and the orientation the diclectric interface. Specifically, E = V/d, and this field magnitude remains unchanged regardless of the liquid height because the voltage is constant.
In Gaussian units, the expression for is derived by minimizing F with respect to h, leading
2 1)
8T
This result consistent with force balance approach, where electrostatic force per unit area at liquid-air interface balances the hydrostatic pressure due to the raised liquid column.
The symbols used are: E (clectric of the liquid, relative permiutivit)), P (density of the liquid), g (acceleration due gravity), and (mathematical constant). The factor arises from use of Gaussian units the formulation.
field magnitude in air when liquid is raised), (dielectric constant
Thus, the height & is given by the above expression.
36

---

Published as conference paper ICLR 2026
Ea
STp
SEED score:
100
Grok-4-0709 Answer:
(s 1)E²
8m
SEED score:
100
F
USE OF LLMS
During the preparation this work, the author(s) used LLMs to improve the language and readability. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.
37

---

Published as a conference paper ICLR 2026
G FUTURE DIRECTION
The error categories we have defined in Sec. and shown in Fig. provide clear guide for future research directions.
Addressing "Concept and Model Misuse" through domain-specific fine-tuning, Our finding that "Concept and Model Misuse" is the most frequent error type strongly motivates the use of our benchmark's 's source materials for domain-specific fine-tuning because this may largely come from insufficient understanding of background knowledge like physical concept and assumption. Creating a high-quality dataset from these lradel textbooks can impart the requisite foundational knowledge directly. Alternatively, these materials are perfectly suited for Retrieval-Augmented Generation (RAG), enabling the model to ground its reasoning in authoritative domain knowledge at inference time (Liu al. 2024), thereby directly addressing the primary failure mode we observed.
Mitigating "Mathematical Logical Errors" with neuro-symbolic methods. The high rate of these errorshighlights well-documentee limitation in the symbolic reasoning capabilities of current LLMs and this type of error may originate from the bottleneck of inference ability. This motivates using LLM + Symbolic approaches Yang et al. (025). like Program-AVe -iieram--iiddd Language Models (PAL) (Gao et al. (023) and Program of Thoughts (PoT) (Chen al. 2022). where the LLM translates the problem into a formal language (like
Python), and deterministic symbolic engine handles the exact mathematical execution.
Correcting "System Limitations" with instruction finetuning. Failures in following output constraints can be directly addressed using our benchmark. The highly structured and consistent format of the ground-truth solutions in CMPhysBench makes it perfect resource for instruction finetuning to better align models with specific task requirements (Wei et al. 202 Wang et al. 2023
Furthermore, unlike binary accuracy, SEED provides fine-grained, non-binary partial credit. This makes ideal dense reward signal for training paradigms like Reinforcement Learning with Verifiable Rewards (RLVR), allowing models to learn incrementally even from imperfect solutions (Gunjal et al.) (0255) and transforming SEED from static evaluation tool into dynamic component for future model training.
H DETAILED INTERPRETATION OF SYMBOLS IN FIGURE
Given that Condensed Matter Physics (CMP) involves specialized terminologies that may lie beyond the general research scope, and acknowledging that the symbolic representations (such as the particle creation/an- nihilation operators and the specific variable shorthands used in the tree diagrams) in Figure I might be confusing to non-experts, we provide comprehensive background reference here. The following table details the fundamental operators, physical parameters, and the specific shorthand notations defined in the Anderson s-d exchange model problem. This supplement aims to bridge the gap for readers from different backgrounds and facilitate clearer understanding of both the physics and the computational graph representation.
38

---

Published as conference paper ICLR 2026
Table 6: Nomenclature and physical interpretations of notations used in Figure 1
Symbol
Physical Interpretation
1. Fundamental Operators & Indices
The Hamiltonian representing the total energy of the quantum many-body system.
ΣkO
Summation over all possible momenta (k) and spins (a)).
0/ō
Electron spin index (f or J). ã denotes the opposite spin of a.
2. Particle Operators (Second Quantization)
/
Creation/AAmiiilation operators for conduction electrons (the mobile electron sea).
dz/d.
Creation/Ammiiilattion operators for the impurity electron (localized state).
ndo
Number operator do) counting the occupation of impurities.
3. Energy & Interaction Parameters
F Fido
Energy levels for conduction electrons and impurity electrons, respectively.
U
Coulomb Repulsion:
The energy penalty for two electrons occupying the same
impurity site.
Vkd
The interaction strength allowing electrons to hop between the conduction band and the impurity.
Hybridization:
go/g:
Landé g-factors (dimensionless magnetic moment) for electrons and impurities.
h/
External magnetic field and Bohr Magneton.
4. Problem-Specifie Shorthands (Variables in Tree Diagrams)
The frequency (energy) variable in the complex plane, appearing leaf node the expression tree.
(()
The Green's 's Function notation. It represents the correlation between state B to state A.
Oka
Shorthand for the mixed Green's function ((Ckr ))m)) This variable appears explicitly in the final answer and the tree structure.
by
Shorthand for the impurity Green's function ((do [dt)). Used to simplify the equation of motion.
39


Reviews:
[{"review_id": "fW33GKOJ9s", "reviewer": "Reviewer_aQN2", "summary": "This manuscript introduces CMPhysBench, a novel benchmark that successfully establishes a rigorous standard for evaluating LLMs on advanced scientific reasoning tasks in physics. The adoption of a fine-grained evaluation protocol is a major contribution.", "questions": "1. please add additional analyses or discussions towards future research directions or the contribution of this paper towards the future improvements of LLM in this domain.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. High Problem Difficulty\n\nFocuses exclusively on graduate-level material, comprising more than 520 meticulously curated questions that require LLMs to generate complete, step-by-step solutions for complex calculation problems. This moves beyond the limitations of high school or undergraduate benchmarks, demanding advanced mathematical rigor and conceptual understanding.\n\n2. Expert-Aligned Metric\n\nThe proposed Scalable Expression Edit Distance (SEED) score provides highly accurate, fine-grained, non-binary partial credit for mathematical responses. SEED exhibits the highest correlation (ρ = 0.90) with human expert ratings, demonstrating superior alignment in evaluating complex symbolic reasoning compared to prior metrics.\n\n3. Extensive Model Analysis\n\nThe paper conducts a comprehensive empirical study evaluating 18 proprietary and open-source LLMs. This extensive analysis identifies a significant capability gap, with the best models achieving only a 36 average SEED score and 28% accuracy, providing quantitative illumination of specific failure modes across the LLM ecosystem.", "weaknesses": "1. It would be better if the authors could discuss how the issues of LLM in this domain identified in the analysis could be mitigated in the future research. Currently, the analyses only show LLM can make multiple types of error and it is still unclear how to improve LLM to avoid such errors. Proposing potential solutions for the identified errors could further improve the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cj56oo5WDr", "reviewer": "Reviewer_nemX", "summary": "This paper presents CMPhysBench, a new benchmark to test LLMs on graduate-level Condensed Matter Physics. It's made of 520+ hard calculation problems from textbooks. They also created a new scoring metric called SEED that gives partial credit for complex math answers (like equations or tuples) using tree-based analysis. Their tests on 18 LLMs show that even the best models, like Grok 4, perform poorly (36 SEED score, 28.9% accuracy), showing a big gap in this specific domain.", "questions": "For the error analysis, how do you know GPT-4o's categorizations are correct? Did you have any humans double-check its work? Does your evaluation look at the reasoning steps, or just the final answer in the box? It seems possible for a model to get the right answer by luck or by making mistakes that cancel out.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper's main strength is tackling a new, hard domain: graduate-level condensed matter physics. Most benchmarks are easier, so this is a needed step up. The SEED metric is also a big plus; it's a smart way to give partial credit on complex math answers instead of just right/wrong. This metric seems useful for other science benchmarks too. The testing of 18 models is thorough, and the error analysis in Figure 6 gives a good breakdown of why models fail, with \"Concept and Model Misuse\" being the biggest problem.", "weaknesses": "The main weakness I see is in the error analysis. The authors used GPT-4o to categorize all the model mistakes. While this is fast, it's not clear how accurate GPT-4o is at this task. It would be better if they had human experts check a sample of these to confirm the error breakdown. Also, the SEED score focuses on the final boxed answer. The prompt asks for step-by-step solutions, but it's not clear if the steps themselves are evaluated. A model could get the right answer with the wrong steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WaypmEnGKZ", "reviewer": "Reviewer_K4Jm", "summary": "Graduate level condensed matter physics benchmark manually curated in high detail. The authors main contributions are (1) mix of LLM and expert question generation (2) presentation of a new metric SEED which helps to reward both partial correctness as well as (intended) to help with symbolic equivalence matching of answers, correlating well with human preference, (3) evaluation of several SoTA models on CMPhysBench showing that models currently struggle with condensed matter physics.", "questions": "Figure 6 very difficult to read even while zoomed in (Model text and also the colors)\n\nFigure 4: I’m unclear on how SEED is actually computed here, where does the 60 in Model Response 1 come from?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1, Scope and subject: Graduate level benchmark based on standard graduate textbooks, requiring complex step-by-step solutions across diverse answer types. Specifically, CMYPhysBench is a non MC/QA benchmark, so much more difficult.\n\n2. Diversity and coverage: Clear balance between categories, and clear explanation and validation of the source material from which benchmark is derived from. The authors also perform strong analysis on failure modes, which are possibly actionable and of interest to members of the AI for science community trying to understand LM capabilities on scientific domains (and when to (or not) use models for specific scientific tasks).\n\n3. SOTA LLMs moderately struggle on this benchmark making it of key interest to the community and interesting for marking LLM progress over time.", "weaknesses": "1. Relevance of SEED as an benchmark evaluation metric versus actual accuracy.\n\nThe goal seems to reward partial correctness (which is understandable from an RL or intermediate reward feedback perspective), however in practice: does SEED actually properly weight when LMs make minor incorrect reasoning steps (or does it only purely give partial credit when LMs fail to decode a final correct answer)? Some more discussion on this would be helpful.\n\nRelated, from model thinking trajectories, how well does SEED correlate with minor incorrect steps in reasoning: I’m worried that SEED partial correctness of answer may not correlate with minor reasoning incorrectness, which may defeat some of the benefit of this partial correctness.\n\nRelated, is it true that partial misses in edit distance should be only partially penalized just at an AST level?\n\n2. Human preference seems to be binary (0 or 1): are there studies on how accurate human labelling is here? Were questions + ground truth answers verified by multiple raters? As such would the human grading here be close to an “accuracy” or a “preference” type of statistic?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2sC539xB5D", "reviewer": "Reviewer_TCqD", "summary": "The paper introduces CMPhysBench, a benchmark designed to evaluate LLMs for Condensed Matter Physics (CMP) problems. The benchmark consists of 520 graduate-level calculation problems, spanning subfields such as magnetism, superconductivity, semiconductors, and strongly correlated systems. Compared to prior benchmarks, CMPhysBench emphasizes symbolic reasoning. A key contribution is the SEED metric, which extends existing expression-based similarity measures to support multiple answer types through syntax tree representations. The authors evaluate 18 proprietary and open-source LLMs, finding that even top models like Grok-4 achieve only 28% accuracy, suggesting a significant gap in domain-specific reasoning for CMP.", "questions": "- Could the SEED metric be adapted to multimodal problems involving diagrams or figures in physics textbooks?\n\n- Would you expect domain-specific fine-tuned LLMs (e.g., physics-trained models) to exhibit qualitatively different failure modes than general LLMs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The dataset is a good contribution, as it is manually curated by domain experts and spans diverse CMP subfields.\n\n- SEED provides a novel measure of partial correctness, addressing a gap in symbolic assessment.\n\n- The evaluation across 18 models is comprehensive, offering generalizability and comparative insight.\n\n- Evaluation with human alignment validation and the error analysis with detailed categorization of reasoning failures helps to verify the findings.", "weaknesses": "- Although the error analysis is informative, the reasoning behind LLM failures remains speculative rather than causal.\n\n- Maybe it's because CMP is a very specific area beyond my research scope. I find the dot examples in the figures very confusing. If a bit more background can be provided to explain the meaning of those symbols, it will help non-experts to better understand the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)