CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers
Authors: Haining Pan
James V Roggeveen
Erez Berg
Juan Felipe Carrasquilla Alvarez
Debanjan Chowdhury
Surya Ganguli
Federico Ghimenti
Juraj Hasik
Henry S. Hunt
Hong-Chen Jiang
Mason Kamb
Ying-Jer Kao
Ehsan Khatami
Michael J Lawler
Di Luo
Titus Neupert
Xiaoliang Qi
Michael Brenner
Eun-Ah Kim
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
statistical mechanics
benchmark
evaluation
numerical methods
scientific problem solving
condensed matter physics
quantum physics
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have demonstrated remarkable progress in coding and mathematical problem-solving; however, evaluation on advanced research-level problems in the hard sciences remains scarce. 
To fill this gap, we present \cmt, a dataset of 50 original problems covering condensed matter theory (CMT) at the level of an expert researcher. The solution for these problems involve analytical and computational approaches commonly used in quantum many-body physics and classical statistical mechanics. The dataset has been designed and verified by a worldwide panel of expert researchers through a collaborative environment. 
Topics in the dataset include Hartree-Fock mean-field theory, exact diagonalization methods, quantum Monte Carlo sampling, density matrix renormalization group, quantum statistical mechanics, classical statistical mechanics, and model building. We evaluate different LLMs by programmatically checking LLM-generated solutions against expert-supplied ground truth. 
To verify LLMs performance at scale, we developed an automated machine-grading pipeline suitable for advanced physics research problems. 
For example, we handle non-commuting operators that are essential for quantum many-body problems by symbolic manipulation and normal ordering. 
Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting a gap in the physical reasoning skills of current LLMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. 
While the highest-performing model, GPT5, correctly solves 30\% of the problems, average performance across 17 models (GPT, Gemini, Claude, DeepSeek, and Llama classes) is only 11.4$\pm$2.1\%.  Moreover, our benchmark contains 18 problems that not a single one of the 17 models considered here can correctly solve, and 26 problems that are solved by at most one model. 
These currently unsolvable problems span the fields of Quantum Monte Carlo, Variational Monte Carlo, and Density Matrix Renormalization Group. 
Furthermore, we illustrate how incorrect answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe that this benchmark set provides valuable guidance for the future development of language models, aiming to achieve the goal of AI research assistants and tutors.

CMT-BENCHMARK: A BENCHMARK FOR CONDENSED MATTER THEORY BUILT BY EXPERT RESEARCHERS
Haining Pan
James V. Roggeveen
Department of Physics and Astronomy Rutgers University
School of Engineering and Applied Sciences
Harvard University
Erez Berg
Juan Carrasquilla
Debanjan Chowdhury
Department of Condensed Matter Physics
Department of Physics
Department of Physics
Weizmann Institute of Science
ETH Zürich
Coreell University
Surya Ganguli
Federico Ghimenti Department of Applied Physics Stanford University
Juraj Hasik
Department of Applied Physics Stanford University
Department of Physics University of Zürich
Henry Hunt
Hong- Chen Jiang
Department of Applied Physics Stanford University
Stanford Institute for Materials and Energy Sciences SLAC National Accelerator Laboratory
Mason Kamb
Ying-Jer Kao
Department of Applied Physics Stanford University
Department of Physics National Taiwan University
Ehsan Khatami
Michael J. Lawler Department Physics
Department of Physics and Astronomy San José State University
Comell University
Di Luo
Titus Neupert
Department of Electrical and Computer Engineering University of California, Los Angeles
Department Physics University of Zürich
Xiaoliang Qi Department of Physics Stanford University
Michael P. Brenner
School of Engineering and Applied Sciences
Harvard University
Google Research
Eun-Ah Kim
Department of Physics Cornell University e64360 edu
ABSTRACT
Large language models (LLMss) have demonstrated remarkable progress in coding and mathematical problem-solving: however, evaluation on advanced research- level problems the hard sciences remains scarce. To this gap, we present CMT-Benchmark, dataset of 50 original problems covering condensed mat- ter theory (CMT) at the level an expert researcher. The solution for these problems involve analytical and computational approaches commonly used in quantum many-body physics and classical statistical mechanics. The dataset has been designed and verified by worldwide panel of expert researchers through

---

collaborative environment. Topics in the dataset include Hartree- mean- -ield theory, exact diagonalization methods, quantum Monte Carlo sampling. density matrix renormalization group, quantum statistical mechanics, classical statistical mechanics, and model building. We evaluate different L.LMs by programmati- cally checking L.LM-generated solutions against expert-supplied ground truth. To verify LLMs performance at scale, we developed an automated machine-grading pipeline suitable for advanced physies research problems. For example, we handle con-oonuuiing operators that are essential for quantum many-body problems by symbolic manipulation and normal ordering. Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting gap in the physical reasoning skills of current L.MMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. While the highest- perthrrsttprrfooinngg model, GPTS, correctly solves 30% of the problems, average performance across 17 models (GPT, Gemini, Claude, DeepSeek, and Llama classes) is only 11.4+22.1 1%. Moreover, our benchmark contains 18 problems that not a single one of the 17 models considered here can correctly solve, and 26 problems that are solved by at most one model. These currently unsolvable problems span the fields of Quantum Monte Carlo, Variational Monte Carlo, and Density Matrix Renormalization Group. Furthermore, we illustrate how incorrect answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe that this benchmark set pro- vides valuable guidance for the future development of language models, aiming to achieve the goal of Al research assistants and tutors.
INTRODUCTION
The progress of Frontier LLMs has been stunning. Whereas a few years ago models struggled on high school mathematics problems (Hendrycks et al.) [02T), today's 's LLM-based systems achieve Gold medals in the International Math Olympiad (Trinh et al. 2024) and competitive coding competitions
inventing solutions that humans are unable to discover. Benchmarks for assessing LLMs against
expert-level mathematics have flourished (Liu et al. 2024 Fan et al. 2024 Roggeveen et al. 2025
including expert-level benchmarks made by professional mathematicians (Glazer et al. 2025) At the
same time, there is an intense interest in LLMs for science, with significant literature focused on creating benchmarks to evaluate agent capabilities against hard science problems (Laurent et al. 2024
Mitchener et al. 2025 Feng et al. 2025| Cui et al. 2025 Pan et al., 2025 Pramanick et al. 2025)
However, existing hard science benchmarks measure knowledge or skill for carrying out textbook
problems for students at varying levels, and do not assess whether models can function as a research assistant on cutting-edge scientific tasks. Typical crowdsourcing strategies will not work in highly technical fields like theoretical condensed matter physics, since the required expertise focused on small communities. While the mathematics community built such benchmarks by assembling groups of leading experts to create research- research-grade problems (Glazer et al. 2025 Balunović et al.) 2025), a systematic counterpart is missing in hard scientific domains.
We created CMT-Benchmark to address this gap so that the LLM research community can hill-climb towards competent AI research assistant. For LLMs to serve as scientific research assistants, they must demonstrate rigorous critical judgment and the ability to synthesize existing knowledge with theoretical principles established in specific scientific domain. We designed the problems and rigorous yet automated evaluation scheme, focusing on condensed matter theory (CMT), a subfield of physics reveals how collective interactions among particles generate emergent phenomena such as supereonductivity and topological phases, while also providing the theoretical foundation for advanced materials and quantum technologies. Research of CMT requires synthesizing microscopic knowledge of material systems with macroscopic observations in a manner that adheres to the theoretical principles,
consists of 50 original, select, high-value problems covering seven computational and theoretical methods, as well as sound model-building (categorized as "Other"), as shown in Fig. Ia The novelty of our problems lies in the design principles we adopted from the principles of trustworthy and impactful scientific research. For this. we assembled international panel of expert researchers to write original problems and provide critiques of each other's 's problems. Each contributor crafted problems they would expect strong graduate student or research assistant to
2

---

answer correctly, measuring critical skills for performing research in their field. They then iteratively refined the problems to identify gaps in critical judgments and insights in LLM reasoning.
The design principle of our evaluation scheme is also based on the mission of aiding the development of competent AI research assistant. Scientific research must push the knowledge frontier so that other researchers in the community can build the outcome. Correctness must be absolute, and results should be deterministically reproducible. Hence, unlike the typical homework grading setting where the grader issues partial credit, we apply the rigorous standards we hold ourselves to: demand that the answers be deterninistically and objectively correct. We designed problems in multiple answer formats that can be automatically graded, including multiple-choice, numerical values, algebraic expressions, and non commomtaiiiv operator expressions. We score the LLM--ennrral correct or incorrect against the ground truth answer supplied by the author of the problem. Even the most advanced models exhibit low performance on our problems, with the highest pass rate of 30%. Our results reveal that current L.MS cannot function as research assistants.
solutions as
We highlight that HMTT-Bnchmark makes the following important contributions:
1. Benchmark for Analytic and Computational Reasoning. It is the first benchmark explic- itly designed to jointly test analytic and computational reasoning in LLMs, assessing their potential as scientific research assistants in CMT field central to understanding emergent quantum phenomena and foundational to quantum materials and technologies.
2. High-Value, Expert-Curated Research Level Dataset. Our dataset comprises 50 original and rigorously designed problems spanning seven computational and theoretical methods, plus moodel-bilddin. Problems were authored and refined by an international panel of expert researchers, including postdocs and professors in top universities, ensuring that each reflects the level of reasoning expected from a strong graduate student or research assistant.
3. Rigorous Evaluation Revealing Fundamental Gaps in LLM Reasoning. Even frontier models struggle on CMT-Benchmark: GPT-5 solves only 30%, with the average across 17 models 21.42..1% Moreover, 18 problems are unsolved by any model, and 26 are solved by at most one. We further diagnose common failure modes, including violations of fundamental symmetries and unphysical scaling dimensions, highlighting critical reasoning gaps and establishing SMMT-Bnncharrk as roadmap for advancing Al scientific assistants.
2 RELATED WORKS
Progress in evaluating expert-level scientific reasoning has been lagging behind mathematical reason- ing. The standard metric for measuring scientific prowess remains the 2023 benchmark Graduate- Level Google Proof Question and Answer (GPQA) (Rein et al. 2023 although the performance of Frontier LLMs has nearly saturated. Recently, Humanity's ' Last Exam (HLE) (Phan et al. 2025) raised the bar, with a crowdsourced collection that incentivized hard problems, with some fraction focusing on the sciences, spread across wide range of categories. Although the difficulty of the benchmark is appealing, neither HLE nor any of the existing benchmarks measure the qualification of LLM to serve as research assistants in specific scientific domains.
Nevertheless, a recent benchmark, (Wang et al. 2025). is notable as it focused on condensed matter physies textbook problems. with difficulty levels ranging from undergraduate to advanced graduate coursework. The dataset contains calculation problems extracted from textbooks on topics spanning Magnetism, Superconduuttvitty, Strongly Correlated Systems, and Semiconductors. The LLM responses were evaluated using abstract syntax trees to offer partial credit, using a custom metric dubbed SEED, which exhibits 90% correlation with human experts. While impressive, this benchmark focuses questions for students, rather than those at the cutting edge of research.
2.1 THE NEED FOR CONDENSED MATTER THEORY
We choose CMT both because of the importance the domain and of its underrepresentation in widely used benchmarks such HLE. As the largest branch of modern physics, condensed matter provides the foundation for understanding emergent quantum phenomena in materials and underpins transformative advances in quantum materials, quantum computation, and quantum technologies. The domain's 's close ties to material science, chemistry, and quantum technologies imply a high demand for AI research assistants in this field. However, the subject is challenging teach algorithmically. The
3

---

Table Comparison of selected advanced benchmarks. We list focus, sourcing method, evaluation,
and size.
Dataser
Focus
Problem Sourcing
Evaluation Method
GPQA RRei al. 2023
Graduate-leve science Q&& (hiol-
Domain experts
Multiple-choice accuracy (google- poogll--rroff
448
egy, physies, chemisiry)
Humaniit's
Last
Broad coverage (mulliple-hhicee
Subject-matied experts worldwide
Automated grading for multiple-hhoice and short-answo wer
2,500
Exam (Phan et
2025
sbott answer)
MathAreaa
Baninovii
Olympiad-style math competitions
Expest creation
Automated formula parsing
96
(exact answers)
ScI o0d- Benchh Bench Tian et al.
Scientific code generation across natural sciences
Scientist curaated
re-
Unit lests and domain specific test cases
80
2024
search scripts
TPBanch
Chung ot al
Theorctical physics (high-cnergy. cosmology)
Researcher-authore novel problems
Combination of auto-verifible checks and rubric-driven grading
57
2025
PnySense Xu cta 2025
Principle-based physics reasoning
Expert curation
Automated grading for multipe-choicee and evaluation of token efficiency
380+
(thenretical physics)
CMPhysBench Wang el al.
Condensed mailer physics (calcula- problems)
Expert curation
Expression tree edit distunce with partial credit (Scalable Expression Edit Distance)
520+
2025
CMI-Benchmark
Condeased matter theory (numerical and analytical)
Expert pancl authorship
Automatic parsing with numerical and sym- bolic equivalence
50
and curation
theoretical framing of CMT involves modeling many interacting entities in accordance with strict set of physical rules. Problems that quickly reach the limits of computational complexity abound. Cutting- edge research in the area requires a multifaceted approach that combines mathematics. theoretical formalisms such as field theory and non-commutative operator algebra, computational methods, a geometric understanding of the physical system, and fundamental concepts, including notions of symmetry. LLMs tremendous progress in coding and mathematics, along with their possession of knowledge far exceeding that of an individual human, suggests that an Al research assistant may be attainable.
2.2 INNOVATIONS IN BENCHMARK CREATION
CMTT-enchmark, with its aim evaluate research-readiness, goes beyond textbook knowledge or skills and applies rigorous standard of correctness. In research, it is crucial to discern what not to do, because whether meaningful answer can be reached is not known a priori. Our problems test such judgments and the ability to synthesize knowledge and skills to define and solve meaningful problem. Moreover, research output must be absolutely correct and reproducible. Hence, our problems are designed to be deterministically evaluated with binary outcome. Since it took experts hours to write original problems, the number of problems in CMT-Benchmark is small compared to crowd- or textbook-soureed benchmarks. However, it is the first hard science equivalent of FrontierMath.
Another innovation we introduced automatic parsing that can handle equivalent expressions of non-commuting operators. Our evaluation scheme was built on the framework for automatically parsing LaTeX introduced in (Roggeveen et al., 2025). However, with problems in quantum many- body physics, needed the parser to recogmize equivalent expressions based operator algebra correctly. The formalism of quantum mechanics uses the algebra of non-commutative operators. A competent researcher readily dismiss an operator expression violating fundamental principles, one of the basic approaches for checking the "sanity" of calculations. While LLMs can carry out algorithmic manipulations,
a different skill to inspect problems, statements, and reasoning from the fundamental principles underlying the formalism. Problems requiring operator expressions in their answer test such skills. Our parser handles operator algebra and identifies equivalent expressions, through symbolic manipulations and normal ordering, as described in Section 3.1.I
DATASET
This dataset was constructed by an international panel of condensed matter theorists including postdocs and faculties from top universities, who were tasked to contribute original problems they would expect their group members to answer correctly. We required the problems to be unambiguous and lead to single, verifiable solution that could be parsed and machine-graded by our evaluation software. In addition to the problem and solution, authors were required to provide a written explanation of their solution to facilitate easy verification of the problem's 's correctness by other panel members. Our dataset covers a broad range of solution modalities, including algebraic expressions, numerical values, multiple-choice questions, and operator expressions, shown in Fig. For
4

---

Probiem type diseribationn (50 problems)
Type
Example question
VMC
PEPS
ED
Which among N. S" .7" L, negping and
9.0%
6.0%
E,, o cl Cutt, are good quanium numbers for H
-t Es (ol, c441, +.c.) +U E, Nepmer + E, huS
Other
32.0%
HF
QMC
Which methods are provably efficient for 1D vs 2D ground-state properties given sign problem constraints?
SM
What the critical coupling Ke for synchronization stability coupled large-N N soft-spin systems?
12.0%
HF
Which HF order parameters preserve translational symmetry 2D triangular lattice?
RM
12 0%
DMRG
What is the ground state degenerucy of an open Kitaev alteraating chain?
RD
ONC
VMC
Which projections restere C4 rotation symmetty of a J1-J2 variational wavefamction?
PEPS
In momentun-superrpoed singlo-defect iPEPS excitation B w. express ff using only Ne and B.
(a) Distribution of problem types in CMT- Benchmark.
(b) Representative example questions by problem type.
Figure 1: Problem type distribution and representative example questions in each type.
multiple-choice problems, we ask for one or more choices from among many options (over 5 for most problems) to avoid 'lucky guesses'. All solutions were evaluated using a uniform evaluation framework with strict passing standard of correctness without partial credit. Our dataset can be found in the Hugging Face repository (Huggingface) 2025).
3.1 PROBLEM SUBMISSION AND VERIFICATION PIPELINE
We built CMT-Benchmark using Google Sheet running custom-built extension. Using the extension the panel could test their solutions against the parsing infrastructure before evaluations, ensuring that ground truth solutions were compatible with our grading framework. The extension also allowed writers to run their prompts through subset the LLMs used in the final evaluation, including providing machine grading of the LLM solutions. The authors used this feature to increase the difficulty of their problems iteratively. This iterative approach also helped authors remove any ambiguities in their problems and significantly improve the quality of the dataset.
When problem fails every model available on the sheet, another author would review the problem accepted into the final benchmark dataset. On
and solution for correctness before the problem the spreadsheet, authors had access to Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT-4o. This iterative problem-buildingg approach, using custom Google Sheet integration, mirrors the success of other recent benchmarks (Roggeveen et al. 2025).
3.1.1 INFRASTRUCTURE FOR AUTOMATING PARSING AND EVALUATION
To enable automated grading of mathematical expressions for correctness, we implement a ETTXtto Sympy parser that converts raw expression provided by either an author or LLM into an expression that can be evaluated. The parser used in this benchmark builds on that used for standard algebraic problems previously used in mathematics benchmarks (Roggeveen et al. (025). To enable parsing, authors are required follow certain guidelines in formatting their answers, along with providing list of the parameters, variables, and functions they expect to appear in their solution. These must be defined in the prompt. The model instructed to return final answer boxed LTT.Xenvirommn and not to introduce any new variables as part of their solution. Although models may generate arbitrary intermediate reasoning, our evaluation pipeline deliberately discards these traces and evaluates only the final boxed expression, keeping the benchmark focused on final research conclusions while remaining compatible with future extensions that explicitly evaluate reasoning traces.
novel component of the parsing logic for this benchmark was the introduction of monn-ommtting operators. While standard algebraic expressions may be evaluated by substituting scalar values for
5

---

Answer modality:
Numerical value
Consider classical O(3) spin Hamiltonian spatial dimensions on triangular lattice: - Su Sj. where J4j = for x-directed bonds and J,s otherwise. AT = find the number gapless Goldstone modes, N and
Question:
Jor lerronagnetic couplings (J D ar and RAF in latex environment separated by aniferrenageeic couplings (J U, < J' < 0), Return
Answer: 1:3
Answer modality: Multiple choice
Consider the dynamies dimensions the following madified active Brownian particle: vig where top postive constant, and u is a vector of unit norm, whose orientation & with respect to the axis evolves according to the overdumped dynamics::
Question:
VV. Au x(t), where V (xx is an external potential that depends only on x, and is positive constant. The matrix A
fully antisymmetric two-dimensional matrix. Consider perturbation of the
potential V - V + (x)) Consider the steady state lincar respouse function R(s) (ol and the steady state sutocorrelation function C(a) (d(())))(((( +))}. Is the fluctuation dissipation theerem between these correlation and
d8(t)
response functions violated! Choose one of the tollowing options: (a) Yes. because the dymumics has positive entropy production
No, hecause dynamics iime-reversiblee (c) No, because the Roltzmann distribation is the stationary distribution. (d)
Yes, because there a nonzero self-propulsion speed. Return your choice among the options 'a","b" "b""""""" and "d" enclosed in a LaTex cnvironment.
Auswer:
Answer modality: algehruic expressions
Question: Consider peculiar example Kitnev alternating chain, whose Hamiltonian given > ( 02 02 02 1). where of and of are Pauli matrices open chain in terms of N and the value of the central charge c. Denote these degencracies as function of N and the value of the charge
and is number of sites. Calculate its ground degenerncy for
return your answer LalleX
Answer: 2 N/2 1.
Answer modality:
non-commutative operator expressions
Question: Consider the Fenni Hubbard Hamiltonian with neurest neighbor hopping in its particle hole symmetric form hipurtite lattice with chemical potential term. Express the Hamiltonian after the following transformation: <t, = P and sublattice or the sublattice, taking m,, be the new density operator. The Σ (t.2).e fi.j.e Σ. Ss. where the only operators in fiujo are the operatoes and only
el. P.., dopending on whether answer will take the form operators Si are the m operators. Return the expression for fi.j.e + 9: in IaTeX enviromennt, Your answer should not include any > notation or the Hermitian conjugate (H.c.) abbreviation.
Answer: -(pp, Ps.n + p].. pt,)) U(m,,
(mn () p(ms, ml.))
Figure 2:
Example questions in CMT-Benchmark by four answer modalities: numerical value, multiple choice, algebraic expressions, non-commutative operator expressions.
variables and evaluating a single numeric value, non-commuting operators are not amenable this treatment. As these non-commuting operators play key role in quantum condensed matter physics problems (e.g., Hamiltonians I! -';p buj c/c, U U, n4,), benchmark incapable of correctly evaluating these expressions would be missing a core component of the field.
We handle such expressions by having authors declare whether any such expressions exist in the problem. These declarations are known only to the parser and are not passed to the model for its evaluation. these cases, we replace any non-commutative expression with non-commutative Sympy symbol and then invoke standard physics simplifications (e.g., {q, cf} = d.y for fermions) to reduce both the specified solution and the model's response canonical, order-sensitiv form, such as the "normal ordering" from condensed matter. We then verify equivalence using standard Sympy symbolic equivalence checks. We will release the code to perform machine grading in the future.

---

3.2
PROBLEM TYPES
The dataset includes 8 problem types in terms of the contents, as shown in Fig. la covering 7 computa- tional and theoretical methods and sound model building. The computational and theoretical methods covered are Hartree-Fock (HF), Exact Diagonalization (ED), Density Matrix Renormalization Group (DMRG), Quantum Monte Carlo (QMC), Variational Monte Carlo (VMC), Projected Entangled Pair States (PEPS), and Statistical Mechanics (SM). Problems that test sound model-building and the use fundamental principles are labeled as the Other type. Example questions from each problem type shown in Fig. Tb and detailed descriptions follow. We employ a diverse range of answer formats, including algebraic expressions, numerical values, multiple-choic questions, and operator expressions, illustrated in Fig. 2 evaluate LLM's capabilities from multiple angles while ensuring automatic and deterministic evaluations.
3.2.1 HARTREE-FOOG (HF)
Problems cover self-consistent mean-field decouplings and ground-state characterization on lat- tices, classification of order parameters consistent with symmetries, Brillouin-zone folding under commensurate charge-density waves, and numerical complexity estimates for plane-wave repre- sentations. For example. we consider solving the self-consistency equation for mean-field theory on triangular lattice associated with the following mean-field Hamilto- nian with mean-field terms being Hartree = E.,' Exy (()) (k2) (A:2) and
Hrwck - E.. En,,, U(k k2) (ct (ka))e(, (k2) (k2), where U(k: Σn Une-ikn the repulsive interaction strength (U, > 0) in the momentum basis. What are the possible order parameters that preserve translational symmetry for a Hartree- Fock mean-field -ield Hamiltonian on two-dimensional triangular lattice?
3.2.2 EXACT DIAGONALIZATION (ED)
Problems cover finite-size many-body spectra and symmetry resolution, including identification of good quantum numbers and block-diagonal sectors, counting symmetry-distinct momentum and point- group blocks, diagnosing exact versus asymptotic degeneracies, scaling of low-lying level spacings, small-cluster combinatorics for model building, and translational or gauge-structure consequences for expectation values and band minima. For example, consider Hamiltonian for N fermions, + H.c.) + E, nigna + >E h,57, and ask which of the following are good quantum numbers: (a) N; (b) S² (c) Ei, cf Ci41, (d) 1j² = = (n + (11²)²
H Ei,
with 7 = (-1) 7 = n__ and IN L); (e) Σ, 1o4751.
3.2.3 DENSITY MATRIX RENORMALIZATION GROUP (DMRG)
Problems cover bond-dimension scaling versus system size, extraction and comparison of correlation lengths (e.g., bulk correlators versus boundary-pinne -pinned responses), effects of boundary conditions and geometry (chains, ladders, cylinders), and phase identification in concrete lattice models. For example, N/2 (oh-1107 + -2 where we consider a Kitaev alternating chain with Hamiltonian H = -IN? of and of are Pauli matrices on site i and N is the number of sites, and ask for the ground-state degeneracy for an open chain in terms of N and the central charge c.
3.2.4 QUANTUM MONTE CARLO (QMC)
Problems cover phase transitions in frustrated transverse-fiee Ising models on the triangular and 4-8 lattices (emergent U(1) versus Ising behavior), sign-problem diagnostics in stochastic series expan- sion on square versus kagome lattices with J. and J+ terms, and determinant- QMC sign-problem conditions fermionic settings (spinless fermions at half filling, two-band model with onsite interac- tions and spin-mixing hoppings, and long-range-interacton lattice model under specified densities, fields, and complex next--narret--eiihhbe hoppings). For example, we consider the transverse- field Ising model in one dimension and on the 4-8 lattice with antiferromannetic and strong ferromagnetic bonds arranged so that every plaquette carries π flux. We ask which standard methods are provably efficient for computing ground-state properties in 1D and 2D for large system sizes (more than 200
7

---

spins), given the two-dimensional sign-problem constraints and the availability of Fordan-Wigner mappings, DMRG, transfer-matrix approaches, or simple variational constructions in one dimension.
3.2.5 VARIATIONAL MONTE CARLO (VMC)
Problems cover symmetry restoration/projection in lattice spin models and the correctness and variance of Monte Carlo estimators for neural-network wavefunctions. For example, we consider the 2D Heisenberg J1-J2 model with a wavefunction v²(x) that breaks rotation symmetry and ask which constructions restore C₁ rotation symmetry using the rotation operator R: (a) Σn & (R"x);
(b) II, 3 (R"x)); x); (c) En (RR)); (d) En (-1)" (R"x)) (e) Σn eaan" x)).
3.2.6 PROJECTED ENTANGLED PAIR STATES (PEPS)
Problems cover iPEPS excitation ansätze built by locally replacing ground state tensor and forming momentum superpositions, coarse- coarse-graining pipelines for extracting CFT data in classical 2D tensor networks, and SU(2)-yymmete iPEPS design with rotational (C4) constraints and parameter counting. For example, we start from an iPEPS with ground-state tensor A, form a defect state by replacing one A with B at position
(f that (A)) - ( B)xx), define the momentum superposition | ())x = D> e sk-z (B))),, and consider the generalized eigenvalue problem H2 B = wu.f. We ask for the form of in terms of the normalization N₂- (B( (B)xx, using only N₂. and B.
3.2.7 STATISTICAL MECHANICS (SM)
Problems cover nonequilibrium stochastic dynamics and combinatorial models, including odd diffu- sivity in chiral active Oomstein-Uhleneee processes with inertia, cectuuaiiondissipaaln checks for torque-driven active Brownian motion, synchronization thresholds coupled random-tenmnn soft-spin networks, cavity-variable choices in the d tions for muultiplicative-niis Langevin equations, and counting fully packed dimers on cylinders with defect-density optimization. For example, consider two coupled systems with N soft spins {2;} and {yi} obeying (Fournier et al. 2025) à; = >(x)x;; + N- J(h 1y34 + K(y; xx)) and ú: -yy)) - J.H yjık + K(:;; y:), where X(x) N
limit for molecular liquids, Onsager-Machlup ac-
with y > and Jtl is symmetric random tensor of zero mean and variance o². In the N + oo limit and defining synchronization by V-'E,aa )² = 0 in steady state, we ask for the critical coupling Ko(y, o above which the synchronous state is stable.
3.2.8 OTHER
Problems cover building and application of fundamental principles: particle-hol hole map- pings and operator rewrites in Hubbard-type models; strong-coupligg correlators in dimerized chains; linked-cluster expansions via inclusion-exclusion: correlation decay and transition claims in long-range Ising models; transport and compressibility statements in frustrated boson models; and zero-temperature phase and correlation properties quantum Ising-tyre type systems. For example, ask the LLM to choose from the following options for classical Ising model with Hamilto- nian - Eizj J(i ji)a,, where (n) = |Jol/(L (1 + n²) We ask the LLM to choose from: (a) For a = 2, there is a non-zero critical temperature T., (b) For a 1, at sufficiently low temperature, (a (o)
(c) For a there is temperature for which (o,o)tn) - (o decays as the inverse of the logarithm distance. (d) For a = 4, (ayo)tm) decays exponentially with distance. (e) For = 2, (a)ōjtn) decays exponentially with distance.
m² 0
EVALUATION
Our experimental goal is to probe the out-of-the-box, zero-shot, closed-book performance of general- purpose frontier models without any domain-specific fine-tuning on CMT- CMTT-Bnchmaak. We evaluate 17 models on the full benchmark, 7 OpenAl's model (GPT-40, GPT-4.1, GPT-5, GPT-5-mini, GPT- 5-nano, GPT-o3, GPT-o4-mini); 3 Google Gemini's 's models (Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro); 5 Anthropic's Claude models (Claude 3.7 Sonnet, Claude 4.0 Sonnet, Claude 4.1 Sonnet, Claude 4.0 Opus, Claude 4.1 Opus); and two open source models (DeepSeek v3, LLaMA
8

---

Model
Overall
HF
ED
DMRG
QMC
VMC
PEPS
SM
Other
GPT-4o
2.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
6.2
GPT-4.1
4.0
0.0
12.5
0.0
0.0
0.0
0.0
GPT-5
30.0
20.0
37.5
0.0
16.7
0.0
66.7
33.3
37.5
GPT-5-mini
24.0
20.0
37.5
0.0
16.7
0.0
33.3
50.0
18.8
GPT-5-nano
14.0
20.0
12.5
0.0
16.7
0.0
33.3
0.0
18.8
GPT-o3
26.0
20.0
50.0
25.0
16.7
0.0
66.7
16.7
18.8
GPT-o4-mini
18.0
20.0
25.0
0.0
16.7
0.0
33.3
33.3
12.5
Gemini 2.0 Flash
10.0
20.0
25.0
0.0
0.0
0.0
0.0
16.7
6.2
Gemini 2.5 Flash
4.0
20.0
12.5
0.0
0.0
0.O
0.0
0.0
0.0
Gemini 2.5 Pro
14.0
20.0
12.5
0.0
0.0
0.0
33.3
0.0
25.0
Claude 3.7 Sonnet
6.0
20.0
0.0
0.0
0.0
0.0
0.0
0.0
12.5
Claude 4.1 Sonnet
2.0
20.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Claude Sonnet
6.0
20.0
12.5
0.0
0.0
16.7
Claude 4.1 Opus
8.0
20.0
12.5
0.0
0.0
0.0
33.3
0.0
6.2
Claude 4.0 Opus
10.0
20.0
0.0
25.0
0.0
0.0
33.3
16.7
6.2
DeepSeek v3
4.0
20.0
12.5
0.0
0.0
0.0
0.0
0.0
0.0
LLaaM Maverick
12.0
20.0
25.0
0.0
0.0
0.0
33.3
16.7
6.2
Table 2: Pass @ rates (%) by model and question type.
Maverick). For the detailed case studies, refer the reader to Appendix A Each model is queried with the prompt written by the author, along with fixed component specifying formatting instructions. models that support it, we passed system instructions requiring the model to provide a solution in the form of boxed 151EXexpression. We evaluate the last boxed expression from in response and grade using the parsing and evaluation code described in Section 3.I.I The LLM's s solution is marked correct if the parser determines is equivalent to the solution provided by the problem author. Some models, in particular Gemini 2.5 Pro, occasionally disregard the formatting instructions and produce responses that cannot parsed without boxed L3TpXeenironment. The parser failed for small number of problems, which were human-gradee
We summarize Pass@1 both overall and by topic inTable 2 which reports the results a percentage correct grouped by model and problem type. Overall accuracy is significantly lower than other physics-related benchmarks (Qiu et al. 2025 Wang et 2025). The best performing models were GPT-5 (30.0%). GPT-o3 (26.0%), and GPT-5-mini (24.0%) the only three to score above 20%. The second tier, which we defined as any models with overall > 10% but below 20%, was comprised of: GPT-o4-mini (18.0%), GPT-5-nano (14.0%), Gemini 2.5 Pro (14.0%), and I.LaMA Maverick (12.0%). All other models scored at or below 10%: Gemini 2.0 Flash (10.0%), Claude 4.0 Opus (10.0%), Claude 4.1 Opus (8.0%), Claude 3,7 Sonnet (6.0%), Claude 4.0 Somnet (6.0%), Gemini 2.5 Flash (4.0%), GPT-4.1 (4.0%), DeepSeek v3 (4.0%), Claude 4.1 Sonnet (2.0%), and GPT-4o (2.0%).
Looking at the results by question type, it is clear that several areas remain extremely challenging for the models to solve. Every model scored 0.0% on VMC problems that required critical judgements while QMC peaks only 16.7% for the first tier models. Only two models, GPT-o3 and Claude 4.0 Opus, achieved a non-zero score on the DMRG questions (25.0%). In contrast, models did comparatively better with technical questions using PEPS: the top models reach 66.7% (GPT-5, GPT-o3), and several others achieve 33.3%.
Overall, the results in Fig. 3 show that No model approaches mastery across topics. These trends highlight persistent gaps in sign-problem reasoning (QMC), long-range entanglement and boundary effects (DMRG), and symmetry- variational projections (VMC).
is difficult for even the strongest models.
CONCLUSION
In this paper, we introduce CMMT-Benchmark. a benchmark built by expert researchers to mirror real research practice in CMT. The dataset comprises 50 original, expert-authored problems that address methods currently used in quantum and classical CMT research and model-building strategies. The methods covered are Sonttre--Fokk exact diagonalization, density matrix renormalization group, quantum Monte Carlo, variational Monte Carlo, Projected Entangled Pair States, and statistical mechanics. SMTT-Benchmar expands the current landscape of scientific benchmarks through

---

102
Oreena Mele by Nodde
Poss Rate yer Probcrs Sype Mode
VMC
00-
00-
20-
ordda orver cord
ore orssusa broun
arpai
carr antel
dimmaa 2.5 Plani Commas Photd creann frre Drinser
Clante peosrr
4. bontat ch Cyosto Clasde gpes Clamie o open Dempores Llads Movernaa
arsar
ard
Grenn 2. Fish
Greant 2 yaast
clastle 2 sontot Clande cassil 4 someet crosde & cpea Oooo canate Peopi of
Sencett
LlastA Mitweria
Figure 3: Model performance on CMT-Benchmark (a) Overall success rate on benchmark by
model. (b) Success rate per model divided by problem type.
research-grade problems that test LLMs' readiness to work as a research assistant. We tested critical judgment and the ability to synthesize different modalities of information, including mathematical, lunguage-based (conceptual), geometric, and fundamental laws of physics. We evaluated all LLM- generated responses using a deterministic machine- based grading system that supports both numerical and symbolic evaluation, including non-commuting operator algebra. We find that current state-of- the-art LLMs struggle with CM Berh-Ben with the strongest models achieving only 24-30% overall accuracy and no model demonstrating mastery across different problem types.
We gained unique insights through the problem development process that surfaced limitations in frontier LLMs. Since our infrastructure provided global, real-time view of how models called into the Google Sheet were performing across all the problems, the authors could iteratively identify the angle that caused all the LLMs in the Google Sheet to fail. Firstly, LLMs struggle in connecting 'verbal" expressions to accurate algebraic expressions or geometric ideas. Researchers in the domain can readily translate verbal descriptions, such "fermionic Hubbard model near half-filling Kagome lattice," into an operator algebraic expression for the Hamiltonian. We think in language but calculate using precise mathematical notation. The inability readily and precisely switch gears between language and mathematics results in LLMs making trivial mistakes and working with expressions that break the laws of physics. This weakness revealed when problems require answers based on the calculations LLMs must design. Another common struggle is in geometric reasoning. Researchers often sketch geometric view of the problem as a key part of reasoning, as in considering the number of Fermi surfaces in Sec. Sec.[A.3 The LLM will need to be connected to tool that can plot the Fermi surface and be instructed on how to do so. Secondly, LLMs struggle with applying fundamental principles such as symmetry to operator algebraic expressions. When relevant terminologies are given, LLMs use the terminologies as a handle to recall the textbook examples. However, slight departure from the textbook example will trip up the LLM and reveal its limited appreciation for fundamental principles as the foundation of critical judgment. For instance, in a mixed-field Ising model with no 22 symmetry to break, some LLMs still predicted symmetry- -breakig transition as function of the transverse field, misdiagnosing the most basic symmetry structure of the problem. Thirdly, LLMs rely on heuristics when a problem requires a judgment call. For example, in a quantum Monte Carlo efficiency question, LLMs often misattribute the bottleneck in a problem to the so-called 'sign problem'; when the prompt explicitly states the absence of the sign problem, some L.MSs can then identify the real bottleneck. Finally, L.LSs often fail to recognize the underlying structure or mapping that allows one to leverage known results to simplify the problem. This was revealed in problems that can be mapped to free fermion problem or problems with an underlying conformal field theory.
ACKNOWLEDGMENTS
E.-A.K. acknowledges funding support from NSF Award 2433348 and OAC-2118310. H.P. ac- knowledges the support of US-ONR grant No. N00014-23-1- 1-2357. D. C. is supported in part by NSF CAREER grant RMR-22235222, and a Sloan Research Fellowship from the Alfred P. Sloan foundation. T.N. and J.H. acknowledge support from the Swiss National Science Foundation through a Consolidator Grant (iTQC, TMCG-2213805). S.G. acknowledges support from NTT research, the Simons foundation, and Schmidt sciences polymath award. F.G. acknowledges support from postdoctoral fellowship of the Stanford's 's Leinweber Institute of Theoretical Physics. E.K. acknowl- edges support from the U.S. Department of Energy, Office of Basic Energy Sciences under Grant No.
10

---

DE-SC0022311, Y.J.K.. acknowledge support from the NSTC of Taiwan under Grant No. 113-2112- M-002-033-MYY3. H.-C.J. is supported by the US Department of Energy, Office of Basic Energy Sciences, Division of Materials Sciences and Engineering, under Contract No. DE-AC02-76SF00555 15. J.V.R. and M.P.B. acknowledge support from the NSF Al Institute of Dynamic Systems 21 2085.
REFERENCES
Mislav Balunović, Jasper Dekoninek, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on Uncontaminated Math Competitions, May 2025. URL http::/arxx/
org/abs abs/ 2505. 23281
Daniel J. H. Chung, Zhiqi Gao, Yurii Kvasiuk, Tianyi Li, Moritz Münchmeyer, Maja Rudolph, Frederic Sala, and Sai Chaitanya Tadepalli. Theoretical Physics Benchmark (TPBench) Dataset and Study of Al Reasoning Capabilities in Theoretical Physics, February 2025. URL
Sett://arxiin www.orgg.//..oooo..oo http://.b/222225555
Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian Rohr, Ekin Dogus Cubuk, Muratahan Aykol, Amil Merchant, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael P. Brenner, Viren Jain, Sameera Ponda, and Subhashini Venugopalan. CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning. In ICLR 2025 Main Conference. arXiv, May 2025. URL htt:: /ttp:://aaxivii org abs 13517
Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng. Corey Wang, and Michael P. Brenner. HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics, December 2024. URL / http://axxvvvoooo
0los/241000995
Kaiyue Feng Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. PHYSICS: Benchmarking Foundation Models on -niversity-Level Physics Problem Solving, March 2025. BRLphttpp///xxx/
wwtp://wwwrraaa//////0// 2503. 21821
Samantha J. Fournier, Alessandro Pacco, Valentina Ros, and Pierfrancesco Urbani. Non-reciprocal interactions and high-dimensional chaos: Comparing dynamics and statistics of equilibria in solvable class models, December 2025.
20908
Federico Ghimenti, Ludovic Berthier, Grzegorz Szamel, and Frédéric van Wijland. Irreversible Boltzmann samplers in dense liquids: Weak-coupling approximation and mode-coupling theory.
Phys. Rev. E. 1334)4434 September 2024. ISSN 2470-0045, 2470-0053. URL http:
7/arxiv. org/ abw//a/4...2 14863
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caro- line Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, and Mark Wildon. FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI, August 2025. StRthhttt://axiiieee www.iiiwwwg/abaa.oooooooe abs/ 24 04872
S. R. De Groot and P. Mazur. Non- -ou-iquili Thermodynanics. Courier Corporation, January 2013. ISBN 978-0-486-15350-6
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset, November 2021. URL htllttttp///aaat wwtpaaawwwww..ra////aa/aaaa httgg//b/211033
Huggingface. ber 2025. URL
AronBenchmark3222ennmhmaa data Datasets at Hugging Face, Septem-
co /ttp//ttsstsss/////oso
benchhmar rk canccharrk_dtaa
11

---

Ryogo Kubo, Morikazu Toda, and Natsuki Hashitsume. Statistical Physics II: Nonequilibrium Statistical Mechanics. Springer Science & Business Media, December 2012. ISBN 978-3-642- 58244-8.
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB- Bench: Measuring Capabilitiies of Language Models for Biology Research, July 2024. URL
http://arxi: org/ attp:/bb//2400) /2407 10362
Benno Liebchen and Demian Levis. Chiral active matter. EPL, 139(6))67001, September 2022. ISSN 0295-5075.
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark, May 2024. URL
http://arxivv httt:://arxiv.org/bbbssss www.orgg////o.o..ee.../eeeeeee/e .00..//200..1229 12209
Ludovico Mitchener, Jon M. Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P. Wellawatte, Andrew White. Lorenzo Sani, and Samuel G. Rodriques. BixBench: A Comprehensive Benchmark for LLM-based Agents in Computational Biology, March 2025. URL
abs/2503. 00096
J. Tailleur, and F. van Wijland. Time irreversibility in active matter, from micro to macro. Nat Rev Phys, 4()):177-883, March 2022. ISSN 2522-5820. URL https
O'Byme, Y. Kafri,
/www. nature. com/ art werticclsseee 021---2225-000000----000000-0000
Haining Pan, Nayantara Mudur, William Taranto, Maria Tikhanovskaya, Subhashini Venugopalan, Yasaman Bahri, Michael P. Brenner, and Bun-Ah Kim. Quantum many-body physics calculations with large language models. Commun Phys, 8 8()):99, January 2025. ISSN 2399-3650. URL
https: www. nature. www.cooaaraaa. ccan/atticess/ 4 0000000025-00000 01956-6
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Tung Nguyen, Daron Anderson, Imad Ali Shah, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Jaeho Lee, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, Robert Gerbicz, John- Clark Levin, Serguei Popov, Fiona Feng. Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Mstyslav Kazakov, Geoff' Galgon, Johannes Schmitt, Alvaro Sanchez, Yongki Lee, Will Yeadon, Scott Sauers, Mare Roth, Chidozie Agu, Søren Riis, Fabian Giska, Saiteja Utpala, Antrell Cheatom, Zachary Giboney, Gashaw M. Goshu, Sarah-Jane Crowson, Mohinder Maheshbhai Naiya, Noah Burns, Lennart Finke, Zerui Cheng, Hyunwoo Park, Francesco Fournier-Facio. Jennifer Zampese, John B. Wydallis, Ryan G. Hoerr Mark Nandor, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Junghae Nam, Edwin Taylor, Jun Jin, Gautier Abou Loume, Hangrui Cao, Alexis C. Garretson, Damien Sileo, Qiuyu Ren, Doru Cojoc, Pavel Arkhipov, Usman Qazi, Aras Bacho, Lianghui Li. Sumeet Motwani, Christian Schroeder de Witt, Alexei Kopylov, Johannes Veith, Eric Singer, Paolo Rissone, Jachyeok Jin, Jack Wei Lun Shi, Chris G. Willcocks, Ameya Prabhu, Longke Tang, Kevin Zhou, Emily de Oliveira Santos, Andrey Pupasov Maksimov, Edward Vendrow, Kengo Zenitani, Joshua Robinson, Aleksandar Mikov, Julien Guillod, Yuqi Li, Ben Pageler, Joshua Vendrow, Vladyslav Kuchkin, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Andrew Gritsevskiy, Dakotah Martinez, Nick Crispino, Dimitri Zvonkine, Natanael Wild- ner Fraga, Saeed Soori, Ori Press, Henry Tang, Julian Salazar, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, T. Ryan Rogers, Wenjin Zhang, Ross Finocchio, Bikun Li, Jinzhou Yang. Arun Rao, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Ariel Ghislain Kemogne Kamdoum, Tad Hogg, Alvin Jin, Carlo Bosio, Gongbo Sun, Brian P. Coppola, Haline Heidinger, Rafael Sayous, Stefan Ivanov, Joseph M. Cavanagh, Jiawei Shen, Joseph Marvin Imperial, Philippe Schwaller, Shaipranesh Senthilkuma, Andres M. Bran, Andres Algaba, Brecht Verbeken, Kelsey Van den Houte, Lynn Van Der Sypt, David Noever, Lisa Schut, Ilia Sucholutsky, Evgenii Zheltonozhskii, Qiaochu Yuan, Derek Lim, Richard Stanley, Shankar Sivarajan, Tong Yang, John Maar, Julian Wykowski, Martí Oller, Jennifer Sandlin, Anmol Sahu, Cesare Giulio Ardito, Yuzheng Hu, Felipe Meneguitti Dias, Tobias Kreiman, Kaivalya Rawal, Tobias Gareia Vilchis, Yuexuan Zu. Martin Lackner, James
12

---

Koppel, Jeremy Nguyen, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao, Pierrot Arsene, Sergey Ivanov, Rafat Pošwiata, Chenguang Wang, Daofeng Li, Donato Crisostomi, Ali Dehghan, Andrea Achilleos, John Amoll Ambay, Benjamin Myklebust, Archan Sen, David Perrella, Nurdin Ka- parov, Mark H. Inlow, Allen Zang, Kalyan Ramakrishnan, Daniil Orel, Vladislav Poritski, Shalev Ben-David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Dan Bar Hava, Aleksey Kuchkin, Robert Lauff. David Holmes, Frank Sommerhage, Anji Zhang, Richard Moat, Keith Schneider. Daniel Pyda. Zakayo Kazibwe, Mukhwinder Singh, Don Clarke, Dae Hyun Kim, Sara Fish, Veit Elser, Victor Efren Guadarrama Vilchis, Immo Klose, Christoph Demian, Ujjwala Anantheswaran, Adam Zweiger, Guglielmo Albani, Jeffery Li, Nicolas Daans, Maksim Radionov, Václav Rozhoñ, Vincent Ginis, Ziqiao Ma, Christian Stump, Jacob Platnick, Volodymyr Nevirkovets, Luke Basler, Marco Piccardo, Niv Cohen, Virendra Singh, Josef Tkadlec, Paul Rosu, Alan Goldfarb, Piotr Padlewski, Stanislaw Barzowski, Kyle Montgomery, Aline Menezes, Arkil Patel, Zixuan Wang. Jamie fucker-Foltz. Jack Stade, Declan Grabb, Tom Goertzen, Fereshteh Kazemi. Jeremiah Milbauer, Abhishek Shukla, Hossam Elgnainy, Yan Carlos Leyva Labrador, Hao He, Ling Zhang, Alan Givré, Hew Wolff, Gözdenur Demir, Muhammad Fayez Aziz, Younesse Kaddar, Ivar Ängquist, Yanxu Chen, Elliott Thorney,, Robin Zhang, Jiayi Pan, Antonio Terpin, Niklas Muennighoff, Hailey Schoelkopf, Eric Zheng, Avishy Carmi, Jainam Shah, Ethan D. L. Brown, Kelin Zhu, Max Bartolo, Richard Wheeler, Andrew Ho, Shaul Barkan, Jiaqi Wang, Martin Stehberger, Egor Kretov, Peter Bradshaw. J. P. Heimonen, Kaustubh Sridhar, Zaki Hossain, Ido Akov, Yury Makarychev, Joanna Tam, Hieu Hoang. David M. Cunningham, Vladimir Goryachev, Demosthenes Patramanis, Michael Krause, Andrew Redenti, David Aldous, Jesyin Lai. Shannon Coleman, Jiangnan Xu, Sangwon Lee, Ilias Magoulas, Sandy Zhao, Ning Tang. Michael K. Cohen, Micah Carroll, Orr Paradise, Jan Hendrik Kirchner, Stefan Steinerberger, Maksym Ovchynnikov, Jason O. Matos, Adithya Shenoy, Michael Wang. Yuzhou Nie, Paolo Giordano, Philipp Petersen, Anna Sztyber-Betley, Paolo Faraboschi, Robin Riblet, Jonathan Crozier, Shiv Halasyamani, An- tonella Pinto, Shreyas Verma, Prashant Joshi, Eli Meril, Zheng- Xin Yong, Allison Tee, Jérémy Andréoletti, Orion Weller, Raghav Singhal, Gang Zhang, Alexander Ivanov, Seri Khoury, Nils Gustafsson, Hamid Mostaghimi, Kunvar Thaman, Qijia Chen, Tran Quoe Khánh, Jacob Loader, Stefano Cavalleri, Hannah Szlyk, Zachary Brown, Himanshu Narayan, Jonathan Roberts, William Alley, Kunyang Sun, Ryan Stendall, Max Lamparth, Anka Reuel, Ting Wang, Hanmeng Xu, Pablo Hernández-Cámara Cámara, Freddie Martin, Thomas Preu, Tomek Korbak, Marcus Abramovitch, Dominic Williamson, Ida Bosio, Ziye Chen, Biró Bálint, Eve J. Y. Lo, Maria Inês S. Nunes, Yibo Jiang. M. Saiful Bari, Peyman Kassani, Zihao Wang, Behzad Ansarinejad, Yewen Sun, Stephane Durand, Guillaume Douville, Daniel Tordera, George Balabanian, Earth Anderson, Lynna Kvis- tad, Alejandro José Moyano, Hsiaoyun Milliron, Ahmad Sakor, Murat Eron, Isaac C. McAlister, Andrew Favre D. O. Shailesh Shah, Xiaoxiang Zhou, Firuz Kamalov, Ronald Clark, Sherwin Abdoli, Tim Santens, Harrison K. Wang, Evan Chen, Alessandro Tomasiello, G. Bruno De Luca, Shi-Zhuo Looi, Kinh-Kha Le, Noam Kolt, Niels Mündler, Avi Semler, Emma Rodman, Jacob Drori, Carl J. Fossum, Luk Gloor, Milind Jagota, Ronak Pradeep. Honglu Fan, Tej Shah, Jonathan Eicher, Michael Chen, Kushal Thaman, William Merrill, Moritz Firsching, Carter Harris, Stefan Ciobâcä, Jason Gross, Rohan Pandey, Ilya Gusev, Adam Jones, Shashank Agnihotri, Pavel Zhelnov, Siranut Usawasutsakorn, Mohammadreza Mofayezi, Alexander Piperski, Marc Carauleanu, David K. Zhang, Kostiantyn Dobarskyi, Dylan Ler, Roman Leventov, Ignat Soroko, Thorben Jansen, Scott Creighton, Pascal Lauer, Joshua Duersch. Vage Taamazyan, Dario Bezzi, Wiktor Morak, Wenjie Ma, William Held, Tran Duc Huy, Ruicheng Xian, Armel Randy Zebaze, Mohanad Mohamed, Julian Noah Leser, Michelle X. Yuan, Laila Yacar, Johannes Lengler, Katarzyna Olszewska, Hos- sein Shahrtash, Edson Oliveira, Joseph W. Jackson, Daniel Espinosa Gonzalez, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Dasouqi, Alexander Shen, Emilien Duc, Bita Golshani, David Stap, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Lukas Lewark, Miguel Orbegozo Rodriguez, Mátyás Vincze, Dustin Wehr, Colin Tang, Shaun Phillips, Fortuna Samuele, Jiang Muzhen, Fredrik Ekström, Angela Hammon, Oam Patel, Faraz Farhidi, George Medley, Forough Mohammadzadeh, Madellene Peñaflor, Haile Kassahun, Alena Friedrich, Claire Sparrow, Rayner Hernandez Perez, Taom Sakal, Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Kenchi Okutsu, Mike Battaglia, Mohammad Maghsoudimehrabani, Alon Amit, Dave Hulbert, Roberto Pereira, Simon Weber, Handoko, Anton Peristyy, Stephen Malina, Samuel Albanie, Will Cai, Mustafa Mchkary, Rami Aly, Frank Reidegeld, Anna-Katharina Dick, Cary Friday, Jasdeep Sidhu, Hassan Shapourian, Wanyoung Kim, Mariana Costa, Hubeyb Gurdogan, Brian Weber, Harsh Kumar, Tong Jiang, Arunim Agarwal, Chiara Ceconello, Warren S. Vaz, Chao Zhuang, Haon Park, Andrew R. Tawfeek, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan
13

---

Kim, Johan Ferret, Yuzhou Wang, Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Abram Jackson, Shreen Gul, Gunjan Chhablani, Zhehang Du, Adrian Cosma, Jesus Colino, Colin White, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Petr Spelda, Vit Stritecky, Syed M. Shahid, Jean-Christophe Mourrat, Lavr Vetoshkin, Koen Sponselee, Renas Bacho, Florencia de la Rosa, Xiuyu Li, Guillaume Malod, Leon Lang, Julien Laurendeau, Dmitry Kazakov, Fatimah Adesanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Julien Degorre, Yigit Yalı,, Gbenga Daniel Obikoya, Luca Amnbolli, Rai, Filippo Bigi, M. C. Boscá, Oleg Shumar, Kaniuar Bacho, Pierre Clavier, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mildred Tanwie, Denis Peskoff, Thomas C. H. Lux, Ben Rank, Colin Ni, Matthew Brooks, Alesia Yakimchyk, Huanxu, Liu, Olle Häggström, Emil Verkama, Hans Gundlach, Leonor Brito-Santana, Brian Amaro, Vivek Vajipey, Rynaa Grover, Yiyang Fan, Gabriel Poesia Reis e Silva, Linwei Xin, Yosi Kratish, Jakub Lucki, Wen-Ding Li, Sivakanth Gopi, Andrea Caciolai, Justin Xu, Kevin Joseph Scaria, Freddie Vargus, Farzad Habibi, Long, Lian, Emanuele Rodolå, Jules Robins, Vincent Cheng, Tony Fruhauff, Brad Raynor, Hao Qi. Xi Jiang, Ben Segev, Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Michael P. Brenner, Mao Mao, Xinyu Zhang, David Avagian, Eshawn Jessica Scipio, Alon Ragoler, Justin Tan, Blake Sims, Rebeka Plecnik, Aaron Kirtland, Omer Faruk Bodur, D. P. Shinde, Zahra Adoul, Mohamed Zekry, Ali Karakoe, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina. Juan Carlos Gonzalez. Gabe Maayan, Sarah Hoback, Rodrigo De Oliveira Pena, Glen Sherman, Elizabeth Kelley, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu, Sandra Mendoza. Ismail Alarab, Joshua Cole. Danyelle Ferreira, Bryan Johnson, Mohammad Safdari, Liangti Dai, Siriphan Arthornthurasuk, Alexey Pronin, Jing Fan, Angel Ramirez-Trinidad Ashley Cartwright, Daphiny Pottmaier, Omid Taheri, David Outevsky, Stanley Stepanic, Samuel Perry, Luke Askew, Raúl Adrián Huerta Rodríguez, Ali M. R. Minissi, Sam Ali, Ricardo Lorena, Krishnamurthy Iyer, Arshad Anil Fasiludeen, Sk Md Salauddin, Murat Islam, Juan Gonzalez, Josh Ducey, Maja Somrak, Vasilios Mavroudis, Eric Vergo, Juehang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Anil Radhakrishnan, Antoine Jallon, I. M. J. McInnis, Pawan Kumar, Laxman Prasad Goswami, Daniel Bugas, Nasser Heydari, Ferene Jeanplong, Archimedes Apronti, Abdallah Galal, Ng Ze-An, Ankit Singh, Joan of Arc Xavier, Kanu Priya Agarwal, Mohammed Berkani, Benedito Alves de Oliveira Junior, Dmitry Malishev, Nicolas Remy, Taylor D. Hartman, Tim Tarver, Stephen Mensah, Javier Gimenez, Roselynn Grace Montecillo, Russell Campbell, Asankhaya Sharma, Khalida Meer, Xavier Alapont, Deepakkumar Patil, Rajat Maheshwari, Abdelkader Dendane, Priti Shukla, Sergei Bogdanov, Sören Möller, Muhammad Rehan Siddiqi, Prajvi Saxena, Himanshu Gupta, Innocent Enyekwe. Ragavendran P. V. Zienab EL- Wasif, Aleksandr Maksapetyan, Vivien Rossbach, Chris Harjadi, Mohsen Bahaloohoreh, Song Bian, John Lai, Justine Leon Uro, Greg Bateman, Mohamed Sayed, Ahmed Menshawy, Darling Duclosel, Yashaswini Jain, Ashley Aaron, Murat Tiryakioglu, Sheeshram Siddh, Keith Krenek, Alex Hoover, Joseph McGowan, Tejal Pat- wardhan, Summer Yue, Alexandr Wang. and Dan Hendrycks. Humanity's Last Exam, February 2025. URL BRLLttt:://ae
org/ abs 2501 2541.
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. SPIQA: A Dataset for Multi- modal Question Answering on Scientific Papers, January 2025. URL http:://axxiv.oor
abs/ 09413
Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunho Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yixuan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Jiaming Ji, Weike Wang. Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing-Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia, Jiawei Lin, Zheyu Shen, Jiahang Chen, Qiuhao Xiong, Binran Wang, Fengyuan Wang. Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing-Hong Cao, Ming-xing Luo, Yaodong Yang, Muhan Zhang, and Hua Xing Zhu, PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models, May 2025. URL /ttp://arxivv org/ hbt://bs /bw..//52.....55 16074
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A Graduate-Level Google Proof Q&A Benchmark, November 2023. URL
org abs/ /atps///311... 12022
James V. Roggeveen, Erik Y. Wang, Will Flintoft, Peter Donets, Lucy S. Nathwani, Nickholas Gutier- rez, David Ettel, Anton Marius Graf, Siddharth Dandavate, Arjun Nageswaran, Raglan Ward, Ava
14

---

Williamson, Anne Mykland, Kacper K. Migacz, Yijun Wang, Egemen Bostan, Duy Thuc Nguyen, Zhe He. Mare L. Descoteaux, Felix Yeung, Shida Liu, Jorge García Ponce, Luke Zhu, Yuyang Chen, Ekaterina S. Ivshina, Miguel Fernandez, Minjae Kim, Kennan Gumbs, Matthew Scott Tan, Russell Yang. Mai Hoang, David Brown, Isabella A. Silveira, Lavon Sykes, Ahmed Roman, William Fredenberg, Yiming Chen, Lucas Martin, Yixing Tang, Kelly Werker Smith, Hongyu Liao, Logan G. Wilson, Alexander Dazhen Cai, Andrea Elizabeth Biju, and Michael P. Brenner. HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of Graduate Class, May 2025. URL hattp://azziie
250/ abs 2505. 11774
Michael te Vrugt, Benno Liebchen, and Michael E. Cates, What exactly is 'active matter"?, July 2025.
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma. Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng. SciCode: A Research Coding Benchmark Curated by Scientists, July 2024. URL www.rxxvv.rgggb.... /b0p//////222220.. 13168
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong, Solving olympiad geometry without human demonstrations. Nature, 625(7995);476-482 January 2024. ISSN 1476-4687.
URL https://www www.ur..ooooaaaoaa..aooooo wtt.aat....oooorr....eee///aaaa.
Weida Wang, Dongchen Huang, Jiatong Li, Tengehao Yang, Ziyang Zheng, Di Zhang, Dong Han, Benteng Chen, Binzhao Luo, Zhiyu Liu, Kunling Liu, Zhiyuan Gao, Shiqi Geng, Wei Ma, Jiaming Su, Xin Li, Shuchen Pu, Yuhan Shui, Qianjia Cheng, Zhihao Dou, Dongfei Cui, Changyong He, Jin Zeng, Zeke Xie, Mao Su, Dongzhan Zhou, Yuqiang Li, Wanli Ouyang, Yunqi Cai, Xi Dai, Shufei Zhang, Lei Bai, Jinguang Cheng, Zhong Fang, and Hongming Weng. CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics, August 2025.
URL htthhhtt://aatt
org/ abs/ /2bss//55... 18124
Yinggan Xu, Yue Liu, Zhiqiang Gao, Changnan Peng. and Di Luo. PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models, May 2025. URL http://
wxxiv.org/aie www.v..rggabb//55..coo 24823
A INSIGHTS
We share some insights from the problem authors on how they designed their problems and the typical failure modes of the LLMs for four different problems below.
A.1 CASE STUDY: CHARGE DENSITY WAVE IN TRIANGULAR LATTICE
In the first HF problem from our spreadsheet, we asked for the ground-state energy per site of commensurate charge density wave on a 2D triangular lattice half filling with onsite and nearest- neighbor repulsions Uo and U1 in the strong-coupling limit (interaction energy scaling >> hopping). This problem requires the LLM first to understand the charge ordering in superlattice embedded in base triangular lattice. Then it should propose several possible candidate states at half-filling. Finally, it should correctly compute the energy per site and determine the lowest energy state as the ground state. Frontier LLMs consistently failed at the first step identify the correct ordering in the first place. We show outputs of the four models, Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro. and GPT-4o, in the color box below.
15

---

Model outputs (excerpts) and failure diagnosis
Gemini 2.0 Flash: the strong coupling limit, treat hopping as a perturbation and minimize
E = Uo Ei Deg + U1 Eūss) neng, with n4 nutt + na4- At half-filling on the triangular lattice, a v3x / CDW with occupancies (0, 1,,1,2) per tri-sublattice gives / -Vo/ and (by balanced-neighbor count) Ev,/N 2U1, hence E/N Uo/3 + 2U1. The
uniform state has E//N 3U1, so the CDW is favored.
Un 3 + 2U1
Gemini 2.5 Flash: We need to consider possible ground states that minimize the repulsion. Case I: Uniform charge density (Mott insulator). No double occupancy, so Eu, = 0. (and thus 3N bonds), Ev, = 3N giving E site 3U,. Case 2: Commensurate CDW (v3 v3). Partition into A,B, sublattices with occupancies (2, L,0) maintain average 1. Then EU, (N/3)Up and Eu, 2NU so E site Ug/3 2U2. Comparing, the CDW is favored if Uo 3U1. Therefore the commensurate U/ 3
With coordination
CDW energy per site is Uo/3 2U,.
+ 2U1
Gemini 2.5 Pro: We need compare the energies of plausible charge configurations. Let N₂ be the total number of sites. Uniform Mott Insulating State: One electron per site (n4 for all i). 2. Charge Density Wave (CDW) State: The electron density is spatially modulated. At half-filling, a CDW state involves having some sites with zero electrons (empty, n, =0) and some sites with two electrons (doubly occupied, n, 2). To maintain the average filling of one, the number of empty sites must equal the number of doubly-occupied sites (No- N2). There can also be singly occupied sites (n; = 1). The ground state CDW on triangular lattice for this model is the v3 v3 ordered state. This structure arises from partitioning the lattice into three interpenetrating sublattices (A, B, C), each containing
Un 2U1
N sites... ... Therefore, this the ground state energy.
GPT-4o: Each site, being half-filled, will have average occupation of electron. For the onsite repulsion energy, each electron contributes an energy Up because the remaining fraction corresponds to preventing double occupancy that provides the remaining 2 per site.
Uo
All models still fail to display the correct stripe ordering. Gemini 2.0 and GPT-4o proposes v3x v3: Gemini 2.5 Flash and Gemini 2.5 Pro compare uniform vs CDW and outputs the same expression; The correct ground state is commensurate stripe CDW at half filling with
no double occupancy; in the strong-coupling limit, only Un contributes, giving E/N = 1/1 2
We attribute this to two factors: (i) a language-geometry gap, wherein the model reasons fluently over symbols but does not natively reconstruct 2D lattice structure and commensurability (leading to suggest incompatible two-sublattice or v3x v3 patterns), and (ii) distributional bias in training data, as trianiular-lattice examples are less common than square-lattice ones in standard expositions, and the even the 'stripe phase' embedded in the triangular lattice at half-filling is even less common to appear in the textbook compared to extended v x v3 pattern phase, so the model does not robustly internalize the associated counting logic and constraints.
A.2 CASE STUDY: PHYSICS-INFORIE GROUP THEORY
The second example falls under the ED category. We asked the model to count the number of symmetry-distinct sectors in a Hilbert space, considering two-dimensional lattice system with 3x3 unit cells, where each unit cell consists of one site that can be empty, occupied by a spin-1/2 electron with either spin, doubly occupied with two electrons of opposite spin. The system has periodic boundary conditions in both directions. The system has translation symmetry, time-reversal symmetry, and fourfold rotation symmetry. The LLMSs responses shown below.
16

---

Model outputs (excerpts) and failure diagnosis
Gemini 2.0 Flash:
..TTime-reversa symmetry means that (kk) (-k This doesn't i
give any additional symmetry-distinct blocks...
3
Gemini 2.5 Flash:
...H contains subspaces with both even and odd N
9
Gemini 2.5 Pro: We assume that N₂ is conserved, so the Hamiltonian is blockdiaggoal in Ne.
86
GPT-4o: Counts 9 translation momenta (Zg Zz), includes fourfold rotations (CA), and then heuristically halves by time reversal, concluding there are 9x4 18 symmetry-distinct
blocks.
18
In Gemini 2.0 Flash, the LLM completely ignored one of the symmetries (rotation symmetry) and thus could not reason with time-reversal symmetry either. This is rather blunt failure. In Gemini 2.5 Flash and Pro, both LLMs ignore that particle number or parity conservation was not mentioned symmetry in the problem statement, and it was explicitly excluded that non- mentioned symmetries are to added. Between the two, adding fermion parity conservation more subtle mistake, as it is often underlying assumption in quantum mechanical descriptions without being explicitly stated. The fermion number, however, is frequently not conserved in Hamiltonians of interest, for instance, when describing superconductors. Gemini 2.5 Pro thus fails more drastically. GPT-4o instead multiplies translation and rotation counts and halves by time reversal to obtain 18, which also does not correspond to the correct block structure.
The LLMs gravitate towards solving changing its assumptions to cases that are more prevalent in the literature (concretely, they have added symmetries). Furthermore, they frequently respond to problem hardness with lengthy output of dense, nearly cryptic text.
standard problem, if problem appears hard, by
A.3 CASE STUDY: ENTROPY PRODUCTION AND FLUCTUATION DISSIPATION THEOREM
third example falls under the Statistical Mechanics category. Here, LLM is asked to consider peculiar type of overdamped Langevin equation in two dimensions under the action of external potential, and to determine whether the fluctuation dissipation theorem (Kubo et al. 2012 Groot Mazur 2013) is violated. The options available the model are
(a) Yes, because the dynamics has a positive entropy production rate.
(b) No, because the dynamics time-reversible.
No, because the Boltzmann distribution is the stationary distribution.
(d) Yes, because there is nonzero self-propulsion speed.
The dynamics proposed resemble those of a chiral active Brownian particle (Liebchen & Levis 2022 which has net-nonzero entropy production rate and is therefore iime-ireeersibbe. However, the frequency of rotation of the self-propulsion direction is chosen in such a way to allow sampling from the Boltzmann distribution with a target temperature and potential energy in the steady state (Ghimenti et al. 2024). Thus, instantaneous snapshots of the particle position look like equilibrium snapshots, even though the dynamics of the particle are out of equilibrium. The simultaneous appearance of these two seemingly contradictory properties is what makes the problem subtle. Indeed, theorem
the steady state of the system has the Boltzmann form, then the flucuation-dissipptii obeyed. This is standard result in nonequilibrimm thermodynamics (Groot & Mazur (013). The correct answer then (c). However, the models fail to appreciate this connection and focus on the nonreversibility of the dynamics and the presence of net nonzero entropy production rate. These aspects are ubiquitous in the field of active matter (0 (O'Byree et al.) 2022 te Vrugt et al., (2025), where the fluctuation -issiipaioon theorem is often violated. Instead of carefully analyzing the proposed dynamics, the models appeal directly to general facts known in the field, leading to an incorrect conclusion.
17

---

Model outputs (excerpts) and failure diagnosis
The given dynamics is that of an active Brownian particle in two di- mensions... Because active Brownian particles consume energy from their environment to maintain their pelf-propulsion, they are intrinsically out of equilibrium. As such, they have non-zero entropy production rate in the steady state. This nonequilibrium nature means that the standard centtuationddisiiaaaii theorem (FDT) violated.
Gemini 2.0 Flash:
Gemini 2.5 Flash: The problem describes active Brownian particle (ABP) model... the particle consumes energy to maintain its persistent motion. Systems driven by active forces inherently *out of thermal equilibrium* ... the stationary distribution of active Brownian particle is generally not the Boltzmann distribution.. described is unequivocally out of thermal equilibrium. For any nonequilibrium system, the Fluctuation Disiiinniiiiiiiiiinnnn Theorem is generally violated.
the system
Gemini 2.5 Pro: A system in thermal equilibrium if satisfies the principle of detailed balance. In steady state, this implies that all microscopic probability currents are zero. Active matter systems, by their very nature, are not in thermal equilibrium... Systems with broken detailed balance in a steady state have continuously positive rate of entropy production... Since the active particle system a NESS with a positive entropy production rate, it does not meet the fundamental requirements for the FDT to hold. Therefore, the FDT violated.
GPT-4o: Parses the ABP equations and emphasizes that self self-propulsion (tu > 0) drives a non-equilibrium steady state with broken detailed balance and a non non-Boltzmane stationary distribution; the antisymmetric A term renders the dynamics non-time teve-reversible. Concludes the FDT is violated.
All four models directly appeal to general conditions and known properties of active matter systems, without analyzing in depth the specific dynamics proposed. Moreover all the models claim that a positive entropy production rate leads to the violation of the fluctuation- dissipation theorem. Most cases appearing in the literature adhere to this statement, but the implication is not always true, as the proposed problem exemplifies.
A.4 CASE STUDY: FERMI SURFACE TOPOLOGY
The model is asked about the number of connected components in the Fermi surface of given tight-binding model. Specifically, it is a two-dimensional square-lattice tight-binding model with staggered flux of 0 per plaquette. Although such a tight-binding model has been studied in the literature, the Al models do not seem to be familiar with it. To solve this problem, one needs to write down the Hamiltonian in k-space and then diagonalize models we tried failed at this step. Although they do realize that we need a unit cell with two sites, resulting in 2 x Hamiltonian the momentum space, no model has correctly written down the Hamiltonian. It appears that the models fail to visualize the connectivity of the lattice (i.e., which sites are neighbors of each other) and also fail to recognize that the unit vector the lattice needs to be modified due to the breaking of translation symmetry. When attempting other similar questions, we also observed that the system sometimes cannot correctly identify the topology of the Fermi surface. Solving such problems typically requires humans visualize the energy dispersion and equal-energy contours, either mentally or by plotting them on a computer. It appears that this kind of spatial reasoning remains weakness of current models.
obtain the energy dispersion. All
18

---

Model outputs (excerpts) and failure diagnosis
Gemini 2.0 Flash: Consider the unit cell as two sites, A and B. The sites are (nz, By) = (2m, 2n.) and (2m 1.) Then A (2m, 2n), B (2m + 2n). Then A + A I, A + A+11 Ú ÿj B + B+ I. B + B + D. The first nearest neighbor to A is A+x. second nearest A+y which satisfies that R2 + nly even. The first nearest neighbor to B is B+x. second nearest is B+y which satisfies that f2 + ny is even. Let's 's try this: 4
1,0 n/3. The
dispersion is E(k) - cos Kig- 2 cos ky.
Gemini 2.5 Flash: This problem exhibits a checkerboard pattern for the y y-hopping. which means the unit cell needs be doubled. We can define unit cell containing two sites, for example, (Nz, nyy) and (Nz; ny However, the y y-hopping depends
as well, implying 2 unit cell in general. common approach for this rype of staggered flux problem is to use gauge transformation Hamiltonian.
to double the unit cell in one direction, yielding 2 x 2
by considering a 2 x superlattice with primitive vectors (1,0) and (0,2), the problem simplifies to two-band model. The specific gauge transformation Cz.y dr. e" (-11)"y can simplify the y- hhopiin. After performing the Fourier transform, the resulting 2 Hamiltonian has the -12(1 + e c-0k-))))) -24 cos kiz
By applying suitable gauge transformation (e.g., Cx.y - eddx.vd..)
-211 CO8 kn form: (k)) -122(1 e-10 (2thg)
Gemini 2.5 Pro: find h(k)), we sum the hopping terms from a site on sublattice A to its neighboring sites sublattice weighted by phase factors. Let's place A-site at the origin (0, 0). Its neighbors are at (1,0), (-1,0), (0, 1), (0, -1), which are all B-sites. The hopping terms are:
From (0, 0) to (1,0): -11- Phase factor: etkez.
From (0, 0) to (-1,0): 0): -t1. Phase factor; e iko.
From 0) to (0, 1): The starting site (0, 0) has n2 + + = 0 (even), so the hopping
is t2. Phase factor: etky.
From (0, 0) (0, 1)): This the Hermitian conjugate of the hopping from (0. -1
to (0, 0). The site (0, -1) } has + ny = -1 (odd), so the hopping from (0, -1) to
(0,0) 0) is - +0 Thus, the hopping from (0, to (0, -1) is -12 Phuse factor: e-
Summing these up gives: h(k) +e iks - the -0 eikk iky -21 cos(k()) tz(e.hy (kk,
GPT-40: To solve this problem, we first need to write down the dispersion relation for the given hight-binding model. The tight-binding Hamiltonian on the square lattice with the specified hopping terms can be given as:
H ==
(elcfore + h.c.) t
(-+ (efor-o + ( Σ ofep
1 (1+
2
The dispersion relation is obtained by performing a Fourier transform:
1 (1 (-1)ax ei
c(kz, -2t, cos(Riz) 2tz
cos(Riy) 1
2
Gemini 2.0 Flash directly writes down the energy dispersion of the tight-binding model without flux, which suggests the more familiar results misled it. Gemini 2.5 Flash and Gemini 2.5 Pro made the right attempt to construct 2 x 2 Hamiltonian, but failed to identify the correct terms. For example, the hopping from the A-site at (0,0) 0) to the B-site at (1,0)
should correspond to an off-diagonal term (9
leekk in the Hamiltonian, which no model
produced correctly. GPT-4o confused about more basic fact: the coordinates n2z, Fl should not appear after the Fourier transformation to momentum space.
19


Reviews:
[{"review_id": "Y25cFAF07T", "reviewer": "Reviewer_Fftv", "summary": "The paper curated a benchmark including 50 problems on condensed matter that are generated by domain experts. The benchmark covers a wide range of topics in the field and requires high levels of understanding and expertise to solve the problems. The LLMs perform badly on the benchmark.", "questions": "1. Have the authors tried to fine tune some LLMs to learn to solve problems as challenging as the benchmark?\n\n2. Would it possible to include reasoning process into the benchmark that can serve as the prompt or a reasoning benchmark?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The curated benchmark involves experts inputs and is carefully designed and evaluated.\n\n2. The benchmark covers a wide range of topics in the condensed matter field.", "weaknesses": "1. The models evaluated are all general purpose LLMs that have not been fine tuned in the field of condensed matter. Therefore, they are unlikely to perform well on this challenging benchmark. The authors may want to fine tune a model on relevant tasks and then evaluate it on the benchmark to see if performance can be improved.\n\n2. The benchmark only includes question and answer, without reasoning process. For such complicated tasks, few-shot prompts and a CoT guide may improve the performance. I think this would be valuable to test.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozt8kfPQ26", "reviewer": "Reviewer_u4aj", "summary": "This paper presents CMT-Benchmark, a dataset of 50 problems and expert-annotated answers in condensed matter theory. A variety of large language models are evaluated on this benchmark dataset, and they show a high gap to a satisfying physical reasoning skills.", "questions": "Though it is released close to the ICLR submission date, I am interested in the comparison of CMT-Benchmark to another public benchmark CMPhysBench [1], which is also used to evaluate LLM in condensed matter physics. What are the major differences between them in the collected problems and answer annotation process?\n\n[1] Wang, Weida, et al. \"CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics.\" arXiv preprint arXiv:2508.18124 (2025).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Create a very high-quality dataset in condensed matter domain. This dataset could potentially be very useful in evaluating LLM's capability in solving scientific research problems.\n- Provide comprehensive benchmarking of multiple LLMs including GPT, Claude, DeepSeek and Llama family models.\n- The writing of this paper is good, providing details in data curation process and necessary background knowledge.", "weaknesses": "- To improve the quality of experiments, authors are encouraged to analyze the failure cases, demonstrating LLM models consistently fails on which types of problems and makes which types of mistakes.\n- As the benchmark is highly domain-specific, authors are encouraged to also evaluate the performance of web search agents (e.g., OpenAI & Tongyi DeepResearch) to show if LLM models could solve the problem through searching public Internet knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R3A61Z3W2f", "reviewer": "Reviewer_qany", "summary": "This paper introduces CMT-Benchmark, a dataset of 50 expert-level problems in condensed matter theory (CMT). The benchmark was built by an international panel of expert researchers at the level expected of strong grad students. The authors then built automated evaluation infrastructure, including a novel parser that can handle non-commutative operator expressions in CMT. The paper found that all frontier models have uniformly low performance in the benchmark.", "questions": "1. Can you design experiments to show that how much of the poor performance is due to formatting constraints versus actual physics reasoning failures?\n\n2. Can tool-augmented agents substantially improve performance on this benchmark?\n\n3. Is there evidence of training data contamination for any problems? How to design the experiments to test this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. I really like the creation of the benchmark process. By asking the human experts from different countries to submit the questions, i think this benchmark truly captures what it means to be an expert in CMT. Thus, it will be more convincing to believe that the progress in this benchmark will imply the progress in CMT research. I think this is a significant contribution to the community. \n\n2. The automated parsing and grading system is carefully-designed and quite impressive, particularly the handling of non-commutative operator algebra through symbolic manipulation and normal ordering.\n\n3. The four detailed case studies (Sections A.1-A.4) provide valuable insights into specific LLM limitations: language-geometry gaps, over-reliance on textbook heuristics, failure to apply fundamental principles, and weak spatial reasoning.", "weaknesses": "1. The paper only tests LLM capabilities without access to tools like web search, code executions or symbolic/numerical computation packages. However,  we know that even human research assistants don't work in isolation without any tool use. It would be interesting to test whether tool-augmented agents improve performance.\n\n2. Requiring answers in boxed LaTeX environments and prohibiting new variables may artificially hurt model performance. The authors note some models (particularly Gemini 2.5 Pro) \"occasionally disregard the formatting instructions,\" leading to parsing failures. How much does the strict format requirement degrade performance compared to free-form responses that could be human-evaluated on a subset?\n\n3. While the authors claim problems are original, there's no systematic verification that similar problems (or solution strategies) don't appear in training data or online.\n\n4. With only 50 problems and some categories having very few examples (PEPS has 3, VMC has 2), per-category performance can have large uncertainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJklbOlL2F", "reviewer": "Reviewer_VTk8", "summary": "The paper presents a new and very difficult benchmark for physics problems in condensed matter theory. It evaluates state-of-the-art LLMs and finds that they cannot solve most of these problems.The problems are categorized into different categories and created by experts from across the world. The benchmark is intended to help develop a research assistant grade AI assistant in this field.", "questions": "- do the problems have different difficulty levels or are they all at approximately the same level?\n- do the problems generalize well, is an LLM that is able to solve these problems expected to generally be good at solving physics problems?  - How do you assess the coverage of these 50 problems, are there redundancies or gaps?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- new benchmark dataset for a field in which data is lacking\n- should enable the development of stronger models for physics problems\n- the problems are checked and created by experts", "weaknesses": "- small dataset: there are only 50 problems because they are manually created by experts\n- no other weaknesses to be found", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)