PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning
Authors: Wanjia Zhao
Qinwei Ma
Jingzhe Shi
Shirley Wu
Jiaqi Han
Yijia Xiao
Si-Yuan Chen
Xiao Luo
Ludwig Schmidt
James Zou
Venue: ICLR 2026 Poster (2026)
Keywords: Physics Reasoning
Process-Level Evaluation
Symbolic Equivalence
Scientific Problem Solving
Primary area: datasets and benchmarks
TL;DR: We present PRISM-Physics, a benchmark and a process-level evaluation framework that encodes physics solutions as DAGs and employs rule-based symbolic equivalence checking for reliable, fine-grained scoring.

Abstract: Benchmarks for competition-style reasoning have advanced evaluation in mathematics and programming, yet physics remains comparatively underexplored. Most existing physics benchmarks evaluate only final answers, which fail to capture reasoning processes, while recent stepwise methods rely on heuristic LLM-as-judge scoring or restrictive linear assumptions, limiting reliability and diagnostic validity.
We introduce PRISM-Physics, a process-level evaluation framework and benchmark for complex physics reasoning problems. Solutions are represented as directed acyclic graphs (DAGs) of formulas, explicitly encoding causal dependencies among intermediate steps to enable fine-grained, interpretable, and theoretically grounded scoring. 
We prove the optimality of the DAG representation and the corresponding scoring policy. Combining with a fully rule-based method for symbolic formula equivalence matching that we developed, we ensure consistent validation across diverse formulations without heuristic judgments. Results show that our evaluation framework is more aligned with human experts' scoring. 
Experiments on state-of-the-art LLMs reveal persistent reasoning failures in physics, while step-level scoring offers both diagnostic insight and rich signals for later training. By combining structural rigor, theoretical guarantees, and symbolic validation, PRISM-Physics provides a principled foundation for advancing process-level evaluation and guiding the development of models with deeper scientific reasoning capabilities.

Published as a conference paper at ICLR 2026
PRISM-PHYSICS: CAUSAL DAG-BASED PROCESS EVALUATION FOR PHYSICS REASONING
Wanjia !
Qinwei Ma Jingzhe Shi Shirley Wu'
Jiaqi Han'
Yijia Xiao
Si-Yuan Chen
Xiao Luo
Ludwig Schmidt' James Zou'
Sttanford University
TTsinghua University University of California Los Angeles
'Harvard University
University of Wisconsin-Maddion
ABSTRACT
Benchmarks for competition-style reasoning have advanced evaluation in math- ematics and programming, yet physics remains comparatively explored. Most existing physics benchmarks evaluate only final answers, which fail to capture reasoning processes, while recent stepwise methods rely on heuristic LLM-as- judge scoring or restrictive linear assumptions, limiting reliability and diagnostic validity. We introduce PRISM-PHYYSICE and benchmark for complex physics reasoning problems. Solutions are repre- sented directed acyclic graphs (DAGs) of formulas, explicitly encoding causal dependencies among intermediate steps to enable fine-grained, interpretable, and theoretically grounded scoring. We prove the optimality of the DAG represen- tation and the corresponding scoring policy. Combining with fully nule-based method for symbolic formula equivalence matching that developed, consistent validation across diverse formulations without heuristic judgments. Re- sults show that our evaluation framework is more aligned with human experts' scoring. Experiments on state-of-the the-hhe-rt LLMs reveal persistent reasoning fail- ures in physics, while step-level scoring offers both diagnostic insight and rich signals for later training. By combining structural rigor, theoretical guarantees, and symbolic validation, PRISM-PHYSIER provides principled foundation for advancing process-level evaluation and guiding the development of models with deeper scientific reasoning capabilities.
process-level evaluation framework
ensure
Project Page: https: //op::///ppnnt prism.
10/PRISM wttph/u...o//RIM--PPeeeeeee
INTRODUCTION
Benchmarks for competition-style reasoning have advanced rapidly in mathematics (e.g., IMO) (Zheng et al., 2021; He et al., 2024; Gao et al., 2024) and programming (e.g.. IOI) (Shi al., 2024; Zhu al., 2025; El-Kishky al., 2025), providing comprehensive testbeds for evaluat- ing large language models (LLMs). In contrast, physics competitions remain comparatively under- served, despite requiring not only deep domain knowledge, but also advanced analytical modeling, multi-step symbolic derivation, and precise numerical computation. These skills fundamen- tal indicators of scientific reasoning ability, as they integrate conceptual understanding, modeling assumptions, and rigorous problem-solving under complex constraint (Jaiswal et al., 2024). Con- sequently, developing fine-grained evaluation frameworks and benchmarks for competition-level physics is essential for systematically assessing and advancing LLMs' capabilities in this critical domain (Chang et al., 2024; Song et al., 2025).
major obstacle that physics derivations are inherently non-linear: solutions frequently branch, merge, or reuse intermediate results. Existing benchmarks fail to capture this structure due to three key limitations:
Most physics benchmarks rely on multiple-choice or short-answer formats (Wang et al.; Rein et al., 2024), evaluating only the final answer and ignoring the reasoning process. Such formats obscure the reasoning process and provide limited diagnostic value for understanding model capabilities. 2. Most also rely on LLM-as-judge scoring (He et al., 2024; Xiang al., 2025), which is prone to hallucinations, prompt sensitivity, and inconsistent grading. 3. While some recent
"Equal contribution. Correspondence httpiiiih@@ssssaaffdd...o

---

Published as a conference paper at ICLR 2026
Free Ansaar Acdelic Risoonse Theee
53
Hoiseees CM6
11.7
244.5
211
5128
11.
211
cen Suntsphanl
Depp dt!
Gee Gees A27
Goode
meent
Figure I: Model performance on PRISM-Physics. We reported both Final-Answer Accuracy, Step- level Accuracy and Response Time,
work (Zhang et al., 2025) has made initial attempts at step-by-step scoring, these approaches often rely on strong assumptions such as strictly linear step ordering or shallow expression matching that limit the validity and generalizability of the framework. As a result, current methodologies remain inadequate for revealing the systematic reasoning failures of L.MMs in physics problem solving.
To address these limitations, we introduce PRISM-Physics, process-level evaluation framework and benchmark that represents physies solutions as directed acyelic graphs (DAGs) of formulas. This graph-based structure explicitly encodes causal dependencies among intermediate steps, enabling fine-grained scoring that is theoretically grounded and interpretable. To ensure consistency, we further develop a fully uulebbsedd symbolic equivalence checker, which provides robust validation across diverse formulations and removes dependence on heuristic judgments.
Our main contributions are summarized follows: 1. We construct large-scale benchmark of competition-level physics problems with carefully curated, DAG-structured solutions. 2. We pro- pose a DAG-based scoring policy that explicitly models causal dependencies among formulas, abling fine eee-grained and interpretable process-level evaluation. We further provide a theoretical proof of its optimality, showing that it minimizes evaluation ambiguity and aligns naturally with the logi- cal structure of physics derivations. 3. We develop a fully rule-based symbolic formula equivalence checker to reliably validate diverse mathematical expressions, ensuring consistent comparison across alternative formulations and climinating reliance on heuristic LLM-as-judge' scoring. 4. We con- duct extensive experiments on a broad range of LLMs, revealing persistent challenges in sustaining coherent reasoning chains and in correctly applying physical principles. Furthermore, we system- atically compare our evaluation framework with existing approaches, demonstrating its superior reliability, interpretability, and diagnostic power for evaluating process-level reasoning capabilities.
Taken together, PRISM-Physte -hysiiss establishes the first principled foundation for process-level evalua- tion in physics, bridging structural rigor, theoretical guarantees, and symbolic validation.
2 RELATED WORK
Physics problems, as a proxy of how LLMs understand Physics, have been used as benchmarks for LLMs in recent years. Especially, previous work has been using Physics Olympiad problems for benchmarking LLM reasoning and problem solving abilities. For instance, OlympiadBench (He al., 2024) aggregates problems from multiple Olympiads; SeePhys (Xiang et al., 2025) incorporates visual problems to study how visual ability improves LLM performance; PhyBench (Qiu et al., 2025) focuses on rigor and originality. While such benchmarks propose metrics (e.g., EED score (Qiu et al., 2025)), they still focus primarily on final answers and fail to represent or provide more fine- ree-ggrined process scores. More recently, process-base mrocess-based evaluation of LLM reasoning has become focus. PhysReason (Zhang et al., 2025) evaluates intermediate steps by checking correctness of expressions and assigning linear scores, but this approach is restricted to expressions (rather than equations) and cannot represent the dependency logic among steps.
Physics Benchmark.
2

---

Published as a conference paper at ICLR 2026
Diagram
Contest
DAG
Creader recengaks sakibbick wilk loggh S8 (deea ScS-dradian) oeciaal SAS the ars ofince paraled to applied teee cheir meddala Srdl (forta blocdt rustral) cenety Siiteh. tengstal stcring firs cxpppie areee fass, dssenga sanperiid dispocemenn
apper
pooducng
Sy5 d--ieccliout SxS and fine Sik You sloxed use SPS fo allaiing Sove. SAS Se for Sud for the slewe modaka, SS for the blok lengls Sub for kangetial diplaarenn Strappiis for shai eagle for family Stapaiked the speed and clutic ware.
Sub-questions
appike
laage inploeeee Sell surlic and the appindd livce 5PS
thy The ol pergerion of so sepy-n chaak waver houmm taninenno airr wave propagams in the fick diectionn with Barkons the SyS-dincctin Derna the apuativ moxion for the Sy-- diaddan Saplkarr and Sa0s whene the mormettil
so tied the tlor madike sod and dearty Stited, the speat Stapolard mono wake dicat parnty
Solutions
Hibuc' lms for decumayiiie (h Coarphis angental arcncc
Frurphis berte aegental boy wdiio tvepats Salinn
Median
Sirepled Siveeeght "ther 2014 1198 co,, outeeuunn sunuung dhi nonk perena mgarive give Sha Sneppes S0 (Swet x150
(1) Glack usago roraar, ju athealt11 uniu Strerppi Stre Ergertaat od pertta Then 501 1110 Viaii Sdoeciapnntt apripprte longerrdss
posetall... la (ittrwitti innapper
Plysks Demain
(parrnn aiffarnur loigha cas Aeunilagi ta Manddna TLJ1 (L.1").41 ass woe 1&S
Machunica
Feense a Inngaoris aves bo seters Rot naa 1119 33 anrr (e (newel pertad Expertal (nngarr (dtwet/abee dana vlowa Fearmal iy Engaor2 Vingag du (11) bipbigg the Exli Eagrage equatis S00 Spartaaa app (11) (athee patsc/parraaa 24 puntaa 1")
Koowtedgs
wytheeewwegeea w. 30 qpe wtp-taa (tra) pe pytttti Eqpenurrrg The mooeeenii vpeii Strapeales "agp (x
Figure 2: A data example with the proposed DAG structure.
LLM-as-Judge for Problem Solving, Reliable evaluation of physics problem solving requires assessing not only final answer correctness, but also the validity of intermediate reasoning steps. Human expert annotation, while generally reliable, is costly and unscalable in large-scale (Gu et al., 2024; Liu et al., 2023; Ye et al., 2025; Petrov et al., 2025; Mao et al., 2024). Automated LLM- as-judge methods have shown potential in mathematical and physics tasks, but are still susceptible to errors from implicit assumptions, symbolic manipulation errors, and misinterpretation of domain concepts (Zheng et al., 2023: Gu et al., 2024; Ye et al., 2025; Hendrycks et al., 2021; Gulati et al., 2024; Liu et al., 2024; Lu et al., 2024). This challenge is amplified in physics, where various physical concepts, constants, and equivalent formulations create many valid variations of the same expression, making judgment more difficult (Wang et al.; Xu et al., 2025; Zhang et al., 2025).
To address these issues, we introduce formula-based verification framework that directly compares symbolic expressions for physical and mathematical consistency, offering faster and more reliable alternative costly human annotation (Brence et al., 2023; Gao al., 2023; Chen al., 2023; Cobbe et al., 2021; Hendrycks et al., 2021; Xia et al., 2024; Gao et al., 2025; Li et al., 2024).
PRELIMINARY AND FORMULATION
3.1 FORMULA-MATCHINC BASED EVALUATION
A crucial component that distinguishes evaluation framework from most LLM-as-judge coun- terparts is that, ours is purely based on rule-based Formula Equivalence Matching, also called as Formula Matching. More specifically, we say two formulas matched if they are mathemati- cally equivalent. In Section 4.1 we will show how we actually match two formulas, and discuss our improvement against previous work.
3.2 MOTIVATION FOR THE DAG STRUCTURE
Naive process-scoring policies present inherent limitations: strict matching fails to recognize correct outcomes obtained through alternative derivations, whereas prefix credit overestimates performance by indiscriminately assigning credit to all prior steps once single formula is matched. To address these issues, we represent the reference solution as a directed acyelic graph (DAG) of formulas, where the edges encode explicit prerequisite relations. With this structure, credit propagates only along causal chains from matched nodes to their ancestors. This approach (i) avoids the harshness of strict matching by rewarding justified intermediate work, (ii) prevents the over-crediting of prefix policies by restricting propagation to prerequisites only, and (iii) offers a representation intuitive for human reasoning, reliable for L.M annotation, and theoretically complete under mild assumptions.
3

---

Published as a conference paper at ICLR 2026
3.3 DAG REPRESENTATION OF SOLUTIONS.
Formally, a directed acyclic graph (DAG) is pair G = (V, E), where V is finite set of nodes and E E is set of directed edges such that there no directed cycle G. That is, there is no sequence of distinct nodes 11, 02,.... Uy with k 2 2 satisfying (e,, 2/+1) E E for all 1 I i < & and (Uk, 01) E DAG thus encodes partial order over its nodes, which is particularly suitable for representing stepwise logical or computational dependencies.
In our setting. each solution is systematically converted into such a DAG: Nodes (formulas). Each V denotes a canonicalized 88T expression representing mathematically key step (e-g., physical law, derived intermediate equation, simplified relation). Canonicalization guarantees syn- tactic and semantic consistency across solutions. 2. Edges (dependencies). For (u, E E, formula v is derived from formula u. By construction, the edges reference only prior nodes, thereby ensur- ing temporal causal consistency and valid topological ordering aligned with natural reasoning. 3. Minimality, Redundant algebraic steps are removed, retaining only essential formulas, therehy yielding concise yet sufficient structure that captures the core reasoning trajectory. 4. Complete- ness. Every node must be connected by a directed path to at least one designated final answer node, Thus, all preserved formulas contribute causally the derivation of the final solution, ensuring that there are no dangling or irrelevant steps.
The resulting DAG captures the logical skeleton of the solution. where nodes formalize reason- ing steps and edges encode causal dependencies. In this way, the derivation is made machine- interpretable, with correctness evaluable both locally (per node) and globally (via entire dependency chains). This structure provides the foundation for our scoring mechanics (see Section 3.4).
3.4 ANCESTOR CLOSURE SCORING
Definition 1 (Ancestor Closure). Let M S. F be the set of matched reference formulas, and let Auc(S)= == (A E F/S: 3B E S and a path A x ... + B}. We define ancestor closure of M as:
Ach(M) = M Anc(M):
i.e, all matched nodes and all their DAG ancestors (reverse reachability).
By this definition, we formally propose our Ancestor Closure Scoring Policy:
Definition (Ancestor Closure Scoring Policy). Let M be the of formulas in the reference DAG that are directly matched by the submission (student solution), then Ancestor Closure Scoring gives the final score as
Ach(MM)
S
FF
where the set of all formulas the DAG.
Intuition. In solution DAG, the edges point from the prerequisite formulas to their dependents. If a dependent formula is achieved (matched), then every formula that lies on any directed path into is considered achieved as well, because those predecessors are logically required for deriving it. Thus, we count the union of each matched formula together with all of its prerequisites; the score is simply the fraction of reference formulas covered by this union.
3.5 JUSTIFICATION SYSTEM AND OPTIMALITY OF DAG
We formulate a good formula-based scoring policy with the following intuition: a scoring policy should first verify which formulas in the reference solution are matched by the student solution, then see some other formulas can be justiffied by the matched reference formulas. We restrict attention to complete justifications; partial approximate justifications outside the scope of this formulation. Under such a formulation, we can formally discuss the optimality of our provided DAG structure representation of the formula relations, well our scoring policy.
Definition 3 (Justification System). Let F be the set of reference formulas. justification relation a relation J 2 x F, where X + B means: once every formula in X is matched, the formula B is automatically warranted, in other words, adding formula B into the student solution will
4

---

Published as a conference paper at ICLR 2026
make any further progress the final answer. The F and all justification relations within form the justification system (F.
We define the minimal justification relation + by
A A+ B = A + B and no proper subset Y C A satisfies Y + B. In this case, A is called a minimal justifier of B.
The minimal justification kernel of justification system (F, =) is the set
K [(A, B) E 2 F A B}.
Two justification systems (F, minimal justification kernel, i.e.
1) (F,+1) and (F, said be equivalent if they have the same
[(A : A B} [(A,B) B).
Assumption (Singleton Minimal Justifiers). For every B E F, every minimal justifier of B has size 1. Equivalently, the justification system can be represented with a binary relation R x F such that A B (A) is minimal justifier of B. Remark. Intuitively,
means is a prerequisite of B: if B is awarded, must also be awarded. Scoring thus flows upward in the DAG: from every scored node B, all <-ancestors A also scored.
Assumption 2 (Causality). If A B, then A occurs earlier than B in the reference solution' logical order.
Theorem 1 (Bijection between order-keeping justifications and DAGs). Fix F (F...,FF with the index order < ... < N. Let Just be the class of justification kernels + F that satisfy Assumptions and (so F; Fj
j). Let DAG be the class of directed acyclic graphs G (F, E) whose edges point forward in the index order (i.e. (Fi, E i<<)).
Define the maps
$ Just + DAG,
+ (F, E == {(A, B): A B)), (IF,, (F)) = where Fe B < (A, B) E.
V DAG - Just,
Then:
(i) & is injective: if (f-)) (b(1 then H 1=
(ii) injective: if (G1) = (G2), then G = G2-
Consequently, t and V are mutual inverses and yield a bijection Just ~ DAG.
Remark. By Theorem we can see that the DAG representation is precisely the minimal encoding of justification system: it records only the minimal justification kernel and thus contains no redun- dant rules. At the same time, closure recovers the full justification system, so the DAG captures exactly the necessary structure with no loss of information and no superfluous complexity. The proof of Theorem given in Appendix A.
3.6 ADMISSIBILITY OF ANCESTOR CLOSURE SCORING
Intuitively, good scoring policy would map each formula in the DAG to I (achieved) or 0 (not achieved), then provide a score accordingly. More formally, we define an admissible scoring policy as follows:
Definition 4 (Admissible Scoring Policy). mapping 2F < 2F where S(M) is the set of scored formulas for matched set M, is admissible if it satisfies: 1.Matched Inclusion: M C S(M).
2.Ancestor Closure: If B E S(M) and A B, then A E S(M))" justifies A" > back-credit A)
3. S.Sunndees: S(M) S Ach(M). (no over-credit beyond justified ancestors)
Theorem 2 (Exact Characterization of Scored Formulas). For any matched set M S F and any admissible scoring policy S.
S(M) Ach(M) M U Anc(M).
Remark. Theorem 2 shows Ancestor Closure Scoring is equivalent to any admissible policy.
5

---

Published as a conference paper at ICLR 2026
4 EVALUATION FRAMEWORK: PRISM-DAG
4.1 RULE-BASED PHYSICS FORMULA EQUIVALENCE MATCHING
A key component of PRISM-DAG is decide whether two formulas equivalent. However, checking equivalence between physics formulas presents three key challenges: (1) Equivalence of equations. Checking formula equivalence is more difficult than expression comparison; (2) Con- stant substitutions. Two equivalent variables might be written in different forms; (3) Unit conver- sion. Values can be expressed in different units. Prior work often avoids these issues by checking only final expressions, enforcing specific formats, or relying LLM-as-Judge for comparison, but such approaches either miss process-leve evaluation or suffer from hallucination.
To enable fine-grained and rigorous process-based scoring, we propose two-stage algorithm for physics formula equivalence checking:
[Stage 1] Constant Substitution. We substitute certain variables with their expressions. Variables, constants, and units are normalized to predefined form for consistency.
(Stage 2] Solution Set Equivalence Check. For two equations with N variables, one variable is randomly chosen as the target, the remaining N are assigned random values, then the target solved to compare whether the solution sets equivalent. This process is repeated for multiple iterations. Solution set equivalence serves as a proxy for equation equivalence.
Details of the equivalence matching procedure and algorithm are given in Appendix B.1 and Algo- rithm 1 in Appendix B.2, which successfully resolve these difficulties.
4.2 SCORING PIPELINE
Given student solution and problem with annotated DAG, can summarize evaluation process PRISM-DAG as three steps, details shown in Algorithm 2 in Appendix B.3.
Formula Extraction and Normalization. Given student's solution, all mathematical expressions are first extracted and rewritten into our dataset's standardized canonical form, discarding invalid expressions such as syntactically malformed formulas or irrelevant numerical fragments.
Formula Matching, Each standardized student formula is compared against the reference DAG of the solution according to Section 4. which outputs a of matched formulas in the DAG.
Scoring. Finally, we score the student solution according to the Ancestor Closure Scoring Policy in Section 3.4 with the DAG and the matched formulas.
PRISM-PHYSICS BENCHMARK
5.1 BENCHMARK OVERVIEW
We first collect and preprocess the data to satisfy the need of our scoring pipeline, then do difficulty annotation and domain categorization to give more fine-grained results for analysis. An example of our dataset is shown in Figure 2.
5.2 DATA COLLECTION AND PREPROCESSING
Three-Step Rewriting Pipeline. To guarantee both internal consistency and external evaluability, every sample in the dataset is processed through structured three-stage rewriting pipeline. Each stage focuses on eliminating ambiguity and enforeing standardization, while preserving the fidelity of the original content: (I) Formula normalization. All mathematical expressions standardized in ЕТTX, with uniform rules for symbolic equivalence and numerical precision; (2) Context clarifi- cation. Problem statements
rewritten to define all variables and answer requirements explicitly, removing ambiguities; (3) DAG construction. Each solution is represented as a directed acyclic graph (DAG) of formulas, verified by rule-based and LLM- based checks.
Verification and Quality Control. At each stage, an LLM-based module verifies the formatting, clarity, and dependency rules; failures trigger corrective feedback and regeneration.

---

Published as a conference paper at ICLR 2026
Fine-Grained Enhancements. Beyond the main pipeline, applied several refinements: forcing significant figure rules, explicitly defining all constants and variables, and unifying answer formatting.
See Appendix for further details and prompts.
5.3 ANNOTATION & ANALYSIS
Difficulty Annotation. Each problem is assigned a composite difficulty label that integrates LLM-based ratings of conceptual depth and computational bur- den with an antropy-based DAG complexity mea- sure. The three components are combined into a uni- fied score, which is mapped to Easy, Medium, Hard, capturing both the content difficulty and the reason- ing complexity of the solution.
Electh
PRISM
Physics Domain Categorization. Each problem is categorized into of seven key physics domains: (1) Mechanics, (2) Electromagnetism, (3) Optics, (4) Atomic, Nuclear, and Particle Physics, (5) Thermo- dynamics and Statistical Physics, (6) Quantum Me- chanics, and (7) Solid State Physics and Miscella- neous Topics. Further details and prompts are pro- vided Appendix D.
Felt
Figure 3: Statistics of PRISM-PHYSICS hi-
EXPERIMENTS
erarchical topics and difficulty level.
6.1 SETTING
We evaluate our proposed PRISM-DAG evaluation
framework on the benchmark. We consider two experimental settings: a text-only setting, where the problem statement is presented as plain text, and multimodal setting, where relevant diagrams or figures are included alongside the text.
Models. We evaluate diverse set of frontier LLMs. To guide the models in generating reasoning- augmented responses, we design zero-shot COTzheg prompts that encourage step-by-step deriva- tions. All models mum generation length, and identical prompt templates, to ensure fair comparison across settings.
with unified inference configuration, including fixed temperature, maxi-
Evaluation Framework Baselines. For comparison,
evaluate against: (1) LLM-as-Judge
Scoring, where an LLM evaluates both the final answer and the solution process given grading prompts(following the evaluation setting of SeePhy (Xiang et al., 2025)); (2) PSAS-S (Zhang al., 2025), an existing process-based framework with strong step-format and ordering assumptions, replicated per its original implementation for fair comparison. Details are provided in Appendix E.
6.2 MAIN RESULTS
Table shows Step-level Accuracy and Final-Answer Accuracy across difficulty levels. Perfor- mance consistently declines and response time increases with problem difficulty, reflecting LLMs' sensitivity to longer reasoning chains, more demanding modeling, and higher computational effort.
Step-level vs. Answer-level Evaluation. shown Table 1, final-answer and step-level evalua- tions diverge sharply with problem difficulty. Final-answer accuracy drops by over 40% from easy to medium tasks and falls below 10% on hard problems, indicating that models struggle to sustain reliability through long reasoning chains. In contrast, step-level evaluation shows that models still gain partial credit by correctly handling significant parts the derivation. Even on hard problems, they apply key principles or derive valid intermediate equations before failing at later stages.
These results demonstrate that insler scoring alone severely underestimates reasoning ability, whereas step-level evaluation provides a more faithful measure of process competence under com-

---

Published as a conference paper at ICLR 2026
Table 1: Step-level Accuracy and Final-Answer Accuracy across difficulty levels (Easy, Medium, Hard, and Avg.) for evaluated models.
Model
Reasoning
Easy
Medium
Hard
Avg.
Step
Step
Step
Time
Step
Time
Open-source Chat LLMs
Deepseek-chat
F
31.42
48.4
45.22
21.12
39.84
67.87
15.6
33,99
84.02
23,4
41.36
64.04
F
21.34
35.17
23.76
7.19
19.92
29.92
3.07
13.53
32.54
11.32
23.81
28.36
3.3ma-3.3-7B InsttBisstuuttTTee
11.63
10.21
I 1ama--4Sae 4-Scont- 178-16F
19.72
30.54
57.22
8.16
15.83
58.36
4.13
10.33
59.99
11.35
19.78
58.40
Open-source Reasoning
Despsolk-Rassenee
T
30.43
48.68
200.93
21.57
42.79
262.44
15.86
37.22
311.34
23.25
43.339
253.49
GPI-OSS-208
T
25.2
10:21
14.46
12.63
16:27
11.57
GPT-OSS-120E
T 17.39
28.49
16.07
8.54
20.13
23:40
4.35
13.4
26.01
10.66
21.32
21.39
Qwen3 A220-AA22--H0 Instruct
T
21.74
33.7
22.29
8.31
18.07
29.95
3.07
9.45
34.25
11.85
21.45
28.31
Proprietary Chat LLMS
Claude-sonett
F
24.7
38.94
23.88
13.71
28.41
26.95
9.21
20.88
27.32
16.54
30.19
25.90
GPT-4o-mini
17.79
7.64
16.31
27.11
14.86
GPT-4.1
F
24.51
40.8
18.14
11.69
23.36
28.95
4.35
14.67
35.56
14.39
27.4
26.80
Proprictary Roavoning
Gemini-2,5- 1amini--..5-Fach Flasshh
T
23.52
36.5
41.08
14.61
27.52
58.40
10.74
24.72
72.86
16.84
30.00
56.08
Gomini-2.5-Pro
71.37
74.28
GPT:5
Low
33.0
54.76
26.77
22.92
44.67
33.86
15.86
36.76
40.22
24.66
46.17
33.05
GPT-5
Medium
30.83
48.03
50.12
17.3
34,92
66.96
10.49
26.97
83.67
20.42
37.55
65.48
GPT:5
High
37.75
103.59
26.29
52.7
127.59
21.99
50.28
153.32
29.36
54.13
126.04
GPT-S-mini
Low
25,3
43.15
20.78
17.98
38.1
27.10
11.0
27.85
30.30
18.71
37.02
25.65
GPT-5-mim
Medium
30.43
49.51
45.41
16.63
36.46
60.85
9.46
26.45
73.56
19.74
38.45
58.73
GPT-S-miii
High
34.78
56.82
149.47
23.82
48.64
178.56
17.14
38.53
210.08
26.01
48.78
176.78
Grok4
T
30.63
52.48
181 66
22.47
46.91
235.58
14.87
41.0
28.166
23 34
47.29
228.6
33.20
36.50
31.72
Multimodal Large Language Models
Claude-sonnet.4
27.72
19:46
16.06
22.77
11.63
23.37
19:19
Gemini-2.5-Flas
T
35.18
55.96
45.19
24.27
48.41
65.65
19.95
39.7
81.96
27.12
48.72
62.69
Gemini-2.5-Pro
T
36.96
63.03
80.71
21.74
43.03
93.58
77.79
GPT-4.1
32.02
55.41
23.10
20.9
44.55
33.15
11.25
36.36
40.82
22.28
46.26
31.60
GPT:S
Medium Medium
37.45
60.24
63.29
26.02
52.46
87.
21.56
49.12
104 99
29.05
54.43
83.49
GPT-S-mini
64.30
75.42
Grok-4
T
31.55
52.73
180.19
21.8
47.73
231.76
15.13
43.64
316.77
23.53
48.42
237.11
plex tasks. Moreover, step-level signals open promising avenues for training and data curation: evaluation relies solely on final answers, rewards difficult problems become extremely sparse. Instead, step-level scoring provides rich intermediate reward signals, offering valuable guidance for reinforcement learning and principled basis for constructing higher-quality training data.
Physics Domain Category Analysis. We analyze LLM performance across physics domains and difficulty levels, shown in Figure 4. Models exhibit varying accuracy across different types, with the highest performance observed in Thermodynamics and Statistical Physics and the lowest in Quantum Mechanics. Step-level evaluation further exposes weaknesses in reasoning coherence, and accuracy consistently drops from Easy to Hard problems across all domains.
CFSS High
Demples-41
Cur EI
Patrik
Polunts
Sniie Pryma
Budee
Plppet
Figure 4: Step-level and final-answer accuracy across Physies Domain Categories and Difficulty Levels.
6.3 MODALITY AND LEAVOL COMPARISONS
Text Models vs. Multimodal Models. The effect of multimodal input varies across model families. In general, adding images provides stronger gains at the step level than at the final- anawer level, highlighting its role in supporting intermediate reasoning. However, for smaller or weaker models,
8

---

Published as a conference paper at ICLR 2026
24
0PS3
6r84
Derilea3
Cadedered Sen332Mi-erat
Figure 5: Comparison of accuracy and response time across reasoning levels.
Figure 6: Distribution of primary error types across models
multimodal input can even be detrimental, diagrams in physics problems often serve presen- tational rather than informational role, with the critical content already conveyed text. Figure provided in Appendix E.2.
Across Different Reasoning Level. We observe that reasoning-oriented models exhibit consis- tently higher accuracy than chat-oriented models, but this improvement comes with substantially longer response times. We further evaluate GPT-5 and GPT-5-mini for three reasoning modes (low, medium, high), shown in Figure 5. In general, results indicate improvement in accuracy with increasing reasoning effort, along with reasoning latency. For GPT-5-mini, we observe performance improvement in final-acurrayy and step-accuracy when increasing reasoning budget; meanwhile, the average latency of medium mode is 129.0% higher than the low mode, and the high mode is 589.2% higher. GPT-5 shows similar trend, though the medium mode performance drops below the low mode: this might be caused by over-thinking that is not thorough enough, while the high mode shows higher and more consistent performance. Notably, while 04-mini was previously claimed to be a good reasoning model, its performance here is relatively poor; one possible explanation is that, as distilled model, suffers from limited generalization and thus struggles with complex reasoning tasks beyond its training distribution.
6.3.1 ERROR ANALYSIS
We perform error analysis on the first incorrect step detected in each solution as shown in Figure 6, using unified taxonomy that integrates process-level physics reasoning errors with formula-level derivation errors. The classification covers seven categories (detailed definitions are provided in Appendix G.2): (1) Diagram Analysis Error (DAE), (2) Physies Theorem Application Error (PTAE), (3) Modeling and Process Understanding Error (MPUE), (4) Condition or Assumption Error (CAE), (5) Variable Relationship Error (VRE), (6) Derivation and Computation Error (DCE), and (7) Unit Dimension Error (UDE).
The dominant error types across models are Condition/Assumption Errors (CAE), which arise when models set up inconsistent incorrect physical assumptions: Derivation & Computation Errors (DCE), which oceur when models make mistakes algebraic manipulation or calculation; and Modeling & Process Understanding Errors (MPUE), which reflect failures in mapping the problem into the correct physical model or reasoning process. This indicates that LLMs often fail both in establishing consistent physical conditions and in executing algebraic reasoning.
6.4 EVALUATION FRAMEWORK ANALYSIS
We further evaluated PRISM-DAG with human annotations to assess effectiveness.
Annotation Setup. We randomly sampled 70 problems (10 from each domain) along with their corresponding DeepSeek-Ve V33 (text-only) solutions. Each problem-solution pair was independently evaluated by human experts reduce variance, with annotators including IPhO Gold Medal- ist and Top- Tier Physics PhD. In cases where the two experts' scores differed substantially, third annotator was invited to adjudicate and determine the final score.
We quantified the agreement between tramework-generatte scores and human annota- tions using Kendall's T, correlation coefficient, along with statistical significance testing via both asymptotic and permutation- - p-values (see Appendix F.2 for details). Higher values indicate stronger concordance, with significance levels verifying the robustness of the observed correlations.
Results.

---

Published as a conference paper at ICLR 2026
Table demonstrates the clear superiority of PRISM-DAG, which achieves the high- est тh and lowest p-values. LLM-as-Judge purely outcome-based, assigning only bi- nary 0/1 scores, while PSAS-S, though process- based, evaluates Steps independently without modeling causal dependencies. Both baselines are LLM-based, whereas our non-LLM PRISM-DAC explicitly accounts for causality across steps, leading to stronger alignment with human judgments. We analyzed failure cases from our evaluator and two baselines to understand strengths and limitations. Details are provided in Appendix F.3.
Table 2: Comparison of annotation alignment.
Method
Asymptodic p-value
p-value
LLM-asJudge
0.294
6.90x
6,00x10
PSAS-S
0.213
2.20x
2.09x100
PRISM-DAG 0.346 1.31 10
1.00 10
7 CONCLUSION AND FUTURE WORK
We introduced PRISM-PHYSICS, benchmark and process-level evaluation framework that en- codes physies solutions as DAGs and employs rule-based symbolic equivalence checking for reli- able, fine-grained scoring.
Experiments reveal persistent reasoning failures in frontier LLMs, underscoring the challenge of sustaining coherent derivations in physics. PRISM-PIIYSICS establishes principled and inter- pretable foundation for process-level evaluation, enabling more robust benchmarks and advancing LLMs toward deeper scientific reasoning, while its step-level signals provide both training guidance and principled basis for higher-quality data.
currently focuses physics, evaluation framework is domain- agnostic and can be readily extended to other subjects such as mathematics, chemistry, and biology. In future work, also plan to use the benchmark for post-training LLMs, particularly study the benefits of incorporating process-level signals during RL-based fine-tuning.
Although PRISM-PHYSHIN
Furthermore, framework designed be easily adapted existing datasets. We therefore encourage both current and future benchmark developers to adopt our framework alongside their original evaluation methods, in order to provide more comprehensive and consistent assessments.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Meta AI. Llama 3.1 8b. 2024. Accessed: 2025-05-15. 30
http:///uuggingaeooco//tetsaasas 1ataaaaaaa.----- 1B,
Anthropic. Claude 3.7 Sonnet. Anthropic model card. 30
2025.
Jure Brence, Saso Dzeroski, and Ljuuco Todorovski. Dimensionnally-consistten equation discovery through probabilistic attribute grammars. Inf. Sci., 2023. 3
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology. 15(3):1-45, 2024.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res., 2023. 3
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. 3
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 30
2025. URL https: /awpvva/a/ivvv//////////aa./iiva org.orgggooooo.o.. /abs////////
Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprini arXiv:2502.0680)) 2025.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Bofei Gao. Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- MATH: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. 3
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Prodram-aided
Graham Neubig. PAL: Program aided language models. In ICML, 2023. 3
Google DeepMind. Gemini 2.5 Flash.
http::////ooooooooee.eeeee cwm/ hemn//eetex-ill. setnerativeaii/dooe moodeesmm http://eeellssmmmm///e/// flash, 2025a. Vertex Al model card. 30
Google DeepMind. Gemini 2.5 Pro. http://///eeeennnnoooooooeeeeeea
2025. Google DeepMind model card. 30
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. 2024.
Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, and Sanmi Koyejo. Putnam-AXIOM A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NewrIPS 24, 2024. 3
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual mul- timodal scientific problems, 2024. URL https: www.wi.wwwo/.b..eeeeeooeeeeseeee
14008. 1,2

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Raj Jaiswal, Dhruv Jain, Harsh Parimal Popat, Avinash Anand, Abhishek Dharmadhikari, Atharva Marathe, and Rajiv Rain Shah. Improving physics reasoning in large language models using mixture of refinement agents, arXiv preprint arXivv2411..00621, 2024.
Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. Evaluating mathe- matical reasoning large language models: focus on error identification and correction. arXiv preprint arXiv:2406.000 00755, 2024. 3
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your llms capable of stable reasoning? arXiv preprint
arXiv:2412.13147, 2024. 3
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu, G-eval: NLG evaluation using gpt-4 with better human alignment. In EMNLP, 2023. 3
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations (ICLR), 2024. 3
Yujun Mao, Yoon Kim, and Yilun Zhou CHAMP: A competition-level dataset for fine-grained analyses of L.LMs' mathematical reasoning capabilities. In Findings of the Association for Com- putational Linguistics: ACL 2024, pp. 19256-13274 Bangkok, Thailand, 2024. Association for Computational Linguistics.
Meta Platforms, Inc. Llama-4-Scout-In 17B-16E. https: http::////uggggaaceeee co meeo://etaaae. / Llama-4-Scout-122 2025. Hugging Face model card; accessed 2025-05-12. 30
OpenAI. gpt o
GPT-4o mini
https:/ com/ coom/ndde// ndex/
2024. OpenAI
blog post. 30
OpenAI. GPT-4.1. http:pr//ggeeas..cc///nrre/gg/g/ 2025a. OpenAl model an- nouncement. 30
GPT-5. oft.::/ttssseeeeaaaoooo///////aa
OpenAI
model announcement. 30
OpenAL.
OpenAl 04-mini.
https://openai.comm
introducing--3-anee -aw----3-aad---mii-n------ - 2025c. OpenAl official announcement, 30
OpenAI. GPT-OSS. https////ppaaal.cmeeeeee//eennte
2025d.
OpenAl model announcement. 30
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating Ilms on 2025 usa math olympiad. arXiv preprint MoXiv:250332/93442 2025.
Shi Qiu, Shaoyang Guo, Zhuo- Yang Song, Yunbo Sun, Zeyu Cai, Jiashen Wei, Tianyu Luo, Yix- uan Yin, Haoxu Zhang, Yi Hu, Chenyang Wang, Chencheng Tang, Haoling Chang, Qi Liu, Ziheng Zhou, Tianyu Zhang, Jingtian Zhang, Zhangyi Liu, Minghao Li, Yuku Zhang, Boxuan Jing, Xianqi Yin, Yutong Ren, Zizhuo Fu, Jiaming Ji, Weike Wang, Xudong Tian, Anqi Lv, Laifu Man, Jianxiang Li, Feiyu Tao, Qihua Sun, Zhou Liang, Yushu Mu, Zhongxuan Li, Jing- Jun Zhang, Shutao Zhang, Xiaotian Li, Xingqi Xia. Jiawei Lin, Zheyu Shen, Jiahang Chen, Qi- uhao Xiong, Binran Wang, Fengyuan Wang, Ziyang Ni, Bohan Zhang, Fan Cui, Changkun Shao, Qing- Hong Cao, Ming xing Luo, Yaodong Yang, Muhan Zhang, and Hua Xing Zhu. Phybench: Holistic evaluation physical perception and reasoning in large language models, 2025. URL https: //tpx//aavvvv///////e///ee//.wvs wrg. abs 2504. 6074. 2, 15
12

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen2.5-72B. httpps//hmngigggaeeee.e//e///ee2 2024.
Hugging Face model card. 30
Qwen
Team.
Qwen3-235B-A22BB.
https://huggingfccccc////
Qwen3- 235B- A22B, 2025. Hugging Face model card. 30
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024. 1
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint 0rXiv:2404.10952 2024.
Peiyang Song, Pengrui Han, and Noah Goodman. A survey large language model reasoning failures. In 2nd Al for Math @orkshop@ ICML 2025, 2025.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R.
Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level sci- entific problem-solving abilities of large language models. In ICML. 1,3
XAL. Grok 4. https: ai aataaa////////. http://ww///oo/eeee 2025. xAI official announcement. 30
Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. CoRR, abs/2400.05662,, 2024. 3
Kun Xiang, Heng Li, Terry Jingchen Zhang, Yinya Huang, Zirong Liu, Peixin Qu. Jixi Hc, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, Hang Xu, Hanhui Li, Mrinmaya Sachan, and Xiaodan Liang- Seephys: Does seeing help thinking? benchmarking dision-based physics reasoning, 2025.
URL httph:t///// /waxivivvvv...////a org/ abs/ 2500 19099. 2.
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. 2025. 3
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang. Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Ye Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge. In ICLR, 2025. 3
Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning, 2025. URL sttps://arxivv /ttp//axivv//..////////////aaava org abs/ 12054. 3.7
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.001 10, 2021. 1
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with me-bench and chatbot arena. NeurlPS, 2023.
Yaoming Zhu, Junxin Wang, Yiyang Li, Lin Qiu, Zong Wang, Jun Xu, Xuezhi Cao, Yuhuai Wei, Mingshi Wang. Xunliang Cai, et al. Oibench: Benchmarking strong reasoning models with olympiad in informatics. arXiv preprint arXiv:2506. 10481, 2025.
13

---

Published as a conference paper at ICLR 2026
Supplementary Materials for PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning
DAG Structure Details
15
A.1 Proof of Theorem
15
A.2 Proof of Theorem
15
B Evaluation Framework Details
15
B.1 Equivalence Matching Details
15
B.2 Algorithm of Equivalence Matching
17
B.3 Algorithm of Scoring Pipeline
18
B.4 Scoring Example
18
Dataset Curation Details
19
C.I Data Source and Collection
19
C.2 Three-Step Rewriting Pipeline Details
19
C33 Prompts for Three-Step Rewriting Pipeline
20
C.4 Rewriting Examples..
23
D Dataset Statistics
28
D.1 Difficulty Annotation Details
28
D.2 Domain Categorization Details
29
Experimental Details for PRISM-PHYSICS
30
E.1 Experimental Setups
30
E,2
Additional results
Evaluation Framework Analysis Details
31
F.I
Annotator information
31
F22 Kendall's Tau-b and p-test
F3 Failure Analysis for Evaluation Framework
32
Error Analysis Details
33
G.1 Prompts for Error Analysis
33
G.2 Error Taxonomy Definitions
33
G.3 Model Failure Solution Examples
34
14

---

Given a standard solution (composed of multiple formulas in DAG structure) and an LLM solution (composed of multiple formulas), we compare every possible pair of one formula and one LLM-formula. In practice, this process, which runs on CPU cores, can be efficiently parallelized, Published as a conference paper at ICLR 2026
A DAG STRUCTURE DETAILS
A.1 PROOF OF THEOREM 1
Proof. (i) If (+H)) (F-2), then their edge coincide: F Fr-g- By definition of Er,, this is equivalent to
[(A A B} = [(A, B) A 2 B},
hence
1-
(ii) If (GG)) = (2)), then their kernels coincide:
EE, - Ez. Unwinding the definition,
(A, B) E E1 = At B + A A B + () E E2,
so E = Eg and thus 11 G2.
Order-keeping ensures that (+-) DAG (acyclic by Assumption 2; all edges point forward), and that (G) E Just (singletons and forward edges by construction). Finally,
V (((()) = and )(V(()) = G
hold by definition of &. and I E.
A.2 PROOF OF THEOREM 2
Proof. (C). Soundness gives
S(M) S Ach(MM.
(2). Matched Inclusion gives
M M S(M).
By Ancestor Closure, if B e S(M) and A < B, then A E S(M). Applying this repeatedly from every B E M along all reverse paths implies
Anc(M) _ S(M).
Hence
Ach(M) S S(M)..
Combining both directions yields the equality:
S(M) Ach(M).
B EVALUATION FRAMEWORK DETAILS
B. EQUIVALENCE MATCHING DETAILS
hence its time consumption is very small compared other steps (e.g. serialized generation of tokens with LLMs) in our benchmarking pipeline. The rest of this section discusses in detail how we compare two physics formulas.
As described, checking whether two physics formulas are equivalent faces several critical difficul- ties, where we provide a more detailed discussion:
1. Equivalence of equations. Formula Matching (i.e. checking equivalence of two formulas) is harder than Expression Matching (i.e. checking equivalence of two expressions). Previ- ous work for expression matching mainly use tree-based formula parsing (Qiu al., 2025). However, such tree-based parsing is not powerful enough to compare formulas.
For result-based judges or expression-be saaeessio--bbeedd judges, one could check equivalent expressions formulas only, but checking equivalent formulas is critical for more fine-grained process-based score.
15

---

Published as a conference paper at ICLR 2026
2. Constant substitutions. In physics two equivalent variables might be written in different forms. For example, the coulomb force F = kQq/r" /rr can be written as F Qq/ Q4/(4reer)) (4Teor"): these two expressions are equivalent, but they in different forms. Sometimes universe constants can be expressed in detailed numbers, for example: = mc² m (3.0 10²m/s)²
3. Unit conversion.
Values can be expressed in different units. For example, f 50 Hz 50 s using different units results in equivalent but seemingly different formulas.
We show our proposed algorithm for formula equivalence matching (i.e., comparing whether two equations are equivalent) in Algorithm B.2.
First, we conduct substitution of certain variables: this includes Math constants (e.g. т, e,etc.), /Euuio ,etc.,, constants or values provided in the problem
Physics constants (e.g. h = Amse (e.g. provided length, etc.), described as 'Stage I' in the algorithm.
1
After unifying all variables across formulas, we move forward to the iterated process of choosing target variable randomly assign values other variables solving for the target variable. The equivalence of solution sets are used as proxy for comparing equivalence of these two equations. In practice, we conduct at most Nmaz = 40 iterations. In each iteration, one target variable is selected randomly, and other variables are assigned random values in [2, 20j. Each iteration would generate one out of three possible outcomes: (1) not-rejecting equivalence trial; (2) rejecting equivalence trial: and (3) failure trial (in cases when both equations have solutions). For each iteration, both equations have solutions and their solution sets are equal within some tolerance threshold (relevant difference e = 10 10- it is classified (1) not-rejecting equivalence trial; if their solution number is different or their solution sets are not equal within certain threshold, this trial is classified as (2) rejecting equivalence trial; while if both equations provide solutions, such trial is classified as (3) failure trial. We continue to run iterations until we have enough successful solutions (at least Nauce 10) or reach the maximum number of iterations (N 40). Then reject the cquivalence if the number of successful trials is less than N Nauce = 10 or there exists a trial that is rejection equivalence trial." !.
Here we discuss some of these design choices.
More-than-one Iterations. In common cases, given sufficient tolerance, the p - value of one iteration is small enough to reject the equivalence of non-equivalent equations. In practice, the tolerance is set to 10 6 considering possible computing errors. However, this would lead to false non-rejecting in special cases, when using only one iteration. Consider these two formulas: fl x = An Ayt2 5 and f2 x = Ag + 2A,t25 where Ag ~ A1 ~ 1, ~ 1, ~ 10-8, If we select x or Au as target variable and randomly assign values to other variables, the difference in solutions of these two formulas would lie within the tolerance, and hence it cannot reject nonequivalence of these two formulas. In this case, using A₁ or / the target variable would reject the equivalence, making false non-rejecting rate of one iteration be around 0.5, which is way larger than what we expect. In our multi-iteration settings, the false non-rejecting rate of the 10-iteration examine process would be around 10 in this case, which is satisfactory enough.
Positive Sampling Intervals. Here we randomly sample most variables in a positive interval, i.e., [2, 20). This is because most variables in physics problems are provided as positive ones, and using negative intervals for sampling may lead to false rejections. For example, T 2 2T a² /GM and 2 / G (Kepler's Third Law) are considered equivalent in most physics settings: the semi-major axis of planet orbits should always be positive. Here, simply set sampling interval to be positive to avoid false rejections caused by similar reasons.
'Some optimizations for reducing time consumption are also used in practice (e.g. rejecting equivalence of formulas once after the first rejecting equivalence trial), which are trivial and have no impact the output of this algorithm, hence we omit them here for clarity
16

---

Published as a conference paper at ICLR 2026
B.2 ALGORITHM OF EQUIVALENCE MATCHING
Algorithm 1: Equivalence Check for Two Physics Equations
Input Two equations E1, E2 (symbolic); constants map C. (symbol + symbol value); sampling range R [a, b]; tolerance E; limits Taxx (time), Nam (trials), Nsuce (min successful trials), Ney (min equalities).
Output: Equivalent E EQUIVALENT, INEQUIVALENT; diagnostics (Neq, Queq: Tifail).
Stage 1: Constant Variable Substitution; Procedure SUSSTIUTECCONSSATSSC..C.CC
Substitute all string-valued entries in into E: // urits/expressions; one
pass, no recursion
Substitute all numeric-valued entries in into E;
e. e. e, co. Iʃ
return updated E
E1
SUBSTIIUTTCCONSTANTS 11 C); Eg SUBSTITUTECONSTANTSS (F,, C2,
Stage 2: Equivalent Check by Solving in Random Conditions;
Reg + Runq + 0, Riall < 0, k < 0; V +- free variables appearing in Ej U Eg;
Function SOLVETTARGET(E, x=, 0,,,T max)
return solution set S of E' for target xt under assignment 0 to V xx within time Tax
Function ALLCCLOSE(S,, E2
return TRUE iff the two finite sets match pairwise within relative/absolute tolerance e while Nmax do
:: + k + 1++1, pick x² ~ V
// random target variable // random values for
sample 0(v) Uni for each V xx=;
non non-target variable
S SOLVETARGET (1,, T Tmax)
S2 + SOLVETARGET
Tmax)
if or S2 nonempty then if ALL.CLOSH((S,
then
neq + neg + +1: // current trial does not reject equivalence
else
nneq f Dneq + 1:
// current trial rejects equivalence
end
else
Miil f Aiail +1; // current trial fails and cannot be used for judging equivalence
end
(eg Mneq) 2 N Nomce and k Nswc then
break end
end
Decision; if Req 2 Neg and Qneq = 0 then
return EQUIVALENT; // Enough valid trials and no trial rejects equivalence
end else
return INEQUIVALENT; // Exists trial rejecting equivalence encugh valid trials
end
17

---

Published as a conference paper at ICLR 2026
B.3 ALGORITHM OF SCORING PIPELINE
Algorithm 2: PRISM-DAG: Evaluation via DAG- MAG-Structured Rubric (3 Steps)
Input Reference DAG G (V, E);; each node E V has formula ((v) and prerequisite set Pred(u); student solution text S. Output: Score E [0, 1]: matched set M C V; achieved set A V.
Step 1: Extract and Normalize Student Formulas;
F
ExtractFormulss())
// raw math expressions
F
0;
foreach f do
g Canonicalize(f)
if' ls then
F (u end
end
Step 2: Match to Reference DAG (Reference Nodes Only); 0;
M
foreach e E V do
if 3 g E F s.f. (quivalent(((V)..)) then
M + M (v}
end
end
Step 3: Dependency Tracing and Scoring:
+ 0;
Procedure MARKANCESTORS(UI)
u & A then
A A (u):
foreach E Pred(u) do MARKANCESTORSSP)
end
end
foreach E M do MARKANCESTORS(O)
end
| V
return (s, M, A);
B.4 SCORING EXAMPLE
Figure 7 illustrates an example of how student's (or LLM's) solution is scored using formula matching and DAG-based back-propagation scoring.
Step (Formula Matching): each formula the student's solution is aligned one-to-one with reference solution, with equivalent formulas connected by green lines. For clarity, not all formula pairs are drawn. Grey arrows denote the dependency relations (DAG) between formulas in the reference solution.
Step Stack-Propagation scoring): once a derived formula in the DAG is matched, correctness propagated backwards along the dependency graph, allowing upstream formulas be credited as well. Correctly credited formulas are highlighted in orange, and the back-propagation path is indicated by orange arrows.
Step (Score Calculation): the final score is calculated by adding the points of correctly credited formulas. In this example, the student achieves score of 90/ 900
18

---

Published as a conference paper at ICLR 2026
A. Formela Matching
B. Back- Packk-Proaagation Scoring
C. Score
Studeat solarion
Reference solution
1- (fx + myy -22
$ (6g + my) -pi
(20))
5 m m (20)
20'
Tg = Bx my
(20)
Tx Rg my (20)
20'
2nyl, in (Sty 2eyy? (ey MN?
Dhe Imn"" me!
(Myy (ea)
20
Tg
(80)
Paloady (30)
teg
+0'
(10)
Plowe (10)
90V1.00'
Figure 7: Scoring pipeline example. A) Formula matching aligns student and reference formulas. B) Back-propagation grading highlights correctly credited formulas along the dependency DAG. C) The final score is computed as the sum of credited points, yielding 90/ 90/100 100 in this case.
C DATASET CURATION DETAILS
C.I DATA SOURCE AND COLLECTION
Our primary source of questions and detailed step-by-step solutions is the book Major American Universities Ph. Qualifying Questions and Solutions. The problems were extracted from PDF format, reorganized in Markdown for readability, and further converted into JSON for structured storage. Notably, this book has been widely adopted as a training resource for advanced physics competitions, ensuring both the difficulty and the pedagogical value of the collected problems. Problems requiring purely textual answers (e.g., "Describe ... ...". "Is ... stable?") were set aside in separate collection and are excluded from the current framework.
C.2 THREE-STER STEP REWRITING PIPELINE DETAILS
To ensure consistency and evaluability, each sample is processed through a three-stage pipeline:
Formula normalization. All mathematical expressions are rewritten into a uniform canon- ical format in ИТEX. This normalization supports precise symbolic equivalence checking, preventing mismatches due to notational variation. For numerical problems, explicit rules the number of significant figures are enforced, ensuring consistent standards across all answers.
2. Context clarification. Each problem statement is rewritten to make all variable definitions and final answer requirements explicit. Where the original text leaves meanings implicit (e.g.. undefined symbols or missing constants), clarifications are added to resolve ambi- guities. The result self-contained problem statement that can be understood without external assumptions.
3. DAG construction. Each worked-out solution is converted into directed acyclic graph (DAG) of formulas according to the requirement in Section 3.3. We first verify if satisfies the requirements via rulc-based methods before the LLM-based verification.
Verification and Quality Control At the end of each stage, we employ an LLM- based verifica- tion module check compliance with formatting, clarity, and dependency rules. If the verifica- tion fails-such as when ambiguities persist in variable definitions or the DAG contains extraneous steps-the stage is repeated with targeted corrective feedback. This iterative loop of generation and verification ensures robustness, yielding uniformly high-quality results across the dataset.
Fine-Grained Enhancements Beyond the main pipeline, several additional refinements were sys- tematically applied:
19

---

Published as a conference paper at ICLR 2026
Numerical precision. For problems with numerical answers, explicit enforcement of significant-figure rules was introduced the problem statement.
Explicit constants. All physical constants and context-dependent variables appearing in either the problem or the solution were explicitly defined in the rewritten version.
Answer formatting. Uniform formatting standards for final answers were applied to the problem context, including required units and symbolic representations.
Individually, these refinements may appear minor; collectively, they substantially improve the machine- aathbin--ctionaailityy. reliability, and pedagogical clarity of the dataset.
F Formula Normalization
Context Clarification
Construction
Pan Veriller
Pao Veriller
fronts reprotion ven (1 frrnce
Defies ruritces
Conven orlatin
systed opirokoce clockin
vehigaties
2. Vardy DAGS rate cledis
Rejectal Verilier
Verifier
Rejestel Verifeer
Figure 8: Overview of the Three-Step Rewriting Pipeline
C.3 PROMPTS FOR THREE-STER STEP REWRITING PIPELINE
Here are all the prompts we adopt for rewriting.
Format Instructions
1) One Formula Per Block
Each formula must be wrapped in its . block.
-Avid chaining multiple equalities or expressions a single block.
-Exception: Chained variable comparisons in inequalities are allowed only if explicitly required.
2) No Terminal Punctuation
Doo end any formula block with punctuation marks (e.g. ..)..).
ST Unit Format
-Always write units using (unit() } to ensure proper parsing (e.g. Nunit (m), ( (m/s)). Notice that the numbers should be put outside the unit, i.c. use $3lunit (km/h}$ instead of $ \unit 3km/s1
Strip Extra Formatting Commands
REMOVE the following:
-Delimitors: \left, \right
-Fonts/Styls:: \mathrm, Amathit, \mathbf, \text
Spacing: \; \1, Aquad, \gquad
MMlti-line: \
Standard Calculus Notation
Use canonical forms for all calculus expressions:
Derivatives: \ \frac dy dxx)
Partial derivatives: \fraci \partial f1 Apartial x)
Integrals: Vint 0 v dt (omit spacing commands)
Summations:
\sum_ 1/11-11 x_i
Rewriting Prompt- First Stage
You expert in Physics, and you going to rewrite given problem and solution into standard form. The formatting requirements are below:
Format Instructions
Moreover, you should make that the answers be graded correctly. you should make the written form of the final answer is unified, which means: 1) You should make sure variables needed the solution, no matter in the final answer or in the intermediate steps, are defined or specified in the problem, either in the context or in the subproblems. e.g. 'You should use Es for kinetic energy, Ep for potential energy, E for total energy, M for the mass of the central body, for the mass of the satellite, R for the radius of the orbit.'
20

---

Published as a conference paper at ICLR 2026
2) You should make sure all variables and concepts are defined clearly in the problem.
3) For all accurate values, don't write them in decimals but fractions instead. For example instead of
y = 2.25x you should write y = z However you believe the value approximate (for example you think 2.25 is not an accurate value) then you should leave as decimals.
4) You should try to avoid putting a representation a formula block, but instead use equations or inequal- ities. For example instead of writing "The maximum energy is 2 you should write Emax 2E and put the definition of Emaz in the problem context. Even if it hard represent it with single vari- able, you should write ans ... and mention that the final answer should be written this form in the problem context.
Now according to the requirements, please rewrite the problem and solution below.
The sample
Your output should be the same json format, keeping entries even if unmodified. Don't forget any single entry or your output would be invalid. Your response should be formatted as jison' Auyyurr wwwycuurrewritteeeeeeee wwwritten_jsenn>yteeeeeeee * and nothing other than the rewritten json should be your output.
Rewriting Prompt- Second Stage
You are expert in Physics, and you going to rewrite given problem make it clearer. More specifically, we wish clarify the requirement for the final answer of each problem, which means:
1) You should add an entry 'final Lasswer form which has three options "algebraic', 'numerical" and 'text-based' representing the form of the final answer. 'text-based' means the final answer is not cal- culation result or derived formula, but instead text description or statement. Notice that this only depends the final answer in the standard solution, therefore each subquestion can only have fi- nal answer form. If this is the same problem, else you may add an entry for each subproblem separately. In any way, this should be separate entry as iteml final Lasser. form') ] for the whole problem or item[ subquestions'' '[i][[" [[i]l""inala. manalanswerrfoo ffrm'm) for the i-th subquestion.
all subproblems, you may write as an entry
whole
hint how to decide if an answer is algehraic numerical: if the answer is formula with mul- tiple variables
each side the formula has variables, is algebraic; if there is only one vari- able or variables only exist in one side, it is numerical (e.g. (r__s=3Xtimm 10 10"7Aunit [m/s ) $ \rho__. (min_ \аppгox 1. 5. Sitines 10"3 \unit (kg/m" are both numerical answers.) Only the final formula would decide if is algebraic numerical.
2) You should add an entry (finalaassee instructions' based on the final answer form. Similar 'fi- nal wllaaasssrrforoe whole problem) or separate entries for cach subquestion. The instruction should contain the following information based the final answer form:
should be either an entry of the whole problem (if the instruction is suitable for the
the final answer algebraic, you should specify the format of the final answer in the problem, e.g., 'Your final answer should be given the form Umin = and your response should be written with H, T, m. g" The variable requirement for the final answer should fully match the final answer in the standard solution and make there's ' no redundant variables (for example, if Es 1 me" then you should most provide variables among Es, m, for the final answer)
b) final answer numerical, you should instead write "Your final answer should express ... as a numerical value', and if the final answer is an approximate value, you should also specify the number of significant figures needed according to the standard answer. Moreover you should add an entry like 'significant. figures': 3 to the problem or subproblem.
If the final answer text-based, you should try restrict the form of final statement, for example you should write 'Your final answer should 'The equilibrium stable/unstable." so that we can seek for the sentence to judge the correctness of the student answer. Now according to the requirements, please rewrite the problem and solution below.
The sample
Your output should be in the same json format, keeping entries even unmodified, and add your new entries as required. Don't t forget any single entry or your output would be invalid. Make sure you output a valid json. DO NOT put any hint for the final answer in the instruction! Only give some information to regularize the format!
Your response should be formatted as "json' An<your's rewritten, Sasettten__jjsss_eeee json>> 'n "*, and noth- ing other than the rewritten json should be in your output.
21

---

Published as a conference paper at ICLR 2026
Rewriting Prompt- Third Stage
You are expert in Physics, and you setting up grading standard for given problem. More specifically, your job is find the "core formulas' in the solution. These core formulas should reflect some significant progress in solving the problem
thus you think they are worth some credit. Moreover, you have to organize them in given format to set grading standard.
1) You should organize the core formulas in a list, each formula represented by dict including the following entries:
"index ": the index the formula, counting from "formula": the content of the formula, which should be a string wrapped with double-dollar (S$) symbols.
"dependency"; this is the crucial part of the grading standard. The indices showing which previous
dependency should be list formulas this one depends on, which indicates without those previous formulas this formula can be derived. Notice that only direct causalities should be considered, for example if A leads to and B leads C. A formula can never depend on another formula after it w...iiaallanserr. (optional) this is true if this formula is the final answer of subproblem or the whole problem. The last formula among all should always be final answer
you don 't need put in the dependency of
Every formula in grading standard must exist the original solution, you should not create any new formula the grading standard, and you should not make any modification to the formulas: just directly copy them from the solution.
3) You should ensure that for isolated question subquestion, if the final answer is correct, the student should receive full score it. That means, any formula should 'reachable' from final answer (or at least the final answer of one subproblem) in the dependency graph. If the final answer contains more than one formuls, you may simply mark multiple formulas as final answers. Below some examples for you to better understand the requirement.
Now, according the requirement, please write grading standard for the problem and solution below. The sample
Reviewing Prompt- First Stage
You are a professor, and now you should review whether your assistant correctly rewrote the problem and solution into standard form. More specifically, he should make sure the written form of the final answer is unified, so that the student answers can be graded correctly. Below your instruction to him which he should satisfy:
Problem Requirement as in the rewriting prompt Moreover, the formulas should also satisfy the following formatting requirements: Format Instructions
You should return something like x <jjjudc>>aallkk//ugpeeeeo
/tt////raasa reason>> . ......//reaae.......... </reason>', You may think before your final out-
put. Keep the reason concise.
Below are some examples for you to understand it hetter:
Now, according the instructions, you should decide if your assistant has rewritten the problem and solution correctly. Below the problem and solution rewritten by your assistant: The Sample
The reviewing prompt for the second and third stage are basically the same with the first stage, only changing the requirement according to that in the rewriting prompt.
22

---

Published as a conference paper at ICLR 2026
C.4 REWRITING EXAMPLES
PRISM-PHYSICS Rewriting Example- Stage 1
Problem: A bucket of mass M (when empty) initially at rest and containing mass of water being pulled up a well by a rope exerting steady force The water is leaking out of the bucket at a steady rate such that the bucket is empty after time T Findd the velocity of the bucket at the instant it becomes empty.
Original OCR Solution:
Let the total mass of the bucket and water be M' Then
mt
M' M + T
where is the initial mass of the water. As the leaking water has zero velocity relative to the bhcket,the equation of motion is
M du = P M.
dt
P
M29
du
dt
dl.
M
M
+ m
m
The velocity of the bucket at the instant it becomes empty is
== /
Pdt
PT
+ m
gT
In
gT
M +
m1
M
Error:
1. The problem statement uses T, but the recognized solution incorrectly shows T
2. In the final equation, the minus sign was misrecognized as multiplication symbol :.
Rewritten Solution
Let the total mass of the bucket and water at time be M'.
mt
M' M m
The equation of motion for the bucket (plus remaining water) is
M' du dt
P - M'g
Rearranging gives
P
du =
M + m 9
dt
The velocity v of the bucket at time = (when empty) starting from rest is
Gemptyy L/"
Pdt
gr
M m
Evaluating the integral yields
PT
+ m
Cempty
In
gT
M
where Temply is the velocity of the bucket at the instant becomes empty.
23

---

Published as a conference paper at ICLR 2026
PRISM-PHYSICS Full Sample Example
"id": 1001,
"context": "A static charge distribution produces a radial electric field given by) mathb[[E] A (e [r} ( nwhere SA$ constant with units (umit(V),cou
m). $bS is a constant with units [ and $r$ is the radial distance from the origin in (uni((m). You should use $ rhhoS for the charge density in unit/C/mi [C/m"3], $ varepsilon O$ for the vacuum permittivity in delta()) mathbf(( ) for the Dirac delta function, and SQ$ for the total charge in (unit(C). When asked for the total charge, provide your answer as nSS((Q==.... \ inS))nwher SQ$ denotes the total charge.". "source"; "images":
St
"caption": "Fig. 1.1",
"location":": "..""
"subquestions": [
"letter": "a",
"subproblem": "Find the charge density rhooSSS (in (ummt(C/m"3)) that produces the given electric field, including both the regular part as a function of $rS and any singular delta function) contributions at the origin. State your result for ri explicitly. You may sketch its form with reference to Fig. 1.1.".
"solution": "The charge density $1,rhOS given by Gauss's s law in differential form:) n rho varepsilon - \,ablla /cdot (mathbf(E) (SS' n'The electric field is' (S)nS''nn ()) math = (frac((((-br))() \mathbffe) [elhbfffe]_r_m nSS n'The divergence in spherical
coordinates for radial function Sf(r) \mathbf(e] is' (SS nabla (edot (f(r) matthbf{
e}_r)= (rr (drr (r'2 ((5) nFoor Sf(r) = A (rbr) we (SSS nS$ nr f(r) A (-b (nS
compute),nSS nSS (n((r) (n(()))())))))))())(())))))))))) =A (-bbr) (nSS n'
| (nSS (r
nnSS ( (dar (S$,,
(-br))
nSS (SS ( (dr
Abre" (SS e (n |edot
/ddot
(A (Aer(-br)-A Abr ( (fracc
H(} (srr)\)sss'''eeee Howeevrr, the Laplacian of (rrc in three dimensions also gives a delta function: (nnn::nnSSSmmmmm in \left( (frac(1)(r) (right) -4 \ppi
rs)nsS nSo the divergence, including the singularity at the origin, is is)nnS)) n) edot E) = -A
+ \pi A ddelta( mathhf((])) TTherefore, the charge density is' i rh = varepsilon A b (-br) + 4 Api varepsilon 0 (delta() (r)) n'The final answer should be written as (S$)) ()) rho = varapssioooon A (e (r 4 \pp \vrrepsioon_0 A \deeta(( (deela((mmahbb(( n)nns
answer form": "algebraic",
"final answer. instructions" "Your final answer should be given in the form Si,rho and your response should be written only with $A, b, r. \varepsilon 0, )delta(() deetta()mathbfff (mathbf(r })S "
"letter": "b",
"subproblem": "What is the total charge SQ$ (in \unit(CC) present for the above charge density? Express SQS using the variables defined. Write the final answer using the form $Q=...",
"solution": "The total charge SQ$ is given by nSk = \int. (atammmmaabbRR----.. \rho
dV |nSSS r\ varepsilon_0 0 A fracc(" r) (r +4 \/pi Aarrpsilon O A
ddelta() mathbf()) we have pnSS nQ = \int_ mathbb (R) "3}
A b
24

---

Published as a conference paper at ICLR 2026
( {r Bigr] dV + (utt(((aatbb(R)]33 4 \pi (varessioo A /delta
mathbf(r)) (nSS The first term becomes (working in spherical coordinates): (SS)) n ( (-b (r dV - varepsilon - r'2 \sin (heta d - A b (inn e
int ( - (nt:(0)"(\inftt))
phi d'' theta \int.(0)>{(pppp) (iniiii))))mmmmm)])1ppp)p> pi} ssin' theta theth (0) (ra) ( (nns)) ssin thera dh/thaa 2| |nn d' nS$ nSS - pi (varessioon 0 A L nTTh second term is SSSS mathbbb varepsilon (delta() mathhf((r) dV = Api varepsilon 0 A n8SS) T \nSS)
(0n (in
[-[bbr] dr (0)
ccdot [fr |ddo 2 cdot pi=
lpi
pi \varepsioon 0 A 4'pi (varepsilon_0 A AASSS nS$ nQ nS$ "final form"" "numerical",
"fnnalanswer.instrcciion "Your final answer should be $Q=0S."
"gadinn standard": [
"index": 1,
"formula": "$S),roo Varrpsiloon00 nnabla /cdot [E]sh, "dependency": []
"index": 2,
"formula": "$S" mathbffF] = A (-br(() (mathbf(e) rSS". "dependency": []
"index": 3.
formula": "SS), (edot (f(r) \mathbf(e)_n) = (frac[1) (12) (drr) (r22 f(r
))$s"
"dependency": []
"index":
"formula": "$Sf(r) = (bb))(
"dependency": 2
"index":
"formula": "(5r2()))=rrr(")r
"dependency":
4
}.
"index": 6,
"formula": "$S) (rao(d)/dd((Arr""(=bt)) (tte://eeecotreeefffcccccottaaa
"dependency":
}.
"index": 7, "formula": "$S" ((s -beebbrr)rr
"dependency": []
25

---

Published as a conference paper at ICLR 2026
"index":8. 8.
formulla: "SS" fss
"dependency":
7
}.
"index": 9,
"ormull": nabla (edot (mathbf(E) (rr)
(A --beett--brrrt Abre
D)S "dependency::
3.
"index": 10.
"formula": nabla /edot (mathhf(E) ( (faacAAeee b (r) "epenndecy"::
nabla
"index": 11,
"foruul":: nabla'22 \left(( (rrac(1)(r) \right) = -4 \pi /delta() (delta()mmahbi (m)thbf(r))SS) "dependency": []
"index": 12,
"formula" "$S),nabla nabla (odot (mathhf([] -A b (fracfe (-br)) +4 \pi delta()) mathbf (r))(())) )SS"
"dependency":
10. 11
"index": 13,
"formula": rho varepsilon A (fbac(e((brr))( + +4 Api varepsilon O A Adelta(' (rahhb((])SSS) "dependency"::
=
12
1.
is final answer": true
"index": 14.
"formula": "$SQ = iint (i (\mathbb/R)'3) \rho dVS$". "dependency": []
"index": 15,
26

---

Published as a conference paper at ICLR 2026
A (frac("[--b]]) r Bigr] dV \int ( (R] 'pp varepsilon 0 A (delta((mathb((r)) dVSS", "dependency":: 13,
"formula": "$SQ = \int_ mat._///)atbbb//
14
"index": 16.
"formula": "$S) int ([mmatbb(RR)33) -varepssilon A (fblfff((((((rr)((] dV =
-| @vrepsilon_ A b ((iiitt) (O) (int_())(2\ppi (-br)]
-2 r2 'sin' theta d phi theta drS$", "dependency":
15
"index": : 17, "formula": : "$$= varepsilon b Lint_(0)[(initt e'(-br] dr (0n=((00((ppp) \
sin theta dl) theta (0)= d',phisS"" "dependency":
16
"index": 18.
"formula": "$S) int (0)"((infty)) r) dr \frac (b.. "dependency": []
"index": : 19, "formula": "$S* (nt)) \sink d' theta 2$8",
"dependency":
"index": : 20,
"oormula": "$S) (2)p) d d'\phi = pissS""
"dependency": []
Index" 21,
formula": "$S "$S- varepsilon 0 A /edot (rac /edor 2 /edot 2) = -i varepsilon 0 ASS".
"dependency":
17.
18.
19.
20
"index": ;: 22,
"formula": "SS" int (n mamtbbbb()))33) \pp Avarepsilon. A (delta(()mahbff] dV
4 \pi (vrepssloo_ 0 ASS".
"dependency":
15
27

---

Published as a conference paper at ICLR 2026
"index": 23,
"formula": "$SQ pi (varepsilon_0. A 4/pi \varepsioon_ AS$", "dependency":
21.
22
}.
"index": 24,
"formula": "$SQ 0S8",
"dependency": 23
1.
Is final answer": true
D DATASET STATISTICS
D.1 DIFFICULTY ANNOTATION DETAILS
We assign composite difficulty label by combining LLM-based annotations and structural DAG complexity:
LLM-Labelled Conceptual and Computational Scores. Each problem evaluated along two dimensions using an LLM: C1 which measures conceptual depth (the underlying physical principles and modeling complexity), and C2, which captures the computational burden (the extent of algebraic or numerical effort required). Both dimensions are rated on a three-level ordinal scale (1-3), with higher values indicating greater complexity.
Prompt for Difficulty Annotation
You are an experienced Physics Olympiad coach and grader. Classify Olympiad-levo physics problems using TWO dimensions (1-3 cach): C1 Conceptual depth (principles & modeling complexity)
C2 Computation burden (algebra/numeric length, error-prone)
Rules:
Do NOT solve or judge correctness; only estimate difficulty.
Use the provided SOLUTION only to estimate step depth/concepts.
Do not use outside tools or knowledge beyond the given text.
Keep outputs concise.
Output STRICT ISON:
"scores": ("C1"; 1-3, "C2": 1-3),
"rationales": ("CI": <= 20 words", "C2": <= 20 words"),
"reasoning": "2-3 concise sentences",
"confidence"; 0.0-1.0
PROBLEM: (problem)
SOLUTION (only for estimating steps/concepts; do NOT grade correctness): {solution]
28

---

Published as a conference paper at ICLR 2026
Entropy complexity of the solution DAG. We compute entropy- basopy-based structural complexity by treating the branching at each layer as the search space:
Depth
log (Widthh)).
This value is discretized into C3 E (1, using thresholds T2.
Composite difficulty, We define a composite score C1 + C2 + Cg. and map it into three difficulty levels: Easy, Medium,, and Hard. This composite annotation captures both physics/content difficulty and structural reasoning complexity, providing a stratified view model performance.
D.2 DOMAIN CATEGORIZATION DETAILS
The dataset covers a wide range of topies, organized hierarchically into key physics domains and 28 subtopics:
Mechanics:
Newtonian Mechanics, Analytical Mechanics, Special Relativity
Electromagnetism: Electrostatics, Magnetostatics and Staai-Stationary Fields, Electro- magnetic Waves
Optics: Geometrical Optics, Wave Optics, Quantum Optics
Atomic, Nuclear, and Particle Physics: Atomic and Molecular Physics, Nuclear Physics, Particle Physies, Other
Thermodynamics and Statistical Physies: Thermodynamics, Statistical Physics
Basic Principles and One-Dimensional Motions, Central Poten- tials, Spin and Angular Momentum, Motion Flectromagnetic Fields, Perturbation The- ory, Scattering and Transitions, Many-Partion Systems, Other
Quantum Mechanics:
Solid State Physics and Miscellaneous Topics: Solid State Physics, Relativity, Other
29

---

Published as a conference paper at ICLR 2026
E EXPERIMENTAL DETAILS FOR PRISM-PHYSICS
E. EXPERIMENTAL SETUPS
We evaluate diverse of 17 leading LLMs, listed in Table 3. Each model is accessed via its official API using standardized decoding parameters. By default, we set the maximum token output 8096, temperature ure 0.0, for all models where these settings are applicable. For reasoning models, the default reasoning effort is chosen as medium. Model-specific parameters are specified the table.
Model
Reasoning
Model Engine Name
Source
Open-source
Deepseek-V3 DeepSock-AI (2025))
F
deepaeek-v3
Sink
2 Qwen22.5-72B Instruct-Turie Turbo Qwen Team (2024)
F
Link
1lama-4-Scout-178 17 Meta Platforms, Inc. (2025)
F
Llama 4 Scout 17B 16B Instruct
Sink
Ilama 3.3.70B 70B Instruct Turbo Al (2024)
Link
Open-source Reasoning LLMs
Deepseek RI DeepSeek Al (2025)
DeepSock-R1 GPT 058 20B
Sink
GPT-OSS-20B OpenAl (2025d)
T
link
GPT-OSS- 120B OpenAl (2025d)
T
CPT-0SS-1209
Link
Qwen2-235B-A22B-nmae Qwen Team (2025)
T
Qwen3
Link
Pruprictary Chat claude-sonmet-42222555.
Claude sondee--onne--4 Anthmpic (2025)
Sink
10
GPT-4o-mini OpenAl (2024)
F
gpt-4o-mini
link
11
GPT-4.1 OpenAT (2025a)
F
qpt-4.1
Sink
Proprictary Reasoning LLMs
Gemini-2.5 Flahash Google DeepMind (2025a)
gemini-2 22.5-
Link
13
Gemini-2.5.PP Pro Google DeepMind (2025k)
T
gemt gemini-2.5-pre
1ink
14
GPT-5 OpenAl (2025b)
Low/MMediu/Htigh
gpt-5
Link
15
GPT-S-mini OpenAT (20256)
Low/MediumHHigh
opt-b-mini
Sink
16
Grok xAI (2025)
T
grok-4
Link
04-mini OpenAl (2025c)
T
mini
Link
Multimodal Large Language Models
18
Gemini- 2.5 Pro Google DeepMind (2025b)
T
gemini-2.S-pro
Jink
19
Gemini-2.5-Flash Google DeepMind (2025a)
T
gemini 2.5 flash
Sink
GPT.5 OpenAl (2025b)
Medium Medium
gpt-5
Link
GPT-5-mini OpenAl (2025b)
gpt mini
Link
Table 3: List of LLMs evaluated in our experiments.
Inference Prompt. Below is the prompt we use for inference:
Inference Prompt
You are a Physics expert. are going to solve physics problem and graded accordingly. Here are some instructions you should follow to make sure your answer is graded correctly: 1. You answer should be written in markdown format. 2. You should provide your key steps and final answer in a clear and concise manner using double-dollar signs for formulas (e.g..
Eg mgll m
). However, put your definitions or uncrucial steps in single dollar signs (e.g., F. is the energy the system', or "according to Newtons second law, F ma'). 3. Use less text and more formulas to explain your reasoning. 4. Your answer should satisfy the format helow: Format Instructions Here is the problem context: the problem Please provide your salution step by step, and then give your final answer. You should try use the variables given in problem and avoid using new variables unless necessary. You should strictly follow the formatting requirements introduced the problem for the final answer.
30

---

Published as a conference paper at ICLR 2026
E.2 ADDITIONAL RESULTS
Text Models vs. Multimodal Models We compute the performance gap between multimodal and text-only settings and visualize the differences in Figure
Fite
Figure 9: Performance differences between multimodal and text-only settings across models and difficulty levels.
F EVALUATION FRAMEWORK ANALYSIS DETAILS
F.1 ANNOTATOR INFORMATION
The majority of the student coauthors of this work have prior experience in Physics Olympiad com- petitions and are currently pursuing, or about to begin PhD studies in computer science with a focus on AI and LLM research. Almost every member has attained first prize in regional physics Olympiads, with more than half earning medals at the National Physics Olympiads, including one International Physics Olympiad (IPhO) gold medalist (Rank 10 Globally). This unique combination of competitive physics training and advanced LLM research experience offers a valuable perspective for the development of this work.
F.2 KENDALL'S TAU-B AND P-TEST
We evaluate the agreement between model-derived scores and human annotations using Kendall's 's ты correlation coefficient, nonparametric rank-based statistic that extends Kendall's by correcting for ties. Let {xx, b be a set of paired observations, where x; represents the score assigned by the model and y, the corresponding human annotation. Kendall's measures the degree to which the rankings induced by x and y agree.
pair, define: Definition. Consider all unordered pairs of distinct indices (1,5) with 1 S í < j S n. For each
concordant if (z: 11)( v,) 0, discordant if (I; 2,y (1) 81) 0. tied in x if 24; = 2i but yi # yj-
tied in y if y = yy but x, + Ij.
tied in both r; Ij and yi = Yj-
Let n. and n. denote the number of concordant and discordant pairs, respectively. Define
iy nnnn - 1), n1 =
Σ
2 (tla - 1),
122 = 2
Σ
1 uu( - 1),
k
31

---

Published as a conference paper at ICLR 2026
where ty is the size of the k-th tie group in and is the size of the 1-th tie group in y. Then Kendall's Tb is
The Rdd
7h
(mo 11) 122)
By construction, T6 [-,,1], with T6 indicating perfect agreement (all pairs concordant), To = -1 perfect disagreement (all pairs discordant), and 7 = representing no association beyond what would be expected by chance.
Statistical significance. To assess whether the observed correlation is statistically significant, we test the null hypothesis Ho Tb = against the two-sided alternative H1 Tb * 0. Two approaches are employed:
1. Asymptotic test. Under Ho, the sampling distribution of ть approximately normal for large with variance given by a closed-form expression that accounts for ties. standard- ized statistic Z Th is used to compute an asymptotic p-value.
2. Permutation test. To avoid reliance on asymptotic approximations, we perform a nonpara- metric randomization procedure: one ranking (c.g.. y) is permuted uniformly at random while x remains fixed, and 76 is recomputed. Repeating this procedure yields an empirical null distribution for Tb, from which p-value is estimated. This approach is robust small sample sizes and ties.
Together, ть provides a rigorous, tie-adjusted measure of ordinal association, and the combination of asymptotic and permutation- based tests ensures robust inference on the agreement between model predictions and human judgments.
F.3 FAILURE ANALYSIS FOR EVALUATION FRAMEWORK
We analyzed failure cases from our evaluator and baselines understand strengths and limita- tions. When scoring strictly by formula matching, causality-aware evaluation is essential: requir- ing every reference formula match is overly rigid and penalizes otherwise correct reasoning. We observe three recurrent failure modes:
Contextual vs. literal equivalence. Two expressions can be equivalent given the problem context but not algebraically identical (e.g., re-parameterized integrals or vector identities).
Textual answers. Description- text-only responses fall outside the scope of strict sym- bolic matching.
Parsing gaps. Some ИЛЕ symbols/commands are not reliably recognized as operators, yielding spurious mismatches (e.g., integrals with differentials, vector/tensor notation).
In order to solve these issues, we propose the following roadmap:
Context-aware matching. We plan to develop context-sensitive equivalence checker; this is left for future work.
2. Text evaluation. We will integrate lightweight LLM "helpers" to assess description- answers, making the framework more complete while keeping the core scorer deterministic.
3. Robust parsing. Despite many fixes, long-tail 18TEX idiosynerasies remain. We will re- lease the formula matcher first and iteratively expand operator coverage based on commu- nity feedback.
32

---

Published as a conference paper at ICLR 2026
G ERROR ANALYSIS DETAILS
G.1 PROMPTS FOR ERROR ANALYSIS
Prompt for Error Analysis
You are Physics Olympiad grader. Your task to analyze a student's solution against standard solution, using the provided detailed scoring breakdown, and determine the PRIMARY enror cause (and optional secondary causes) from the taxonomy below. You do NOT need align map steps the scored expressions already indicate where the solution is correct incorrect. Focus WHY the incorrect parts
are wrong.
Error taxonomy (choose labels exactly):
DAE: Diagram Analysis Error incorrect interpretation of diagrams//igurs/cchhettsss
PTAE: Physics Theorem/AAppicationn Error misuse/missapliiaaiion of physical laws/principles.
-PPEE: Modeling Process Understanding Error - imcorrec/iicompplte physical model process understanding.
CAE: Condition/Assunpiia Error invali/unjissiiic/mmispppiddd conditions, including bound-
ary/initial conditioon.
<RRE Variahle Relationship Error incorrect relationships between physical quantitics (e.g., con- straints, kinematic relations).
Derivation & Computation Error - ceaee--llgeebiicc algebraic/symbolic manipulation mistakes, arithmetic/sign/ substitution errors.
DCE:
UDE: Unit/Dimension Error unit inconsistency dimensional mismatch.
General guidance: 1) Use the scoring breakdown to identify which expressions are incorrect.
2) For the incorrect expressions, determine the earliest fundamental cause from the taxonomy. If multiple causes apply, select ONE primary label and list the secondary. diagrams are referenced missing, do not assume their content; judge only from given text/expres-
sions.
Output STRICT ISON (no extra text, markdown):
"rriaayy errorr:: "DAE PTAE MPUE CAE VRE DCE UDE", ["DA3] PTAE MPUE CAE VRE DCE JDE"),
"Incerccttcoppcssioons"""
"strings of the incorrect student expressions"
},
drelattd_correat_expree
1, "strings correct related student expressions any"
"rationale": "2-5 concise sentences explaining the diagnosis", confidence"; 0.0-1.0
PROBLEM CONTEXT: (problem)
STUDENT ANSWER: [student.answer)
AUTO-GRADER EQUATION MATCHES: (matches)
SCORE: (score: .3t] (out 1.0)
G.2 ERROR TAXONOMY DEFINITIONS
We provide detailed definitions of the seven enror categories used in analysis:
Diagram Analysis Error (DAE) incorrect interpretation of diagrams, figures, or
schematic representations.
33

---

dV dr The Published as a conference paper at ICLR 2026
Physics Theorem Application Error (PTAE) incorrect inappropriate use of physical
laws, theorems, or principles.
Modeling and Process Understanding Error (MPUE) incorrect incomplete con- struction of the physical model, including misunderstanding of the physical process being analyzed.
Condition Assumption Error (CAE) invalid, unjustified, misapplied physical conditions, including boundary or initial conditions.
Variable Relationship Error (VRE) incorrect establishment or use of relationships be- tween physical quantities.
Derivation and Computation Error (DCE) incorrect algebraic manipulation, symbolic transformation, arithmetic mistakes, sign errors, or incorrect numerical substitutions.
Unit Dimension Error (UDE) inconsistency in physical units failure maintain dimensional correctness.
G.3 MODEL FAILURE SOLUTION EXAMPLES
PRISM-PHYSICS Failure Example: Condition or Assumption Error
A particle of mass moves under the influence of an attractive central force f(r). The potential energy associated with this force is (r)), defined so that f(r) angular momentum of the partiele J. The effective potential denoted by V Vr(r). The total mechanical energy is E. The instantaneous radial coordinate of the particle is r, and the radial velocity
Problem:
The radius of the circular orbit is ru- For the stability analysis, the student should write their answer for the maximum n as ams where n is the exponent in the force law [(r) p with K a positive constant. (c) Determine the maximum value of for which the circular orbit is stable under the force law
f(r) K where is positive constant.
GroundTruth Solution:
For f(r) = pm we have
0f
K Ör p1+1
n
From the equilibrium condition,
0V J2
Ör mr3
Also, @V fr(),
-f(r) J²
3r
which gives
K
J2
n
3
Thus,
J mKr 3-n
Insert this value for f2 into the stability condition:
3.
0ʃ
>
0
m 4
Dr
Substitute J² m/Kr3-5 3-n and
pm
3K
>
1++1
So
3 - n 0
34

---

Published as a conference paper at ICLR 2026
Therefore,
ans = nt < 3
Student Solution Snippet:
(c): Maximum Value of For the force law f(r) pm
0ʃ nK
Ör
p
Substituting into the stability condition:
nk
3( 3(-K/ra)
pn+1
r
nk
3K
< 0
1h+1
p
K(n 3)
<0
rn+1
Since K and ntt1 >
we need:
<
The maximum value of n for stability is therefore n. = 3.
ans =
Error Analysis Output:
"primary_error"; "CAE",
"Incorrect.expressiirrr [
"The maximum value of for stability is therefore 3.",
),
"reelateelcorrret..errrsssssss!
n K(n- - 3) -=
},
"rationale": "All algebra and the stability condition were derived correctly, yielding < 3. The student then incorrectly interpreted this strict inequality as allowing the maximum stable value. This is mistake: = 3 is the marginal (neutral) case, not a stable one, so the conclusion ans is unjustified."
35

---

Published as a conference paper at ICLR 2026
PRISM-PHYSICS Failure Example: Diagram Analysis Error
Problem: An aquarium is being filled with water of refractive index 1. A stationary fish inside the aquarium looks vertically upward through the rising surface of the water at a stationary monochromatic plane wave light source outside the aquarium. The water surface rises upward at constant speed V relative to both the fish and the source, while the rest of the water and the fish are stationary with respect to the aquarium. The fish observes the incoming light through the moving water interface as the aquarium fills. Use the following variable definitions:-
the refractive index of water- V is the upward speed of the rising water surface with respect to both fish and light source- c is the speed of light in vacuum- u is the frequency of light as emitted by the source in vacuum- w/ the frequency of light as observed by the fish inside the water- of light in waterExpress your answer for u' in terms of u, V, n, and c. Also state, using an equation, whether u' is less than, greater than, equal to
the speed of light in water measured by the fish- Aw is the wavelength
GroundTruth Solution:
The speed of light in water as measured by the fish is
1
V
n²
The wavelength of the light in water is
Amo nv
The observed frequency by the fish is
v u
A
Substituting for yields
V (1 +
v/ =
Aw
Substituting for mw yields
2 V (1 +
1/
n
This simplifies to Expanding the terms gives
V (1 11-
1
v/
71
c
V = v - V (1 1
nv
n
For n and V O,
1/ < v
Therefore, the fish observes the light source to be red-shifted.
Student Solution Snippet:
We analyze the scenario using the Doppler effect for light at a moving interface. The light source is stationary in air (refractive index 1), emitting light of frequency u and speed e. The water surface rises at speed V upward, and the fish is stationary in water refractive index n.. The fish observes the frequency u' of light that has entered the water.
36

---

Published as a conference paper at ICLR 2026
The speed of light in water as measured by the fish is
C
The incident light in air approaches the moving water surface. Since the interface is moving toward the source (upward, while light comes from above), the frequency perceived just after crossing into water is Doppler shifted. Let the incident plane wave in air be
E; Eg cos(wt kz), 2TU,
k
At the interface z VI, the phase
dt(1) - wt k(V)) wt s0(1 (1-1)))
The transmitted wave in water is
E, Ej cos(w't - 1222,, k' nw
At Vt, its phase is Continuity of phase requires (x(t) = (t). hence
(1(1) = w'l KVL.
w 1 V
= w/ RVV.
Substituting 1' na gives
nV
w/ 1
= w
1 -
Thus,
1
w/ w
Equivalently, in terms of frequency:
V
V
nV
C nV
Now compare u' and ur:
V - u u
c
nVV 1 = (n 1)V
nV
Since > 1, V>> > 0, and c nV > 0 (non-relativistic case), we have u > 0. Therefore,
u' > u.
Final result:
u'
d' >
Error Analysis Output:
"primary_error": "DAE",
37

---

Published as a conference paper at ICLR 2026
eet.orrrectexppesssion.
[
E; Bo coalut ka), = 25v, == E: 6,(0) wt k(Vi) wt =v== wt(1 E):
J' K'V m(1 k)
n J'
V
= nw
continuity of phase the interface: d:(2 Vt) d:(2 Vt)
1, "rationale": "The student wrote the incident wave wt kz, which corresponds to a wave propagating upward; for light coming from above toward the interface (downward propagation) the sign of the spatial term is wrong. This incorrect sign Vt and
gives the wrong expression for the incident phase at therefore leads to the incorrect algebraic relation and final formula for v'. The use of phase continuity and V' = nw/c and c/n are otherwise appropriate, the error is an incorrect interpretation of the wave propagation direction (diagran/sign error)
PRISM-PHYSICS Failure Example: Derivation and Computation Error
Error Analysis Output:
"primary_error": "DCE",
"incorrect.exprrsssoonns""
aL Br
mr
7 3
mf
mr
,
k
k
},
"relaatedcorreeteeperessnnss [
87.
mr
Or
p3 p
mF mrdd
b.
k
},
"rationale": "The student made a sign/algebra error when differentiating the potential terms in L with respect to E, flipping the signs of the k and k' contributions. This is a
38

---

Published as a conference paper at ICLR 2026
pure perrivttio/coompiiiii physics), since the correct radial equation has the opposite signs and follows from the correctly differentiated Lagrangian or directly from Newton's form. The later orbit derivation uses the correct form of F(r), the error is localized to the symbolic differentiation step."
mistake (not misapplication of
PRISM-PHYSICE Failure Example: Modeling and Process Understanding Error
Problem: A particle of mass m moves under a restoring force -Kr and a resistive force -Rv where the displacement from equilibrium and is the particle's velocity. Here, m is the mass of the particle, K is the spring constant, R is the damping coefficient, æ is the displacement from equilibrium, and the velocity. For fixed к and arbitrary initial conditions, find the value R = R. that gives the most rapid approach to equilibrium. Also, state whether is possible to choose initial conditions (other than the approach to equilibrium is more rapid for R > Re or R < R.. Specifically, R. should be defined in terms of K and m. Express the rate of approach and clearly specify all variables throughout. The final answer for the most rapid approach value should be given as Rg in terms of and m.
and v = 0) such that
GoundTruth Solution:
The equation of motion for the particle can be written
d² dr + Kx
m
+
d12
Thus, for certain special (non-generic) initial conditions, the approach equilibrium may be even faster than in the critically damped case.If R. < Re, the roots are complex:
R + R R²
2m
The general solution is
Rt
VR R²
Rt
V R2
R?
I
Aexp
cos
B exp
sin
2m
2m
2m
2m
The rate of decay towards equilibrium is determined by the exponential factor
R
R
2m
2m
Thus, the approach to equilibrium is always slower than for critical damping when R < R.. In summary, the value of that gives the most rapid approach equilibrium for arbitrary initial conditions is
R. 2/Km
For R > Re, in principle, one can choose initial conditions such that the approach is even more rapid, but this occurs only for special initial conditions (such as setting 0 in overdamped general solution). In general, for arbitrary initial conditions, critical damping (R = R.) gives the most rapid approach. For < Rg, the approach to equilibrium always slower than for critical damping.
Student Solution Snippet:
The equation motion for damped harmonic oscillator is:
dxx
dx
Kr.
R
dt?
dt
39

---

Published as a conference paper at ICLR 2026
Rearranging to standard form:
dxx
R dr
K
0
df2
m dt
m
Let w K and 24 n
so the equation becomes:
m
dx
27 dr
wHs 0
dt2
dt
The characteristic equation is: The roots are:
22 + 22r + w/ 0
ー= 1/2 wi
The behavior depends on the discriminant 7:2 wg: Case I: Underdamped ( wo R < 2 Km)
z(t)= e (A cos + B sin(wat))
where Wa / -/
Case 2: Critically damped (7 = wp or R = 2v/km Km)
axt) = - "(A (A+ Bt)
Case 3: Overdamped (ry > wo or R > 2/ Km)
xxl) At Tit + Be
where -- - and = - 7/2 For the most rapid approach to equilibrium, we need to minimize the time for z(t) +0 In the underdamped case, the envelope decays as e In the critically damped case, the de- cay is 7t, In the overdamped case, the slowest decay rate is determined by rı -y + //2 - wp2-wwh-Sn -Snneennee v72 w > 0 when > wp. we have |ral < making overdamped systems slower than critically damped.
R For critically damped systems, For underdamped systems, the envelope decay rate is y 2m
the decay rate is also 7 f However, the critically damped case has no oscillations and provides the fastest approach without Serrshot.TTherefoo the critical damping condition gives the most rapid approach:
R. 2v Km
Regarding initial conditions: For any given R, the approach rate is determined by the system parameters, not the initial conditions. The initial conditions only affect the amplitude coeffi- cients A and B. but not the fundamental decay rates. Therefore, it is not possible to choose initial conditions to make R > R2 or R < R. approach equilibrium more rapidly than R.-
R. 2/ Km
Error Analysis Output:
"primary_error": "MPUE",
"incrreet.exppessioons" I "Therefore, it is not possible
to choose initial conditions to make R > R. or < R. approach equilibrium more rapidly than R R..", "However, the critically damped case has no oscillations and provides the fastest approach without overshoot."
40

---

Published as a conference paper at ICLR 2026
"related "related.correct www..rrree....eessssoo......
dx = = R
dt
2V Km,
72 u.g
Lnderdamped: =(t) = "(Acos(wad)) B Bsin(waf),,
critically damped: a(t) = ((1 Bt), overdamped: a(t) Ae B
V 7 -2
1, "rationale": "The student correctly derived the equation of motion root structure, and critical damping value Rer but then asserted that initial conditions cannot alter which R gives the fastest approach. That is modelling process misunderstanding: transient approach time can depend on initial conditions (e.g. certain initial displacements or velocities can make overdamped underdamped responses reach zero sooner for some measures). The blanket statement that critical damping is always fastest for every initial condition is therefore incorrect."
PRISM-PHYSICS Failure Example: Physies Theorem Application Error
Problem: A bucket of mass M (when empty) initially at rest and containing mass of water being pulled up well by a rope exerting steady force The water is leaking out of the bucket at steady rate such that the bucket is empty after time FFind the velocity of the bucket at the instant it becomes empty.
GroundTruth Solution
Let the total mass of the bucket and water at time be M'.
ml
M M m
The equation motion for the bucket (plus remaining water)
M du dt
P - M'g
Rearranging gives
P
du =
M + mt 9
dt
The velocity v of the bucket at time = (when empty) starting from rest is
Pdt
Cempty
gr
M
m
Evaluating the integral yields
PT
M + m
Cempty
In
M
gT
where Tempty is the velocity of the bucket at the instant it becomes empty.
Student Solution Snippet:
41

---

Published as a conference paper at ICLR 2026
I need to find the velocity of the bucket when it becomes empty at time / = Setting up the
problem The mass of water decreases linearly from m to 0 over time т:
Mwater (f) m(1 -
The total mass of the system at time / is:
Mtotal (1) M m(1
The rate of mass loss is:
dilwaater
dt
T
Applying Newton's 's Second Law
Since the leaked water has zero velocity relative to the bucket, when it leaves the bucket, it has the same velocity
the bucket. The forces acting the system are:
Upward force: P Weight: (M + M.waee (t)}g
Using Newton's 's second law for variable mass systems:
P (M + m(1 5)19 (M + m(1 - 1) du
+ dmrwatter
di
dt
Error Analysis Output:
\begin(verbat.im)
"primary_error": "PTAE",
"incorrectteppprsssoonss"
P- (M m(1 1)19 (M m(1 +)) de + detter
1,
"reelated.correec.exvpesssss
dmtater/dt -m/r (student's expression, correct),
(M + m(1 - t/T)) dy P (M + m(1 t/r))s.
(since escaping water zero velocity relative bucket,the thrust term vanishes)
Vempty
Px
gr
}, "rationale"; "The student misapplied the pariable-mase form of Newton's second law: the net external force equal to d/dt (Mv) and thus kept dM/dt term on the RHS (or equivalently omitted the momentum-flux term on the RHS), which is incorrect bookkeeping for mass leaving with zero velocity relative to the bucket. For escaping water with velocity equal to the bucket, the relative-velocies
term vanishes and the correct ODE is M(t) dv/dt 2 M(t) g, leading to different integral (logarithmic) result. All subsequent algebra and the final numeric expression therefore follow from this incorrect application."
42

---

Published as a conference paper at ICLR 2026
PRISM-PHYSICS Failure Example: Unit Dimension Error
Error Analysis Output:
"primary_error": "ODE",
"Incorrecttexpresssionn""" [
A A
N N
1/2
2) of
(27707)
(b 200000057730
10-12.6.123x104
2200)
TA AB =0
A
22"
N
22
10 IA
2
},
V"eelaatedccoreettexxprrsssss"!" [
A AN.
-AA,
À Jn2
1//2
N Noc
No = 10 12
),
"rationale": "The student mix mixed time units: À and o are in years while the counting time was used in hours, so formulas combining À and T (e.g. A = 1/ (APoTT) and subsequent N and x expressions) are dimensionally incorsistent. The algebraic manipulations themselves are otherwise coherent, but the unit mismatch renders the numerical/physical results incorrect. The same unit inconsistency propagates into the background-case quadratic and its solutions.",
PRISM-PHYSICS Failure Example: Variable Relationship Error
Error Analysis Output:
"primary_error": "VRE",
"inorreett.xppeessionn""" [
6(R A),
d= 1.22 (R +R++))
1.22 x TOmO 4.217 10?,
d 5.145 10
51450
43

---

Published as a conference paper at ICLR 2026
"eellatelcccrreet.xpppessso"""""
p. n+ (04)
3.580 10 m
},
"rationale"; "The student used R + h (distance from Earth's s center to the satellite) as the propagation distance for the beam instead of the correct path length from satellite to ground (the height h). That an incorrect relationship between physical quantities (distance to apply diffraction). The diffraction formula and numerical algebra are otherwise applied correctly, so the error is conceptual about which length variable to use.",
44


Reviews:
[{"review_id": "03me9o7w3B", "reviewer": "Reviewer_Fpc6", "summary": "This paper introduces PRISM-Physics, a physics benchmark with particular focus on process-level evaluation. The authors model reasoning process as a Directed Acyclic Graph, where each node represents a formula in the resoning chain, and edges denote logical dependencies. They then propose to score a student answer by counting the ancestor closure of student's nodes within that of reference answer. Besides the scoring scheme, the authors established a way to reliably transform natural language reasoning into DAG with normalized formulas, allowing for robust formula matching during evaluation. Lastly, they curated PRISM-Physics, contributing to process-level evaluation of physics problem solving.", "questions": "1. Since you assume all ancestor nodes in the reference DAG are scored, have you checked how often a student answer actually misses those ancestor nodes? Would missing these nodes break the logical soundness of the student's answer?\n2. In Section 6.4, did human experts score with Ancestor Closure Scoring as well? If yes, have you tried alternative evaluation methods (that is based on true logic instead of formula matching) to see how Ancestor Closure Scoring aligns with that scheme?\n3. In Appendix E.1, did you use 8K context and zero temperature for evaluation of reasoning LLMs?\n4. Have you tried extracting the student's answer as a DAG as well, and compare it to the reference DAG for a more rigorous scoring?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and very detailed.\n2. Experiments and analysis are comprehensive and insightful, covering a wide range of LLMs and problem types.\n3. The proposed DAG-based scoring scheme is novel and captures certain logical dependencies in the reasoning path.", "weaknesses": "1. The Ancestor Closure Scoring Policy seemed too forgiving for rigorous process-level evaluation. If I understand correctly, this scoring scheme overlooks skipped steps in the students reasoning path, as long as a targeted formula is attained downstream. Neither does this scheme check for validity of derivation, since it does not check whether assumptions leading to the formula are correct.\n2. Following point 2, it would be nice to verify how well the scoring scheme aligns with true logical evaluation.\n3. While the DAG-based scoring system does not require a student's answer to have the same sequential steps, it also ignores the structure in the student's answer. If I understand correctly, the student's answer is extracted as a bag of formulas with no logical dependencies extracted from context; only the reference answer is represented as DAG with logical dependency. Again, this seemed quite forgiving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "umUVtNSp82", "reviewer": "Reviewer_4ABk", "summary": "This paper introduces PRISM-Physics, a large-scale physics reasoning benchmark with a proposed DAG-based evaluation protocol that addresses the limitations of existing LLM-as-judge scoring methods. The evaluation framework includes a fully rule-based symbolic formula equivalence checker to ensure consistent validation across diverse mathematical formulations, thereby eliminating reliance on subjective judgments. In the experiments, the paper investigates a diverse set of leading LLMs on PRISM-Physics and demonstrates the superiority of the proposed evaluation protocol compared to the LLM-as-judge method.", "questions": "See Weaknesses 1, 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of using DAG to judge the correctness of the final answer and intermediate steps is reasonable, and I agree that using the LLM-as-judge method to evaluate the correctness of physics problems is challenging and prone to errors.\n2. The theoretical analysis part of the paper is solid.\n3. The experiment is comprehensive and convincing.", "weaknesses": "1. My main concern is that, although PRISM-Physics can conduct rule-based judgments to determine the correctness of the final answer and intermediate steps using a DAG, the construction of the DAG still heavily relies on LLM-based extraction and rewriting. Compared to the existing LLM-as-judge method, the uncertainty introduced by LLMs seems to have merely shifted from the judgment stage to the preprocessing stage.\n2. Another concern lies in the scalability and additional computational cost of the proposed evaluation protocol. Compared to existing benchmarks, PRISM-Physics requires an annotated DAG in addition to the final answer for each question in order to perform a more rigorous evaluation. Thus, the scalability of the proposed protocol appears limited. If we aim to extend this rigorous protocol to other existing benchmarks, what additional requirements would those questions need to meet? Furthermore, if we intend to construct a DAG for a new physics problem, how much extra computational cost would this introduce in the preprocessing stage?\n3. Typo: in Line 362: \"zero-shot **COTzheg** prompts\". In Table 1, it would be better to retain the same number of digits after the decimal point and to bold the best results.\n4. The results in Figure 8 (Appendix E.2) are difficult to understand. The authors should at least explain the meaning of each rectangle in the text and clarify whether the difference shown represents \"multimodal – text\" or \"text – multimodal\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKzfqWWsYD", "reviewer": "Reviewer_PRSU", "summary": "This paper introduces PRISM-PHYSICS, a framework for evaluating physics problems at the process level. Solutions are represented as directed acyclic graphs (DAGs) to capture dependencies between steps. The key innovation is the ancestor-closure scoring, which allocates partial credit based on intermediate steps. A rule-based symbolic equivalence checker ensures accurate comparison of formulas. Experimental results show that PRISM-PHYSICS provides detailed and reliable evaluations.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- PRISM-PHYSICS provides a large-scale, competition-level benchmark with carefully curated, DAG-structured solutions to complex physics problems.\n- A fully rule-based symbolic equivalence checker ensures consistent validation of diverse mathematical expressions, eliminating reliance on heuristic LLM scoring and offering a more reliable comparison across alternative formulations.\n- The ancestor-closure scoring policy allows for partial credit on intermediate steps, offering a more nuanced and fair assessment of student reasoning.", "weaknesses": "- Does the system account for context-dependent variations in formulas? (e.g., solving a problem from both kinematics and dynamics perspectives, or analyzing it through momentum and energy considerations)\n- In physics problems, certain expressions may be contextually equivalent, but the strict analysis in this algorithm might overlook such context-dependent equivalence.   Does the current framework account for these context-sensitive variations?\n- If skips over intermediate, simpler steps during the solution process, would this result in incorrect evaluation by the proposed method?  \n- The summary of the experiment is not clear enough. It is hoped that the author will have a more clear and organized discussion of the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gKfT4qqLPF", "reviewer": "Reviewer_4yRy", "summary": "This paper introduces PRISM-PHYSICS, a benchmark and a process-level evaluation framework that encodes physics solutions as DAGs and employs rule-based symbolic equivalence checking for reliable, fine-grained scoring.", "questions": "Refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A large-scale benchmark of competition-level physics problems with carefully curated, DAG-structured solutions. \n2. A DAG-based scoring policy that explicitly models causal dependencies among formulas, enabling fine-grained and interpretable process-level evaluation.\n3. A fully rule-based symbolic formula equivalence checker to reliably validate diverse mathematical expressions, ensuring consistent comparison across alternative formulations and eliminating reliance on heuristic LLM-as-judge scoring.", "weaknesses": "1. My main concern is that Figures 1 through 5 are very unclear, and even when enlarged twice, they are still hard to read. These figures should ideally be the most direct representation of the data analysis in this study, PRISM-PHYSICS. I hope the authors can improve the clarity of these figures.\n2. Figure 1 appears on page 2, but there is no corresponding content on the first two pages. Is its placement here inappropriate? Additionally, Figure 1 is not referenced anywhere in the text. The same issue applies to Figure 2.\n3. I feel that the originality of the article is somewhat limited. Could the authors provide an explanation of the connection between the challenges presented in this study and the research methods? At the moment, the challenges and methods do not seem to align well.\n4. Regarding the analysis of the experimental section, I hope the authors can summarize it more clearly. The current summary of the experiments is not very clear.\n5. How was the step level notation done? I would appreciate clarification on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)