Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
Authors: Aryan Gulati
Brando Miranda
Eric Chen
Emily Xia
Kai Fronsdal
Bruno de Moraes Dumont
Sanmi Koyejo
Venue: ICML 2025 poster (2025)
Keywords: Benchmarks
Large Language Models
Mathematical Reasoning
Mathematics
Reasoning
Machine Learning
Primary area: deep_learning->robustness
TL;DR: Putnam-AXIOM is a challenging mathematical reasoning benchmark for LLMs, revealing significant reasoning performance gaps and the impact of data contamination.

Abstract: Current mathematical reasoning benchmarks for large language models (LLMs) are approaching saturation, with some achieving $>$ 90% accuracy, and are increasingly compromised by training-set contamination.
We introduce Putnam-AXIOM, a benchmark of 522 university-level competition problems drawn from the prestigious William Lowell Putnam Mathematical Competition, and Putnam-AXIOM Variation, an unseen companion set of 100 functional variants generated by programmatically perturbing variables, and constants.
The variation protocol produces an unlimited stream of equally difficult, unseen instances -- yielding a contamination-resilient test bed.
On the Original set, OpenAI's o1-preview – the strongest evaluated model – scores 41.9%, but its accuracy drops by 19.6 % (46.8% relative decrease) on the paired Variations.
The remaining eighteen models show the same downward trend, ten of them with non-overlapping 95% confidence intervals.
These gaps suggest memorization and highlight the necessity of dynamic benchmarks. 
We complement ("boxed") accuracy with Teacher-Forced Accuracy (TFA), a lightweight metric that directly scores reasoning traces and automates natural language proof evaluations.
Putnam-AXIOM therefore provides a rigorous, contamination-resilient evaluation framework for assessing advanced mathematical reasoning of LLMs.
Data and evaluation code are publicly available at
https://github.com/brando90/putnam-axiom.

Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
Aryan Gulati" |
Brando Miranda
Eric Chen
Emily Xia 1
Kai Fronsdal" 1
Bruno Dumont
Sanmi Koyejo
Abstract
1. Introduction
Current mathematical reasoning benchmarks for large language models (LLMs) are approaching saturation, with some achieving > 90% accuracy, and are increasingly compromised by training-set contamination. We introduce Putnam-AXIOM, benchmark of 522 university-level competition problems drawn from the prestigious William Lowell Putnam Mathematical Competition, and Putnam-AXION Variation, an unseen compan- ion set of 100 functional variants generated by programmatically perturbing variables, and con- stants. The variation protocol produces an un- limited stream of equally difficult, unseen in- test
The ability for Large Language Models (LLMs) to reason about complex problems has plethora of applications in fields such as economics (Zhang et al., 2024), drug discovery (Bran et al., 2023), and even simulations of human behavior and society (Park et al., 2023). Rapid adoption of LLMs for reasoning has, in turn, spurred rapid gains on standard reasoning benchmarks (OpenAI, 2023; Gemini, 2023; Yang et al., 2024a).
Outpacing Current Evaluations. Indeed, advanced mod- els like GPT-4 (OpenAI, 2023) and Gemini Ultra (Gemini, 2023) have reported human-level performance on many benchmarks like MMLU (Hendrycks et al., 2020) and MMMU (Yue et al., 2023). Similarly, L.MSs have seen progress in other challenging benchmarks like GSM8K (Chen et al., 2022) and MATH (Hendrycks et al., 2021), with SOTA models attaining nearly 90% accuracy on MATH (Lei, 2024) and nearly perfect accuracy on GSM8K (Zhong et al., 2024). Although this progress demonstrates rapidly evolv- ing LLM capabilities, it creates an evaluation ceiling effect: current benchmarks lack sufficient difficulty to discriminate between state-of-the-as models' reasoning abilities.
stances yielding a contamination-resilne bed. On the Original set, OpenAl's ol -preview the strongest evaluated model - scores 41.9%, but its accuracy drops by 19.6 % (46.8% rela- tive decrease) on the paired Variations. The re- maining eighteen models show the same down- ward trend, ten of them with non-overlapping 95% confidence intervals. These gaps suggest memorization and highlight the necessity of dy- namic benchmarks. We complement ("boxed") accuracy with Teacher-Forced Accuracy (TFA), a lightweight metric that directly scores reasoning traces and automates natural language proof eval- uations. Putnam-AXION therefore provides rig- orous, contamination-resileen evaluation frame- work for assessing advanced mathematical rea- soning of LLMs. Data and evaluation code are publicly available at https: / //g/ttttte wiighttt.hc... com/ brando 90/ http:/utnaamaaattaaaa
Data Contamination. by data contamination, which remains major concern for current evaluation benchmarks. By training LLMs on larger portions of the internet, researchers are incorporating an increasing number of open-source benchmark data into the models' pretraining. Therefore, a model can display ar- tificially high "reasoning ability" by simply memorizing the answers it has seen, undermining the integrity of the evaluation.
The problem is further complicated
To address these limitations, we introduce the Putnam- AXIOM (Advanced eXamination of Intelligence in Operational Mathematics) dataset, a novel and challenging compilation of high-level mathematics problems sourced from the William Lowell Putnam Mathematical Competi- tion, annual mathematics competition for undergradu- ate college students in North America which requires ad- vanced mathematical reasoning and covers a wide range of university-level mathematical concepts. In addition, we
'Equal contribution DDeparmment of Computer Sci- ence, Stanford University. Stanford, USA. Correspondence Brando Sanmi Koyejo
to: Aryan Gulati wrraannl@@cs.sttfor...coo wew.dddo@ee.sttfoo...do.o
Miranda <sanmi @ www.sstooordddedleooo.... edu>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Patnam-AXIOM: A Functional and Static Benchmark
also introduce functional variations of the Putnam-AXIOM dataset to combat data contamination, taking inspiration from the solution employed by (Srivastava et al., 2024). Functional variations adjust variables, constants, and the phrasing of problems through Python scripts, allowing us to generate an unlimited number of new problems that are not found on the Web but still retain their mathematical complexity and validity, Putnam-AXION, enables fully au- tomated evaluations by requiring models to provide final answers within "V ")boxed()" ( brackets which can then be extracted and compared to the ground truth final solution us- ing an equivalence function ! as used for the MATH dataset (Hendrycks et al., 2021). This approach eliminates the need for human evaluation, and avoids the limitations of multiple- choice formats (Schaeffer et al., 2024), thus maintaining soundness while enabling scalability.
The Putnam-AXIONs a new evaluation benchmark of 522 challenging mathematical problems sourced from the William Lowell Putnam Competition, designed to assess advanced mathematical reasoning in LLMs.
Functional variations for 100 of these problems us- ing Python scripts, altering variables, constants, and problem phrasing to generate unlimited novel prob- lems while preserving their mathematical complexity, effectively avoiding data contamination.
Teacher-forced accuracy (TFA), to provide a more com- plete assessment of LLMs reasoning abilities be- yond traditional boxed answers.
2. Related Work
2.1. Mathematies benchmarks
Initial evaluations on Putnam-AXIOM demonstrate its dif- ficulty with ol -previeww scoring less than half at 41. 1.94%, while GPT-4o achieves only 19.35%. Even math-specialized models such as Qwen2-Math- 7B and Owen2-Math-77 7B Instruct perform poorly, scoring 5.51 % and 11.8% respec- tively. Performance further declines on functional variations of Putnam-AXIOM, which include significant drops for most models, decreasing by 20-30% in relative performance. "These low accuracies demonstrate Futnam-AXIOM' s ef- fectiveness as a challenging benchmark, while variations expose models' reliance on memorization.
Numerous benchmarks exist to assess the mathematical ca- pabilities of models, each typically focusing on a specific task. Two notable examples are MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021). The MATH dataset contains questions sourced from American high school mathematics competitions such as the AMC 10, AMC 12, and AIME (Hendrycks et al., 2021), while the GSM8K dataset contains 8.5K handwritten elementary school level questions (Cobbe et al., 2021). Both contain questions and answers with detailed rationale explanations.
As models have become larger and more powerful, even the most difficult existing benchmarks have become less chal- lenging. For instance, while the MATH dataset saw 6.9% accuracy on its release, it now sees 87.92% accuracy with GPT-4 MACM (Lei, 2024). Similarly, GPT4 has attained 97.1% accuracy on the GSM8K (Zhong al., 2024). This saturation necessitates the development of more challenging benchmarks.
Proof-based Evaluation Metrics. In addition to introduc- ing the Putnam-AXIOM Original and Variation benchmarks, we identified the need for more sophisticated L.M reason- ing evaluation metrics. Current evaluation metrics for rea- soning are inadequate, as they rely solely on final "boxable" answer without assessing the actual reasoning process. For problems with only a few possible final answers - such as with true/false or modular arithmetic - this means mod- els will often get the final boxed answer correct by random chance. Additionally, evaluations are then limited to the sub- set of problems with simple boxable answers, completely ignoring theorem proving and problems with complex for- mulae as the solution. In open-ended evaluations areas, like theorem proving, the current standard is either human eval- uations (He et al., 2024), which are expensive, or using a formal language, which requires translating theorems and setting up complex environments and dependencies (Yang et al., 2024b). We therefore explore alternative automatic metrics to boxed answers and find that a simple and cheap method, Teacher-Forced Accuracy (TFA), is a promising approach.
Many contemporary data sets have been created to combat the saturation of existing benchmarks. For instance, the ARB dataset includes hundreds of challenging problems in high school and college-level math, physics, and chemistry (Sawada et al., 2023). Similarly OlympiadBench contains nearly 9,000 problems from the International Mathematics Olympiad (IMO), the Chinese GaoKao, and more (He et al., 2024). Finally, SciBench is a similar reasoning benchmark that includes hundreds of college-level scientific reasoning questions from instructional textbooks (Wang et al., 2023).
Although these datasets alleviate the saturation problem, they come with many limitations. For instance, ARB (Sawada et al., 2023) and Olympiadßench (He et al., 2024) both contain several symbolic and proof-based questions which cannot be graded automatically and require a costly and lengthy human evaluation process. Though ARB at- tempts to utilize LLMS to grade their own responses with a rubric, this process is often unreliable and self-referential
Our contributions are:
For instance, the equivalence function would evaluate the answers 0. 5. 1/2, and (1) as equal.
2

---

Patnam-AXIOM: A Functional and Static Benchmark
(Huang al., 2024a). Our Purnam-AXIOM dataset ad- dresses these limitations by offering challenging Putnam problems with fully-written solutions and easily evaluable answers. It enables efficient automated assessment via frameworks like LM Harness (Gao et al., 2024), avoiding costly human evaluation or unreliable self-grading.
2.3. Evaluation Metries
Several approaches have been proposed to reduce the re- liance of model evaluations box- answers, particu- larly in domains like free-form writing translation where unique answers do not exist (Leiter et al., 2022; Opitz & Frank, 2021). Historically, tasks such as translation and natural language generation, which lack a single correct answer, have used more flexible metries, including n-gram match (Lin, 2004), model- (Guerreiro et al., 2023), embedding proximity (Zhang al., 2020), paraphrasing (Thompson & Post, 2020), generation as evaluator (Yuan et al., 2021), and information alignment (Deng et al., 2021). However, these metrics are not designed to assess reasoning ability or the correctness of mathematical statements.
PutnamBench (Tsoukalas et al., 2024) is a related bench- mark that primarily focuses on formal theorem proving. Its main objective is to derive formalized proofs of mathemat- ical statements and it provides formalizations in systems such as Lean, Isabelle. and Coq, all sourced from the pres- tigious Putnam competition. PutnamBench also includes 640 natural language statements and their corresponding answers where applicable. While both benchmarks draw from the same competition, Putnam-AXIOM focuses on the curation of natural language problems for final answer verification and introduces automatic functional variations to generate additional benchmarks addressing potential data contamination. Further through Putnam-AXIOM we go be- yond just the final answer by assessing the model outputted solution through evaluation proxy metrics.
When relying on boxed answers, we simply do not know how often the generated reasoning steps actually support the final answer. For evaluating reasoning abilities, the ROSCOE suite of metrics is noteworthy as it measures vari- ous fine- -iniigg aspects of reasoning steps such as semantic consistency, logicality, informativeness, fluency, and factu- ality (Golovneva et al., 2023). We omit descriptions of each metric, but highlight that most of them rely on sentence em- bedding models and operate on step-by-step level. Unfor- tunately, the original ROSCOE metrics were predominantly tested on GPT-3 generations, and we find that these metries do not provide evaluations that are comparable across dif- ferent models. Although fine-grained metries like ROSCOE can be useful for interpreting specific aspects of a model's 's capabilities, an ideal reasoning benchmark would employ a single metric that comparable across models and highly correlated with the correctness of the generated reasoning.
2.2. F Functional Benchmarks
Data contamination is a significant problem in creating eval- uation benchmarks, as many of these problems are openly available on the Internet and are likely included in the train- ing data for large models (Schaeffer, 2023; Sainz et al., 2023). Thus, the MATH (Hendrycks et al., 2021), AGIEval (Zhong et al., 2023), OlympiadBench (He et al., 2024), and ARB (Sawada et al., 2023) benchmarks (which are all sourced from problems on the Internet) could potentially be contaminated. Therefore, models may achieve artificially high performance on an evaluation benchmark by memoriz- ing the answers the problems Magar & Schwartz (2022); Ranaldi al. (2023).
In Huang et al. (2024b), authors drew upon equivalence be- tween language modelling and compression. They demon- strated that using bits per character (BPC) to measure a model's compression rate on several external large corpora is highly correlated with model performance on various benchmarks. However, this approach has drawbacks: eval- uating compression on large corpora is expensive, and the equivalence only holds for base models, as fine-tuned mod- els are not general -prrpose compressors for arbitrary text. Despite this, we suspect there would still be a relatively high correlation for most fine-tuned models. Relatedly, Yuan et al. (2023) found that pre-training loss is strongly correlated with mathematical ability for the LLaMA family (Touvron et al., 2023a;b). Unfortunately, creating an open benchmark using this metric is impractical due to the dependence of pre- training loss on differences in pre-training data, tokenizers, and other training-specet parameters.
A straightforward way of avoiding data contamination issues is to utilize problems unavailable on the Internet. However, even if problems are not currently part of model training data, it is unrealistic to expect them to remain inaccessible. At the same time, is costly to rely on the continuous human development of new datasets.
Srivastava et al. (2024) attempts to alleviate this data con- tamination issue by creating functional variations of the MATH dataset, where new problems can be generated sim- ply by changing numeric parameters, yielding different so- lutions. They observe a significant discrepancy in models' performance between standard benchmarks and these new variations. We recognize the potential of this idea and have adapted it to our more challenging dataset. We have al- tered the variables, constants, and phrasing of many Putnam questions while preserving their overall difficulty and re- quirements for logical and mathematical reasoning.
Relationship to process-superrision metrics. Teacher- Forced Accuracy (TFA) complements recent step-level eval- uation methods based on process supervision. PRM-style approaches label each intermediate step with learned re-
3

---

Patnam-AXIOM: A Functional and Static Benchmark
ward model trained on either expert annotations (Lightman et al., 2023) large-scale automated traces (Luo et al., 2024a). While PRMs are powerful, they require (i) millions of step-level labels, (ii) an additional model to learn the reward, and (iii) non-trivial calibration at inference time. By contrast, TFA dispenses with reward learning entirely: given a reference proof, we condition the LLM on the gold prefix and measure whether it predicts the next step. This teacher- forcing procedure yields direct, noise-free estimate of reasoning fidelity, is agnostic to model size, and incurs only a single forward pass per step. Empirically, we find that TFA correlates with final-answer accuracy but, like PRM scores, can still penalize solutions that "get the right box" " via spurious reasoning, thereby providing a lightweight yet alternative for ("boxed"") final answer accuracy.
modified these questions by adding trivial next step the original questions, changing the solution accordingly. This additional step was designed so to ensure that solvers reached the same conclusions and insights necessary to solve the problem, but then output a single boxed final an- swer. We provide an example of such a change in Figure I. By incorporating this minor modification, we preserved the inherent difficulty and complexity of the original problems while making the answers suitable for automated evalu- ation. Furthermore, since Putnam proof-based problems often test different reasoning abilities than Putnam answer- based problems, modified boxing allows us to provide a more comprehensive test. Of the original 522 problems, 221 required modified boxing, representing 42.3% of the dataset.
3. Methods
3.2. Putnam-AXIOM Variation Dataset
Models trained on snapshots of the internet have likely en- countered Putnam questions, potentially inflating their per- formance the Putnam-AXIOM Original dataset. There- fore, drawing inspiration from (Srivastava et al., 2024), we introduce functional variations of select problems from Putnam-AXIOM Original providing an effective way of evaluating models that have been trained on the entire inter- net by taking advantage of weaknesses in model memoriza- tion. These variations are classified into two types.
3.1. Putnam-AXIOM Original Dataset
Dataset. The Putnam-AXIOM Original Dataset contains 522 problems curated from the William Lowell Putnam Mathematical Competition posed between 1938 and 2023. These problems were selected based on their ability to yield unique, numerically evaluable final answer, enabling au- tomated assessment while preserving mathematical rigor. The dataset encompasses various topics within university- level mathematics categorized into 11 distinct domains Geometry, Algebra, Trigonometry, Calculus, Linear Alge- bra, Combinatorics, Probability, Number Theory, Complex Numbers, Differential Equations and Analysis.
Variable Change. The simplest variation is a variable change, where variable names are altered and the final an- swer is unvaried. Variable changes slightly modify the prob- lem from its original statement, which models could have trained on.
To maintain a consistent and rigorous evaluation, each prob- lem retains its original exam ID, which indicates its diffi- culty level and the topic categories. The ID format includes the exam sitting (A or B) and a number (1-6) representing increasing complexity, with being easiest and 6 being most difficult. The dataset is formatted using L5TEX to accurately capture the complex equations and symbols the problems employ. Additionally, we utilize Asymptote vector graphics for encoding mathematical figures and diagrams ensure language models can process visual elements directly. Fur- ther, we standardized the placement of boxed answers by relocating them to the end of each solution string to mini- mize unintended emergent behaviors leading to evaluations that are less "harsh" or prone to penalizing the model for formatting deviations rather than actual comprehension.
Constant Change. Constant changes modify numeric prop- erties of the question, altering constants within the step- by-step solution and the final answer. Constant changes significantly transform the problem from its original state- ment, challenging models to perform complex reasoning on how the changes affect the solution and final answer, as in the example from Figure 2.
Variational Dataset Description. We created functional variations for 100 Putnam-AXIOM questions (19.2% of the full dataset), selected to maximize coverage across mathematical domains while ensuring variation feasibil- ity geroblem--peciffii constants, non-generalizable solutions, and questions lacking constants or boxable answers were left). The dataset includes 37 constant+variable and 63 variable-only changes. We rephrased problem statements while maintaining the core task to prevent pattern recogni- tion by LLMs. Each variation can generate infinite unique, equally difficult snapshots, offering a sustainable evaluation method. To evaluate various SOTA models, evaluators are expected to generate snapshots (instances of the infinite po- tential variations) of the variation dataset by running the
Modified Boxing. Given the complex nature of certain Putnam questions, some problems do not lend themselves to simple, singular boxed final answers. Instead, they of- ten include conditions, multiple possible answers, varied answer formats and elaborate proofs. These original ques- tions would have necessitated costly and difficult human evaluations which we seek avoid. To address this, we
4

---

Patnam-AXIOM: A Functional and Static Benchmark
Determine which positive integers have the following property: For all integers m that are relatively prime to n., there exists a permutatton w: (1.2.....") - n !!, such that (TT(k)) = mk (mod n) for all E m..
Problem:
Determine the sum of the first positive inte- gers n (in terms of k) which have the following property: For all integers me that are relatively prime to n. there ex- ists permutation и: {1, ...,.... n + {1, 2....... n) such that (x())) mk (mod n) for all & E 21,2..... n)).
Problem:
Solution: The desired property holds if and only if 1or 2 (mod Let dr.m be the permuta- tion of Z//nZ induced by multiplication by vn; original problem asks for which n does du,m always have a square root.
Solution: Let a.mm be the permutation of 2/nZ induced by multiplication by m; the original problem asks for which n does oa,m always have a square root.
The desired property holds if and only n 1 or n = 2 (mod 4), hence making the required sum
By Lemma amm does not have square root.
2k 4k +
Year: 2016
ID: Al
Final Answer: ??
Year: 2016
ID: Al
Final Answer: 2k 4/
Figure 1. A modified boxing example in Putnam-AXIOM. Here see that the original problem holds true for number of values of conditioned on a specific property making it hard find boxable expression. thus modify the solution to still require the solver to get to that conclusion and add further computation of summing up the first & such values of giving a boxable solution while keeping the core the problem the same.
generation code.
tions rose sharply to 80%, while accuracy on the variations increased only modestly to 33%. These results suggest that the model rapidly memorized the original problems while continuing to struggle with functionally equivalent varia- tions, highlighting the importance of contamination-resilleen benchmarks.
3.3. Model Evaluations
Using the LM Harness Evaluation framework (Gao et al., 2024), we evaluated several open-source and proprietary SOTA LLMs. We rely on LM Harness Evaluation because its widely-used, vetted codebase that lets us reuse the same tokenizer-awarc boxing extractor and MATH (Hendrycks al., 2021) equivalence function, giving our results im- mediately reproducible and trustworthy evaluation pipeline. Models were prompted to provide answers in \boxed for- mat, which were then compared to Putnam ground truths with an exact final answer match. We evaluated the 522- question Putnam-AXIOM Original dataset once. For the variation dataset, we conducted five trials, each using a ran- domly selected variation snapshot and its corresponding 100 original questions. We then calculated mean accuracy and 95% confidence intervals.
3.5. Proxy Reasoning Metrics
Final-answer ("boxed") accuracy treats LLM''s chain of thought as black box: it cannot penalize lucky guesses on binary questions, detect spurious derivations, or compare the quality of two correct proofs. Recent propess-supervvision work mitigates this by training reward models that score every intermediate step, but requires vast step-level anno- tations and an extra model inference time (?Luo et al., 2024b). To obtain step-level insight without additional an- notation, introduce Teacher-Foree Accuracy (TFA) deterministic metric that measures how well model predicts each reference step under teacher forcing. We then compare TEA and its relatives with the 18 automatic metrics in ROSCOE (Golovneva al., 2023).
3.4. Fine-Tuning Experiments with Variations
To simulate data contamination and evaluate the robustness of the Putnam-AXIOM Variations dataset, we performed LoRA fine-tuning using the next-token prediction objective and trained until convergence. The models were then evalu- ated before and after fine-tuning on a 100-question subset of the Tutnam-AXION Variations benchmark and its corre- sponding original problems. Prior to fine-tuning, the model achieved 12% accuracy the variation set and 23% the originals. After fine-tuning, accuracy on the original ques-
Teacher Forcing: In teacher forcing (Jiang et al., 2023; Lamb al., 2016), the model is conditioned on the ground truth solution tokens rather than its previous predictions. Given question and its ground truth solution tokenized as 81, ..,... s,, let 81, , & be the tokens predicted by the model under teacher forcing. We explore the following teacher forcing metrics:
5

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem: Define a growing spiral in the plane to be sequence of points with integer coordinates Po (0, (),0), .Pn such that v 2 2 and:
Problem: Consider a growing spiral in the plane, defined
as sequence of points Lg = (0, 0), Li...., .....,,,. each having integer coordinates, where va 2 and:
How many of the points (x, 2) with integer coordinates I a I 2011,0 0 S S 2011 camnot be the last point, P.. of any growing spiral?
Determine the number of points (w, v) with integer coor- dinates 0 S w I 1680, 0 S v I 1680 that cannot be the final point, L, of any such growing spiral.
Solution: We claim that the set of points with 0 S x I 2011 and 0 S S 2011 that cannot be the last point of a
Solution: We claim that the set of points with 0 I w S
4680 and 0 S v S 4680 that cannot be the last point of
growing spiral
as follows: (0, y) for y - 2011:
growing spiral are follows: (0, u) for 0 S 4680: (w, 0) and (w, for < w I 4680; (w0, for 2 S w S 4680; and (3),3 for S w S 4680.
(z, 0) and (a, for
2011; (ar, for 2
2011; and (x,3) 3) for S x S 2011.
This gives a total
This gives a total
2012 + 2011 2011
4681 + 4680 + 4680
+2010 + 2009 = 10053
+4679 + 4678
23398
excluded points.
excluded points.
Year: 2011
ID:
Final Answer: 10053
Year: 2011
ID: A1
Final Answer: 23398
Figure 2. Constant and variable change in Putnam-AXIOM. Here, we perform variable change on the original problem/solution on the left by changing variables 'x' "w," 'y" to 'e," and 'P" 'L.' We also perform constant change by altering the constant '2011' '4680'. The constant change affects the final answer, changing it from 10053 to 23398. Finally, rephrase the problem.
1.
Accuracy (TFA) measures the propor- tion of tokens that the model predicts correctly when conditioned on the ground truth tokens.
with benchmarks when evaluated on very large corpora. The idea is that due to differences in tokenization, aver-
age bits per token are not directly comparable. Instead
we use
1 N
TFA
Σ
1 s.]
1 BPC
T Σ log P(s: = 1 81, 82, ......... 84-1)
2. Teacher-Forcel Cross Entropy (TFCE) measures the average negative log likelihood of the ground truth tokens under the model's predicted probability distri- bution.
where T' is the number of characters in the solution string rather than the number of tokens.
The main limitation of the teacher forcing approach is the dependency the ground truth solution. Models are often finetuned for specific style or problem solving approach (such as tool use or code generation). In this case, we would expect that teacher forcing metrics would under represent the models' abilities.
TFCE Σ log P(s = 8i 1 82.11.8..11. 82. 88-1)
3. Perplexity is a measure of how well a probability dis- tribution predicts a sample. In the context of teacher forcing, it an exponentiation of the cross entropy.
ROSCOE: The ROSCOE suite offers 18 distinct metrics, each tailored to assess different facet of reasoning de- scribed by (Golovneva et al., 2023). These metrics are broadly categorized into four groups. The first category, semantic alignment, focuses on identifying relationships between concepts that share the same or similar meanings. Metries in this category typically examine reasoning on
Perplexity = exp (TFCE)
4. Bits Per Character (BPC) (Huang et al., 2024b) is very similar to TFCE and has been shown to correlate well
6

---

Patnam-AXIOM: A Functional and Static Benchmark
step-by-step basis. In contrast, semantic similarity met- rics evaluate the problem and solution holistically. Logical inference metrics, utilizing a specially trained model (Lau- rer et al., 2024), detect contradictions between reasoning steps. Lastly, language coherence is assessed by evaluating model outputs using the perplexity score from GPT-2 Large (Radford et al., 2019) and a grammar model (Krishna et al., 2020). We use the code provided by the authors as is to evaluate these metries.
Model
Score
%
TFA
Gemma-2B 2a--na----Baeee
15/522
2.87
0.717
Gemma-7B-Bae
24/522
4.60
0.784
DeepSeck-Mat-77B-BBB
21/522
4.02
0.779
Qwen2-Math-7B-BB
50/522
9.57
0.770
NuminaaMath-7B-BBB
54/522
10.34
0.742
Mistral--7B--0.3-BBal
21/522
4.02
0.735
Llama-3-8B-Bas
17/522
3.25
0.748
Gemma- Commma-2B-Ine BB-nntructt
5/522
0.95
0.634
Metric Evaluation: Given the challenging nature of Putnam-AXIOM and the poor performance of existing mod- els, we opted to test the proposed proxy metrics on the MATH dataset instead. For a metric to be effective as a benchmark, its evaluations must be comparable across dif- ferent models. To generate evaluation data, we utilized 15 open-source models, ranging from 7 billion 70 billion pa- rameters, which exhibit a wide range of performance across the? 7 different MATH datasets. We then compared the proxy metric evaluations with each model's boxed accuracy for each dataset. A high correlation between the proxy metric and boxed accuracy indicates better proxy. 2 Our results, including the raw correlations for each metric in Table 4, are presented in the Appendix.
Gemma-7B: 7
24/522
4.60
0.702
Qwen2-Math-7B- BB-uh--BBBInstuut
60/522
11.49
0.758
DeepSeek- Maate----aat------ (eahh-BBIInsstte In-ntutrttt
36/522
6.89
0.750
Bestral-7B-Instrutt--v
21/522
4.02
0.735
Llama-3-8b Instruct
30/522
5.75
0.738
BerSSeek-Math7BB..
45/522
8.62
0.740
Claude-3.5 Sonnet
83/522
15.96
GPT-4
59/522
11.30
GPT-4o
101/522
19.35
0l -previiw
219/522
41.94
Table 1. Putnam-AXIOM Original Results and New TFA Scores. TFA Scores showcase pereentage of model next-token predictions matching ground truth. We cannot evaluate TFA for proprietary models because we do not have access to their log probabilities as seen in Appendix Section A.5
4. Results
4.1. Putnam-AXIOM Model Performance
Table presents Putnam-AXIOM Original dataset accu- racies. Most models score below 10%, with even Nu- minaMath, the AI Mathematics Olympiad winner (Invest- ments, 2024), achieving only 10.34%. These low accura- cies demonstrate Putnam-AXIO's effectiveness as chal- lenging benchmark. Figure 3 contrasts Putnam-AXIOM Variation dataset mean accuracies with the 100 correspond- ing original questions, along with the confidence intervals across the five variation snapshots with the average accu- racies Table 2. Original accuracies typically surpass variation accuracies. For models like ol -peevie,,, GPT-4o, and DeepSeek-R Distilled Qwen-32B, non-overlapping confidence intervals reveal statistically significant differ- ences, indicating artificially inflated performance on original questions due to data contamination. Looking at the num- bers highlights significant accuracy declines across models: DeepSeek- -Ieck-RR-QQwaar R--Qeen-33BB shows the steepest drop at 37.5 %, followed by GPT-4o at 36% and op-evvee at 17 %.
racy (219/522) while other models received less than 20%. Analyzing the answers, we see that most of the OpenAI ol ol-preview responses followed generally the same logical path as the ground truth solution. However, several of these questions contained logical mistakes and inconsistencies. The biggest discrepancy between model responses and the ground-truth solution was general lack of mathematical rigor. Whereas the ground truth solution will make claims to advance its solution then prove those claims step-by-step. ol -prevviww will often make and use claims without justifica- tion. While this does succeed in getting to the correct boxed final answer, these unjustified claims would receive little credit when marked by a human grader. A large part of the difficulty of mathematical reasoning is being logically air- tight throughout the entire solution; thus, though ol -preview shows promise, there are still evident flaws in its mathemat- ical reasoning abilities. In several solutions like Figure 5, for instance, ol -preview correctly identified the maximal or minimal value of variable, but failed to provide sufficient proof that the value it provided was indeed the maximum or minimum.
4.2. LLM Error Analysis
OpenAl ol-preview Performance: Out of all models, we see that OpenAl ol-preview performed the best on Putnam-AXIOM Original, receiving 41 19.4% boxed accu-
GPT-4o Performance: Like the ol -p-revie GPT-4o mostly followed correct logical reasoning for most of its solutions. For GPT-4o, the biggest discrepancy between model re- sponses and the ground-truth solution is the same general
We note that care must made before optimizing any models using a proxy metric as otherwise Goodhart's Law may take effect.
7

---

Putnam-AXIOM: A Functional and Static Benchmark
Original Variation Accuracies
OpenAl ol Preview
Haude-3.5-5onne
Deepseek-RL-Distil---eeeeee -iistill-Qwen-322B
Chat GPT 4o
DeepSeek-R1-Dee Boggseek-R--Dsttll-LLaa-----
Qwen2-math- mahh-7--nnstrut
Deepseek-math-7--nnssnuuu
D
Mistral-Codestral222--v.111
Qwen2-math-7b
Meta-Llamaa3.1-88--nes
Meta-Llama-3.1-8B
Mistral-7B-vO.1
Google-Gemma-7b
Uama-3.1-708
Deapseek-mathh7-bbaase
Intervals
Google-Gemma-2b
Variation
7istral-7B-
Original
15
25
35
45
55
Accuracy
Figure The drop of accuracies on Putnam-AXIOM Variation from corresponding Original questions statistically significant for nearly all models. This figure shows the mean accuracies for models along with 95% confidence intervals drawn.
lack of mathematical rigor throughout most of the solutions. An example of this lack of rigor is shown in Figure 6, where GPT-4o makes the claim that rectangle gives the minimal area subject to set of constraints without any justification. In addition to issues with rigor, GPT-4o also displayed logi- cal leaps and incoherent reasoning, as displayed Figure where the model simply assumes that an answer is correct. These logical leaps symptomatic of issue in the GPT- 4o's s CoT reasoning, the model prioritizes reaching the final answer rather providing rigorous logical output.
of the problem and proceeds an incorrect final solution.
4.3. Proxy Metrics
To evaluate the performance of our proxy metrics, we first test each of them on MATH, easier benchmark, can find models that achieve both very good and poor per- formance. In Table 3 we compare how our chosen metrics are correlated with the boxed accuracy of the answer on MATH. For the sake brevity we only include the three most notable metrics from the ROSCOE suite: Informative- ness Chain, Semantic Coverage Chain, and Perplexity Step. While it might be possible to combine the ROSCOE metries together and obtain stronger proxy metric, the straightfor- ward approaches failed. Simple averaging performed poorly, and we could not find a weighted or sparse combination of the ROSCOE metrics without overfitting to the specific mod- els that the weights were fit on. See Table the Appendix for the full results. Despite it's simplicity, TFA outperforms (i.e. is more correlated with boxed accuracy) all of the other metrics including all of the ROSCOE metries every cat- egory in MATH. Interestingly, the ROSCOE methods that correlate best with boxed accuracy are semantic similarity metrics quantifying the degree of semantic equivalence be- tween pieces of text. BPC performs reasonably well, but still trails behind TFA.
General Analysis:
Beyond GPT-4o and the ol op-preve we wanted a general overview the reasoning behaviors of models. To do so, we chose the best-performing open- source models, DeepSeck-Mer Maat----tthhB-Bttt Qwen2-Math Mat2--7a.B.B... and NuminaMath-7B. We tend that open-source mod- els are much more error-prone than the proprietary models we evaluated earlier. In general, we notice that open-souree souree models are subject to the same lack of mathematical rigor. However, this rigor issue is overshadowed by major calcu- lation errors, hallucinated//rreleaat information, misunder- standings of the problem, and logical jumps. For instance, in Figure 8, NutinaMath simultaneously makes calcula- tion, irrelevancy, and misunderstanding error when writing the last step of its solution; in Figure 9, the model makes false assumptions about functions defined in the problem; in Figure 10, the model completely removes a crucial part
8

---

Putnam-AXIOM: A Functional and Static Benchmark
6. Implications and Future Work
Implications for model development and evaluation. The sharp accuracy drop we observe when models are con- fronted with Putnam-AXIOM Variation (e.g.. -19.6 pp for ol ol-preview) indicates that many current LLMs still rely memorized artifacts rather than genuine mathematical reasoning. Consequently, leaderboard gains obtained on static benchmarks can overstate true capability. We recom- mend that practitioners report performance on dynamic or cnntamination-chekkee splits- suchh as our functional vari- ants-alongside traditional scores obtain a more faithful picture of progress.
Decy th
4.21
0.3
Figure 4. TFA against boxed accuracy with respect to model choice on Putnam-AXIOM. We see general positive relation- ship (correlation of 0.52) between the two metrics with a couple outliers. Without the two outliers the correlation is 0.62. We see an especially strong positive trend between models within in the same class (i.e. base and instruct models) which intuitively makes sense since these models are trained in similar ways.
Guidance for benchmark designers. Our results demon- strate two practical design principles. (i) Functional vari- ation: programmatic perturbations of constants and vari- able names create an unbounded supply of unseen, equally difficult items that resist contamination while preserving automated evaluation. (ii) Step-level metrics: lightweight measures such as Teacher-Forced Accuracy (TFA) expose reasoning errors invisible to final-answer checks, require extra annotation or verifier model, and enable automatic evaluation of natural-language proofs. We encourage fu- ture benchmarks to adopt both ideas and to refine step-level metrics so they better measure genuine reasoning.
Thus we select TFA as our proxy metric of choice for Putnam-AXIOM for both its correlation with accuracy and because of its low evaluation cost. In Table are the results of TFA on Putnam-AXIOM Original. Figure 4 showcases the relationship between TFA and accuracy on Putnam- AXIOM. One potential reason for the outliers QWen2-Math 7B-Instruct and DeepSeek-Math-e might be because they were trained with reinforcement learning and thus have a different style of writing compared to other models. Un- fortunately we can't evaluate TFA on proprictary models as we require the log probabilities of the input tokens. It would be possible to feed the input the proprietary model incrementally, but this would require API request for every token in the input.
Future work. We identify four immediate extensions. (i) Human correlation. A systematic comparison between TFA (and other proxies) and expert grading of proof traces will quantify how faithfully automatic scores reflect hu- man judgment. (ii) API-efficient step metrics. While TFA only needs sequential token predictions, access to log- probabilities allows a single forward pass; designing equally informative metries that operate on generated text alone would make step-wis evaluation practical for closed-source systems. (iii) Scaling functional variation. Extending our variation engine from 100 problems to the full 522-problem corpus- corpus-and  to proof-based questions- -will increase sta- tistical power and broaden coverage. (iv) Formal-proof targets. Incorporating items that require complete formal proofs (e.g., Lean or Isabelle) will push models toward deeper mathematical understanding and create a bridge to automated theorem proving.
5. Limitations
Problem coverage. The present release contains only those Putnam items that yield a unique numeric or algebraic an- swer after modified boxing procedure; mon-modii problems are excluded and thus some forms of mathematical reasoning remain untested.
Functional variation scope. Programmatic perturbations are implemented for 100 of 522 tasks (19.2 %), provid- ing strong evidence of contamination yet limiting statistical power. Expanding variation coverage is future work espe- cially with AI assisted methods.
Evaluation style dependence. TFA conditions a ref- erence solution; models that follow substantially differ- ent-bn but reasoning paths may receive lower scores, and models fine-tuned for tool use or code generation may be under-represented.
A Acknowledgments
Authors acknowledge support by NSF 2046795 and 2205329, IES R305C240046. ARPA-H, the MacArthur Foundation, Schmidt Sciences, OpenAl, and Stanford HAL.

---

Patnam-AXIOM: A Functional and Static Benchmark
Impact Statement
Guerreiro, N. M., Rei, R., Stigt, D., Coheur, L., Colombo, P., and Martins, A. F. T. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv: 2310. 10482, 2023.
Putnam-AXIOM supplies the community with a tougher, contaminatiin-resillee benchmark- and a lightweight step- level metric-for evaluating mathematical reasoning in large language models. By revealing performance drops that rote memorization masks, it enables more reliable tracking of genuine reasoning progress and guides future research toward models that truly solve, rather than recall, complex problems.
He, C., Luo, R., Bai, Y., Hu, S., Thai, L.,, Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi. Liu, Z., and Sun, M. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv: 2402. 14008, 2024.
References
Hendryeks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. International Conference Learning Representations, 2020.
United states copyright act. U.S. Code Title 17, 1976. Available at https:/ /www. copyri ght. gov/title17/.
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, Chemerow: Augmenting large- language models with chemistry tools, 2023. URL https: www.aiwww.gaaaa.aaaaaaaaaaoooaaa 20...////33...0 05376.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. In Van- schoren, and Yeung, S. (eds.), Proceedings of the Neu- ral Information Processing Systems Track Datasets and Benchmarks, volume 1. Curran, 2021.
Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., and Wang, W. Y. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv: 2210.03849, 2022.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models can- not self-correct reasoning yet, 2024a. URL https: / //arxiv.
Cobbe, K., Kosaraju, V. Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 2110.14168, 2021.
Huang, Y., Zhang, J., Shan, Z., and He, J. Compression represents intelligence linearly. arXiv preprint arXiv: 2404.09937, 2024b.
Investments, X.
Ai mathematical olympiad
Deng, M., Tan, B., Liu, Z., Xing, E. and Hu, Z. Compres- sion, transduction, and creation: A unified framework for evaluating natural language generation. In EMNLP, 2021. URL ht / aaaaccaanttt acclathologyy.rrg wrg.:y2...g.2..... mammainn la..mii..9... 599. .df..
progress prize 1, 2024. URL https:
/ //kaggle...// com cttm///tmptttins//////m
ai-mat al-mathemmatiicallllyyyaaaaaaap
Jiang, A. Q., Li, W., and Jamnik, M. Multilingual mathemat-
ical autoformalization. arXiv preprint arXiv: 2311.03755, 2023.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, I.,, Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: wttp///eeooooo//e///////e. wwwp://ord..o/o/er..... 12608602.
Krishna, K., Wieting, J., and lyyer, M. Reformulating un- supervised style transfer as paraphrase generation. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference ods in Natural Language Processing (EMNLP), pp. 737- 762, Online, November 2020. Association for Computa- tional Linguistics. doi: 10. 10.16653w1/20000eemml..mom 55. URL https: haapsss//acaaathhlllltoo hwtpttthhholo..oooo...oo manlp-main.55 55.
Empirical Meth-
Gemini. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv: 2312.11805, 2023.
or2/ 2020.
Golovneva, O. Y., Chen, M., Poff, S., Corredor, M., Zettlemoyer, L., Fazel-Zarandii, M., and Celikyilmaz, A. Roscoe: A suite of metrics for scoring step-by-step rea- soning. International Conference on Learning Represen- tations, 2023.
Lamb, A. M., ALIAS PARTH GOYAL, A. G., Zhang, Y., Zhang, S., Courville, A. C., and Bengio, Y. Professor forcing: A new algorithm for training recurrent networks. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information
10

---

Patnam-AXIOM: A Functional and Static Benchmark
Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https::/prroceedingss neurips. oc htppaaprr____ httppppe.filleeeppppr////////// 1602 1022600ff665410004440oaaaa.....2 pdf.
doi: 10.18653/112222.eeed--m 129, URL https: //acllrtthology.orrg///// wttp:////............o.eoe.e..ee aatac-mmaing -main. 129.
Park, J. S., O'Brien, J. C.,, Cai, C. Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023, URL https:// arxiv mxxiv.org//vo.o.ooo abs 2304. 03442.
Laurer, M., van Atteveldt, W., Casas, A., and Welbers, K. Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert-nli. Political Analysis, 32(1): 84-100, 2024. doi: 10.101 1010/ppm..023..... 2023.20.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Lei, B. Maem: Utilizing a multi system for condition mining in solving complex mathematical problems, 2024.
Ranaldi, L., Ruzzetti, E. S., and Zanzotto, F. M. PreCog: Exploring the relation between memorization and perfor- mance in pre-trained language models. In Mitkov, R. and Angelova, G. (eds.), Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing. pp. 961-967, Varna, Bulgaria, September 2023. INCOMA Lid., Shoumen, Bulgaria. https:: /aaclantoologe wwg//o/......... ranip- 1.103.
URL httpt:t///// /ttx/vixvv/v/////////e////aaaal/ org/
04735.
Leiter, C., Lertvittayakumjon, P., Fomicheva, M., Zhao, W.,
Gao, Y., and Eger, S. Towards explainable evaluation metrics for natural language generation. arXiv preprint arXiv: 2203. 1131,, 2022.
Lightman, H., Kosaraju, V., Burda, Y.. Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step, 2023. URL https: //arxiv. org/ abs 2305 20050.
Sainz, O., Campos, J. A., Ferefa-Ferrero, I., Etxaniz, J., Lacalle, O. L.,, and Agirre, E. Nlp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark, 2023. URL wrw.i..vivvv///.vi//...aaaaaaavv swexiiv.or org/ abs/ 2310 18018.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Sunmarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: //aclanthologh org/w04-1011
Sawada, T., Paleka, D., Havrilla, A., Tadepalli, P., Vidas, P., Kranias, A., Nay, J. J., Gupta, K., and Komatsuzaki, A. Arb: Advanced reasoning benchmark for large lan- guage models, 2023. URI. httttss:/arxxvv.rrg
Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng. L., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision, 2024a. URL https: //a farxiiv vv o abs /2lbs/22400 ..0.//44666000 06922.
abs/2307.13622
Schaeffer, R. Pretraining on the test set all you need, 2023. URL https: ort/abs/////..oooooooo....o....r /2009. 08632.
Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., Bradley, H., Biderman, S., and Koyejo, S. Why has predicting downstream capabilities of frontier ai models with scale remained elusive?, 2024. URL https::/p//. arxiv perxii.orr. orgg /bss//////// 22222446.60 04391.
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv: 2406.06592, 2024b.
Srivastava, S., B, A. M., V, A. P., Menon, S., Sukumar, A., T, A. S., Philipose, A., Prince, S., and Thomas, S.
Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation. Annual Meeting of the Association for Computational Linguistics, 2022. doi: 00.48850/axxiv2223..0022
Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv: 2402.19450, 2024.
OpenAl. Gpt-4 technical report. Preprint, 2023.
Thompson, and Post, M. Automatic machine translation evaluation in many languages via zero-shot paraphras- ing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 90-121, Online, November 2020. Association for Compu- tational Linguistics. doi: 10. 10.11653/v1/2220.emmm/ main.
Opitz, J. and Frank, A. Towards decomposable met- ric for explainable evaluation of text generation from AMR. In Merlo, P., Tiedemann, J., and Tsarfaty, R. (eds.), Proceedings of the 16th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics: Main Volume, pp. 1504-1518, Online, April 2021. Association for Computational Linguistics.
8. URL https: http::ss//aaaaathhhlicc .rg.//..rr/........ or2/2020. 2020. emn emnlp-man 1--maiin. 8.

---

Putnam-AXIOM: A Functional and Static Benchmark
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2302. 13971, 2023a.
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv: 1308,00825, 2023.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv: 2311.16502, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee. J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 2307.09288, 2023b.
Zhang, T., Kishore, V., Wu", F., Weinberger, K. Q. and Artzi, Y. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: petttt:://ppereeee wptp/ewinneeeeeeeeeeee net forum? id=SkeHuCVEDr.
Zhang, Y., Mao, S., Ge, T., Wang, X., de Wynter, A., Xia, Y., Wu, W., Song, T., Lan, M., and Wei, F. Lim as a mastermind: A survey of strategic reasoning with large language models, 2024. URL https: httttps://axii.orrgg/ abs
Tsoukalas, G., Lee, J., Jennings, J., Xin, J., Ding, M., Jennings, M., Thakur, A., and Chaudhuri, S. Putnam- bench: Evaluating neural theorem-proven on the put- nam mathematical competition, 2024. URL https: //arxive org/ abs 2407.1121-
Zhong, Q., Wang, K., Xu, Z., Liu, J., Ding, L., Du, B., and Tao, D. Achieving ;97 URL https: aattps://areet /arxiv..rgg/ abs 2404.2 14963.
Zhong. W., Cui, R., Guo, Y., Liang, Y.. Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A numan-centren benchmark for evaluating foundation mod- els, 2023. URL https: //arxiv....//// www.vvorgggooeeee. brg//bb/// abs 2204. 06364.
Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models. arXiv preprint arXiv: 10635, 2023.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. Qwen2.5-math techni- cal report: Toward mathematical expert model via self- improvement, 2024a. URL attps://aat http://axxvvvvoo/ org/ abs/2409.121B 12122.
Yang, K., Swope, A., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R. J., and Anandkumar, A. Leandojo: Theorem proving with retrieval-augmenned language models. Advances in Neural Information Pro- cessing Systems, 36, 2024b.
Yuan, W., Neubig, G., and Liu, P. Bartscore: Evaluating generated text as text generation. In Ranzato. M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27263-27277 Curran Asso- ciates, Inc., 2021. URL https: http::/ss////ccceiigg..... neurips. ot/ pouripps.ccpapprr/22//LLLs elcbeeffaca3e00of1aaaeeeeedddd
pdf.
12

---

Putnam-AXIOM: A Functional and Static Benchmark
A. Legal Compliance
collect and modify various problems from the William Lowell Putnam Competition to create the original and variation datasets of Pumaa-AAXOOM Putnam problems are created by the Mathematical Association of America (MAA), which also the source of the AMC and AIME problems used in the MATH dataset (Hendrycks al., 2021). Like (Hendrycks al., 2021), we do not in any form seek to monetize or commercialize Putnam problems- only utilize them for academic purposes.
Our use of the Putnam problems to create an evaluation dataset completely falls under the "research" section of Fair Use. Indeed, according to Section 107, of the U.S. Copyright Act (USC, 1976), our work certainly qualifies as Fair Use for the following reasons:
1. Our use of MAA problems is only for academic research purposes. We do not monetize commercialize the problems.
2. Our use of Putnam problems as a reasoning evaluation benehmark for large language models is significantly different from their original use competition problems.
3. Our use of Putnam problems is transformative. As detailed in Section 3 above, we have transformed the questions to be answered with single numerical or algebraic "boxed answer" as well created variations. We have altered all of the solutions so that the final boxed answer lies at the end of the solution (so as encourage models to explain their rationale before outputting solution). We have also standardized the solutions: If there are many solutions given, only the first; if there are any references irrelevant mathematics necessary to understand and solve the problem (such as comments like "Communicated by ..."), we have removed those.
4. Our use of Putnam problems to construct a benchmark has no effect on the demand for or supply of Putnam problems in the William Lowell Putnam Competition. The existence of our dataset does alter the value of the original problems as those already freely available online-noe nor does it influence the market of future competitorspproblem writers.
13

---

Putnam-AXIOM: A Functional and Static Benchmark
A.1. Full Table of Accuracies for Putnam-AXIOM Variation and corresponding Original questions
Table demonstrates TFA has the highest correlation with final boxed accuracy in statistically significant way. For example, TFA significantly outperformed it seen in table 5 in statistically significant way, 0.66 + 0.02 compared to 0.53 + 0.05 for the best ROSCOE score (Infrrmativeness Chai). Similarly, TFA 0.66 + 0.02 beats BPC 0.54 + 0.2.
Variation
Original
Model
Score
Percentage (%)
Score
Percentage (%
Gemma-2B-Bin
3/ 100
3
5/ 100
Gemma-7B-Ban
6/ 100
6
7/ 100
DeepSeck-Math-1 Math-7B-Basc
6/ 100
6
10/ 100
10
Mistral-7B-vO.3---
7 100
7
7/
Llama-3. 3.-BBB
7/ 100
7
100
6
Qwen2-Math-785 --B--aB-------
9/ 100
9
100 100
Meta Llama 3.1-70B
6/ 100
6
10/
Gemma-2B-Instruue
0/ 100
0
100
2
Gemma-77 Baiiaaa-BBInstttee 7B-Instruct
2/ 100
2
11 100
11
2/ 100
2
100
8
Llama-3.1-88 Instruct
7 100
7
10 100
10
Qwwe2--Math-7B-Insteee
14/ 100
/ 100
18
DeepSeck-Mauh-77-nnsrrcc
10/100
11
12/ 100
12
DeepSeek-Math-RL.
10/ 100
10
12 100
12
DeepSeck-R1--istill-- -i-esstill-Qwen
12 100
12
23 / 100
23
DeepSeck-RR RI--istilll-Laan
7/ 100
7
19 100
19
DeepSeek DeepSeek-Distilled- Seae--een-333
24/100
24
33/ 100
Claude-3.5 Sonnet
27 100
27
28 / 100
28
GPT-4o
22 100
22
30 100
30
ol-preview
43/ 100
43
51 100
Mistral-CodeAA122B
9/ 100
9
16 100
16
Table 2. Accuracy drops significantly on Putnam-AXIOM Variation compared to corresponding Original questions for nearly all models, These are mean accuracies over five trials.
14

---

Pulnam-AXIOM: A Functional and Static Benchmark
Number Theory
Counting and
Intermediate
Probability
Prealgebra
Precalculus
Geometry
Algebra
Algebra
Average
Metric
TFA
0.718
0.632
0.663
0.645
0.644
0.660
0.669
0.662
TFCE
0.486
0.442
0,458
0,468
0.501
0.466
0.505
0.475
Perpelexity
0.413
0.385
0.390
0.381
0.441
0.399
0.416
0.403
BPC
0.542
0.519
0.561
0.507
0.568
0.558
0.527
0.540
Info. Chain
0.494
0.536
0.486
0.616
0.550
0.460
0.542
0.526
Sem. Cov. Chain
0.450
0.499
0.437
0.559
0.523
0.449
0.486
0.486
Perp. Step
0.644
0.207
0.252
0.081
0.314
0.224
0.145
0.225
Table Correlation magnitude with respect model choice between proxy metries and boxed accuracy on the MATH dataset. Each entry corresponds to the absolute value of the correlation between the proxy metric and boxed accuracy on cach dataset (higher is highest and second highest correlation magnitude respectively each dataset. Notably,
better). Bold and underline corresponds among the ROSCOE metries. only the Informativeness Chain and Semantic Coverage Chain appear to be somewhat comparable across models. TFA performs best with average correlation around 0.67.
15

---

Putnam-AXIOM: A Functional and Static Benchmark
A.2. Proxy Metric Correlations With Boxed Accuracy
We the facebook/ roscce 512 -rbberta base embedding model for the computation of some the ROSCOE metrics. Everything else is set to the default in the released code.
Counting and Probability
Intermediate Algebra
Number Theory
Precalculus
Geometry
Prealgebra
Algebra
Average
Metric
TFA
0.718
0.632
0.663
0.645
0.644
0.660
0.669
0.662
-TFCE
0.486
0.442
0.458
0.468
0.501
0.466
0.505
0.475
-Perpelexity
0.413
0.385
0.390
0.381
0.441
0.399
0.416
0.403
-PCC
0.542
0.519
0.561
0.507
0.568
0.558
0.527
0.540
Grammar Step Grammar Step Max
0.024
0.007
0.274
0.112
0.109
0,204
0.471
0.165
0.033
0.088
0.103
0.045
0.173
0.134
0.070
0.092
Faithfulness
0.005
0.116
0.092
0.159
0.102
0.036
0.125
0.089
Informativeness Step
0.146
0.268
0.183
0.338
0.268
0.201
0.315
0.246
Informativeness Chain
0.494
0.536
0.486
0.616
0.550
0.460
0.542
0.526
Repetition Step
0.006
0.110
0.035
0.134
0.248
0.014
0.224
0.008
Reasoning Alignment
0.176
0.078
0.109
0.054
0.050
0.108
0.135
0.
-External Hallucination
0.055
0.131
0.093
0.168
0.179
0.111
0.109
0. 12.21
Redundancy
0.035
0.040
0.035
0.074
0.118
0.016
0.006
0.035
-Common Sense Error
0.324
0.489
0.347
0.289
0.456
0.425
0.248
0.368
-Missing Step
0.168
0.334
0,243
0.105
0.298
0.163
0.030
0.192
-Semantic Coverage Step
0.039
0.163
0.124
0.228
0.172
0.111
0.196
148
-Semantic Coverage Chain
0.450
0.499
0.437
0.559
0.523
0.449
0.486
0.486
Discourse Representation
0.080
0.086
0.162
0.115
0.221
0.142
0.029
0.119
Coherence Step vs Step
0.159
0.232
0.265
0.165
0.367
0.210
0.085
0.212
Perplexity Step
0.644
0.207
0.252
0.081
0.314
0.224
0.145
0.225
-Perplexity Chain
0.025
0.093
0.117
0.054
0.190
0.146
0.029
0.085
-Perplexity Step Max
0.256
0.024
0.388
0.171
0.107
0.182
0.323
0.097
Table 4. Correlation magnitudes between proxy metrics and boxed accuracy on the MATH dataset. Each entry corresponds to the absolute value the correlation between the proxy metric boxed accuracy on each dataset (higher better). Bold underline corresponds the highest and second highest correlation magnitude respectively for each dataset. The proxy metrics split into five categories: teacher forcing based, grammar based, embedding based, consistency based, and perplexity based. With exception of teacher forcing category, the remaining categories are all ROSCOE metrics. We refer to the ROSCOE metrics by the names used in the released code base, which differ slightly from those in the original paper. Notably, among the ROSCOE metrics, only the Informativeness Chain and Semantie Coverage Chain appear to be somewhat comparable across models. TFA performs the best with average correlation around 0.67. All metrics that were errors were negated so that all correlations are positive in the table.
16

---

Putnam-AXIOM: A Functional and Static Benchmark
Metric
Average + 95% CI
TFA
0.66 = 0.02
-TFCE
0.48 = 0.02
-Perplexity
0,40 = 0.01
-BPC
0.54 + 0.02
Grammar Step Grammar Step Max
0.16 = 0.11
0.09 = 0.02
Faithfulness
0.09 = 0.03
Informativeness Step
0.25 0.06
Informativeness Chain
0.53 0.05
Repetition Step
0.01 = 0.10
Reasoning Alignment
0.10 = 0.02
External Hallucination
0.12 = 0.02
Redundancy
0.04 + 0.02
Common Sense Error
0.37 0.06
-Missing Step
0
-Semantic Coverage Step
0.15 # 0.03
-Semantic Coverage Chain
0.49 = 0.04
Discourse Representation
0.12 + 0.05
Coherence Step vs Step
0.21 + 0.06
Perplexity Step
0.23 + 0.09
Perplexity Chain
0.09 # 0.04
Perrleexity Step Max
0.10 + 0.05
Table Demonstrates TFA the proxy metric most correlated with boxed accuracy on the MATH benchmark. Average correlations with 95% confidence intervals for proxy metrics on the MATH dataset. The proxy metrics are split into five categories: teacher forcing based, grammar based, embedding based, consistency based, and perplexity based (the last four are ROSCOE metries). refer to the ROSCOE metrics by the names used in the released code base, which differ slightly from those in the original paper.
Metric Category
Average 95% CI
TFA
0.66 + 0.02
Grammar Based (ROSCOE)
0.132 0.071
Embedding Based (ROSCOE)
0.288 0.088
Consistency Based (ROSCOE)
0.190 0.035
Perplexity Based (ROSCOE)
0.189 + 0.066
Global ROSCOE
0.200 0.028
Table Demonstrates TEA is significantly correlated than any of the ROSCOE metrics by category and globally. The average correlations with 95% confidence intervals for ROSCOE metrics on the MATH dataset. The ROSCOE metric categories are: grammar based, embedding based, consistency based, and perplexity based. The global ROSCOE score is provided an overall summary of all ROSCOE metrics.
17

---

Putnam-AXIOM: A Functional and Static Benchmark
A.3. Model Analysis Examples
Let n be positive integer. Determine, in terms of n,, the largest integer m with the following property: There exist real numbers 21,..., 22n with -1 < the intervals
Problem:
32n such that the sum of the lengths of
[s] 1 22 - [ag] 1=-1-1 . 1 1, ..... 2n 26-1
is equal to for all integers with - k S m.
Model's Response: are given positive integers and need determine the largest integer m, in terms such that there exist real numbers #1, .2....: 22n satisfying: 1. The are strictly increasing hetween -1 and 1:
32
2. For each integer with 1 S v m. the sum of the lengths of r. intervals is equal to 1:
Our goal is to find the largest integer m (in terms of n) satisfying these conditions.
**Gnneral C We have 2n variables and m equations.
Each equation is independent due to the distinct exponents.
To have solution, must have least many variables equations.
**Teerefore, the maximum number m is bounded by 2n.**
However, since we have 2n variables but they constrained by the ordering and the interval (-1, 1), the effective degrees of freedom are less than 2n.
*Conclusion:
Given the constraints and the inercasing number of equations with the largest integer for which the conditions can be satisfied is:
n
**Final **
The final answer is n 1 hope is correct.
Year: 2022
ID:
Final Answer:
Figure 5. An example response where OpenAl ol ol-preview gets the final answer correct, but made an unjustified claim. OpenAl ol-preview claims that the largest possible value for m is n. However, its only justification for this claim is that bounded by 2n. OpenAl ol- ol-preview does not justify why a value of m between n and 2n cannot work.
18

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem:
Find the least possible of in the plane that intersects both branches of the hyperbola ay and both branches the hyperbola zy (A set S plane called convex any two points the line segment connecting them is contained in S.)
Model's Response: find the least possible area convex in the plane that intersects both branches of the hyperbolas zy = 1 and zy = 1, we need to consider the geometric constraints. Let's start by visualizing these hyperbolas:
The hyperbola zy consists of branches in the first and third quadrants:
y
The hyperbola zy -1 consists of two branches in the second and fourth quadrants:
We aim to place convex set that intersects both branches of each hyperbola. One useful geometric fig- for this rectangle aligned along the hyperbolic We guarantee that the rectangle will intersect both branches zy and ry 1 by making its sides tangent to hyperbolas.
The geometric configuration simplifies the calculation:
minimum enclosing convex area tangency ideally is of square structure, yielding the unit hyperbola intersection:
area 4 (Since side 2 2 4formation in closed square).
This geometric construction ensures least, minimum convex interaction:
Final Answer: The final answer is 4 hope it is correct.
Year: 2007
ID: A2
Final Answer: 4
Figure example response where GPT-4o gets final answer correct, but made an unjustified logical step, resulting in non-rigorous solution. GPT-4o logically jumps the idea that the geometric shape that minimizes the area is rectangle. However, it does not attempt to justify this claim any way, and proceeds as though this is obvious.
19

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem: Given
11: 10iz + 10iz 11 = 0,
find the value of [z]. (Here is complex number and i² -1.)
Model's Response: To solve the equation
11: 10 + 10iz + 10iz 11 = 0
and find the value of [=]. where is complex number, we'll proceed step by step.
**Step 2: Assume |a| 148
Let's assume that |a| 1, which means : = e0 for some real number
"Step Substitute e into the Equation** Substituting z et into the equation, we get:
1 + 10i(e )0 + 10ie 11 = 0
Simplify each term: (e 16 10xe
)e 000
So the equation becomes:
11e 1010 + 10ie 918 + 100ie 11
**Step 6: Analyze the Equations**
The trigonometric equations complex and difficult to solve directly. However, notice that if 1, then s lies on unit circle, and its magnitude indeed Moreover, since we couldn't 't find any specific that satisfies both equations simultaneously (as shown by trying several angles), this suggests that specific angle 0 other than those corresponding to |a| 1 satisfy the equation. **Conclusion::**
Since manipulations suggest that circle), we conclude that:
necessary (and given the nature of complex numbers and the unit
*Final **
* The final answer
hope is correct.
Year: 1989
ID:
Final Answer:
Figure 7. An example response where GPT--o make logical leap and is completely incoherent. In this solution, the model jumps assume that |z = and performs series calculations. Then, concludes final answer simply because the equations made were too "complex and difficult" to solve.
20

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem: Let & be the smallest positive integer for which there exist distinct integers mi, Ma, Mag Ma, mas such the polynomial
p(a) = (x m1)(z m2) - ma) ma)( - ms)
has exactly nonzero cnefficients. Find set of integers m2, ma, ma, my for which this minimum is achieved and given the sum & ma + m2 + + ma + was the final answer.
The polynomial p(æ) (u ma)(2 m²) ma)(a m4) ms) has exactly & nonzero coefficients when the values of m, distinct integers. We want minimize &s and find set of values for m, that achieves this minimum.
Model's Response:
The expansion p(z) will include all possible products of the differences (x m.). For each term the form (x a), this generates a coefficient for that term, and this will happen for each combination of and j. including when i j.
The minimum possible value for l be calculated by considering the sparsity of nonzero coefficients in the polynomial's expansion. This achieved when the polynomial written in the form that eliminates as many terms as possible, leading to minimal number of nonzero coefficients.
For (r m=)(= (x= - (2 m²)(2 m4)(x ms). it can be achieved if the differences (x - m.) produce terms that include maximal cancellation. The problem becomes one of determining the minimal degree of the polynomial that includes maximal cancellation.
key observation here is that minimal nonzero coefficients are achieved when polynomial arranged such a way that most coefficients are zero due to symmetry and cancellation properties.
The smallest positive integer & for such polynomials happens at k: 2, for instance, (1: 1)(x 2)(x 3) (x 5) results in polynomial with precisely coefficients follows:
p(x) = - (1 +1+ 21+2+3++4 5)x²
(sum of products of each combination of 2 distinct elements)æ² etc...
Since the polynomial this problem demands exactly \ nonzero coefficients, the product of differences can lead cancellations resulting in minimal terms. Let's consider the initial observation:
T/a 1, 1/22 = 2, m/g 3, 174 ma
The resulting polynomial, (ar (x 2x - 3) - 4) -)
Final answer: 15
Year: 1985
ID: B1
Final Answer: 3
Figure example response where NuminaMath makes calculation error, includes irrelevant information, and misunderstands the problem. In this solution, solution, its value is 2, so final answer should be + + + 5 = 17, not 15; this is due misunderstanding of the problem and/or calculation error. Furthermore, the model includes the expression of the "resulting polynomial." which is completely irrelevant to the problem it's attempting to solve.
that the final answer is obtained by adding & + mm + m2 ma + m4 In the model's ''s final
21

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem: p(z) = op + ai + ... + Omx is polynomial with real cnefficients a,, then set
['(((z)) = ab + + ...... am
Let f(x) 3x + 7x Find, with proof, polynomial g(x) with real coefficients such that
(i s(0) 1, and
(i) (=))))) = ['(())))
for every integer Ti 2 1.
9+ +9 99 = 62. We want to find polynomial g(x) such that 90) = and ((F()))) [(g(=))) for every integer 2 Suppose
Model's Response: We have that (xx) = 3x 7 and thus =
g(x) x² bæ Then ('() = 1 62 + 11 + we find value for such that 6² 2 = 62, then we have [=)) = ((F(x)"). Solving 6² 2 62 gives 6² = 60, so b /60 = 2v15. Therefore, we can take ((x) 2 + 2 15x + 1. Since g(0) 1, g(x) satisfies both conditions.
Final Answer: The answer
2v15f
hope is correct.
1985
A6
Final Answer: 6x
5z 1
Figure 9. An example response where Qwen2-Math-77 hallucinates and misunderstands the problem. First, Qwen misunderstands that polynomial quadratic-which is not stated anywhere problem statement. Then, Qwen falsely states assumes T(g(z)) (x)())), then [(g(z)") [(f(x)) for all n. These mistakes lead Qwen to the wrong solution.
22

---

Putnam-AXIOM: A Functional and Static Benchmark
Problem: Let zo = 1 and for n 2 0, let In+1 3x, + [ v5J. In particular, = 5, 22 26, 712. Find a closed-form expression for #2007- ([a] means the largest integer S a.)
=136,
Model's Response:
Int1 3rn + In V5l
3rn + 1.,(v5 (v. 22 nx
3rn + nnl vn(v5- (o5vv-22 20n
5un + un (v5 2)
x. v5).
we yr = x2 v5), then v, = 3n/(2 + vB), so
2 v5 24 v5
Mat1 yn
yn
Thus, yn = yo for all n 2 1, so
2 - 2-
2 V5
V5.
Fo
2+ v5
2 + v5
(-2 + v5) v5)
-
Final Answer: The final answer is
2 - v5
hope it is correct.
Year: 2007
ID:
Final Answer:
2006
[+//
3997
1+
3997
v5
Figure 10. An example response where DeepSeek-Math-P7 makes an incorrect logical jump. In the series of equalitics in the beginning, DeepSeck simply removes the floor function and proceeds. However, the floor function is integral part of the problem, and without it, DeepSeek unable to come to the right solution.
23

---

Patnam-AXIOM: A Functional and Static Benchmark
A.4. Binary and Complex Questions
Several questions Putnam-AXIOM are binary, meaning that the question inherently has two possible answers. These include true/false questions, questions about divergence convergence, questions about the winner of two-player game. These questions make up 26 of the 262 question in Putnam-AXIOM Original; of the 60 questions of Putnam-AXIOM Variation, binary questions make up 8. We refer to all questions that are not binary as "complex" questions.
Given the guessable nature of these questions and our answer-matching evaluation method, models have much higher chance of randomly guessing the right answer on these questions. To discern whether the inclusion of these guessable questions significantly affects the overall difficulty of Putnam-AXIOM, we conducted an analysis of the accuracy of various models with and without the binary questions, with the overall accuracies in Figure ??.
We that, with the exception of Qwen2 Math 7B, almost all models have higher accuracy on Putnam-AXIOM with its binary questions than without, meaning that guessing is contributing to their success to some extent. However, we see that on the more advanced models models-Qwent Math 7B, GPT 4, and Claude Sonnet 3.5 the gap between the accuracies on the entire dataset and the accuracies on only complex questions much smaller. This likely because these models are capable enough that they successfully answer a similar percentage of complex questions and binary questions; less advanced models get significantly fewer complex questions correct than binary questions, so we see large accuracy gap. Based on the results of this experiment, we've decided to use only the complex questions for most of our evaluations such in Figure 3.
24

---

Patnam-AXIOM: A Functional and Static Benchmark
A.5. Teacher-Foreed Accuracy (TFA) Under Proprietary APIs
To compute Teacher-Foreed Aceuracy (TFA), we use teacher forcing, where the model predicts each token based on the ground truth sequence up to that token. This requires conditioning each prediction the true preceding tokens, rather than relying on previously generated tokens.
Frontier models under proprietary APIs, however, typically predict the next token only based on previous predictions, conditioning only the first token on the full ground truth. To compute TFA accurately with these models, we can "brute-force" it by generating tokens one at a time, conditioning each new prediction on the entire ground truth sequence. This approach requires re-feeding the full sequence history for every token the ground truth, making computationally intensive.
For string of N tokens, this method requires ((N2) operations because we reprocess the token history at each prediction step. By contrast, open-source models allow us to compute TFA in single forward pass, reducing the complexity to O(N).
25


Reviews:
[{"review_id": "bECNpJVPbG", "reviewer": "Reviewer_6ec1", "summary": "Putnam-AXIOM is a new benchmark designed to assess higher-level mathematical reasoning in large language models (LLMs), using 522 challenging problems from the William Lowell Putnam Mathematical Competition. To address data contamination, the authors introduce functional variations of 100 problems by altering variables, constants, and phrasing, enabling infinite novel yet equally difficult problems. Evaluation shows that even top models like GPT-4o and o1-preview perform significantly worse on these variations, highlighting the limitations of current models. The benchmark also proposes new reasoning metrics like Teacher-Forced Accuracy (TFA) to go beyond final boxed answers and better capture reasoning depth.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and detailed review, particularly for recognizing that our work addresses an important gap by introducing a benchmark capable of measuring higher-level mathematical reasoning, and appreciating the thoroughness of our experimental designs and comprehensive ablation studies. Your positive feedback about the validity and utility of our claims and metrics is greatly valued.\n\n> ***Figure 3 may have the labels for Original and Variation swapped***\n\nThank you for pointing this out, you are correct that the labels for Figure 3 were mistakenly swapped. We will be sure to include the corrected version of the figure in our revised submission.\n\n> ***discussion on to what extent can TFA—or any of the other proposed metrics—address the issue of correct answers with flawed reasoning, which is notably common in leading models like o1***\n\nWe appreciate the reviewer’s interest in how Teacher-Forced Accuracy (TFA) handles scenarios in which the model’s final answer is correct despite a flawed chain of thought. TFA differs from standard auto-regressive (AR) generation because each predicted token is conditioned on the gold-standard solution, thus focusing on whether the model *recognizes* a correct reasoning path rather than whether it can generate one autonomously. From a different perspective, it uses the forward during training instead of during inference for generation and then computes per token accuracy match to the gold reference string.\n\nBecause TFA never lets the model produce its own possibly incorrect steps, it does not directly reveal flawed intermediate reasoning in AR generation. This is one reason we were motivated to rigorously verify TFA’s validity – by analyzing its correlation with boxed-answer accuracy — ensuring that it reliably measures deeper competence across multiple problems (MATH, Putnam-AXIOM), fourteen models and six model families.\n\nIndeed, one could explore hybrid approaches —for instance, partially teacher-forcing the solution, then auto-regressively generating the remainder—but these methods are often computationally expensive or require custom hardware code. In line with Occam’s Razor, we opted for TFA’s simpler, scalable approach first.\n\nWe will clarify these points in the revised manuscript, adding discussion in the TFA section and appendix to make explicit the difference between TFA and AR generation.\n\n# Evaluating the Effectiveness of Variations on Contamination\nDuring the rebuttal phase, we have performed our own fine-tuning experiment with DeepSeek-R1 Distill-Qwen-7B (we chose this model because it was the best-performing model of its parameter size in Table 2). On the Putnam-AXIOM Variation dataset, this baseline model received 12% accuracy on the variation questions and 23% accuracy on the corresponding original questions.\n\nWe performed full-parameter SFT (intended to mimic or exaggerate the effects of data contamination) on the full Putnam-AXIOM Original dataset (with all 522 questions), running for 5 epochs. We then evaluated the fine-tuned model on Putnam-AXIOM Variations (with 100 variation questions and 100 corresponding original questions). After fine-tuning, the model received 33% accuracy on the variations and 80% accuracy on the corresponding original questions. \n\nClearly, the full-parameter SFT successfully contaminated the model such that it was able to attain saturation on the corresponding original problems with a 57% increase in accuracy; however, the variations still proved to be challenging for the model to solve, given a mere 21% increase in accuracy. Our experiment shows that our variations are a useful tool in combating data contamination.", "claims_and_evidence": "Yes. This paper mainly has two claims: 1. variations of the Putnam benchmark are likely to cause trouble to existing model (due to issues like data contamination); 2. the proposed teach-forced accuracy (TFA) is better correlated with boxed accuracy than existing metrics like ROSCOE. To me, both claims have been empirically validated.", "methods_and_evaluation_criteria": "Yes. Putnam is a solid source of dataset for measuring mathematical reasoning. The effectiveness of the newly proposed TFA metric has also been validated on the widely used MATH dataset.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs look good to me. I especially appreciate the comprehensive ablation study in the appendix.", "relation_to_broader_scientific_literature": "Both the new dataset and the automatic proxy metric offer valuable contributions to mathematical reasoning research, especially as many existing benchmarks have become saturated.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- I’d love to see some discussion on to what extent can TFA—or any of the other proposed metrics—address the issue of correct answers with flawed reasoning, which is notably common in leading models like o1.", "other_comments_or_suggestions": "- Figure 3 may have the labels for Original and Variation swapped. Based on my understanding and the data in Table 2, the blue intervals should correspond to the Variation set.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPJexcHhhF", "reviewer": "Reviewer_Lp2w", "summary": "The paper introduces a new dataset comprised of mathematical problems that have appeared at the Putnam contest for university students. To enable automatic evaluation, the problems are selected or have been reprhased to be such that the solution can be checked automatically as a boxed answer. This rephrasing of the problems need not be semantically preserving, but nonetheless aim to capture the main logical intuition of how the original problem is to be solved. Further, to combat contamination, the authors introduce a functional variation dataset where problems can be generated as (very) similar variants of the original problem by for instance changing constants in the original problem. The paper evaluates a number of open and some proprietary models on these datasets, both in terms of scores, but also via other proxy metrics, in the process demonstrating that current models do not perform well on these tests.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and detailed response. We deeply appreciate the time and effort you invested in providing such valuable feedback.\n\n> ***how are the Putnam problems selected exactly?***\n\nAs outlined in Section 3.1, we selected problems based on two primary criteria:\n\n1. Each problem must yield a definitive final answer that can be enclosed in a \\boxed{} format, enabling fully automated evaluation without human grading. These adjustments preserve the original mathematical insights while eliminating the need for human evaluation.\n\n2. In constructing the dataset, we deliberately excluded binary-answer questions (e.g., true/false or yes/no), as these allow models to succeed via random guessing. These accounted for around 10% of the Putnam problems. This filtering reflects our effort to ensure both rigor and meaningful signal in model performance.\n\nOur datasets excluded questions that were not conducive to these standards, similar to the PutnamBench paper. We curated problems to ensure broad coverage across topics (Algebra, Combinatorics, etc.) and difficulty levels (indicated by 1-6). Based on Putnam’s own difficulty rating we had 205 easy, 182 medium, and 135 hard. \n\nThe Variation set is a curated subset of the main dataset, where expert mathematicians systematically alter surface features—like constants or variable names—without changing the problem’s core reasoning. This helps prevent models from relying on memorization.\n\n> ***“experimental evaluation on contamination of the selected Putnam problems.”***\n\nDuring rebuttal, we fine-tuned DeepSeek-R1 Distill-Qwen-7B (chosen as the best-performing model in its size class) on the full Putnam-AXIOM Original dataset for 5 epochs using full-parameter supervised fine-tuning (SFT) intended to mimic or exaggerate the effects of data contamination. The baseline model achieved 12% accuracy on variations versus 23% accuracy on corresponding originals. After fine-tuning, accuracy increased to 33% on variations versus 80% on corresponding originals. This demonstrates that while contamination allows near-saturation on original problems (+57% accuracy), variations remain challenging (+21% accuracy), confirming their effectiveness against contamination. We plan to extend these experiments to additional models.\n\n> ***lack of technical depth...***\n\nOur goal was not to complicate the problem statements themselves but rather to preserve the intrinsic difficulty of the Putnam examination while making the benchmark automatically gradable at scale. The “boxed-answer” approach, though seemingly straightforward, ensures that the dataset is both substantial and tractable for automated evaluation—an approach shared by similar math benchmarks.\n\n> ***“how other semantically-equivalent text paraphrasings...”***\n\nThank you for pointing out this area of exploration. We tried paraphrasing the variation questions with the assistance of LLMs and meticulously verified by humans. This resulted in minimal changes in accuracy and thus didn’t report paraphrasing for simplicity. We will clarify this point in our revised submission.\n\n> ***the fact that the models are not scoring well on...***\n\nWe compared model responses between the Original and Variation datasets. Both responses adopted a “step by step” narrative style. However, original responses tend to follow a systematic, clear approach while responses often reveal internal doubts result in incomplete algebraic details or solutions. One such example is 1947 B4 where the question is:\n\n > Given $P(z) = z^2 + az + b$, a quadratic polynomial of the complex variable $z$...Find the values of $a$ and $b$ and give their sum.\n\nOriginal Solution (shortened for readability):\n> Let's think step by step. Since $|P(z)| = 1$ for all $|z| = 1$, then for $z$ on the unit circle, $|z^2 + az + b| = 1$. If $z \\\\neq 0$, then $|1 + a/z + b/z^2| = 1$, so $|Q(w)| = 1$ where $w = 1/z$ and $Q(w) = w^2 + a w + b$...Therefore, the sum $a + b = b - 1/b$. But since $|b|=1$, we have $b = e^{i\\\\phi}$ for some angle $\\\\phi$, so $a + b = e^{i\\\\phi} - e^{-i\\\\phi} = 2i \\\\sin \\\\phi$.\n\nVariation Solution (shortened):\n> Let's think step by step. Since $P(p)$ is a quadratic polynomial and $|P(p)| = 88$ for all $|p|=88$...Okay, so I've got this problem here where I need to find the angle A in triangle ABC. The conditions given are a bit specific: angle A is less than angle C...So,180 - \\u03b1/2 > 90 Which gives, \\u03b1/2\n\nThe original solution directly addresses the problem, while the variation’s solution “hallucinates” a triangle ABC scenario and omits key steps. We see three main causes:\n1. LLMs often appear to recall answer patterns rather than genuinely solve problems\n2. Small changes to variable names or constants (e.g., from *x* to *p*) can disrupt these memorized patterns, suggesting a lack of true variable abstraction.\n3. Unusual symbol choices or number pairings can misalign the prompt, triggering irrelevant or hallucinatory reasoning steps.", "claims_and_evidence": "Yes, the paper substantiates the claims made in the introduction/abstract, but some questions are open, which I elaborate on below.", "methods_and_evaluation_criteria": "The evaluation makes sense, however, one concern I have, which the authors can hopefully clarify is: how are the Putnam problems selected exactly? Give the low scores of the models, on problems that are very likely in the training data, this is particularly important. For instance, are some easy problems omitted? Or problems for which it is not trivial to convert it to one with a boxed answer? It is important to understand whether a systematic process is being followed here. Similarly for the functional variation dataset.", "theoretical_claims": "This is an experimental paper which introduces a dataset, there are no theoretical claims.", "experimental_designs_or_analyses": "As mentioned earlier, the selection of the problems and how that is done, is a bit unclear and it is important. Further, I would have hoped that the authors would have done a bit more on the contamination part, that is, checking for contamination of the selected Putnam problems.", "relation_to_broader_scientific_literature": "Obtaining meaningful mathematical benchmarks, ones on which models do not yet work well, is important. The work done to produce this dataset is valuable, and the dataset of Putnam problems, can be useful to develop better models. Similarly, for the functional variation dataset.", "essential_references_not_discussed": "Given the mentions of contamination and to better motivate the functional variation dataset, I was hoping to see some experimental evaluation on contamination. of the selected Putnam problems. For instance, using [1]. \n\n[1] ConStat: Performance-Based Contamination Detection in Large Language Models, NeurIPS 2024.", "other_strengths_and_weaknesses": "The datasets introduced by the paper are useful and the paper is well written. \n\nThe main concern I have, aside from the ones mentioned already, is the lack of technical depth. For instance, the rephrasings to get a boxed answer are quite straightforward. Similarly for the functional dataset. It would have been interesting to see how other semantically-equivalent text paraphrasings of the problems (many ways to do so), would have affected the results, and similarly for the contamination analysis I mentioned already. Ultimately, the fact that the models are not scoring well on  (slight variants of) problems that are very likely in the dataset, is interesting, and one would have hoped to see more evaluation here.\n\nStill, I believe the datasets can serve as a useful benchmark for future research.", "other_comments_or_suggestions": "- it was unclear early on whether the rephrasings to get a boxed answer are semantically equivalent; they are not, may be good to state.\n- throughout the paper it was unclear how problems are selected, but this was mentioned already above.\n- line 313, it is unclear what \"to generate evaluation data\" means.\n- may be I am misreading Fig 3, but I would think original should be scoring higher in accuracy than variation?", "questions_for_authors": "Already mentioned in the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6DZeyrumor", "reviewer": "Reviewer_W8M8", "summary": "This paper introduces a new mathematical benchmark made of 522 questions from the William Lowell Putnam math competition from 1938 to 2023, among which 100 can be infinitely modified by changing variable names (100 of them) and constant values (37/100 of them) – called the Variation split.\nThis paper also presents teacher-forced accuracy (TFA) as an additional evaluation metric to give a more complete assessment of LLMs’ reasoning abilities.\n\nExperiments show that current state-of-the-art models struggle to achieve good performance with OpenAI o1-preview achieving ±42% accuracy. Experiments also show that models evaluated on the Variation split are weaker than when evaluated on the corresponding 100 original questions (suggesting some data was present during pre-training).\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and detailed response.\n\n> “This is not the correct approach as you already have a metric that will correlate 100% of the time with accuracy, it is accuracy itself. The whole point of using proxy metrics is to evaluate something else than accuracy, i.e. the reasoning chain logic. The correlation between these pseudo metrics should be evaluated against human labels or a strong LLM as a judge such as GPT4o, not against accuracy.”\n\nWe thank the reviewer for highlighting the need to evaluate chain-of-thought independently from final-answer accuracy. While we chose a metric that correlates with accuracy (as better reasoning should yield correct answers), we recognize teacher-forced accuracy (TFA) serves primarily as a fine-grained measure of how closely a model follows valid derivations token by token. We acknowledge TFA doesn't necessarily equal reasoning quality without correlation to gold-standard step-by-step evaluations. We will supplement our analysis with human or GPT-4 judgments on intermediate steps to verify whether TFA truly tracks logical consistency rather than just final correctness. These results will be updated during the rebuttal period. \n\n> “The claim that changing variable names, constant values, and turn of phrase will tackle data contamination is not convincing (see explanation in Other Strengths And Weaknesses section).”\n\nWe acknowledge that our functional variations don't completely solve data contamination but rather mitigate its effects. Our experimental results show variations consistently decrease accuracy scores across SOTA models, providing strong evidence for mitigation.\nDuring rebuttal, we fine-tuned DeepSeek-R1 Distill-Qwen-7B (chosen as the best-performing model in its size class) on the full Putnam-AXIOM Original dataset for 5 epochs using full-parameter supervised fine-tuning (SFT) intended to mimic or exaggerate the effects of data contamination. The baseline model achieved 12% accuracy on variations versus 23% accuracy on corresponding originals. After fine-tuning, accuracy increased to 33% on variations versus 80% on corresponding originals. This demonstrates that while contamination allows near-saturation on original problems (+57% accuracy), variations remain challenging (+21% accuracy), confirming their effectiveness against contamination. We plan to extend these experiments to additional models.\n\n> The numbers in Table 1 do not match the scatter plot in Figure 4.\n \nThank you for pointing out this discrepancy. The accuracy values for TFA in Table 1 and Figure 4 are from a previous dataset version (236 problems), while Table 1 model accuracies are from the expanded dataset (522 problems). For proper comparison, we plotted TFA scores against accuracies using the 236-problem dataset. Due to resource constraints, we haven't yet run TFA on the expanded dataset but are working on this and will provide updated results during the rebuttal phase.\n\n> Why was “perplexity Step” chosen instead of ‘’Common Sense Error’’?  In \naddition, what does it mean to be “chosen” for a pseudo metric if anyway at the \nend of section 4.3 only the top one (TFA) is “selected” to evaluate models (table 1 \n& figure 4).\n\nWe choose ROSCOE's perplexity step for comparison with our perplexity proxy metric. However we didn't make this selection clear in the paper. The experiments that backed up our rationale for choosing TFA versus baselines is detailed in Section 4.3 of our paper. We did all the experiments to demonstrate our reasoning / rationale to why we chose TFA vs *baselines*. TFA had the highest correlation and performed best on MATH which is why we decided to select it. \n\n> In particular, Table 3 is a pure subset of Table 4 without any other additional \ninsight or information. It is therefore unclear what message it provides compared \nto Table 4. Table 4 is probably enough.\n\nWe appreciate your suggestion and acknowledge that Table 3 provides no additional insights beyond Table 4. While initially included to highlight the highest-scoring metrics, we are amenable to removing this redundant table in the revised version.\n\n> The caption in Figure 3 may be inverted: right now it shows that the \naccuracies on the Original questions are weaker than on the Variation split.\n\nThank you for pointing this out, you are correct that the labels for Figure 3 were mistakenly swapped. We will be sure to include the corrected version of the figure in the finalized version.\n\n> What is the distribution of problem difficulty (1-6) in the original & \nvariation splits?\n\nAvg difficulty: Original (2.46) vs Variation (2.48) datasets - comparable ratings strengthen our claim that accuracy differences stem from data contamination rather than problem difficulty. Distribution (1-6): Original - 68(1), 109(2), 97(3), 93(4), 75(5), and 80(6) problems; Variation - 8(1), 27(2), 24(3), 11(4), 10(5), and 20(6).", "claims_and_evidence": "Claims are somewhat supported by evidence.\n\n(1) The claim that TFA provides a more complete assessment of LLM’s reasoning abilities compared to final answer accuracy makes sense intuitively but is not supported by strong evidence (see explanation in Experimental Designs Or Analyses section).\n\n(2) The claim that changing variable names, constant values, and turn of phrase will tackle data contamination is not convincing (see explanation in Other Strengths And Weaknesses section).", "methods_and_evaluation_criteria": "Pseudo metrics (teacher forced, and ROSCOE) are evaluated based on their correlation with final answer accuracy. They should be evaluated based on other metrics that directly inspect the reasoning ability of LLMs (either human or LLM as a judge labels).", "theoretical_claims": "No theoretical claims found.", "experimental_designs_or_analyses": "1. The motivation for using pseudo metrics like Teacher Force & ROSCOE is to evaluate the models at their reasoning skills because the final \\boxed{} answer may not represent this well enough. However when evaluating these metrics, correlation with the final answer accuracy is used to select the top pseudo metric. This is not the correct approach as you already have a metric that will correlate 100% of the time with accuracy, it is accuracy itself. The whole point of using proxy metrics is to evaluate **something else** than accuracy, i.e. the reasoning chain logic. The correlation between these pseudo metrics should be evaluated against human labels or a strong LLM as a judge such as GPT4o, not against accuracy.\n\n2. The numbers in Table 1 do not match the scatter plot in Figure 4. For instance:\n- Gemma-7B-Base in Table 1: (0.046 acc, **0.784 TFA**) – in Figure 4: (±0.043 acc, **<0.74 TFA**)\n- DeepSeek-Math-7B-Base in Table 1: (**0.0402 acc**, 0.779 TFA) – in Figure 4: (±**0.06 acc**, ±0.772 TFA)\n- Qwen2-Math-7B-Base in Table 1: (**0.0957 acc**, 0.770 TFA) – in figure 4: (**<0.06 acc**, ±0.765 TFA)\n\nIn particular, the numbers in Table 1 do not seem to correlate as strongly as in the figure 4.\n\n3. Section 4.3 says that the “_chosen metrics_” are reported in Table 3 (in the appendix) but Table 4 (in the appendix) shows that ‘’Common Sense Error’’ correlates more (0.368) than the chosen ‘’Perplexity Step’’ (0.225). Why was “perplexity Step” chosen instead of ‘’Common Sense Error’’?\nIn addition, what does it mean to be “chosen” for a pseudo metric if anyway at the end of section 4.3 only the top one (TFA) is “_selected_” to evaluate models (table 1 & figure 4).\n\n4. Experimental results should report model’s performance on some of the most interesting proxy metrics. Note that “interesting” here should not mean high correlation with final answer accuracy, but rather a high correlation with human or LLM judgment of the reasoning chain logic. Intuitively, “Common Sense Error”, “Hallucination”, and “Repetition” or “Redundancy” would be things to avoid in a reasoning chain, and thus point towards interesting pseudo metrics (to be validated by llm or human as a judge experiments).", "relation_to_broader_scientific_literature": "This is a dataset paper. It provides another Math benchmark to evaluate LLMs. This is similar to MATH and GSM8k but the problems are more challenging.\nIn addition, this paper proposes a subset of questions that can be randomly altered to always provide brand new evaluation questions.", "essential_references_not_discussed": "Nothing essential is missing.", "other_strengths_and_weaknesses": "**Strengths:**\n\nThis paper is well-written and organized. It presents a novel set of problems to evaluate LLMs, and importantly a mechanism to randomly change 100 questions so that they are always different, thus limiting data contamination during large pre-training runs.\n\n**Weakness:**\n\nModifying variables in the Variation split does not change the reasoning or final answer of a problem, thus having minimal impact on data contamination.\nChanging constants on the other hand can yield to different steps in the reasoning chain and different answers. Nonetheless, the strategy / logic used to solve this problem will be very similar if not the same. As such, the impact on data contamination is limited as LLMs can memorize the “strategy” to solve these exact problems. It is better than LLMs memorizing the raw values and numbers of each variables, but still not as great as novel questions requiring new reasoning strategies. This is of course hard to achieve programmatically, but that is the only way to truly avoid data contamination during finetuning.\n\nOut of the 100 problems selected for Variation only 37 have different constants.", "other_comments_or_suggestions": "The caption in Figure 3 may be inverted: right now it shows that the accuracies on the Original questions are **weaker** than on the Variation split.", "questions_for_authors": "What is the distribution of problem difficulty (1-6) in the original & variation splits?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bsWrazTTVh", "reviewer": "Reviewer_Vq1J", "summary": "This paper presents Putnam-AXIOM, a benchmark of 522 problems from the Putnam competition along with their ground truth solutions. The paper also proposes some manual modification to the original problems to make evaluation less ambiguous, as well as manual variable and constant substitutions for 100 problems (Putnam-AXIOM Variation). The paper evaluates both open-source and proprietary LLMs on the original and varied problems and show that LLMs achieve higher accuracy on the original problems, suggesting data contamination problem exists.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and detailed response. We deeply appreciate the time and effort you invested in providing such valuable feedback.\n\n> I am under the impression … not proof-based.\n\nWe'd like to clarify that Putnam-AXIOM explicitly addresses the challenge of evaluating symbolic and proof-based questions through several carefully designed methodological choices:\n\nWe retained numerous proof-based questions by applying our Modified Boxing technique (detailed in Section 3.1), which transforms these questions to produce unambiguous, final answers while preserving their mathematical complexity. This approach allows for automatic evaluation without sacrificing the deep reasoning required in the original problems.\n\nOur methodology supports a wide range of answer formats through our equivalence function, similar to the approach used in MATH [1]. For Putnam problems that involve complicated final answers, we added minimal next steps that maintain \"the same mathematical insights and problem-solving approaches required by the original problems\" while ensuring solutions converge to single, automatically evaluable answers.\n\nWe deliberately excluded binary-answer questions (~10% of Putnam problems) to eliminate success through random guessing.\n\n\n[1] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., ... & Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.\n\n>These numbers are misleading … main text.\n\n\nThank you for pointing out that mistake, the corrected values should be that “DeepSeek-R1-Qwen-32B shows the steepest drop at 27.3%, followed by GPT-4o at 26.7% and o1-preview at 15.6%.\" This correction will be reflected in the final version of our paper.\n\nHowever, we emphasize that our variations are highly effective at mitigating contamination. During rebuttal, we fine-tuned DeepSeek-R1 Distill-Qwen-7B (chosen as the best-performing model in its size class) on the full Putnam-AXIOM Original dataset for 5 epochs using full-parameter supervised fine-tuning (SFT) intended to mimic or exaggerate the effects of data contamination. The baseline model achieved 12% accuracy on variations versus 23% accuracy on corresponding originals. After fine-tuning, accuracy increased to 33% on variations versus 80% on corresponding originals. This demonstrates that while contamination allows near-saturation on original problems (+57% accuracy), variations remain challenging (+21% accuracy), confirming their effectiveness against contamination. We plan to extend these experiments to additional models.\n\nHowever, the dataset proposed here has limited novelty due to existing work on PutnamBench.\nThank you for pointing out this possible confusion. As mentioned in our introduction, Putnam-AXIOM focuses on natural-language problems with final answer verification, functional variations to address contamination, and proxy metrics for reasoning. PutnamBench translates Putnam questions into formal statements for theorem-proving languages. Our datasets serve distinct purposes despite being sourced from the same competition.\n\n> The approach to create … less novel.\n\nThank you for pointing out this concern. We agree that manual effort is required to create variations. While we created a script using GPT-4o to assist in variation creation, significant manual verification is still required. We note that most difficult benchmarks (like MATH) also require substantial manual effort.\n\n> Although TFA seems … practical use.\nThank you for pointing this out. We acknowledge that boxed accuracy correlates better with general perception. However, per Goodhart's Law, optimizing too much for any proxy diminishes its value. We aim to create a fast, inexpensive proxy for reasoning, not necessarily the optimal metric.\n\nThe below sentence is confusing and seems to come from nowhere … actual comprehension.\nWe standardized placement of boxed answers at the end of solution strings based on findings from Chain-of-Thought prompting, where models provide more coherent solutions when reasoning through the entire problem before giving the final answer. For TFA, this means models are more penalized for formatting deviations and less penalized for solutions with reasoning similar to benchmark solutions.\n\n>PutnamBench has 640 problems … were excluded?\n\nThis discrepancy is due to the fact that Putnam-AXIOM ultimately aims to serve a different purpose than PutnamBench: we selected problems from the Putnam based on their ability to contribute a final, boxed answer for our automatic evaluation process. PutnamBench also pruned ~60% of Putnam problems. We excluded binary-answer questions but included many proof-based questions by rewording them to produce evaluable final answers while preserving difficulty.\n\n>In Figure 3 … missing something?\n\nThank you for pointing this out. The labels for Figure 3 were indeed mistakenly swapped. We will correct this in the final version.", "claims_and_evidence": "Some claims are inaccurate or exaggerating:\n\n1. > On Line 103-105 of Related Work: \"Our Putnam-AXIOM dataset addresses these limitations by offering challenging Putnam problems with fully-written solutions and easily evaluable answers.\"\n\nI am under the impression that Putnam-AXIOM did not address the costly human evaluation issue for symbolic and proof-based questions. Instead, it tries to avoid this problem by only keeping the problems that are not proof-based.\n\n2. > On Line 323-324 of Section 4.1:  Looking at the numbers highlights significant accuracy declines across models: DeepSeek-R1-Qwen-32B shows the steepest drop at 37.5%, followed by GPT-4o at 36% and o1-preview at 17%.\n\nThese numbers are misleading and it seems that the authors are subtracting the min of the lower accuracy type from the max of the higher accuracy type. From Table 2, it is clear that the accuracy drop is much less than what is reported in the main text.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally sound.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experimental designs and analyses are generally sound.", "relation_to_broader_scientific_literature": "The finds of the paper is interesting. However, the dataset proposed here has limited novelty due to existing work on PutnamBench.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper motivates the problem well and provides a comprehensive evaluation of the benchmark with many models with different tiers of capabilities. The analysis also makes sense.\n2. The modified boxing approach is very interesting and I believe this can be a promising way to sanitize benchmarks.\n\n**Weaknesses**\n1. The approach to create modified boxing and variation dataset especially the constant change seems to be largely a manual effort. The author also mentions that not all problems are suitable for such variation. This makes the approach less generally applicable and less novel.\n\n2. Although TFA seems to correlate well with boxed accuracy, from Table 1 and Figure 4, it is clear that it cannot be used as a proxy for model selection. That is, the model with the highest TFA intuitively is not a strong model (Gemma-7B-Base) and boxed accuracy correlates better with general perception. This makes the TFA less appealing for practical use.\n\n3. The paper has many presentation flaw that significantly affects understanding of the results. See questions below.\n\nOverall, I believe the main novelty of the paper is on the strategies used for creation of variation dataset and alternative evaluation metric. However, the strategies appear to be manual (not very scalable without introducing errors), and the evaluation metric seems to be noisy.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. The below sentence is confusing and seems to come from nowhere. Can the authors explain what this means. Also does this modification affect the correlation performance of TFA?\n\n> . Further, we standardized the placement of boxed answers by relocating them to the end of each solution string to minimize unintended emergent behaviors leading to evaluations that are less “harsh” or prone to penalizing the model for formatting deviations rather than actual comprehension.\n\n2. In Table 1, NuminaMath-7B-Base has accuracy of 10.34%. However, in Figure 4, it is shown to be less than 5% with the pentagon symbol. Can the authors explain the inconsistency?\n\n3. PutnamBench has 640 problems whereas this dataset has 522. What is causing the discrepancy and is it because proof-based problems were excluded?\n\n4. In Figure 3, it seems that the original accuracy is generally lower than the variation accuracy. However, the text suggests original accuracy is higher. Am I missing something?\n\n\n## Update After Rebuttal\nAfter reading the author's rebuttal and the other reviews, I have decided to increase my rating. As noted by several reviewers, there are still issues with the writing and presentation, including errors and occasional misrepresentations. However, I believe that the paper's contributions outweigh these weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper received unanimous accepts from reviewers. The authors are encouraged to incorporate the reviewers' suggestions in the camera-ready version.
Decision: Accept (poster)