Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning

Authors: aryan  gulati
brando  miranda
eric  chen
emily  xia
kai  fronsdal
bruno  de
moraes  dumont
sanmi  koyejo
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-26 06:41:27

Abstract: As large language models (LLMs) continue to advance, many existing benchmarks designed to evaluate their reasoning capabilities are becoming saturated. Therefore, we present the Putnam-AXIOM Original benchmark consisting of 236 mathematical problems from the William Lowell Putnam Mathematical Competition, along with detailed step-by-step solutions. To preserve the Putnam-AXIOM benchmark's validity and mitigate potential data contamination, we created the Putnam-AXIOM Variation benchmark with functional variations of 52 problems. By programmatically altering problem elements like variables and constants, we can generate unlimited novel, equally challenging problems not found online. We see that almost all models have significantly lower accuracy in the variations than the original problems. Our results reveal that OpenAI's o1-preview, the best performing model, achieves merely 41.95\% accuracy on the Putnam-AXIOM Original but experiences around a 30\% reduction in accuracy on the variations' dataset when compared to corresponding original problems.

Main paper: 
 introduction: 
 The ability for Large Language Models (LLMs) to reason about complex problems has a plethora of applications in many fields such as economics [Zhang et al., 2024] , drug discovery [Bran et al., 2023] , and even simulations of human behavior and society [Park et al., 2023] . The prominence of this ability has led to significant development in the performance of LLMs on many reasoning benchmarks.Outpacing Current Evaluations. Indeed, advanced models like GPT-4 [OpenAI, 2023] and Gemini Ultra [Team, 2023] have even surpassed human-level performance on many benchmarks like MMLU [Hendrycks et al., 2020] and MMMU [Yue et al., 2023] . Similarly, LLMs have seen astonishing progress in other challenging benchmarks like GSM8K [Chen et al., 2022] and MATH [Hendrycks et al., 2021] , with SOTA models attaining nearly 90% accuracy on MATH [Lei, 2024] and nearly perfect accuracy on GSM8K [Zhong et al., 2024] . Though this progress is a testament to the rapidly evolving ability and utility of LLMs, it also presents a large problem: Existing datasets are no longer sufficient to evaluate the reasoning abilities of LLMs.Data Contamination. Compounding this issue is one of the most significant problems facing evaluation datasets today, i.e., data contamination. As LLMs are increasingly trained on more of the internet, an increasing number of the open-source problems used in evaluation benchmarks are incorporated in the training data of these models. A model can therefore display artificially high "reasoning ability" by simply memorizing the answers it has seen undermining evaluation integrity.To address these limitations, we introduce the Putnam-AXIOM (Advanced eXamination of Intelligence in Operational Mathematics) dataset, a novel and challenging compilation of highlevel mathematics problems sourced from the prestigious William Lowell Putnam Mathematical Competition, an annual mathematics competition for undergraduate college students in North America which requires advanced mathematical reasoning and covers a wide range of university-level mathematical concepts. Further, we also introduce functional variations of this AXIOM dataset to combat data contamination taking inspiration from the solution employed by Srivastava et al. [2024] .These are small variations of questions on the Putnam that are equally difficult as the Putnam but unavailable anywhere on the internet. AXIOM enables fully automated evaluations by requiring models to provide final answers within "\boxed{}" brackets which can then be extracted and compared to the ground truth final solution using an equivalence function1 . This approach eliminates the need for human evaluation, allows for complex open-ended answers, and avoids the limitations of multiple-choice formats, thus maintaining rigor while enabling scalability. Initial evaluations on Putnam-AXIOM demonstrate its difficulty with OpenAI o1-preview scoring less than half at 41.95%, while GPT-4o achieves only 17.80%. Even math-specialized models such as Qwen2-Math-7B and Qwen2-Math-7B-Instruct perform poorly, scoring 5.51% and 11.8% respectively. Performance further declines on functional variations of Putnam-AXIOM, which include significant drops for most models, decreasing by 20-30% in relative performance. These low scores underscore Putnam-AXIOM's utility for measuring LLMs' advanced reasoning capabilities, while the variations scrutinize true reasoning skills by exposing the models' reliance on memorization. 
putnam-axiom original dataset: 
 Dataset. The Putnam-AXIOM Original Dataset contains 236 problems curated from the William Lowell Putnam Mathematical Competition posed between 1985 and 2023. These problems were selected based on their ability to yield final "\boxed{}" solutions ensuring compatibility with our automated evaluation. The dataset encompasses various subjects within university-level mathematics categorized into 11 distinct domains -Geometry, Algebra, Trigonometry, Calculus, Linear algebra, Combinatorics, Probability, Number theory, Complex numbers, Differential equations and Analysis.To maintain a consistent and rigorous evaluation, each problem retains its original exam ID, which indicates its difficulty level (A or B for sitting, 1-6 for increasing complexity). This categorization helps in evaluating subject-specific understanding and overall problem-solving skills at different levels of complexity. The dataset is formatted using L A T E X to accurately capture the complex equations and symbols the problems employ. Additionally, we utilize Asymptote vector graphics for encoding mathematical figures and diagrams to ensure language models can process visual elements directly.Further, we standardized the placement of boxed answers by relocating them to the end of each solution string to minimize unintended emergent behaviors leading to evaluations that are less "harsh" or prone to penalizing the model for formatting deviations rather than actual comprehension.Model Assessment. Drawing inspiration from the MATH dataset by [Hendrycks et al., 2021] , which demonstrated the effectiveness of using boxed answers for evaluating mathematical understanding in LLMs, we similarly create a dataset with final solutions being wrapped in \boxed{} commands.Boxed answers allow for an exact match criterion rather than relying on approximate heuristics by simply parsing the LLM generated string solution for the value within the box, thereby enhancing reliability and consistency of the evaluation process while being quick and cost-effective. To further ensure fair evaluation, we implemented an equivalence function that homogenizes similar answers, addressing both simple string inconsistencies and complex mathematical equivalences like (x + 1) 2 and x 2 + 2x + 1 or numerical expressions such as \frac{1}{2}, 1/2, and 0.5 and equating them. Modified Boxing. Given the complex nature of certain Putnam questions, some problems do not lend themselves to simple, singular boxed answers. Instead, they often include conditions, multiple possible answers, varied answer formats and elaborate proofs. These original questions would have necessitated costly and difficult human evaluations which we seek to avoid. To address this, we modified these questions by adding a trivial next step to the original questions, changing the solution accordingly. This additional step was designed so as to ensure that solvers reached the same conclusions and insights necessary to solve the problem, but then needed to perform a simpler computation to get a simplified, boxable answer. We provide an example of such a change in Figure 3 . By incorporating this minor modification, we preserved the inherent difficulty and complexity of the original problems while making the answers suitable for our boxed answer evaluation criteria.Models trained on snapshots of the internet have likely encountered Putnam questions, potentially inflating their performance on the Putnam-AXIOM Original dataset. Therefore, drawing inspiration from Srivastava et al. [2024] , we introduce functional variations of select problems from Putnam-AXIOM Original providing an effective way of evaluating models that have been trained on the entire internet by taking advantage of weaknesses in model memorization. These variations are classified into two types.1. Variable Change. The simplest variation is a variable change, where variable names are altered and the final answer is unvaried. Variable changes slightly modify the problem from its original statement, which models could have trained on.2. Constant Change. Constant changes modify numeric properties of the question, altering constants within the step-by-step solution and the final answer. Constant changes significantly transform the problem from its original statement, challenging models to perform complex reasoning on how the changes affect the solution and final answer, as in the example from Figure 4 .Variational Dataset Description. We created functional variations for 52 Putnam-AXIOM questions, considering limitations such as problem-specific constants, non-generalizable solutions, and questions lacking constants or boxable answers. The dataset includes 26 constant+variable and 26 variableonly changes. We rephrased problem statements while maintaining the core task to prevent pattern recognition by LLMs. Each variation can generate infinite unique, equally difficult snapshots, offering a sustainable evaluation method. To evaluate various SOTA models, evaluators are expected to generate snapshots (instances of the infinite potential variations) of the variation dataset by running the generation code.Using the LM Harness Evaluation framework [Gao et al., 2024] , we evaluated several open-source and proprietary SOTA LLMs. Models were prompted to provide answers in \boxed format, which were then compared to Putnam ground truths with an exact final answer match. We evaluated the 236-question Putnam-AXIOM Original dataset once. For the variation dataset, we conducted five trials, each using a randomly selected variation snapshot and its corresponding 52 original questions.We then calculated mean accuracy and 95% confidence intervals. 
putnam-axiom model performance: 
 Table 1 presents Putnam-AXIOM Original dataset accuracies. Most models score below 10%, with even NuminaMath, the AI Mathematics Olympiad winner [Investments, 2024] , achieving only 4.66%. These low accuracies underscore AXIOM's difficulty. Figure 1 contrasts Putnam-AXIOM Variation dataset mean accuracies with the 52 corresponding original questions, along with the confidence intervals across the five variation snapshots with the average accuracies in Table 2 . Original accuracies typically surpass variation accuracies. For models like o1-preview, GPT-4o, Claude-3.5 Sonnet and NuminaMath-7B-TIR, non-overlapping confidence intervals reveal statistically significant differences, indicating artificially inflated performance on original questions due to data contamination. Looking at the numbers highlights significant accuracy declines across models: GPT-4o shows the steepest drop at 44%, followed by o1-preview at 30%, GPT-4 at 29%, and Claude-3.5 Sonnet at 28.5%.Original (Final Accuracy) Though we used automated evaluations for efficiency, a manual review of model responses on Putnam-AXIOM Original provides deeper insights into models' reasoning and errors. We selected the two best-performing models, GPT-4o and OpenAI o1-preview, as they likely exhibit the strongest reasoning abilities. Our goal is to analyze this reasoning in greater depth.OpenAI o1-preview Performance: Out of all models, we see that OpenAI o1-preview performed the best on Putnam-AXIOM Original, receiving 41.9% boxed accuracy (99/236) while other models received less than 20%. Analyzing the answers, we see that most of the OpenAI o1-preview responses followed generally the same logical path as the ground truth solution. However, several of these questions contained logical mistakes and inconsistencies. The biggest discrepancy between model responses and the ground-truth solution was a general lack of mathematical rigor. Whereas the ground truth solution will make claims to advance its solution then prove those claims step-by-step, o1-preview will often make and use claims without justification. While this does succeed in getting to the correct boxed final answer, these unjustified claims would receive little credit when marked by a human grader. A large part of the difficulty of mathematical reasoning is being logically airtight throughout the entire solution; thus, though o1-preview shows promise, there are still evident flaws in its mathematical reasoning abilities. In several solutions like Figure 7 , for instance, o1-preview correctly identified the maximal or minimal value of a variable, but failed to provide sufficient proof that the value it provided was indeed the maximum or minimum.GPT-4o Performance: Like the o1-preview, GPT-4o mostly followed correct logical reasoning for most of its solutions. For GPT-4o, the biggest discrepancy between model responses and the ground-truth solution is the same general lack of mathematical rigor throughout most of the solutions.An example of this lack of rigor is shown in Figure 8 , where GPT-4o makes the claim that a rectangle gives the minimal area subject to a set of constraints without any justification. In addition to issues with rigor, GPT-4o also displayed logical leaps and incoherent reasoning, as displayed in Figure 9 where the model simply assumes that an answer is correct. These logical leaps are symptomatic of an issue in the GPT-4o's CoT reasoning, as the model prioritizes reaching the final answer rather providing a rigorous logical output.General Analysis: Beyond GPT-4o and the o1-preview, we wanted a general overview of the reasoning behaviors of models. To do so, we chose the best-performing open-source models, DeepSeek-Math-7B-RL, Qwen2-Math-7B, and NuminaMath-7B. We tend to see that open-source models are much more error-prone than the proprietary models we evaluated earlier. In general,we notice that open-source models are subject to the same lack of mathematical rigor. However, this rigor issue is overshadowed by major calculation errors, hallucinated/irrelevant information, misunderstandings of the problem, and logical jumps. For instance, in Figure 10 , NuminaMath simultaneously makes a calculation, irrelevancy, and misunderstanding error when writing the last step of its solution; in Figure 11 , the model makes false assumptions about functions defined in the problem; in Figure 12 , the model completely removes a crucial part of the problem and proceeds to an incorrect final solution. 
conclusion: 
 In this paper, we present Putnam-AXIOM, a novel challenging benchmark of 236 problems from the Putnam examination for evaluating reasoning capabilities of large language models. Our dataset allows for automated evaluations with an equivalence function. While SOTA LLMs already have saturated performance on benchmarks like MATH, they still struggle with successfully answering questions in Putnam-AXIOM. To address potential data contamination issues, we introduce Putnam-AXIOM Variations, altering the variable names, constant values, or the phrasing of the question to create a potentially infinite number of problems not found anywhere on the internet. We notice that for most problems, models get significantly worse on the variations than they do the corresponding original questions. Our dataset fills the void opened by rapid progress in model reasoning capabilities.We hope that our benchmark will accelerate future research into artificial reasoning. 
 
 Appendix: We collect and modify various problems from the William Lowell Putnam Competition to create the original and variation datasets of Putnam-AXIOM. Putnam problems are created by the Mathematical Association of America (MAA), which is also the source of the AMC and AIME problems used in the MATH dataset [Hendrycks et al., 2021] . Like Hendrycks et al. [2021] , we do not in any form seek to monetize or commercialize Putnam problems-only to utilize them for academic purposes.Our use of the Putnam problems to create an evaluation dataset completely falls under the "research" section of Fair Use. Indeed, according to Section 107, of the U.S. Copyright Act [USC, 1976] , our work certainly qualifies as Fair Use for the following reasons:1. Our use of MAA problems is only for academic research purposes. We do not monetize or commercialize the problems. 2. Our use of Putnam problems as a reasoning evaluation benchmark for large language models is significantly different from their original use as competition problems.3. Our use of Putnam problems is transformative. As detailed in Section 2 above, we have transformed the questions to be answered with a single numerical or algebraic "boxed answer" We have altered all of the solutions so that the final boxed answer lies at the end of the solution (so as to encourage models to explain their rationale before outputting a solution). We have also standardized the solutions: If there are many solutions given, we only use the first; if there are any references irrelevant to mathematics necessary to understand and solve the problem (such as comments like "Communicated by ..."), we have removed those. 4. Our use of Putnam problems to construct a benchmark has no effect on the demand for or supply of Putnam problems in the William Lowell Putnam Competition. The existence of our dataset does not alter the value of the original problems-as those are already freely available online-nor does it influence the market of future competitors/problem writers.Problem: Let Fm be the mth Fibonacci number, defined by F1 = F2 = 1 and Fm = Fm-1 + Fm-2 for all m ≥ 3. Let p(x) be the polynomial of degree 1008 such that p(2n + 1) = F2n+1 for n = 0, 1, 2, . . . , 1008. Find integers j and k such that p(2019) = Fj -F k and give the answer in the form j/k. Solution: More generally, let p(x) be the polynomial of degree N such that p(2n + 1) = F2n+1 for 0 ≤ n ≤ N . We will show that p(2N + 3) = F2N+3 -FN+2. Define a sequence of polynomials p0(x), . . . , pN (x) by p0(x) = p(x) andp k (x) = p k-1 (x) -p k-1 (x + 2) for k ≥ 1.Then by induction on k, it is the case that p k (2n + 1) = F 2n+1+k for 0 ≤ n ≤ N -k, and also that p k has degree (at most)N -k for k ≥ 1. Thus pN (x) = FN+1 since pN (1) = FN+1 and pN is constant. We now claim that for 0 ≤ k ≤ N , p N -k (2k + 3) = k j=0 FN+1+j.We prove this again by induction on k: for the induction step, we have Solution: Let σn,m be the permutation of Z/nZ induced by multiplication by m; the original problem asks for which n does σn,m always have a square root.p N -k (2k + 3) = p N -k (2k + 1) + p N -k+1 (2k + 1) = F N +1+k + k-1 j=0 FN+1+j.The desired property holds if and only if n = 1 or n ≡ 2 (mod 4), hence making the required sum 2k 2 -4k + 3 .Year: 2016 ID: A1 Final Answer: 2k 2 -4k + 3 Figure 3: A modified boxing example in Putnam-MATH. Here we see that the original problem holds true for a number of values of n conditioned on a specific property making it hard to find a boxable expression. We thus modify the solution to still require the solver to get to that conclusion and add a further computation of summing up the first k such values of n giving a boxable solution while keeping the core of the problem the same.Problem: Define a growing spiral in the plane to be a sequence of points with integer coordinates P 0 = (0, 0), P 1, . . . , P n such that n ≥ 2 and:• • • How many of the points (x, y) with integer coordinates 0 ≤ x ≤ 2011, 0 ≤ y ≤ 2011 cannot be the last point, P n of any growing spiral?Solution: We claim that the set of points with 0 ≤ x ≤ 2011 and 0 ≤ y ≤ 2011 that cannot be the last point of a growing spiral are as follows: (0, y) for 0 ≤ y ≤ 2011; (x, 0) and (x, 1) for 1 ≤ x ≤ 2011; (x, 2) for 2 ≤ x ≤ 2011; and (x, 3) for 3 ≤ x ≤ 2011. Problem: Define a growing spiral in the plane to be a sequence of points with integer coordinates L0 = (0, 0), L1, . . . , Ln such that n ≥ 2 and:• • • How many of the points (w, v) with integer coordinates 0 ≤ w ≤ 4680, 0 ≤ v ≤ 4680 cannot be the last point, Ln of any growing spiral?Solution: We claim that the set of points with 0 ≤ w ≤ 4680 and 0 ≤ v ≤ 4680 that cannot be the last point of a growing spiral are as follows: (0, v) for 0 ≤ v ≤ 4680; (w, 0) and (w, 1) for 1 ≤ w ≤ 4680; (w, 2) for 2 ≤ w ≤ 4680; and (w, 3) for 3 ≤ w ≤ 4680. Determine the greatest possible value of 10 i=1 cos(3xi) for real numbers x1, x2, . . . , x10 satisfying 10 i=1 cos(xi) = 0.Solution: Since cos(3xi) = 4cos(xi) 3 -3cos(xi), it is equivalent to maximize 4 10 i=1 y 3 i for y1, . . . , y10 ∈ [-1, 1] with 10 i=1 yi = 0; note that this domain is compact, so the maximum value is guaranteed to exist.The maximum value is 480/49 .Year: 2018 ID: A3 Final Answer: 480/49Determine the least possible value of 10 i=1 sin(3ci) for real numbers c1, c2, . . . , c10 satisfying 10 i=1 sin(ci) = 0.Solution: Since sin(3ci) = 3sin(ci) -4sin(ci) 3 , it is equivalent to minimize 4 10 i=1 y 3 i for y1, . . . , y10 ∈ [-1, 1] with 10 i=1 yi = 0; note that this domain is compact, so the minimum value is guaranteed to exist.The minimum value is -480/49 . We refer to all questions that are not binary as "complex" questions.Given the guessable nature of these questions and our answer-matching evaluation method, models have a much higher chance of randomly guessing the right answer on these questions.To discern whether the inclusion of these guessable questions significantly affects the overall difficulty of Putnam-AXIOM, we conducted an analysis of the accuracy of various models with and without the binary questions, with the overall accuracies in Figure 6 .We see that, with the exception of Qwen2 Math 7B, almost all models have a higher accuracy on Putnam-AXIOM with its binary questions than without, meaning that guessing is contributing to their success to some extent. However, we see that on the more advanced models-Qwen2 Math 7B, GPT 4, and Claude Sonnet 3.5-the gap between the accuracies on the entire dataset and the accuracies on only complex questions is much smaller. This is likely because these models are capable enoughthat they successfully answer a similar percentage of complex questions and binary questions; less advanced models get significantly fewer complex questions correct than binary questions, so we see a large accuracy gap.Based on the results of this experiment, we've decided to use only the complex questions for most of our evaluations such as in Table 1 and Figure 1 . Problem: Let n be a positive integer. Determine, in terms of n, the largest integer m with the following property: There exist real numbers x1, . . . , x2n with -1 < x1 < x2 < • • • < x2n < 1 such that the sum of the lengths of the n intervals[x 2k-1 1 , x 2k-1 2 ], [x 2k-1 3 , x 2k-1 4 ], . . . , [x 2k-1 2n-1 , x 2k-1 2n ] is equal to 1 for all integers k with 1 ≤ k ≤ m.Model's Response: We are given n positive integers and need to determine the largest integer m, in terms of n, such that there exist real numbers x1, x2, . . . , x2n satisfying:1. The xi are strictly increasing between -1 and 1:-1 < x1 < x2 < • • • < x2n < 1 2.For each integer k with 1 ≤ k ≤ m, the sum of the lengths of n intervals is equal to 1:n i=1 x 2k-1 2i -x 2k-1 2i-1 = 1Our goal is to find the largest integer m (in terms of n) satisfying these conditions.. . . **General Case:** We have 2n variables and m equations.-Each equation is independent due to the distinct exponents.-To have a solution, we must have at least as many variables as equations.-**Therefore, the maximum number m is bounded by 2n.** -However, since we have 2n variables but they are constrained by the ordering and the interval (-1, 1), the effective degrees of freedom are less than 2n.Given the constraints and the increasing number of equations with m, the largest integer m for which the conditions can be satisfied is:m = n **Final Answer:**The final answer is n . I hope it is correct.Year: 2022 ID: A6 Final Answer: n Figure 7 : An example response where OpenAI o1-preview gets the final answer correct, but made an unjustified claim. OpenAI o1-preview claims that the largest possible value for m is n. However, its only justification for this claim is that m is bounded by 2n. OpenAI o1-preview does not justify why a value of m between n and 2n cannot work.Problem: Find the least possible area of a convex set in the plane that intersects both branches of the hyperbola xy = 1 and both branches of the hyperbola xy = -1. (A set S in the plane is called convex if for any two points in S the line segment connecting them is contained in S.)Model's Response: To find the least possible area of a convex set in the plane that intersects both branches of the hyperbolas xy = 1 and xy = -1, we need to consider the geometric constraints. Let's start by visualizing these hyperbolas:The hyperbola xy = 1 consists of two branches in the first and third quadrants:y = 1 x .The hyperbola xy = -1 consists of two branches in the second and fourth quadrants:y = - 1 x .We aim to place a convex set that intersects both branches of each hyperbola. One useful geometric figure for this is a rectangle aligned along the hyperbolic axes. We can guarantee that the rectangle will intersect both branches of xy = 1 and xy = -1 by making its sides tangent to the hyperbolas. . . . The geometric configuration simplifies the calculation: The minimum enclosing convex area tangency ideally is of square structure, yielding the unit hyperbola intersection:area = 4 (Since side 2 • 2 = 4formation in closed square).This geometric construction ensures least, minimum convex interaction:Final Answer: The final answer is 4 . I hope it is correct.ID: A2 Final Answer: 4 Figure 8 : An example response where GPT-4o gets the final answer correct, but made an unjustified logical step, resulting in a non-rigorous solution. OpenAI o1-preview logically jumps to the idea that the geometric shape that minimizes the area is a rectangle. However, it does not attempt to justify this claim in any way, and proceeds as though this is obvious.Problem: Given 11z 10 + 10iz 9 + 10iz -11 = 0, find the value of |z|. (Here z is a complex number and i 2 = -1.)Model's Response: To solve the equation11z 10 + 10iz 9 + 10iz -11 = 0and find the value of |z|, where z is a complex number, we'll proceed step by step. The trigonometric equations are complex and difficult to solve directly. However, notice that if |z| = 1, then z = e iθ lies on the unit circle, and its magnitude is indeed 1. Moreover, since we couldn't find any specific θ that satisfies both equations simultaneously (as shown by trying several angles), this suggests that no specific angle θ other than those corresponding to |z| = 1 satisfy the equation. **Conclusion:** Since our manipulations suggest that |z| = 1 is necessary (and given the nature of complex numbers and the unit circle), we conclude that: *Final Answer:** The final answer is 1 . I hope it is correct.ID: A3 Final Answer: 1 Figure 9 : An example response where GPT-4o make a logical leap and is completely incoherent.In this solution, the model jumps to assume that |z| = 1 and performs a series of calculations. Then, it concludes that the final answer is 1 simply because the equations it made were too "complex and difficult" to solve.Problem: Let k be the smallest positive integer for which there exist distinct integers m1, m2, m3, m4, m5 such that the polynomialp(x) = (x -m1)(x -m2)(x -m3)(x -m4)(x -m5)has exactly k nonzero coefficients. Find a set of integers m1, m2, m3, m4, m5 for which this minimum k is achieved and given the sum k + m1 + m2 + m3 + m4 + m5 as the final answer.Model's Response: The polynomial p(x) = (x-m1)(x-m2)(x-m3)(x-m4)(x-m5) has exactly k nonzero coefficients when the values of mi are distinct integers. We want to minimize k and find a set of values for mi that achieves this minimum. The expansion of p(x) will include all possible products of the differences (x -mi). For each term of the form (x -a), this generates a coefficient for that term, and this will happen for each combination of i and j, including when i = j. The minimum possible value for k can be calculated by considering the sparsity of nonzero coefficients in the polynomial's expansion. This is achieved when the polynomial is written in the form that it eliminates as many terms as possible, leading to a minimal number of nonzero coefficients.For (x -m1)(x -m2)(x -m3)(x -m4)(x -m5), it can be achieved if the differences (x -mi) produce terms that include maximal cancellation. The problem becomes one of determining the minimal degree of the polynomial that includes maximal cancellation.The key observation here is that the minimal nonzero coefficients are achieved when the polynomial is arranged in such a way that most coefficients are zero due to symmetry and cancellation properties.The smallest positive integer k for such polynomials happens at k = 2, for instance, (x -1)(x -2)(x -3)(x -4)(x -5) results in a polynomial with precisely coefficients as follows:p(x) = x 5 -(1 + 2 + 3 + 4 + 5)x 4+(sum of products of each combination of 2 distinct elements)x 3 etc...Since the polynomial in this problem demands exactly k nonzero coefficients, the product of differences can lead to cancellations resulting in minimal terms. Let's consider the initial observation:m1 = 1, m2 = 2, m3 = 3, m4 = 4, m5 = 5The resulting polynomial, (x -1)(x -2)(x -3)(x -4)(x -5) Final answer: 15Year: 1985 ID: B1 Final Answer: 3 Figure 10 : An example response where NuminaMath makes a calculation error, includes irrelevant information, and misunderstands the problem. In this solution, we see that the final answer is obtained by adding k + m 1 + m 2 + m 3 + m 4 + m 5 . In the model's final solution, its value of k is 2, so its final answer should be 2 + 1 + 2 + 3 + 4 + 5 = 17, not 15; this is due a misunderstanding of the problem and/or a calculation error. Furthermore, the model includes the expression of the "resulting polynomial," which is completely irrelevant to the problem it's attempting to solve.Problem: If p(x) = a0 + a1x + • • • + amx m is a polynomial with real coefficients ai, then set Γ(p(x)) = a 2 0 + a 2 1 + • • • + a 2 m . Let f (x) = 3x 2 + 7x + 2.Find, with proof, a polynomial g(x) with real coefficients such that (i) g(0) = 1, and(ii) Γ(f (x) n ) = Γ(g(x) n )for every integer n ≥ 1.Model's Response: We have that F (x) = 3x 2 + 7x + 2 and thus Γ(F (x)) = 3 2 + 7 2 + 2 2 = 9 + 49 + 4 = 62. We want to find a polynomial g(x) such that g(0) = 1 and Γ (F (x) n ) = Γ(g(x) n ) for every integer n ≥ 1. Suppose g(x) = x 2 + bx + 1. Then Γ(g(x)) = 1 + b 2 + 1 = b 2 + 2. Ifxn+1 = 3xn + ⌊xn √ 5⌋ = 3xn + ⌊xn( √ 5 -2) + 2xn⌋ = 3xn + xn( √ 5 -2) + 2xn = 5xn + xn( √ 5 -2) = xn(2 + √ 5). If we let yn = xn(2 + √ 5), then xn = yn/(2 + √ 5), so yn+1 = yn • 2 + √ 5 2 + √ 5 = yn.Thus, yn = y0 for all n ≥ 1, soxn = y0 2 + √ 5 = 1 2 + √ 5 = 2 - √ 5 (-2 + √ 5)(2 + √ 5) = 2 - √ 5 5 -2 = 2 - √ 5.Final Answer: The final answer is 2 -√ 5 . I hope it is correct. In the series of equalities in the beginning, DeepSeek simply removes the floor function and proceeds. However, the floor function is an integral part of the problem, and without it, DeepSeek is unable to come to the right solution.Numerous benchmarks exist to assess the mathematical capabilities of models, each typically focusing on a specific task. Two notable examples are MATH [Hendrycks et al., 2021] and GSM8K [Cobbe et al., 2021] . The MATH dataset contains questions sourced from American high school mathematics competitions such as the AMC 10, AMC 12, and AIME [Hendrycks et al., 2021] , while the GSM8K dataset contains 8.5K handwritten elementary school level questions Cobbe et al. [2021] . Both contain questions and answers with detailed rationale explanations.As models have become larger and more powerful, even the most difficult existing benchmarks have become less challenging. For instance, while the MATH dataset saw 6.9% accuracy on its release, it now sees 87.92% accuracy with GPT-4 MACM [Lei, 2024] . Similarly, GPT4 has attained 97.1% accuracy on the GSM8K [Zhong et al., 2024] . This saturation necessitates the development of more challenging benchmarks.Many contemporary data sets have been created to combat the saturation of existing benchmarks. For instance, the ARB dataset includes hundreds of challenging problems in high school and college-level math, physics, and chemistry Sawada et al. [2023] . Similarly OlympiadBench contains nearly 9,000 problems from the International Mathematics Olympiad (IMO), the Chinese GaoKao, and more He et al. [2024] . Finally, SciBench is a similar reasoning benchmark that includes hundreds of college-level scientific reasoning questions from instructional textbooks Wang et al. [2023] .Although these datasets alleviate the saturation problem, they come with many limitations. For instance, ARB Sawada et al. [2023] and OlympiadBench He et al. [2024] both contain several symbolic and proof-based questions which cannot be graded automatically and require a costly and lengthy human evaluation process. Though ARB attempts to utilize LLMs to grade their own responses with a rubric, this process is often unreliable and self-referential. Our Putnam-AXIOM dataset addresses these limitations by offering challenging Putnam problems with fully-written solutions and easily evaluable answers. It enables efficient automated assessment via frameworks like LM Harness [Gao et al., 2024] , avoiding costly human evaluation or unreliable self-grading.PutnamBench [Tsoukalas et al., 2024] is a related benchmark that primarily focuses on formal theorem proving. Its main objective is to derive formalized proofs of mathematical statements and it provides formalizations in systems such as Lean, Isabelle, and Coq, all sourced from the prestigious Putnam competition. PutnamBench also includes 640 natural language statements and their corresponding answers where applicable. While both benchmarks draw from the same competition, Putnam-AXIOM focuses on the curation of natural language problems for final answer verification and introduces automatic functional variations to generate additional benchmarks addressing potential data contamination. Further we focus on assessing true mathematical reasoning ability and hence take measures to remove easily guessable answers.Data contamination is a significant problem in creating evaluation benchmarks, as many of these problems are openly available on the Internet and are likely included in the training data for large models [Schaeffer, 2023 , Sainz et al., 2023] . Thus, the MATH [Hendrycks et al., 2021] , AGIEval [Zhong et al., 2023] , OlympiadBench [He et al., 2024], and ARB [Sawada et al., 2023] benchmarks (which are all sourced from problems on the Internet) could potentially be contaminated. Therefore, models may achieve artificially high performance on an evaluation benchmark by memorizing the answers to the problems Magar and Schwartz [2022], Ranaldi et al. [2023] .A straightforward way of avoiding data contamination issues is to utilize problems unavailable on the Internet. However, even if problems are not currently part of model training data, it is unrealistic to expect them to remain inaccessible. At the same time, it is costly to rely on the continuous human development of new datasets. Srivastava et al. [2024] attempts to alleviate this data contamination issue by creating functional variations of the MATH dataset, where new problems can be generated simply by changing numeric parameters, yielding different solutions. They observe a significant discrepancy in models' performance between standard benchmarks and these new variations. We recognize the potential of this idea and have adapted it to our more challenging dataset. We have altered the variables, constants, and phrasing of many Putnam questions while preserving their overall difficulty and requirements for logical and mathematical reasoning.For instance, the equivalence function would evaluate the answers 0.5, 1/2, and \frac{1}{2} as equal 



Reviews: review: title: title: Benchmarks suffers from low power and contamination
main_review: main_review: The paper introduces a benchmark based on 236 Putnam problems between 1985 and 2023. The authors meticulously filtered problems that are correctly gradable ensuring minimizing noise from 
- problem formatting (latex equations, images)
- boxed answer construction 
- answer equality checking

Next, the authors identify contamination as a problem in benchmark evaluation considering the time horizon of the problems collected. They proposed functional problem transformations as a potential approach to fix this. However, I believe that this level of contamination prevention might be insufficient as the problem can also memorize problem-specific logic beyond simply memorizing the answer. Additionally, functional transformations are only applied on 53 problems which reduces the power of evaluations and would like result in high error bars during evaluation.
title: title: Official Review
main_review: main_review: The motivation of the paper is interesting. However, the dataset only consists of 236 examples, which is too small to serve as a convincing benchmark. Also, most evaluated models achieve very low accuracy, which questions the effectiveness and robustness - the difficulty of the problems should be more hierarchical.
title: title: Good benchmark contribution
main_review: main_review: This work proposes a mathematical benchmark based on challenging Putnam questions to evaluate the mathematical reasoning abilities of large language models (LLMs). The experimental result show that current strong LLMs perform poorly on this benchmark, and is negatively affected by the reasoning irrelevant variations like variable change or constant change, indicating potential contamination of the data. 

In general, this benchmark would be a good contribution for more challenging problem set to evaluate the mathematical problem solving abilities (with rich examples and explainations regarding potential legal concerns). The problem modifications would be a good strategy to reduce problem loss in terms of the current box-based evaluation scheme (maybe it could be more scalable if detailed explained). Besides, there are some existing work on revealing the potential performance gap that might reflect data contamination like [1], would findings in those works echo or strengthens the takeaways here? Also, how would this dataset being a valuable adding-on to existing benchmarks that are based on challenging examinations like [2]?

[1] Xu et al. Benchmarking Benchmark Leakage in Large Language Models. https://arxiv.org/pdf/2404.18824v1

[2] Huang et al. OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI. NeurIPS Dataset and Benchmark 2024. https://arxiv.org/pdf/2406.12753
title: title: Review of "Putnam-AXIOM"
main_review: main_review: **Summary**

This paper presents a new benchmark, Advanced Examination of Intelligence in Operational Mathematics (Putnam-AXIOMO).
This benchmark consists of 236 problems which can be automatically evaluated as the answers must be provided inside of a 
"\boxed{}" command.
Alongside releasing the anonymous code, the authors also deal with dataset contamination, as these problems have been publicly available, by mutating parts of 53 problems so that they cannot have been memorised by any model.

**Strengths**
- New difficult reasoning benchmark, even very strong models struggle to perform well on this benchmark.
- Deals with training set contamination. One of the largest problems for current benchmarks is train set contamination, it is nice to see it concisely dealt with and the decontamination process yields accuracy decreases for almost all models.

**Weaknesses**
- Requires the models to use "\boxed{}" for their final answer limiting the expressivity of the benchmark.
- Minor point: please use bigger font sizes in the plots.

**Questions**
- How many of the problems were marked incorrect because the model did not use the \boxed command correctly? I would like to understand more the difference between the models ability to follow instructions (use \boxed like it is instructed to) and its ability to reason.
- Is there an algorithmic way to keep editing these problems so that there is always a dataset we can be sure no model has memorised?