From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
Authors: Shubhra Mishra
Gabriel Poesia
Noah Goodman
Venue: COLM 2025 (2025)
Keywords: math reasoning
training dynamics
reasoning
TL;DR: We conduct the first analysis of how math reasoning skills are learned during pre- and post-training using open checkpoint and open weight models.

Abstract: Large Language Models (LLMs) solely trained on next-token prediction learn to solve a wide range of problems involving mathematical reasoning. How does this ability evolve during training? We show the first analysis of how mathematical reasoning abilities of several open-weight LLMs develop during pre-training and post-training. To this end, we construct MathCAMPS, a synthetic dataset of novel mathematical reasoning problems grounded in 44 fine-grained skills taken from the Common Core curriculum from K to 8th grades. In one experiment, we show that mathematical skills are learned during pre-training in an order that measurably correlates with the human-designed curriculum, even though training data are randomly ordered. We also show a detailed analysis of which mathematical abilities benefit from instruction-tuning, a widely used post-training method and, in contrast, which skills suffer. Our work paves the way for an empirical understanding of LLM training dynamics in relation to reasoning.

Published as a conference paper at COLM 2025
From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models
Shubhra Mishra!
Gabriel Poesia'
Noah D. Goodman
Department of Computer Science and Psychology²
Stanford University {shubhra, poesia, ngoodman] n @estnford.eddeeeee
Abstract
Large Language Models (LLMs) solely trained on next-token prediction learn to solve wide range of problems involving mathematical reason- ing. But how does this ability evolve during training? We show the first analysis of how mathematical reasoning abilities of several ween- LLMs develop during pre-training and post-training To this end, we con- struct MathCAMPS 1, synthetic dataset of novel mathematical reasoning problems grounded in 44 fine-grained skills taken from the Common Core curriculum from K to 8th grades. In one experiment, we show that mathe- matical skills are learned during pre-training in an order that measurably correlates with the human-designed curriculum, even though training data are randomly ordered. We also show a detailed analysis of which mathe- matical abilities benefit from instruction tuning, widely used post-training method and, in contrast, which skills suffer. Our work paves the way for an empirical understanding of LLM training dynamics in relation to reasoning.
Introduction
Large language models are pre-trained on simple objective of next-token prediction. From this pretext task alone they acquire the ability to perform tasks requiring surprising range of skills for humans, including answering complex questions, translating between languages, and generating code. Among these tasks, one notable case that has been widely used as proxy for the general capability of new generations of LLM has been their ability to perform mathematical reasoning. This attention to mathematics is natural, since mathematics is a key tool for reasoning underlying many of the most impressive feats of human engineering, such sending rockets to space or building nuclear reactors. Many benchmarks, such as GSMSK (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), or more recently, FrontierMath (Glazer et al., 2024), have been proposed to evaluate LLMs on mathematical problems of various levels. But how this ability develops during training is still poorly understood.
How can we gain insight into the learning dynamics of mathematical reasoning skills? Sev- eral open-weights LLMs have been made available with intermediate checkpoints, including Pythia (Biderman et al., 2023), OLMo (OLMo et al., 2025) and Amber (Liu et al., 2023). How- ever, several challenges arise in designing informative evaluations of these models during their training. First, we would like rule out the potential effect of data contamination: LLMs being able to answer a question simply because it was seen during pre-training. Unfortunately, since several of the well-known mathematical reasoning benchmarks have been available on the Web since before these models were trained, this possibility cannot be ruled out by using existing benchmarks such as GSMSK or MATH. Moreover, while these benchmarks are useful for obtaining aggregate estimates of reasoning capabilities, their diversity also precludes a more fine-grained understanding of how specific mathematical skills might evolve or interact. For instance, while "grade-school math" (e.g., in GSM8K) consists of an extremely broad set of specific skills, existing datasets aggregate all of these
All code and data associated with the project can be accessed at ht //gi thub. com gamm/ppesiia/////
mathcamps

---

Published as a conference paper at COLM 2025
Attribute Grammar For All CC Standards
L.O.A.11
Symbolic Problem
Math Word Problem (MWP)
oydo- conoktency
Stmt in ["eq" Expr Expr]
Problem s- Stmtt Question
Sub
somplng
[[question XI]
[[var y
[[var x= (20 -Ill
B
LLM
How many blacks does she have left to build other otherthings?
Arya has 20 building blocks, and uses of them to build house.
Add on Expr "v" Expr
Theme: "building blocks"
symboic
Arower:
MathCAMPS
Symbolic Edit
Incremental Follow Up (IFUP)
cyde- coroistancy
sampling
[Ivar -VI]
y=
6 new blodks in another toy box.
After using the 9 blocks, Arys found
CC NMP
ITUP
[[question x]]
How many blocks will she have
[[var = [x 6]]]
symbolic
available to build things now?
+ [[question w]]
Answer:
samoling
Symbolic Edit
Counterfactual al Follow Up (CFUP)
cyde- corobtency
[[var y 14]]
LIM
Imagine that instead using 9 blocks, Arya builds taller tower and uses 14 blocks, How many blocks does the have left?
ylll [[question x]]
symbolic
Figure 1: Overview of the MathCAMPS generation pipeline. We start from a grammar (A) that represents problems tied to a Common Core Standard a specific mathematical ability drawn from a human curriculum. We sample problems in a symbolic form (B), and use a language model to realize it in natural language (C), applying a cycle-consistency where we back-translate the problem into symbolic form and ensure the answer remains the same, validating truthfulness. We also synthesize incremental and counterfactual follow-up problems
into a single final accuracy number. This is useful for comparing different models, but less informative about what exactly models learn how they might differ.
In this work, we develop a synthetic dataset and design evaluations to address these questions, and elicit number of insights into how mathematical skills develop during LLM pre- and post-training. Concretely, we introduce MathCAMPS: fine-grained dataset created automatically from the Common Core (CC) mathematics curriculum, including 49 different skills (CC "standards") from grades K-8. The CC curriculum is adopted by thousands of K-12 schools and includes grade-wise standards that indicate specific skills students must be proficient at by grade. By constructing MathCAMPSS in direct relation to the CC, our benchmark enables rich set analyses of mathematical proficiency in language models, allowing direct parallels to abilities that human students are also evaluated on.
We encode the family of problems associated with each CC standard as a grammar, allowing us to sample arbitrarily many symbolic problems, which our pipeline then converts into natural language word problems using GPT-4o (OpenAI et al., 2024). To ensure that the word problems are faithful to the symbolically generated structures, we introduce a method for cycle consistency that allows us to ensure data quality in a fully automatic manner.
Our fine-grained dataset then allows to study the training dynamics of several open-weight language models for which training checkpoints have been made available. We report on series of novel analyses of the learning dynamics of mathematical reasoning skills that our methodology enables. Our contributions are:
1. We construct MathCAMPS, a dataset of 4900 new problems stratified into fine- grained capabilities defined by the Mathematics Common Core Standards for K-8 grades. We also release our pipeline to generate arbitrarily many more.
2. We use OLMo2, Amber, and Pythia (families of open-weight models) to analyze how specific mathematical skills evolve during pre-training We explore alignment between LLM and human learning, skill evolution patterns, and the robustness of reasoning abilities.
2

---

Published as a conference paper at COLM 2025
3. We use OLMo2 and Amber models to explore how instruction tuning, widely used post-training method, affects specific mathematical abilities. We find that the specific instruction tuning methodology used in each model varies widely into how broadly beneficial they are to reasoning, as well as which specific mathematical skills they affect the most both positively and negatively.
2
Related Work
Our work closely relates to (i) current benchmarks of mathematical reasoning in LLMs, (ii) benchmarks constructed using LLMs, (ii) behavioral testing and applications in NLP, and finally (iv) open-weight LLMs that provide training checkpoints.
Benchmarks of mathematical reasoning MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) have been two leading benchmarks for the evaluation of mathematical reasoning in LLMs. Both datasets consist entirely of human-authored problems - a process that is expensive to reproduce their initial releases. Given that LLMs are trained on Web data, it is unclear whether they might have been trained on the test problems of these benchmarks (Bubeck al., 2023) either directly or from other sources (e.g., all problems in MATH come from past public competitions). In fact, GSMIK (Zhang et al., 2024), a new dataset that independently attempted to reproduce the data distribution of GSM8K, has found reduced performance on several models, suggesting the possibility of test set contamination. Moreover, these datasets evaluate very wide range of abilities, without distinction. The GHOSTS dataset provided the first fine-grained evaluation of undergraduuat--lvel mathematical skills in LLMs (Frieder et al., 2023); but given that the reasoning evaluation was performed manually by expert annotators, this unfortunately does not scale to evaluating a large number of training checkpoints. In contrast, the evaluation in MathCAMPS is both fine-grained and fully automated, albeit at the grade school level.
and result, neither benchmarks were updated since
LLM-generated synthetic datasets for LLMs As collecting data from human annotators at scale is expensive (especially in domains requiring expertise), prior work has relied on LLMs to aid the generation of large-scale benchmarks (Hartvigsen et al., 2022). BigToM (Gandhi et al., 2023), a benchmark of social reasoning in LLMs, applied the idea of symbolically scaffolding questions for the LLM to realize in natural language, an approach that we trans- port to mathematics. Dyval (Zhu et al., 2024) proposed a method for generating reasoning problems for LLMs based on a DAG representing the computation. While Dyval contains two mathematical tasks (arithmetic and solving linear equations), MathCAMPS takes this idea further for mathematical reasoning, spanning 44 skills directly grounded on a human curriculum. Other synthetic evaluations focused on mathematical skills include GSMore (Hong et al., 2024) and the concurrent work on GSM-Symbolic (Mirzadeh et al., 2024). Both these works focus on evaluating the robustness of LLMs by perturbing existing problems from an existing dataset, GSM8k, whereas in MathCAMPS we synthesize problems from scratch, grounded on human curriculum (Hong et al. (2024) also proposes perturbations to coding problems, which we do not focus on here).
Behavioral testing in NLP Our goal to provide fine-grained evaluation of mathematical reasoning has parallels with behavioral testing - the idea of testing software systems on specific features, as opposed to just their overall adequacy (Ribeiro et al., 2020). In particular, CieckList (Ribeiro et al., 2020) allowed testing machine translation models for fine-grained failure modes. Dynaboard (Ma et al., 2021) proposed an NLP leaderboard where users can adapt to their own needs by choosing the utility of different metrics; our dataset enables similar user-customizable comparison between models for mathematical reasoning.
Open-Weight Language Models Our analysis of the learning dynamics of is done on top of models for which training checkpoints are made publicly available. These include Pythia (Biderman et al., 2023), OLMo (OLMo et al., 2024), and LLM360 (Liu et al., 2023). Moreover, OLMo and LLM360 also have pust-trained finstruciion-tuned versions, allowing us to also understand the impact of this step in the LLM training pipeline.
3

---

Published as a conference paper at COLM 2025
3 MathCAMPS
We now describe our pipeline for automatically generating mathematical problems and follow-up questions that are grounded in human curriculum- the Mathematics Common Core //ww. thecorestandards .rrgg))).......oooooooo..))).r)) Figure overviews our pipeline. We describe the Common Core, how we represent its standards in a grammar, symbolic problems, generate follow-ups, realize those in natural language, and finally improve quality by checking for cycle consistency.
3.1 The Mathematics Common Core
To ground problems in human curriculum, we turn to the Common Core State Standards for Mathematics. 41 states in the United States adopt the CC as their curriculum. The CC details the mathematical content that students should master from Kindergarten up to 12th grade. Within each grade, the CC elaborates series of individual standards, which detail particular mathematical skill that students should learn at that grade. Each standard has an identifier, such as K. CC. C. and summary description - for K. W.c.C.c.. C. 7, this is "Compare two numbers between and 10 presented as written numerals". ". Here, K indicates that this is standard for the Kindergarten grade level, whereas E. 8 Anallye and solve pairs of simultaneous linear equations" - is an 8th grade standard.
We take 44 standards spanning grades K through 8 to compose MathCAMPS,, focusing on standards that are amenable to automatic problem generation with a final answer in text form. The complete CC curriculum has 229 standards across grades K through 8, bringing our coverage to 19.2% of the curriculum for these grades. Notably, we currently do not cover standards focusing on conceptual understanding (e.g., 3. OA. D. "Identify arithmetic patterns [...], and explain them using properties of operations." "), or standards that emphasize visual reasoning (e.g. A. 4 'Represent three-dimensional figures using nets made up of rectangles and triangles, and use the nets to find the surface area of these figures.") "). All 44 standards covered in MathCAMPS are listed in Appendix C.
Representing Common Core standards We represent CC standards as non-terminals in an attribute grammar (Heine & Kuteva, 2007)- - rich formalism that can encode semantic, context-sensitive rules. Attribute grammars can encode syntax much like context-free grammar, but also allow us to embed information processing (e.g,, setting and testing conditions attributes, such as bounds on constants) in the production rules. We map each standard s to non-terminal Ps, such that all strings produced by expanding P3 using production rules are valid symbolic representations of a problem pertaining to standard i. Figure shows a (simplified) grammar for the standard 1. OA A. 1 - "Use addition and subtraction within 20 to solve word problems involving situations of adding to, taking from, putting together". Here, a word problem, generated by the Problem non-terminal, consists of a sequence of declarative statements expressing equations between expressions. For this standard, an expression consists of addition, subtraction, variables, and constants. After these declarations, the problem ends with a question an expression representing the value that the problem asks for. Concretely, our grammar is implemented in Python: each non-terminal becomes a stochastic function that samples and applies a production rule, recursively expanding non-terminals that it produces. In the grammar in Figure 1 (A), sampling a Problem generates a structure such as the one shown in Figure (B).
Enforcing problem constraints When sampling problems, there is no priori guarantee that all generated statements are necessary to answer the question. To avoid such statements, we remove them by applying a simple graph reachability algorithm on dependency graph between statements, removing statements that the answer does not depend on. This enforces the constraint of only having useful statements in problems. Besides this constraint, which we always enforce, each standard can apply specific constraints. The standard 1. A. 1 has an example of such constraint: it requires that students only be asked to use "addition and subtraction within 20." To be faithful to this standard, we must validate that no intermediate values used in the solution exceed 20. To encode this and other constraints across the curriculum, we implement suite of 6 parameterized filters (detailed in Appendix D)
4

---

Published as a conference paper at COLM 2025
that are selectively applied depending on the standard's specification. Applying rejection sampling from the grammar using the standard's 's filters gives a procedure for generating valid symbolic problems. For all standards that can be formulated as solving system of linear equations, we use SymPy (Meurer et al., 2017) to obtain final answers. For other cases, use two simple custom procedures (to list the factors of numbers and to compare values).
3.2 From symbolic to word problems
To realize the symbolic problems into natural language, we use few-shot prompting with GPT-4 (Figure 1 (C)). For each standard, sampled two valid symbolic problems and manually wrote a problem in natural language that faithfully represents the symbolic structure. For standards involving word problems, which typically contain simple cover story, we also sampled a random theme out of 188 that we crafted (e.g., "Book", "Pirate ship", "Money"). These examples are then given to GPT-4 in-context, along with new symbolic structure (and a random theme, for standards where that is relevant), requesting it to generate a faithful natural language problem for that structure.
Unlike generating problem stories from a fixed set of templates, using language model for generating natural language problems gives us fluid, diverse language. Unfortunately, we also lose any guarantee that the generated word problem represents the original symbolic structure faithfully. To mitigate this issue, we also introduce a cycle consistency method that we have found to drastically improve problem quality. Precisely, we use the same few-shot examples we crafted for each standard in reverse (i.e., with the natural language problem coming first, followed by the symbolic structure) to have GPT-4 translate the word problem wrote into a symbolic structure. In this step, the model is not given the original structure. We then parse and apply the appropriate solver to the generated symbolic problem; we consider the generation cycle-consistent if the answers to the original and recovered problems are the same (illustrated in Figure 1). We then discard problems that fail this test. A more in-depth analysis of the efficacy of cycle-consistency can be found in Appendix F.
3.3 Generating follow-up questions
As human instructors know, follow-up questions often useful way to probe a student's understanding. In MathCAMPS, we leverage our symbolic representation of problems to derive follow-up questions. We propose two kinds of questions: counterfactual questions, where we change a constant in the original problem, and incremental questions, where we add new piece of information. For each CC standard, we mark which (if any) of these two categories of follow-ups are applicable. Symbolically, follow-up questions are represented as a difference to be applied to the original question when we apply the difference, we obtain a new problem. We then use the same solver as the original problem to obtain the ground-trutt answer to the follow-up question. We employ the same few-shot structure to translate this difference into natural language question, and parse it back into a symbolic structure to test for cycle consistency.
4 Results
We use MathCAMPS to evaluate how the ability to reason mathematically evolves during pre- and post-training. We use open models that have released intermediate checkpoints (namely Amber from LLM360, OLMo2 7B from OLMo, and Pythia 12B from Eleuther AI). Then, we evaluate a subset of the available checkpoints on MathCAMPS. We use the evolution of checkpoint accuracy on the whole dataset and on subsets of the dataset to measure how different mathematical reasoning abilities evolve during training. Additionally, OLMo2 and Amber have instruction-tunet versions, compared to their based pre-trained checkpoints, to study the impact of instruction tuning on individual Common Core standards. We also perform a comprehensive fine- grained (overall, per-grade, per-standard) evaluation of 23 popular open (including Llama 3, DeepSeek, Mistral of various sizes) and closed (e.g. GPT-4o, Claude 3, Gemini 1.5) models on MathCAMPS for which checkpoints are not available, and report all the results
versions: we use these and Llama-3.1-883
5

---

Published as a conference paper at COLM 2025
Performance by Grade Group Training LLM360/Amber
EieutherAU/pythia--2Z
allenaa/OLMo-2-1122--8
Grade Group
0.90
23
0.80
456
78
0.70
K1
Accuragi
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0
20
40
60
80
100
20
40
80
100
20
40
60
80
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 2: Model accuracy on problems coming from different grade groups evaluated during training. Each data point corresponds to an LLM checkpoint evaluated on MathCAMPS problems testing skills from the indicated range of grades. Training Progress (X-axis) is measured by percentage of total pre-training tokens seen by the checkpoint. Accuracy is final-answer accuracy on solving the problems with few-shot CoT prompting.
in the Appendix H. In the remainder of this section, present our research questions and observations.
4.1
Pre-training
How aligned are the learning trajectories of mathematical reasoning in LLMs compared to human studentsscurricula? We first analyze how performance on skills from groups of grades evolve over time. Given overlap in the content for each grade, we split the grades K-8 into K-1, 2-3, 4-6, and 7-8. Then, we analyze how performance on these grade groups improves over time. We see the performance of the OLMo, LLM360, and Pythia models in Figure 2.
RQ1:
Notably, these models' training data not ordered according to any human curriculum. Despite this, we see that skills from earlier grades show higher performance earlier on in training, and this trend continues. humans first because these skills implicitly build on each other in the human reasoning data that LLMs observe. We see some evidence for this: when we look the evolution of skills by grade groups, see that the evolution of these groups follow the human curriculum, since on average, skills from earlier grades are also learned faster. However, even though the aggregate trend holds true, we also find that specific skills learned in ways unaligned with the human curriculum. For example, 6.E..A.11, a skill that asks students to evaluate numerical expressions involving whole-number exponents learned very quickly, simply because calculating small exponents for small bases is an easily memorizable task. In contrast, even though human students could also learn these skills by memorization early on, they tend to learn them later only because they' re exposed to them later. Another important note is that we only evaluate the ability to solve "computational" problems using these skills, whereas human students also need to pick up more conceptual skills (like explaining exponentiation, making analogies, illustrating numerical scenarious, etc). These abilities are not captured by simply memorizing the results of operations for small numbers.
possible that LLMs learn skills that are easier for
Next, we analyze the relationship between model performance and the magnitude of the numbers involved in the problem. In Figure 3, we show the accuracy for each training checkpoint for the different numbers of digits seen in the answers. Interestingly, we note how model behavior does not follow exactly what is seen in humans. For example, even though problems with 1-digit answers are easier on average than those with two-digit answers, Amber seems consistently over perform in cases where there are 2-digit answers.

---

Published as a conference paper at COLM 2025
Performance by Answer Training LLM360/Amber
BleutherArpyyhia-122
Digits in Answer
0.90
1-digit
0.80
Z-digit answer
S-digit answer
4-digit ansiwer
5-digit answer
Accuracy
0.50
0.40
0.10
0.00
0
20
40
60
90
100
20
40
80
100
40
60
80
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure Performance on problems of varying number of digits in their final answer, across pre-training checkpoints.
This holds true for the final Pythia model as well, but not for most other training check- points we looked at. This highlights discrepancy that could potentially reveal underlying mechanisms that prevent LLMs from reasoning robustly and consistently.
Do skills emerge certain point training, or do they evolve smoothly? We evaluated each checkpoint
RQ2:
diverse set of 49 skills across grades K-8. We observed that all skills tend to evolve smoothly during training, as opposed to certain skills disctretely emerging at specific training step. In Figure 4, we show how all skills across grades 2 and 7 evolve across the Pythia, Amber, and OLMo models. Given space constraints, we include results for all other grades in Appendix B.
The granularity in MathCAMPS lets us conduct analyses that aren't possible by looking at aggregate accuracies from datasets like #SM8K/MATH over time. For instance, the skills we included in 2nd grade mostly involve adding and subtracting within 100 under different settings (multi-step word problems, problems including currencies and lengths, etc.). Pythia demonstrates particular performance gap between standards 2.0A.A.1 and 2.NBT.B.6. The prior encompasses the skill to solve one- and two-step problems with unknowns in all positions and the latter asks that students be able to add up to four two-digit numbers. higher performance on 2.NBT.B.6 indicates that models performed well in setting with more additions, but the linguistic complexity of 2.0A.A.1 made the problems harder (despite the numerical calculations in 2.0A.A.1 problems being easier). Amber and OLMo also demonstrate this performance gap in a smaller magnitude.
For seventh grade, however, there more overlap which skills models struggle with and which ones they are consistently performant on. 7.NS.A.1 is the skill about adding and subtracting rational numbers, with the fraction and decimal suffixes representing the fractional and decimal subsets of that standard. 7.NS.A.2 is about the multiplication and division of fractions, and 7.NS.A.3 involves solving real-world math problems that include the four operations and rational numbers. Interestingly, involving operations with decimals are learned earlier on and tend to evolve faster. This is likely because operations with decimals are easier to memorize, given their high surface similarity to operations with integers (e.g. 0.28 0.45 is very similar to 28 + 45).
that across all models, skills
However, the skills involving fractions are learned at slower speeds, if they are learned at all. This is likely because operations with fractions require more reasoning steps per operation (e.g. calculating { 7 first requires finding common denominator before conducting the easier step of addition). An alternate explanation involves the tokenization of numbers: it likely that decimals and fractions are tokenized differently, and that the tokenization of decimals is more similar to that of integers. This would enable an easier transfer of mathematical reasoning skills learned for integers to skills applied to decimals.
7

---

Published as a conference paper at COLM 2025
(a) Grade 2
Performance on Common Core Standards Over Training
EleutherAt/pythie-122
allenai/OLMo-22112244
1.00
CC Standard
0.90
2.M..8.5
0.80
2.MD.CD
2.N6...
0.70
2.
NBT.. 6
C0.60
0.50
40.40
0.20
0.00
20
100
20
60
20
Training Progress (%)
Training Progress (%)
Training Progress (%)
(b) Grade
Performance on Common Core Standards Over Training
EleutherAt/pythia
LLM360/Amber
allenal/OLMo 2-1124-78
Standard
0.90
A.1-e.mm)
7.NS.A.1-H 7-Na-A.1-fraction
7.NS A.2
0.70
2.NS.A. 3-decimml
7.N5.A.3
00.60
0.50
0.40
0.20
0.00
60
80
80
100
300
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 4: Learning dynamics of individual Common Core standards in grades and Full results for all grades can be found in Appendix B.
Performance on Follow-up Questions Over Training LLM360/Amber
EeutherAA/pyyhia-122
Mienall/OMM-22-1224
0.55
Evaluation
0.50
Enital Probiem
Problem Follow-ups
0.35
0.30
0.25
0.20
0.10
0.05
0.00
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 5: Performance for models during training when also asked to answer follow-up questions about each problem. Here, we only consider problems that have at least one associated follow-up question (counterfactual or incremental).
Additionally, hypothesize that in some models, skill 7.N.A... is learned relatively quickly because multiplying and dividing fractions requires fewer reasoning steps than adding and subtracting would.

---

Published as a conference paper at COLM 2025
These types of fine-grained analyses enabled by MathCAMPS might inform model pre- training and evaluation, especially for models that are trained with specific end uses in mind (e.g,, in educational applications).
Effect Instruction Tuning on Accuracy by Skill
0.20
Acc. A (Base IIstructt
0.00
allenai allena//OLoo-2-12225
0.20
0.40
-0.00
4.0.
2.40 9 Enraeeg
4.501
3.H0
7.05
Common
Effect of Instruction Tuning Accuracy by Skill
Acc. A (Base Instruck)
LLM360/Amber
4ND A.3.A.-11
7,40
3
2,45
Common Core Standard
Figure 6: Effect of instruction tuning on individual skills in OLMo and LLM360. We show the difference in accuracy between the base and post-trained model: a negative value means that, on that standard, post-training improves performance, whereas a positive value indicates that performance was hurt. We find widely different profiles between the two instruction tuning methods: while the majority skills in OLMo improve, many more are either unaffected or hurt by the instruction tuning method applied to LLM360.
RQ3: Do models learn to apply reasoning skills in robust manner? We evaluate whether models' reasoning skills are robust (and not a result of surface-level reasoning) by asking follow-up questions about each problem, probing for deeper understanding (Section 3.3 explains how we generate such questions). Figure 5 shows performance on problems and follow-up questions as training progresses. We observe that generally, as model ability to solve problems improve, so does their ability to answer follow-up ques- tions about the problems they solve, demonstrating a relatively robust capability increase. However, the gap between the ability to answer one- turn and follow-up questions tends to increase over time. It is likely that this isn't caused by worsening ability reason deeply, but rather because models solve more and more challenging problems as they train, and problems that are challenging are likely to have follow-ups that are also harder. Overall, we find surprisingly robust ability to reason about mathematical problems developing during LLM training: not only the problems we use are novel, the models are increasingly capable of answering subsequent questions about those problems see in existing mathematical reasoning datasets, nor in post-training datasets focused on conversations (e.g., UltraChat (Ding et al., 2023)).
task do not generally

---

Published as a conference paper at COLM 2025
4.2
Post-training
RO4: How does instruction tuning impact reasoning abilities? Here, we use the Amber, OLMo, and Llama models to show how instruction tuning impacts specific mathematical skills. In Figure 6, we highlight Amber and OLMo. We hypothesize that the impact of instruction tuning on model's overall reasoning ability depends on the base model's reasoning ability. We see that OLMo's s reasoning abilities were impacted more positively as compared to Amber. This pattern of predominantly positive impact also holds for Llama-3.1- SB (Figure in the Appendix). We note two potential reasons for this observation. First, the base Amber model is weaker than base OLMo and Llama, which means that it likely doesn't t have the latent mathematical reasoning abilities instruction-tuning would let us observe (by enabling the LLM to answer questions). Second, for OLMo specifically, pre-training consists of two stages with distinct data distributions, with the first stage including 3,9T tokens of broad web-scale data and the second stage including 843B tokens of focused, high-quality mathematical and scientific data. Since this second pre-training stage included high quality reasoning data, improved the base model's ability to reason mathematically further, giving instruction tuning even more of a chance to surface the gains from this second stage of training.
These results contrast with an observation made by the concurrent work of Springer et al. (2025), which measured the impact of the quantity of pre-training data on performance changes due to fine-tuning. They show that the more tokens an LLM sees during pre- training, the less effective instruction tuning is on it. In our case, OLMo and Llama, which were trained on 4T and 15T tokens during pre-training respectively, gained more from instruction tuning compared to Amber, which was only pre-trained on 1.3T tokens.
Although there is broad variety in how specific skills are impacted by instruction tuning, we note a few interesting trends. First, we see how skills that are extremely challenging for the base model tend to stay unimpacted by instruction tuning. For example, 8.EE.C.8, skill that measures the ability to solve systems of two equations in two variables, sees no improvement across all three models we inspect. Additionally, we also see some correlation in which skills improved or were hurt by instruction tuning across models. For example, in Figure 8, we see that skills that show improvements after instruction tuning tend rely on multi-step reasoning, which possibly emerges due to better language understanding and instruction following. Overall, we find that instruction tuning has a variable impact on mathematical reasoning abilities of LLMs, with nuanced interaction between model, specific skill, and the impact derived from post-training.
Conclusion
We introduce MathCAMPS, fine-grained synthetic benchmark of mathematical reasoning in LLMs. MathCAMPS is directly grounded in the Common Core Standards, a widely used curriculum in human education. By tying our problems to human curriculum, we enable a much wider range of analyses to understand how mathematical reasoning abilities evolve. We show analyses of alignment between human and LLM learning, the evolution of skill learning over training, robustness of reasoning skills over training, and the impact of instruction tuning on specific skills, though we believe these are only few examples of analyses that MathCAMPS permits. MathCAMPS can also be used to directly compare how humans and LLMs acquire skills over time by measuring the performance of human participants of various ages on MathCAMPS.
While we currently cover 44 CC standards, our pipeline can be easily extended to cover additional standards where problems have a computational nature, and where answers can be obtained using a computer solver. These can include topics beyond high-school, includ- ing calculus and linear algebra. This framework, however, is difficult to extend to more conceptual problems, including mathematical proofs, or problems that require explanations, as opposed to final computational answer. Judging natural language reasoning reliably, in the absence of an exact answer to compare to, remains an open problem - an important challenge to allow to extend the scope of evaluation of mathematical reasoning in LLMss
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This work was supported by NSF Expeditions Grant, Award Number (FAIN) 1918771. GP was also supported by the Stanford Interdisciplinary Graduate Fellowship (SIGF).
References
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. ar Xiv preprint or Xix:23355.14333 2023.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in neural information processing systems, 36:27699 66627499-27744, 2023.
Kanishk Gandhi, Jan-Phillipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Under- standing social reasoning in language models with language models, 2023.
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, and Mark Wildon. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024. URL https:
//arxiv.org 2b4/1//////11111../ 2411 04872.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection, 2022.
Bernd Heine and Tania Kuteva. The genesis of grammar: A reconstruction, volume 9, Oxford University Press, USA, 2007.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
Pengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya, Rada Mihalcea, and Soujanya Poria. Evaluating Ilms' mathematical and coding competency through ontology-guided interventions. 2024. URL https://appi semanti semanticscholar cscholar. org/CorpusID. 267028311.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et Llm360: Towards fully transparent open-source llms. Xiv preprint arXit=2312.065500 2023.
11

---

Published as a conference paper at COLM 2025
Li Lucy, Tal August, Rose E. Wang, Luca Soldaini, Courtney Allison, and Kyle Lo. Mathfish: Evaluating language model math reasoning via grounding in educational curricula, 2024.
URL https::/aaaaii.orrrr org/ abs//a/0/ 04226.
Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. Dynaboard: An evaluation- as-a-service platform for holistic next-generation benchmarking. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, Pp. 10351-10367. Curran As- sociates, Inc., 2021. URL https: httpps://proceedinggg httpp//iwnnnniiipssc//p/peeeeeee htle//5bb977ddffff33444411133333 0aw.:/5pp6ee00--appeeeeee pdd..
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondłej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. Peer] Computer Science, 3: e103, January 2017. ISSN 2376-599.. doi: 10.7717/peerj-cr 103. URL https://doi org/10. 7717/peerj-cs. 103.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arviv preprint ar arXiv:2410.05229, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint 1Xiv:2501.0065) 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Ji- acheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025. URL https://arxiv.orr wrtp://////b///0000 005500005555
OpenAI, ;, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, An- drej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Win- ter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Fugene
12

---

Published as a conference paper at COLM 2025
Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, Ian Silber, lan Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, llya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-40 system card, 2024. URL https://arxiv. org//a aaas///1.... 21276.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. arxiv preprint ar arXio:2005.04118, 2020.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune, 2025. URL https: /ttpss////////.////e/ae/// wttp:/vv..org/b///5....// 19206.
13

---

Published as a conference paper at COLM 2025
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators. arXio preprint arXio:2455.01724 2024.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic, 2024.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks, 2024.
14

---

Published as a conference paper at COLM 2025
Effect of Instruction Tuning Accuracy by Skill
meta maamm/ -..--33. 8-88
(Base Instruch
0.05
0.49
-0.15
-0.45
1.85 ALs aee.. 3
340 aeseeaaaie sarre
3,00 2.01.. Dar
7.8
Common Core Standard
Figure 7: The impact of instruction-tuming on Llama 3.1 8B.
Standard ID K.C..C.7
Description
Compare two numbers between and 10 presented as written numerals.
K.OA.A.4
For any number from 1 to 9, find the number that makes 10 when added to the given number, e.g., by using objects or drawings, and record the answer with drawing equation. Fluently add and subtract within 5.
K.OAA..5
K.NB.A.1
19 Into ones and some further ones, e.g., by using objects drawings, and record each composition or decomposition by drawing or equation (e.g., 18 10 + 8); understand that these numbers composed of ten ones and one, two, three, four, five, six, seven, eight, or nine ones.
Compose and decompose numbers from
Table CC Standards for Grade K
A Instruction-Tuning Results
In Figure 7, the results for instruction tuning on Llama-3.1 and in Figure 8, a side-by-side comparison of the impact of instruction tuning on all skills.
B Learning Dynamics for Grades K-8
Figures through 17 show the learning dynamics for each grade for Amber, LLM360, and OLMo.
C Common Core Standards in MathCAMPS
We will be releasing the code used to generate MathCAMPS and our first dataset. All of the Common Core standards we implement will be described in a configuration file, commoncore, yaml, where standards are instantiated by composing high-level components from the Common Core attribute grammar. Moreover, we provide our prompts used to generate the dataset and model responses, well as all problems and model responses for all LLMs we evaluated.
We list the Common Core standards represent in MathCAMPS in Tables 1 through 9, segregated by grade, Standards 3.D.D,,8, 4.MD.A.2, A.1, and 7.NS.A.3 are split up into sub-standards. This was done for ease of implementation of the grammar.
15

---

Published as a conference paper at COLM 2025
Effect of Instruction Tuning on Accuracy by Skill
Acc. A (Base- Instruct)
LLM360/Amber
-0.00
-0.40
-0.00
allenai/OLMo 2-1124-7B
Acc. A (Base Instruct)
0.90
0.00
-0.00
-0.40
0.40
Acc. (Base- Instruct)
0.20
meta- meta-llaman waw-lllmaaaaaaa MaamaLLaaaaaaa Llamm-3.1111
0.00
-0
0..40
.0..B.4 4N0 8.
2000 6.6
AND
2.N0T
A.N a
B.g
K.O
A
TOK
2402
SNOT8.
SMO
4MD
7.86
Common Core Standard
Figure Impact (base minus post-trained performance) of instruction tuning across Amber, Llama, and OLMo, sorted by skill rather than by the performance delta. Overall, the direction of impact by skill tends be similar across models, even magnitude differs.
16

---

Published as a conference paper at COLM 2025
Performance on Common Core Standards Over Training LLM360/Amber
EleutherAl/pythia-1
allenaijQLMo---1122--
1.00
CC Standard
0.90
M
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
80
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 9: Learning Dynamics Across Amber, Pythia, OLMo for Kindergarten
Performance on Common Standards Training
EleutherAU/pythla-122 12b
LLM360/Amber
allenna/OLMMo---1224-7
CC Standard
0.90
1.0A.A.1
0.80
1.0A.A.2
1.0A.D.8
0.70
0.66000000
Accuracy
0.60
0.50
0.40
0.20
0.10
0.00
20
40
60
80
100
20
40
80
100
40
80
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 10: Learning Dynamics Across Amber, Pythia, OLMo for First Grade
Performance on Common Core Standards Over Training
EleutherAl/pythie--12
LLM360/Amber
allenaajOLMoo---122--8
1.00
Standard
0.90
2.MD.6.5
2.MD C.8
2.NBB B.S
0.70
2.NBT.B.6 B.G
2.NBT B.7
60.60
2.0A.A.1
0.50
0.40
0.30
0.20
0.10
0.00
40
80
100
40
80
100
40
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 11: Learning Dynamics Across Amber, Pythia, OLMo for Second Grade
17

---

Published as a conference paper at COLM 2025
Performance on Common Core Standards Over Training
EleutherAl/pythia-22
LLM360/Amber
allenai/QLMo--211222
1.00
CC Standard
0.90
3.MD.D.8-po B.ppol--pgygon
0.80
3.MD.D.9-triame
9.8-triangie
arcuacnlateral
0.70
00.60
0.50
0.40
0.30
0.20
0.10
0.00
80
40
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 12: Learning Dynamics Across Amber, Pythia, OLMo for Third Grade
Performance on Common Standards Training
EleutherAU/pythla-12
LLM360/Amber
allenna/OLMo---1124-77
CC Standard
0.90
0.80
4. A.2ffatnnn
4.MD.A.3
0.70
B.
B.
Accurag
0.60
4.NFA.2
4.
0.50
4.0A.A.3
4.0A.B.4
0.44
0.20
0.10
0.00
20
40
60
80
100
20
40
80
100
20 Training Progress (%)
40
80
100
Training Progress (%)
Training Progress (%)
Figure 13: Learning Dynamics Across Amber, Pythia, OLMo for Fourth Grade
Performance on Common Core Standards Over Training
EleutherAl/pythie--22
LLM360/Amber
allenaijOLMo-2-1124--8
1.00
Standard
0.90
5.NBT.6.5
S.NBT.G.6
L.BTB.7
0.70
S.NF.A.1
S.NF...2
60.60
S.NF.B.4
5,0A.A.1
0.50
0.40
0.30
0.20
0.10
0.00
40
80
100
40
80
100
40
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 14: Learning Dynamics Across Amber, Pythia, OLMo for Fifth Grade
18

---

Published as a conference paper at COLM 2025
Performance on Common Core Standards Over Training
EleutherAA//pyhhe--1ZZ
LUM360/Amber
ailenai/OLMno--1122--87
1.00
Standard
0.90
6.EE.A11
0.80
6.EE,
6.N5. B.2
6.N5.B.3
0.60000000
Accurasy
0.60
0.50
0.40
0.30
0.20
0.10
0
20
40
60
80
100
20
40
80
100
40
60
80
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 15: Learning Dynamics Across Amber, Pythia, OLMo for Sixth Grade
Performance on Common Core Standards Over Training
EleutherAl/pythia-1e
LLM360/Amber
allenal/OLMM--211222
1.00
CC Standard
7.N5.A.1-decims
0.80
0.70
3-ddcmal
00.00
7.N5.A.3 3.3-fraction
Accuracy
0.60
0.50
0..40
0.30
0.10
0.00
80
100
40
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 16: Learning Dynamics Across Amber, Pythia, OLMo for Seventh Grade
Performance Common Core Standards Over Training
EleutherA//pythia-12 12b
LLM360/Amber
allenaa/OLMo-2211244
1.00
Standard
0.90
8.EE
0.80
8.EE.C.7
8.EE.C.8
0.70
Accuras
2.660
0.50
0..0
0.30
0.20
0.10
0
20
40
60
80
100
20
40
80
100
40
80
100
Training Progress (%)
Training Progress (%)
Training Progress (%)
Figure 17: Learning Dynamics Across Amber, Pythia, OLMo for Eight Grade
19

---

Published as a conference paper at COLM 2025
Standard ID 1.AAA.11
Description
Use addition and subtraction within 20 to solve word problems involving situations of adding to, taking from, putting together, taking apart, and comparing, with unknowns in all positions, e. by using objects, drawings, and equations with symbol for the unknown number to represent the problem.
1.0A.A.2
Solve word problems that call for addition of three whole num- bers whose sum is less than or equal to 20, e.g., by using objects, drawings, and equations with a symbol for the unknown number to represent the problem.
LOADD
Determine the unknown whole number in an addition or subtrac- tion equation relating three whole numbers.
Table 2: CC Standards for Grade
Standard ID 2.0A.A.1
Description
Use addition and subtraction within 100 to solve one- and two- step word problems involving situations of adding to, taking from, putting together, taking apart, and comparing, with unknowns in all positions, e-g., by using drawings and equations with symbol for the unknown number to represent the problem.
2.NBT.B.5
Fluently add and subtract within 100 using strategies based on place value, properties of operations, and/or the relationship between addition and subtraction.
2.NBT.B.6
Add up to four two-digit numbers using strategies based on place value and properties of operations.
2.NBT.B.7
Add and subtract within 1000, using concrete models or drawings and strategies based on place value, properties of operations, and/or the relationship between addition and subtraction; relate the strategy to a written method. Understand that in adding or subtracting three-digit numbers, one adds or subtracts hundreds and hundreds, tens and tens, ones and ones; and sometimes it is necessary to compose or decompose tens hundreds.
2.M..B.5
Use addition and subtraction within 100 to solve word problems involving lengths that are given in the same units, e.g., by using drawings (such as drawings of rulers) and equations with sym- bol for the unknown number represent the problem.
2.M..C.8
Solve word problems involving dollar bills, quarters, dimes, nick- els, and pennies, using and é symbols appropriately.
Table 3: CC Standards for Grade 2
20

---

Published as a conference paper at COLM 2025
Standard ID 3.0A.A.3
Description
Use multiplication and division within 100 to solve word prob- lems in situations involving equal groups, arrays, and measure- ment quantities, e.g., by using drawings and equations with a symbol for the unknown number to represent the problem.
3.0A.A.4
Determine the unknown whole number in a multiplication or division equation relating three whole numbers.
3.0A.C.7
Fluently multiply and divide within 100, using strategies such as the relationship between multiplication and division (e.g., know- ing that 8 x = 40, one knows 40 + 5 8) properties of opera- tions. By the end of Grade 3, know from memory all products of two one-digit numbers.
3.OA.D8
Solve two-step word problems using the four operations. Repre- sent these problems using equations with letter standing for the unknown quantity. Assess the reasonableness of answers using mental computation and estimation strategies including round- ing.
3.M....- triangle
Solve real world and mathematical problems involving perime- ters of polygons, including finding the perimeter given the side lengths, finding an unknown side length, and exhibiting rectan- gles with the same perimeter and different areas or with the same area and different perimeters.
3.MD.D.8- quadrilateral
Solve real world and mathematical problems involving perime- ters of polygons, including finding the perimeter given the side lengths, finding an unknown side length, and exhibiting rectan- gles with the same perimeter and different areas or with the same area and different perimeters.
3.M..D.8- polygon
Solve real world and mathematical problems involving perime- ters of polygons, including finding the perimeter given the side lengths, finding unknown side length, and exhibiting rectan- gles with the same perimeter and different areas or with the same area and different perimeters.
3.NBT.A.2
Fluently add and subtract within 1000 using strategies and algo- rithms based on place value, properties of operations, and/or or the relationship between addition and subtraction.
Table 4: CC Standards for Grade 3
21

---

Published as a conference paper at COLM 2025
Standard ID 4.0A.A.3
Description
Solve multistep word problems posed with whole numbers and having whole-number answers using the four operations, includ- ing problems in which remainders must be Interpreted. Represent these problems using equations with letter standing for the unknown quantity, Assess the reasonableness of answers using mental computation and estimation strategies including round- ing.
4.0A.B.4
Find all factor pairs for a whole number in the range 1-100. Rec- ognize that whole number is a multiple of each of its factors, Determine whether given whole number in the range 1-100 a multiple of given one-digit number. Determine whether given whole number in the range 1-100 is prime or composite. Fluently add and subtract multi-dight whole numbers using the standard algorithm.
4.NBT.B.4
4.NBI.B.5
Multiply a whole number of up to four digits by one-digit whole number, and multiply two two-digit numbers, using strategies based on place value and the properties of operations. Illustrate and explain the calculation by using equations, rectangular arrays, and/ or area models.
4.NBT.B.6
Find whole-number quotients and remainders with up to four- digit dividends and one-digit divisors, using strategies based on place value, the properties of operations, and/or the relationship between multiplication and division. Illustrate and explain the calculation by using equations, rectangular arrays, and/on area models.
4.NEA.2
Compare two fractions with different numerators and different denominators, e.g., by creating common denominators or nu- merators, or by comparing to a benchmark fraction such as 1/2. Recognize that comparisons
valid only when the two fractions refer to the same whole. Record the results of comparisons with symbols 2, =, ,, and justify the conclusions, e-g., by using visual fraction model.
4.M. decimal
Use the four operations solve word problems involving dis- tances, Intervals of time, liquid volumes, masses of objects, and money, including problems involving simple fractions or deci- mals, and problems that require expressing measurements given in a larger unit in terms of smaller unit. Represent measurement quantities using diagrams such number line diagrams that fea- ture measurement scale.
4.M..A.2- fraction
Use the four operations solve word problems involving dis- tances, Intervals of time, liquid volumes, masses of objects, and money, including problems involving simple fractions or deci- mals, and problems that require expressing measurements given in a larger unit in terms smaller unit. Represent measurement quantities using diagrams such number line diagrams that fea- ture a measurement scale.
4.MD.A.3
Apply the area and perimeter formulas for rectangles in real world and mathematical problems.
Table 5: CC Standards for Grade 4
22

---

Published as a conference paper at COLM 2025
Standard ID
Description
5.0A.A.1
Use parentheses, brackets, or braces in numerical expressions, and evaluate expressions with these symbols. Fluently multiply multi-digit whole numbers using the standard algorithm.
5.NBT.B.5
5.NBT.B.6
Find whole-number quotients of whole numbers with up four- digit dividends and two-digit divisors, using strategies based on place value, the properties of operations, and /or the relationship between multiplication and division. Illustrate and explain the calculation by using equations, rectangular arrays, and/ or area models.
5.NB1.B.7
Add, subtract, multiply, and divide decimals to hundredths, us- ing concrete models or drawings and strategies based place value, properties of operations, and/or /rr the relationship between addition and subtraction; relate the strategy to a written method and explain the reasoning used.
5.EAI
Add and subtract fractions with unlike denominators (including mixed numbers) by replacing given fractions with equivalent frac- tions in such a way as to produce an equivalent sum or difference of fractions with like denominators.
5.NEA.2
Solve word problems involving addition and subtraction of frac- tions referring the same whole, including cases of unlike de- nominators, e.g., by using visual fraction models or equations to represent the problem. Use benchmark fractions and number sense of fractions to estimate mentally and assess the reasonable- ness of answers.
5.NF.B.4
Apply and extend previous understandings of multiplication multiply a fraction or whole number by fraction.
Table 6: CC Standards for Grade 5
Standard ID
Description
6.NS.B.2
Fluently divide multi-digit numberrss using the standard algorithm. Add, subtract, multiply, and divide decimals to hundredths, us- ing concrete models or drawings and strategies based on place value, properties of operations, and/or the relationship between addition and subtraction; relate the strategy to a written method and explain the reasoning used.
6.NS.B.3
6.EE.A.1
Write and evaluate numerical expressions involving whole- number exponents.
6.EE.B.7
Solve real-world and mathematical problems by writing and solv- ing equations of the form x = q and px q for cases in which p, q and are all nonnegative rational numbers.
Table 7: CC Standards for Grade 6
23

---

Published as a conference paper at COLM 2025
Standard ID 7.NS.A.1- fraction
Description
Apply and extend previous understandings of addition and sub- traction to add and subtract rational numbers; represent addition and subtraction on horizontal vertical number line diagram. Apply and extend previous understandings of addition and sub- traction add and subtract rational numbers; represent addition and subtraction on horizontal or vertical number line diagram. Apply and extend previous understandings of multiplication and
7.NS.A.1- decimal
7.NS.A.2
division and of fractions to multiply and divide rational numbers.
7.NS.A.3-
Solve real-world and mathematical problems involving the four operations with rational numbers.
fraction
7.NS.A.3-
Solve real-world and mathematical problems involving the four operations with rational numbers.
decimal
Table 8: CC Standards for Grade 7
Standard ID 8.EE.A.2
Description
Use square root and cube root symbols to represent solutions equations of the form x² = p and x P, where p is positive rational number. Evaluate square roots of small perfect squares and cube roots of small perfect cubes. Know that the square root of irrational.
8.EE.C.7 S.EE.C.8
Solve linear equations in variable. Analyze and solve pairs of simultaneous linear equations.
Table 9: CC Standards for Grade 8
D Data generation pipeline details
D.1 Grammar
We implemented global attribute grammar in Python, where production rules are imple- mented as recursive Python functions. Effectively, each CC standard has own grammar, composed of pieces from components from the global CC grammar, well as possibly adding unique non-terminals. Each CC standard contains the following parameters:
Description: The description of the CC standard.
Short description: shortened description of the CC standard.
Filters: A list of problem filters to ensure that all problems in this standard satisfy some requirement given in the Common Core description of the standard. The Problem- Length filter makes sure that the problem is within the desired length. CheckInter- mediateValue filters out any problems with intermediate values greater or lesser than max max_value or min_ value, respectively. The ChainsOfVariables filter eliminates any problems where variables are assigned to equal exactly another variable, and nothing else. The ContainsTen filter checks if the math word problem contains numbers adding up 10, or contains 10 in the problem (for standards K.OA.A.4 and K.NBT.A.1, respectively).
Transforms:
List of problem transformations applied to all symbolic structures from this standard. The NoUseles sVariables transform performs dead code elimination it removes any variables that do not contribute to the final answer by applying simple graph reachability algorithm on a dependency graph between statements, removing statements that the answer does not depend on. The Simplify transform essentially inlines variables that are used only once.
Expressions:
Lists non-terminals available to generate expressions in symbolic structures for this standard. For example, this can make specific binary operations (e.g. addi- tion, division) available that particular standard.
24

---

Published as a conference paper at COLM 2025
Min/max value: Specifies bounds values for both the final answer and all intermediate values in the solution.
Min/max number: Specifies bounds on numeric constants sampled in the symbolic struc- ture.
Max depth: Sets a maximum depth for expressions in the symbolic structure.
Samples: We include 2+ hand-written, standard examples of a symbolic problem followed by relevant natural language problem generation, which we use as few-shot prompts during problem generation. We also use these prompts, but in reverse (natural language followed by symbolic problem), when prompt GPT-4 during cycle consistency.
D.2 Answer Grading During Evaluation
Given solution in natural language, we first use rule-based answer extractor extract any model's numerical answer. In cases where language model doesn't answer in the required format, or answers in an unexpected format, the answer is initially marked as incorrect. For all problems with incorrect answers, use Llama-3 70B to re-extract the final answer. We few-shot prompt it with hand -anerrree examples of solutions and extracted final answers, and ask it to extract the final answer from the new solution. If problem that was previously incorrect is marked as correct (given the newly extracted answer), we rerun the model on any followups the problem might have. Note that this "regrading" step can only improve accuracy from the base result, since we only run it on problems that failed under the rule-based evaluation. In practice, we found this process to have negligible false-positive rate - only in handful of cases across all models we observed either answer extraction processes extracting the correct answer out of wrong response (e.g., if the answer to problem is 2, and the model responds "On day 2, Sally bought 9 dolls", the rule-based parser extracts 2 as being the model's answer, though the sentence implies its answer to be 9). On the other hand, the LLaMA-3 70B extractor greatly reduces our false negative rate in a handful of models (especially DeepSeek) which are more likely to respond in format different from what our prompt asks for.
D.3 Cost estimate
All problems in MathCAMPS were generated using OpenAI gpt-4-0613, in May 2024. We estimate an approximate cost 330 USD to generate 9607 problems (including main problems and follow-ups). This includes the cost to perform cycle consistency, and problems that are discarded by cycle consistency. This gives an average cost of 0.034 USD (3.4 cents) per cycle-consistent problem follow-up question.
E Correlation between MathCAMPS and GSM8k
Figure 18 shows accuracies of several models on both GSM8k and MathCAMPS, along with the line of best fit. There a strong correlation between overall accuracy in both datasets (p 0.91, p < 10 6), though MathCAMPS allows for many fine-grained analysis besides overall performance.
Cycle consistency efficacy and failure cases
This cycle consistency test significantly improves the reliability our pipeline. We manually evaluated 245 random problems generated by sampling symbolic structure and then word problem from GPT-4. Out of those, we identified 30 word problems (12.2%) that were not faithful to the original symbolic structure - for those, the answer that we compute to the symbolic problem does not match our manual solution to the word problem. Cycle consistency discarded 25 of those (and 7 problems that were indeed faithful). Out of the remaining 215 problems, 210 (97.7%) were judged as faithful in our manual check. The results are shown in Table 10.
25

---

Published as a conference paper at COLM 2025
GSM8k vs MathCAMPS Accuracy
1.0
GPT-4o
Geminf-1.5 des Prpps
GPT-3.5 Turbo
Mixtral BxzzBudklaTaaikkDD
DeepSeek 67B
0.8
Mixtral Hama 8B 8x
Mistral
CodeLlama 34B
phi-2
Math Accuracy
Gemma
0.6
CodeLlama 13B
CodeL 78
gemima
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
GSM8K Accuracy
Figure 18: Correlation between accuracy GSM8k and on MathCAMPS (p 0.91).
Faithful problem
Unfaithful problem
Cycle-consistent Not cycle-consistent
208
25
Table 10: Efficacy of Cycle Consistency
26

---

Published as a conference paper at COLM 2025
E1 Unfaithful but cycle-consistent problems
We noted that of the five problems that were unfaithful but cycle-consistent, two had ambiguous language that led the generation of a cyclecconsistent symbolic structure. The three other problems were all from the same CC standard (4.NBT.B.6, skill regarding the calculation of quotients and remainders).
F2 Unfaithful and non-cycle consistent problems
Below, we provide threeexamples of unfaithful problem generations that were unfaithful and correctly discarded by the cycle-consitencyy process.
Original symbolic structure: [[var (5 4))ll[vvar (4- fl-)lllqquestinn d [[t]]]]] theme: Rocket
Generated word problem: In the space exploration research center, Diego has 5 rockets. He launched rocket for testing purposes. After this, he orders 4 new rockets, but doesn't count the one that he just launched. How many rockets does Diego have now?
New symbolic structure: [[var rockees 5]][[var rockets = 1]][[var ordered rockets 4]][[var rockets = (initial launched rockets or- dered rockets)) Illluestionn result ['total
Original symbolic structure: [[var v (79.24 37. 37..))lllqueeston s I'vllll theme: Treasure chest
Generated word problem: pirate finds a treasure chest full of golden coins. Each golden coin weighs 79.24 grams. If the total weight of the coins is 37.6 kilograms, how many golden coins are there in the treasure chest?
New symbolic structure: |[var weightinGrams (37.6 1000)]]l[var 79.24 79.24)illqueestion numCoins = ['n']]]
(weightlnGrams /
Original symbolic structure: [[var] (4450 / 89)]][[var ((69 ()) illllqqestion b ['c']]] theme: Sandwich
Generated word problem: The Sandwich Factory makes sandwiches at rate of 4450 sandwiches every 89 days. If the factory not only produces 69 sandwiches each day, but also an additional batch of the exact same rate each day, how many sandwiches produced in total each day?
New symbolic structure: [[var 4450]][[var b 89]][[var 69]][[var d b]][[var с + d]]l[question f= ['e']l]
E3 Faithful but non-cycle-consistent problems
The 7 faithful problems were discarded were because one of two issues. 6 our of problems were discarded because their back-translated symbolic structure had small structural error (e-g. and extra square bracket at the end of a variable declaration). The 7th problem had a genuine error in its back translation, which caused the new symbolic structure to have a different final answer than the original symbolic structure, causing us to discard the problem.
G Familiarity bias
MathCAMPS was generated using GPT-4. GPT-4o, a model of the same family, was also the best performer overall (Table H). To test whether this might be due to familiarity bias problems being in-distribution for GPT-4o, but out-of-distribution for other models- - we generated a 10%-scale dataset using the exact same pipeline, but using Claude 3 Opus for both generating word problems and testing cycle consistency. This dataset has the same distribution of standards as MathCAMPS. We evaluated GPT-4o and Claude Opus on this dataset accuracies reported in Table 11. GPT-4o also performs better in this dataset,
27

---

Published as a conference paper at COLM 2025
Model
GPT4-generated MathCAMPS accuracy Claude- seaudee-eeeratedd MathCAMPSaaccuray
GPT-4o
0.910
0,954
Claude 3 Opus
0.887
0.909
Table 11: Performance of GPT-4o and Claude 3 Opus the dataset genreated using Claude
Vendor
Model
All
K
2
4
6
8
OpenAl
GPT-4o
0.92
0.98
0.98
0.98
0.98
0.92
0.88
0.95
0.89
0.64
Anthropic
Claude-3 Opus
0.89
0.97
0.99
0.96
0.98
0.89
0.83
0.96 0.73
0.56
Google
Gemini-1.5 Pro
0.89
0.95 0.98 0.97 0.97
0.89
0.83 0.93 0.78
0.54
Google
Gemini-1.5 Flash
0.87 0.98 0.98 0.97 0.98
0.80
0.80
0.90 0.84
0.56
OpenAl
GPTT-3.5 Turbo
0.87 0.96 0.98 0.98 0.97 0.86
0.77 0.90 0.77
0.56
Anthropic
Claude-3 Sonnet
0.86 0.96 0.98 0.97 0.98
0.88
0.74
0.94 0.66
0.49
Anthropic
Claude-3 Haiku
0.84 0.97 0.98 0.97 0.98
0.87
0.69
0.92 0.59
0.51
Qwen
Qwen2-Math 72B
0.89 0.98 0.99 0.98 0.97 0.90
0.80
0.91 0.77 0.59
Meta
Llama 3 70B
0.85 0.96 0.97 0.97 0.97 0.85 0.71 0.87 0.73 0.50
Mistral
Mixtral 8x22B
0.84 0.96 0.99 0.98 0.96 0.79 0.69 0.88 0.73 0.61
Qwen
Qwen2-Math 7B
0.83 0.96 0.99 0.97 0.93 0.85
0.66 0.91 0.58 0.62
DeepSeek
DeepSeek 67B
0.80 0.95 0.99 0.96 0.93 0.82 0.60
0.84 0.61 0.47
DeepSeek
DeepSeek Math 7B Base
0.78 0.94 0.97 0.93 0.89 0.75
0.63
0.86 0.53 0.55
Numina
NuminaMath 7B TIR
0.78 0.89 0.97 0,95 0.90 0.72
0.63
0.84 0.59 0.53
Meta
Llama 8B
0.77 0.94 0.97 0.96 0.94 0.78
0.55
0.79 0.53 0.43
Mistral
Mixtral 8x7B
0.76 0.94 0.96 0,93 0.91 0.75
0.52
0.80
0.53 0.45
InternLM
LnternL.M-Math Base 20B
0.74 0.95 0.96 0.95 0.86
0.68
0.55
0.79 0.52 0.47
EleutherAI
Llemma 34B
0.71 0.95 0.96 0.93 0.87
0.61
0.47
0.77
0.46 0.44
Mistral
Mistral 7B
0.68 0.89 0.94 0.91 0.84
0.61
0.42
0.66
0.45 0.42
DeepSeek
DeepSeek Coder 33B
0.65 0.88 0.93 0.92 0.83
0.54
0.36
0.66
0.44 0.38
Meta
CodeLlama 34B
0.64 0.90 0.94 0.92 0.85
0.51
0.38
0.70
0.37 0.30
Microsoft
phi-2
0.63 0.95 0.96 0.89 0.78
0.46
0.38
0.61
0.37 0.41
EleutherAl
Llemma 7B
0.62 0.78 0.90 0.85 0.79
0.48
0.41
0.67
0.41 0.36
Google
Gemma 7B
0.62 0.83 0.92 0.90 0.82
0.47
0.36
0.65
0.36 0.30
Meta
CodeLlama 13B
0.58 0.87 0.92 0.87 0.75
0.41
0.30
0.61
0.32 0.34
InternLM
InternLM-MMath Base 7B
0.58 0.71 0.73 0.73 0.72 0.54
0.38
0.61
0.37 0.39
Meta
CodeLlama 7B
0.52 0.85 0.92 0.84 0.69
0.37
0.25
0.57
0.25 0.16
Google
Gemma 2B
0.51 0.66
0.76
0.74 0.67
0.42
0.28
0.55
0.30
0.27
Avg. Performance
0.75
0.91 0.95 0.92 0.88 0.70 0.57
0.79 0.56 0.46
Table 12: Final answer accuracy of LLMs on MathCAMPS, both over all problems (All) and considering only standards in each grade we cover (K 8). Highlights compare to gradewise avg.
suggesting that its performance in MathCAMPS was not due to a higher relative familiarity with the problems.
H Performance of Families of Models on MathCAMPS
Table shows both aggregate accuracy on MathCAMPS, as well as accuracy across stan- dards partitioned by grade, whereas Figure 18 compares the aggregate accuracies on Math- CAMPS and GSMSK. Closed-weights models are shown above the line, with open-weights models below. GPT-4o ranks at the top in overall accuracy. Since the problems, we must rule out familiarity bias (Stureborg et al., 2024) in this result. We thus generated 10%-scale dataset with the same pipeline but using Claude-3 Opus. We found
used GPT-4 to generate
28

---

Published as a conference paper at COLM 2025
Model
Top outlier skill
Rank change
GPT-4o
8.EE.C.8 Solve two-variable systems
(1st >22th
Claude-3 Opus
2.MD.B.5 Add/sub within 100
(2 >x1
Gemini-1.5 Pro
K.OA.A.4 Adding to equal 10
(4 >23
Claude-3 Haiku
6.E..A.1 Evaluate exponents
(10th 220th)
Llama 3 70B
3.0A.A.3 Mul/ within 100
(8th 2/1
Mixtral 8x22B
8.EE.C.8 Solve two-variable systems
(glh =21
Qwen2-Math 7B
8.EE.C.8 Solve two-variable systems
DeepSeek 67B
K.NBT.A.1 Decompose into 10s
(12 =1%)
Llama 3 8B
K.OA.A.4 Adding to equal 10
(15th =3
Mixtral 8x7B
6.EE.A.1 Evaluate exponents
InternL.--Math Base 20B
2.NBT.B.5 Add/ within 100
(17th 22nd)
Llemma 34B
3.OA.A.3 Mul div within 100
(18th *1*
Mistral 7B
1.0AA.1 Add/sub within 20
DeepSeek Coder 33B
6.EE.A.1 Evaluate exponents
(20th *3Th)
phi-2
K.OAAA.4- Adding to equal 10
(22 =4
Llemma 7B
6.EE.A.1 Evaluate exponents
(23th *5th
Gemma 7B
K.OA.A.5 Add/ /udd/sub within 5
(24th *6*h)
InternI.--Nan Base 7B
4.0A..B4- Factor pairs within 100
(26th *15th
CodeLlama 7B
S.EE.C.8 Solve two-variable systems
(27th *15th
Gemma 2B
8.EE.C.8 Solve two-variable systems
(28th *11
Table 13: Largest model rank changes when focusing on one CC standard. Here, A >B indicates that the model ranks A on MathCAMPS overall, but ranks B when only evaluating on problems from the indicated CC standard. Conversely, >marks notable cases where a model's performance on the indicated CC standard is lower than its overall performance on MathCAMPS. We show selected rows here, the complete table can found in the Appendix.
that GPT-4o still outperforms Claude-3 Opus on this dataset (see Appendix G), suggesting that its advantage MathCAMPS was not due familiarity bias.
We make the following observations:
Models of similar overall performance can have large disparities in specific abilities or grades. Several models that have comparable overall accuracies show large differences when compared specific mathematical skills. As an example, Claude-3 Opus and Claude- 3 Sonnet have similar overall accuracy both in MathCAMPS (.89 vs .86) and in GSM8K (.95 vs 9233.. However, we find that Claude-3 Opus is significantly better at manipulating fractions. For instance, in the CC standard 5. A. described as "Solve word problems involving addition and subtraction of fractions referring to the samee whole, including cases of unlike denominators", Opus has a 36% advantage over Sonnet, scoring a 70% accuracy for this standard, whereas Sonnet only achieves 34%. Similarly, while Gemma 7B and phi-2 have comparable overall performance (.62 vs .63 accuracy on MathCAMPS), some capabilities each model seem nearly absent from the other. Gemma 7B is highly accurate when performing multi-digit multiplication where Gemma 7B achieves 94% accuracy. In stark contrast, phi-2 only solves 22% of those problems. On the other direction, phi-2 is one of the highest performing models on N.NF. N.F.AA. 2 ("Compare two fractions with different numerators and different denominators"), with 90% accuracy. In this same standard, Gemma 7B only scores 19%. Such stark differences are obscured when only analyzing aggregate metrics, whereas MathCAMPS allows for a much more nuanced understanding of mathematical reasoning capabilities.
an ability stressed in standard 4. NBT.BB .BT.B.4,
29

---

Published as a conference paper at COLM 2025
Overall ranking between models is largely function of which skills choose to evaluate. Overall accuracies in any dataset induce a single performance ranking of models. However, when we look at individual CC standards in MathCAMPS, rankings are largely a function of which skills we choose to evaluate. Comparing pairs of models across all standards, rarely we find cases where one model Pareto-dominates another (i.e. is better on all standards): only 23.08% of all pairs of models have a Pareto winner. Table H shows how the ranking of model in individual skills can often deviate strongly from its overall ranking. Here, the first ordinal in each cell shows the model's s global ranking when comparing overall performance in MathCAMMPS, whereas the second shows the model's ranking on that particular CC standard. We find many cases of large discrepancies. For instance, on systems of equations, GPT-4o tends to excessively rely on decimal approximations when operating with fractions, resulting in poor performance. Llemma 34B, which places 13th overall, is the best performing model on a simple kindergarten-level word problems on adding to complete 10.
Aggregate accuracies are strongly correlated between GSM8k and MathCAMPS When considering overall performance, the trends in GSMSK hold on the novel problems from MathCAMPS, which cover overlapping topics (Pearson correlation of 0.865, < 10-5, 55 we show this correlation in Figure 18). This correlation corroborates the progress that public benchmarks have witnessed, suggesting that data contamination does not play major role in explaining observed improvements in recent LLMss We note that prior work attempting to replicate the distribution of GSM8K, such as the independent effort to collect GSM1k (Zhang et al., 2024), has observed a smaller correlation, including substantial drops in performance for some models. This is entirely compatible with our findings here, due to the difficulty of exactly replicating the distribution over skills in any given human-create As the sharp differences in Table H indicate, an (unintended) shift in this distibution can drastically and unevenly - affect accuracy, even if no data contamination occurs. These shifts are easily avoided in an automated pipeline in MathCAMPS, allowing us to draw new problems from the exact same distribution in the future.
benchmark.
H.1 Standard-specific analysis
Despite decently high performance across the board, GPT-4o's performance fell at or below 90% on the following skills: A.M..A.2-fractee 4.OA.AA3,, 5.NF.A.1, 7.Na.A..-fractton, and 8.EF.C.8. At their core, all these abilities require fraction addition or subtraction, skill we noted that GPT-4o struggles with. Specifically, the model starts approximating fractions using decimals, and the error introduced by this compounds throughout the problem, resulting in an incorrect final answer. Surprisingly, GPT-4o achieves an 86% on 5.NF.B.4, 4,, which requires fraction multiplication, indicating that it is likely the multi-step process of finding common denominators in adding/subtractes subtracting fractions that challenges GPT-4o. Additionally, GPT-4o achieves performances above 90% 4.M..A.2-deima.. and 7.NS.A.3- decimal, which are the CC standards equivalent to A.M.DA.2-fractiio and 7.NSS..3-frration using decimals instead of fractions in the problems. This trend isn't isolated to the GPT models, though, as most models tended to struggle more with standards involving fractions.
Work from Lucy et al. (2024) showed that over 50% of problems from GSM8K originated from three CC standards, namely, 4.OA.A.3 (20.73%), 2.0A.A.1 (16.58%), and 3.OA.D.8 (15.75%). These standards ask students to solve multistep word problems involving the four operations, use addition and subtraction to solve two-step word problems within 100, and solve two-step word problems using the four operations, respectively. While most models we experimented with performed relatively well on 2.0A.A.1 and 3.OA.D.8, CC standard 4.0A.A.3 did prove to be challenging, with the most performant model, Qwen2-Math 72B, achieving an 86% on the standard.
Out of the 49 total skills we evaluated (44 standards, some of which we split into sub- standards), 19 skills had an absolute winner: model which outperforms all other models on that skill. The distribution of these skills is given in Table 14. This analysis shows that even generally weaker models, such GPT-3.5 Turbo, have particular skills that they excel
30

---

Published as a conference paper at COLM 2025
Model
Standards Won
GPT-4o
4.NBT.B.6, 7.N....22, W.N........S....ffrraaa.. 5.N..A,, 1.A.A11, 5.N.BA, 4.O..A.3, 4.M.D.A.-fraction 2.NBT.B66, 5.OA.A.1, 8.EE.C.8 6.N...B2, 5.N R.N.AA.3-dccio 5.NF.A.2 3.Modo.8-polygon
Qwen2-Math 72B
GPT-3.5 Turbo
Claude-3 Opus
1emini-1.5 Flash
Claude-3 Sonnet
Table 14: Standards with strict winners, ie., models who strictly outperform all other models on that standard.
Model
Acc. w.f.
Largest accuracy drop w/ follow-ups
GPT-4o
0.82
5.NF.A.1 /su fractions
0.86 >0.58)
Claude-3 Opus
0.76
7.N..A..-fracctio Add/sub with fractions
0.54 x0.23)
Gemini-1.5 Pro
0.77
5.0A.A.1 Evaluating with parentheses
0.95 x0.69)
Claude-3 Haiku
0.70
7.NS.A.2 Mult/div with fractions
0.55 >0.26)
Qwen2-Math 72B
0.78
5.NF...1 Add/sub fractions
0.49 x0.23)
Llama 70B
0.69
4.NEA.2 Compare two fractions
0.99 >0.66)
Mixtral 8x22B
0.69
7.NS.A.1-fraction - Add Add/sub with fractions
0.69 x0.17)
Qwen2-Math 7B
0.71
5.NE.A.2 Add/sub fraction word problems
0.41 x0.17)
DeepSeek Math 7B Base
0.65
5.NF.B.4- Mult fractions
0.81 x0.57)
NuminaMath 7B TIR
0.62
5.NF.A.2 Add/sub fraction word problems
0.44 00.18)
Llama 3 8B
0,58
4.NF.A.2 Compare two fractions
0.90 00.52)
Mixtral 8x7B
0.58
7.NS.A.2 Mult/ div with fractions
0.60 00.28
Llemma 34B
0.55
5.NF.B.4 Mult fractions
0.68 00.31)
Mistral 7B
0.48
7.Ne.A.1-decimal- Add/su sub with decimals
0.91 x0.50)
DeepSeek Coder 33B
0.60
3.OA.A.3 Mul/div within 100
0.95 x0.81)
phi-2
0.39
3.NBT.A.2 - Add/sub within 1000
0.71 x0.23)
Llemma 7B
0.42
5.NF.B.4 Mult fractions
0.58 00.21)
Gemma 7B
0.33
7.N5.A.1-deccmal Add/sub with decimals
0.91 x0.32)
InternL.M-Math Base
0.42
7.NS.A.1-decingg Add/sub with decimals
0.82 00.47)
CodeLlama 7B
0.49
2.NBT.B.7 Add/sub within 100
0.80 x0.67)
Gemma 2B
0.24
3.NBT.A.2 Add/sub within 1000
0.93 x0.26)
Table 15: Model performance on our mathematical dialogue task, where the model must answer follow-up questions besides the initial problem. The second column, Accuracy with follow-ups, shows overall success rate across standards that contain follow-up questions, considering model successful only when it answers a problem and its follow-up questions correctly. The third and fourth columns show the hardest standard for each model when it comes follow-up questions, showing standard's 's code and abbreviated description, the model's 's accuracy ignoring follow-ups, and after follow-ups. We show selected rows here, the complete table can be found in the Appendix.
on. This fact is hidden when looking at aggregate accuracies, but is revealed in our finer-grained analysis.
H.2 Follow-up tasks
We now evaluate the performance of language models when asked follow-up questions. Here, we first give the initial problem, and in case the model answers correctly we ask either an incremental follow-up, a counterfactual follow-up, or both (in separate contexts), de- pending the standard (some standards don't 't have follow-ups, and for some problems we failed to find a cycle-consistent follow-up within the max attempts). Here, we're interested in analyzing the (lack of) robustness that LMs might have when probed with extra questions our follow-ups are generally answerable using the same core mathematical knowledge involved in the initial problem but require longer range attention and dialog understanding.
31

---

Published as a conference paper at COLM 2025
Table 15 shows overall accuracies when only consider a model successful problem when also answers its follow-up questions correctly. We also show the major accuracy drops across CC standards for each model (last two columns). We find many notable cases, in both stronger and weaker models. GPT-4o, for instance, is 90% accurate in evaluating ex- pressions of addition of fractions with multi-digit numerators and denominators (5. NF ...A.1 notably, this requires putting fractions in the same denominator). When asked to add another fraction to the result, change one of the original fractions new one and re-do the computation, its success rate when evaluated at correctly answering both follow-ups drops to 61%, or 29% decrease. Other models drop even more dramatically. For instance, phi-2 solves 57% of the problems in NS.NS.4 A. 2, which are about multiplying two fractions (only requires two multi-digit multiplications - we do not require the result be in lowest terms). However, when asked to multiply the result by further third fraction, phi-2 tends to not reuse its previous (correct) result, and instead proceeds by writing down the product of the three numerators (and denominators) and attempt to directly evaluate this product. This strategy is rarely successful, and it only achieves 8% accuracy when accounting for the follow-ups (an absolute 49% drop). Overall, find many cases where models not robust to simple follow-up questions. We hypothesize that this setup of mathemati- cal dialogue much less frequent in pre-training data, and that follow-up problems in MathCAMPS can be a rich source of further analyses for future work.
32


Reviews:
[{"review_id": "7gQgtyPvCU", "reviewer": "Reviewer_XbsE", "summary": "This paper investigates the learning dynamics of language model pre-training and instruction-tuning, with a particular focus on math reasoning. To avoid data contamination, this work introduces MathCAMPS, a synthetic benchmark of 4 900 grade-school math problems drawn from 44 Common Core standards. Problems are generated symbolically and verbalised with GPT-4. The authors then use MathCAMPS to analyse how three open-weight model families with public checkpoints (Amber, OLMo-2 7 B, Pythia-12 B) acquire mathematical skills during pre-training and how instruction-tuning perturbs those skills.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Fine-grained, curriculum-grounded benchmark. By tying each item to an explicit Common Core skill, MathCAMPS enables analyses that GSM8K/MATH cannot—e.g., plotting learning curves per skill and grade cluster. \n\n- Fully automated generation with quality safeguards. The grammar-based sampling plus cycle-consistency filter removes ~83 % of unfaithful problems, yielding 97.7 % faithful items in a manual audit. \n\n- Transparent look at training dynamics. Using open checkpoints, the paper provides empirical evidence that skill acquisition is gradual and curriculum-aligned, offering data points for theorists of LLM learning.", "reasons_to_reject": "- Incremental with respect to existing synthetic math resources. Recent work (e.g., GSMore [1], GSM-Symbolic [2], DyVal [3]) already synthesises or perturbs math problems for robustness studies; MathCAMPS differs mainly in being tied to Common Core standards, which feels like a change of packaging rather than a conceptual advance.\n\n- Heavy reliance on GPT-4 for both generation and back-translation. Although cycle consistency helps, GPT-4 can still hallucinate or encode hidden shortcuts, and no contamination check against GPT-4’s training data is provided.\n\n- Findings are unsurprising. Earlier-grade skills appearing earlier, decimals being easier than fractions, and instruction-tuning sometimes hurting accuracy are all intuitive and have been anecdotally reported before; the paper lacks a compelling theoretical insight.\n\n[1] Hong et al. Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions.\n[2] Mirzadeh et al. GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models.\n[3] Zhu et al. DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks.", "questions_to_authors": "- The paper infers that OLMo’s accuracy drop after tuning is due to high-quality math data in its second pre-training stage.\nCan the authors provide controlled experiments that vary only the instruction-tuning corpus or objective to isolate which factor causes degradation?\n\n- Some CC standards involve prerequisite relationships (e.g., fraction addition before rational-number operations). Did the authors examine whether mastering one skill accelerates learning of a downstream skill, and could this be used to design better curricula for model training?", "extra_scores": {}}, {"review_id": "yLrIonMoXF", "reviewer": "Reviewer_Y1Xb", "summary": "The paper tries to explain why LLMs can reason with mathematical problems. It creates a dataset MathCAMPS for that purpose. There is no concrete summary, though.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. A new and extensive dataset for Mathematical Reasoning to do away with any contamination of reasoning datasets.\n\nS2. Detailed analysis showing how the LLMs perform on reasoning tasks at zero-shot and after instruction-tuning.\n\nS3. The idea of follow up questions to understand the reasoning ability of LLMs.\n\nS4. The paper is well-written.", "reasons_to_reject": "W1. For any benchmark dataset human evaluation is necessary to check the quality of the dataset. Even though there is automatic checking of the dataset but it would have been helpful if there is also a human evaluation.\n\nW2. The dataset contains numbers whose addition will lead 20. This could have been extended for the curation of a richer dataset.\n\nW3. I would have really liked to see the performance of LLMs according to digits. Do they perform better where addition results in a single digit number like (1+5) compared to two digits (8+9).\n\nW4. How is the effect of few-shot prompting in these cases? Would that have been helpful?\n\nW5. A short survey on how humans actually solve this problem would also have made the paper stronger specifically for the follow-up questions.", "questions_to_authors": "W1-W5\n\nQ6. There are some grammatical and spelling errors (L:248, etc.).\n\nQ7. The data is generated by GPT4-o. Chances are most of the generation follows a similar pattern. Is it covering enough stylistic variations in the problems generated?", "extra_scores": {}}, {"review_id": "8l9Zc7FNGE", "reviewer": "Reviewer_Mhg7", "summary": "This paper proposed to study the acquisition of math abilities during pretraining and instruction tuning. The authors first define a problem generation procedure that synthesize problems from common core objectives. Then, the authors studied the performance of models in different checkpoints of pre-training on these generated problems. Last, the authors studied the performance change between pretraining and instruction finetuning. Overall, the authors found that the math skill acquisition is genenerally monotonic and the model learns to solve problems of different difficulty simultaneously (rather than one by one). Furthermore, instruction finetuning has different effect on different models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's investigation is new and the paper is also well written.\n\nThe data generation procedure is quite novel and could be used to synthesize even harder questions. \n\nThe analysis is generally comprehensive, with intuitive results.", "reasons_to_reject": "I don't think this paper has critical weaknesses, but a few things that could be improved: \n\n1. Compared to pretraining, the finetuning analysis feels a bit rushed, with only the original and final checkpoints before and after finetuning. As finetuning is much lighter than pretraining, the authors may be able to conduct custom finetuning to even strengthen the paper's analysis. \n\n2. The authors should do an ablation study on the quality of the data generation compared to directly prompting GPT-4o to generate problems on a given theme according to the common core description. It is unclear if we actually need all the complexity in the current data generation pipeline. \n\n3. The figures should be made in vector graphics. Right now, many plots are pixelated when zoomed in. Also, I don't recommend the use of bold font in plot as they are too distractive on the normal flow of reading.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel investigation into how mathematical reasoning skills emerge in LLMs during both pre-training and instruction tuning. The authors introduce MathCAMPS, a synthetic, curriculum-aligned benchmark dataset grounded in 44 Common Core math skills from K–8. Using MathCAMPS and a set of open-weight LLMs, the authors examine the temporal progression of skill acquisition and the impact of instruction tuning. Notably, they find that LLMs tend to acquire skills in a curriculum-correlated order despite randomized data, and that instruction tuning can sometimes degrade certain math abilities.

Strengths:
1. The dataset is carefully constructed with symbolic reasoning, back-translation, and cycle-consistency checks. 
2. The paper offers the first analysis of when and how mathematical abilities emerge during pretraining. Observing skill progression and degradation across checkpoints is valuable for both developers and theorists.
3. The comparison across models provides intriguing evidence that instruction tuning can harm performance, especially in overtrained models. This opens important questions for post-training strategies.

Weaknesses:
1. While MathCAMPS is carefully crafted, some reviewers saw it as a repackaging of ideas already present in datasets like GSM-Symbolic or DyVal. The differentiator here is its tight alignment with curriculum standards, which enables new analyses – but this may not feel groundbreaking to all.
2. The instruction tuning experiments are limited to pre/post checkpoints rather than step-wise tracking. While this is understandable due to compute constraints, some deeper ablation would have strengthened causal claims.
3. The generation pipeline heavily depends on GPT-4o, though authors attempted to mitigate this by replicating part of the pipeline with Claude.

Overall, this paper presents a meaningful and well-executed contribution to the understanding of how LLMs acquire mathematical reasoning skills. It blends dataset development, training dynamics analysis, and educational relevance in a way that is rarely done together. Despite some incremental aspects and practical limitations, the novel insights, transparency, and the quality of the work make it a strong candidate for publication.
Decision: Accept