Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
Authors: Sadegh Mahdavi
Muchen Li
Kaiwen Liu
Christos Thrampoulidis
Leonid Sigal
Renjie Liao
Venue: ICML 2025 poster (2025)
Keywords: Mathematical Reasoning
Large Language Models
Evaluation
Primary area: deep_learning->large_language_models
TL;DR: We propose a new dataset and benchmark for training and evaluating mathematical reasoning in LLMs.

Abstract: Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. 
However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts.
In addition, current benchmarks are prone to contamination, leading to unreliable evaluations.
In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions.
Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in **AoPS-Instruct**, a dataset of more than 600,000 high-quality QA pairs.
Our experiments demonstrate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. 
Moreover, we build an automatic pipeline that introduces **LiveAoPSBench**, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance.
Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability. 
Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain.

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Reiistaan Evaluation
Sadegh Mahdavi ! Muchen Li
Kaiwen Liu
Christos Thrampoulidis Leonid Sigal 1234 Renjie Liao 123
Abstract
LLMs in this domain. Our benchmark and code is available at /ttps://livend hbench. loabbiio/eeaeebbeee html.
Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. However, the training and evaluation of these models are con- strained by the limited size and quality of avail- able datasets, as creating large-scale data for such advanced problems requires extensive ef- fort from human experts. In addition, current benchmarks are prone to contamination, lead- ing to unreliable evaluations. In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions. Using open-source ILMss, de- velop method to extract question-answert pairs from the forum, resulting in AoPS-Instruct, a dataset of more than 600,000 high-quality QA pairs. Our experiments demonstrate that fine- tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. Moreover, build automatic pipeline that introduces LiveAoPSBench, evolving evalua- tion set with timestamps, derived from the latest forum data, providing a contamination-resistaa benchmark for assessing LLM performance. No- tably, we observe a significant decline LLM performance over time, suggesting their success older examples may stem from pre-training exposure rather than true reasoning ability. Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of
1. Introduction
Large language models (LLMs) have shown tremendous success in solving various tasks such code generation (Li et al., 2022), math reasoning (Shao et al., 2024), and com- monsense reasoning (Zellers et al., 2019; Achiam et al., 2023), suggesting that current models may show signs of artificial general intelligence (AGI) (Bubeck et al., 2023). Math reasoning is perhaps one of the most challenging tasks for the LLMss, since mathematics is inherently structured, requiring not just recall of facts but also rigorous logical in- ference, abstraction, and understanding of formal symbolic systems. As such, there have been grand challenges (Selsam et al., 2019) and million-dollar prizes AIMO (2023) estab- lished for model capable of solving Olympiad-level math problems.
On the training side, despite significant progress certain areas, such as geometry, particularly with the assistance of symbolic methods (Trinh et al., 2024), the performance of LLMs remains limited on Olympiad-level problems (He et al., 2024). One of the key challenges in advancing competition-level math reasoning, compared to other do- mains like coding or grade-school math, is the scarcity of large-scale data. Creating valid and challenging math ques- tions, along with providing correct solutions, is costly. This is especially true for Olympiad-level problems, which can be time-consuming even for experts. This highlights the need for scalable and automated methods to collect high- quality data for Olympiad-leve problems to facilitate further advancements in this field.
On the evaluation side, in contrast to the rapid advancements in LLMs, the evaluation of their math reasoning capabil- ities remains relatively underdeveloped. First, afore- mentioned, the cost of creating and annotating advanced math problems is high. Second, popular datasets such as MATH (Hendrycks et al., 2021b) and GSM8K (Cobbe et al., 2021) have been saturated by both open-source and closed- source models (Achiam et al., 2023; Yang al., 2024b).
Equal Contribution, order decided by a coin flip Unniversity of British Columbia, Vancouver, Canada 'Vector Institute for AI, Toronto, Canada *Canada CIFAR AI Chair "NSERC CRC Chair. Correspondence to: Sadegh Mahdavi <smahdavi woc.ube.ca>. Muchen Li <muchenli www.niiiees...........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination- Evaluation
Third, benchmarks whose test sets are publicty available online (Hendryeks al., b021b; Cobbe et al., 2021; He et al., 2024; Zhang et al., 2023b) are prone to potential con- tamination. Although techniques like n-gram matching and leeality-sensitive hashing have been applied as a common practice (Achiam et al., 2023; Dubey et al., 2024; Yang et al., 2024a) to reduce contamination, they still suffer low accu- racy and would not be able to rule out rephrased questions, as shown by Yang et al. (2023). Given these limitations, it is crucial to develop an evolving evaluation benchmark that contains abundant and up-to-date test samples, and designed with appropriate difficulty to fairly assess a model's 's math reasoning abilities.
as OlympiadBench, Omni-Math, and our LiveAoPS- Bench dataset, verifying the effectiveness of our dataset in enhancing math reasoning capabilities of LLMs.
2. Related Work
In this section, we provide an overview of the existing math- ematical datasets used for evaluation and training purposes. Additionally, we review the latest methods and LLMs for enhancing and evaluating these math datasets.
Evaluation Datasets for Math. The evaluation of the mathematical capabilities of LLMs has traditionally re- lied on well-established and widely-used datasets such as GSM8K and MATH (Cobbe et al., 2021; Hendrycks et al., 2021b), which have served as benchmarks for several years. These datasets typically contain math problems ranging from middle-school to high-school level, providing broad coverage across various problem categories. However, they present two significant limitations: 1) being older, their test sets are more susceptible to contamination from current training data of LLMs (Yang et al., 2023), and 2) they have reached a level of saturation, with state-of- fittheeertttt (SOTA) models achieving over 90% accuracy (Yang et al., 2024b). To address these shortcomings, Zhang et al. (2023b) intro- duced the Gaokao dataset, which includes more challenging high school-level problems from the Chinese college en- trance exam. In addition, newer datasets such as Olympiad- Bench (He et al., 2024), AMC23 (AOPS, 2023), AIME24 (AOPS, 2024), and Omni-Mate (Gao et al., 2024) represent higher levels of difficulty, collecting from more recent high school competition problems. While these datasets tem- porarily mitigate the risk of data contamination, they remain susceptible to this issue as LLMs continue to evolve, partic- ularly with fine-tuning on newer data. To address this, we introduce LineAoPSBench, which utilizes the most recent posts from the AoPS forum and applies techniques to exclude any previously used problems from the new posts. More importantly, our pipeline is fully au- lomated, allowing the evaluation set to evolve with forum posts, thereby significantly decreasing the likelihood of con- tamination.
The Art of Problem Solving' (AoPS) forum is a rich re- source for Olympiad-level math problems, featuring discus- sions topics such algebra, geometry, combinatorics, and number theory from competitions like AMC (AOPS, 2023), AIME (AOPS, 2024), and the International Math- ematical Olympiad (IMO). However, the forum's 's unstruc- tured nature, including irrelevant comments and incomplete solutions, poses challenges in extracting high-quality, struc- tured question-answer (QA) pairs. Developing an effective automatedd pipeline curate these QA pairs is essential to address the scarcity of large-scale, high-quality data for training and evaluating models in Olympiad- math rea- soning. In this paper, we utilize the posts from the AoPS forum to create a large-scale training and a contamination- resistant evaluation set. Our pipeline is designed to run automatically, enabling us to build and maintain evolving train/evaluation datasets. This automated approach is cru- cial, as it allows for continuously updating the datasets, ensuring they are less likely to suffer from contamination, even as existing data potentially becomes compromised over time. In summary, our key contributions are as follows:
We build pipeline to extract questions and solu- tions from raw AoPS forum data, constructing the AoPS-Instruct, a novel large-scale dataset with 652K Olympiad-level math QA pairs.
Using the most recent QA pairs, We build tomatic pipeline that introduces LiveAoPSBench, contamination-resistaan evaluation set for assessing the math reasoning capabilities of LLMs. LiveAoPSBench stays regularly updated.
Training Datasets for Math. Training datasets for math- ematical reasoning can be categorized into two types: pre- training and supervised fine-tuning (SFT) datasets. First, pretraining datasets consist of large-scale math data, e.g., bil- lions of tokens used during the pretraining phase of L.LMs. Notable examples include Open- Web- (Paster et al., 2024) and Minerva (Lewkowycz et al., 2022), which contain 38.5B and 14.7B tokens of math data, respectively. Sccond, SFT datasets focus on high-quality question-ant ansserr pairs. Examples include Open-Math-Inntrute (Toshniwal al., 2024), Orca-math (Mitra al., 2024), MetaMath (Yu et al., 2024), DART-Math (Tong et al., 2024), ScaleQuest (Ding
Our experiments on LiveAoPSBench show a declining performance trend over time for various LLMs, indi- cating potential data contamination, and stressing the need for up-to-date evaluation data.
Fine-tuning various L.MSs on AoPS-Instrune lead to improved performance on standard benchmarks such
2

---

Leveraging Online Olympiad- Level Math Problems for LLMs Training and Contamination-Resittae Evaluation
2023
2024
Drop (%)
Model
(5.2K)
(3.8K)
Qwen2.5-Math-lmn
34.80
33.40
4.02
72B 1-3---een-MMM Math--
37.84
36.15
4.45
25
72B-Qwen2.5-Muthee
42.36
40.45
4.51
22B-NuminaMuth-CoT
25.59
24.14
5.68
Accuracy (%)
30
N0B-Innermlm2-Mutte
17.78
16.03
9.83
MBB-DeepSeeKCoodrV2-mm
22.08
19.80
10.31
25
7B Qwen2 Math-Ins
33.26
29.32
11.85
7B NuminaMath CaT
16.88
14.76
12.55
20
7B- DeepSeek Math RI.
14.35
12.44
13.35
7B-Maatsstral-vO.1
15.25
13.00
14.76
15
7B- 1o-Innemll2-MMtth--a
16.26
13.64
16.16
27B-Gemma2-it
12.78
11.59
9.30
10
70B-Llama-3.1-k
22.02
19.34
12.16
8B-Llamu3, 1-Ins
13.01
10.85
16.55
5
Llama3.2 2-Inn
12.67
10.32
18.51
Mar-tay AmsAug 23
Doc Sep-Noy 23
Man-May Aug 24
9B Gemma2-Ht it
11.63
9.30
20.01
1B Liaaalamaa 3.2 Ins
6.32
4.83
23.62
Figure 1. Accuracy trends of various LLMs Licch over 18- 18-month period, highlighting a consistent decline in performance. We separate the math expert model from the general purpose model on the right. The degradation in accuracy varies across models, ranging from 2.4% 23.6%. Please refer to figure for results on state-of-the reasoning models like Qwen-QwQ and DeepSeek-RI R.1.
et al., 2024), MATH (Shah et al., 2024), and the training sets of widely used benchmarks such as GSM8K (Cobbe al., 2021) and MATH (Hendrycks al., 2021b). How- ever, these datasets are generally limited to grade-school and intermediate high-school level mathematics and do not target more advanced topics like plympiad-leved math. One of the most closely related datasets to ours is Numina (Li et al., 2024), which combines popular SFT datasets like Orca-math, MATH, and GSM8K, along with approximately 190K new Olympiad-level QA pairs. Concurrently, Yue al. (2024) introduced large-scale instruction fine-tuning dataset for math and science, which has also shown improve- ments in mathematical reasoning. Table presents detailed comparison of our dataset with these related datasets.
Table I. Comparison our dataset with other related datasets from the literature. Our dataset uniquely includes timestamp informa- tion and leverages open-source large language models (LLMs)s like Qwen 2.5 72B for solution rewrites. denotes inclusion of ad- ditional training datasets such as GSM8K, Orca-Math, and MATH. Datasets marked with have their solutions entirely generated by
LLM.
Dataset Size
Time Stamp
Olympiad Level
Solution
Dataset
Train
Eval
Rewrite
Numina
850K*
0.IK
X
GPT4-o
OpenMaathinstruct
Mixtral
OlympiadBench
6.1K
X
V
Human
DARI-Math
585K
-K
X
X
DeepSeckMathh
GSMBK
7.5K
1K
X
X
Human
MATH
7.5K
5K
X
Human
Orea-Math
2RUK
X
X
GPT-4"
MAmmoTH2
10M
X
X
Mixtral
Resistant Evaluation. Benchmarks that are publicly accessible are prone to be contaminated due to the potential inadvertent data overlap during training. The typical decontamination method involves using exact substring (e.g., n-gram) matching to detect overlaps with the target evaluation sets (Zhuo et al., 2024). However, this approach fails to catch rephrased examples and can not climinate all overlaps with the test set (Yang et al., 2023). While alternative L..M-based methods for decontamination have been proposed, they often lack guarantees and may result in high false-positive rates (Yang et al., 2023). A reliable way to mitigate contamination is to select data that appeared after LLMs were trained, known as the knowledge cut-off. In the code generation domain, LiveCodeBench (Jain et al., 2024) addresses this issue by categorizing data based on timestamps, setting a cutoff date, and designating
Contamination-
AoPS (Ours)
647.2K
3.8K
/
V
Qwen
2.5
data beyond this point unseen. We adopt a similar strategy in the math domain, partitioning the dataset by timestamps and enabling users to select data based on specific dates. Although this approach may not fully eliminate rephrased existing questions, it ensures that evaluation data remains unseen and less contaminated, providing more accurate and fair assessment of LLMs.
Several specialized models have been developed to improve the mathematical reasoning ca- pabilities of LLMs (Shao et al., 2024; Mistral, 2024; Li et al., 2024; Yang et al., 2024b; Azerbayev et al., 2024). These models are typically initialized from pretrained general-
Math-Specife Models.
3

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resiitae Evaluation
AoPS-Instruct Pipeline
1.07M Topics
550K Topics
550K Topics
652K QA Pairs
Compose 2831
Conzote C(3), 29).
Compuic (131.26).
Compuse C131, 26)
0131, thin sdlr
C(01, 25) thing is s0t
CYSL, 28) is the same thing so si Which ceneeingg out renms gives 6496.
Step F) - al lele *I
Stop 3: Step Canco commno
canceling
cunceling leanta (phcce us 4695
terms (21 30 20)/0 x
givss 4690
Topic
Q4 Extraction > Orcen 32B
Amswer
Final arswer 4496
Hiltering
Rewrting
Thank you!
Thank you!
Owce
Quen
14B
71B
Original A
Rewithee
Original A
matrices vecters
the
remenber IE'I's vary
LiveAoPSBench Pipeline
Repeat Rewriting Muttiple
LAi
Symbolic Rqual?
Mutchhe
LLML-A
Fiber Out Phoof Questions
Tinesturg Cut off
Quality
LIM RewiittnnA
Q2
LLMS-Aj
LLM Rewitten.A,
Fittr
Subsomple
Quality Vinifaosim
OriginAA
Originl
By Human
(training data). First, irrelevant topics are detected using small LLM. We then extract question-ansse answer pairs from relevant discussions, and rewrite cach answer into step-by-step solution. Bottom: LiveAoPSBench Processing pipeline of LiveAoPSBench (evaluation data). We take the most recent posts, use two LLMs to rewrite the solution, filter out the questions without clear answers, and create the final evaluation set.
Figure 2. The overall process of our dataset curation. Top: Processing pipeline of
purpose models, trained on large math datasets, followed by math-specific SFT, and then refined through reinforcement learning with human feedback (RLHF). In this paper, we fine-tune both general and math-specifin models to demon- strate that AoPS-Instruct brings consistent improvements.
author presents math problems (e.g., competition-level prob- lems) or general questions, such as seeking advice or re- sources. Our raw dataset consists of 1, 076, 712 topics. Top- ies posted up until December 2023 are used the training set, while those posted between January and August 2024 are reserved the evaluation dataset.
3. AoPS Dataset
Step 1: Math Question Detection. We then filter out inrele- vant topics, specifically those not containing a mathematical question. To achieve this, we use Qwen 2.5 14B (Yang et al., 2024a) to decide the relevance of each topic. The first post of each topic determines whether the topic a mathematical question or not,
In this section, we first describe the process of extracting and cleaning QA pairs from the AoPS forum to construct our training set. Then we explain how to utilize the latest forum data to create a reliable, contamination- dataset for assessing model performance.
evaluation
manually design a few-shot prompt, provide the first post of the topic to the model, and prompt the model to output if the topic is math question not. This step reduces the dataset to 478, 337 topics with math questions after pruning 598, 375 irrelevant ones.
3.1. Math Instruction Fine-tuning Dataset: AnPS-Instruct
We now describe the five steps of our automated pipeline for constructing the instruction fine-tuning dataset AoPS- Instruct.
Step 2: Question-Answer Extraction. After filtering, we extract the math question from the first post of each topic and identify potential solutions provided in subsequent posts. Since this task requires understanding the entire conversa- tion and determining which responses contain valid solu- tions, we employ the 70B variant of Llama 3.1 for this step,
Step 0: Raw Forum Discussion Collection. We begin by collecting raw discussions from the forum website, where each discussion called a "topic". these topics, the

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and
Evaluation
Question: Find all solutions in N: 2" + 17y
User Response:
The multiplicatiie order of 2 modulo 17 is 8, since 2 = = -1 (mod 17). Therefore, if x 2 3. then 8 2", and so 2 =
(mod 17). So < 2, and so the only solution is (x, y) = (2. 1).
Qwen Rewritten Solution:
1. Determine the multiplicative order of 2 modulo 17. We find that 2 16 (mod 17).
Since 2 1 (mod 17). it follows that 2 (mod 17). Therefore, the multiplicative order of modulo 17 8.
Consider the equation 2
17y. If x 2 3, then 2 is multiple of 8 (since 2 and higher powers of 2 multiples
8).
4. If'2 is multiple of then 2 = 1 (moo 17), because the order of 2 modulo 17 is 8.
5. Substituting 2 = (mod 17) into the equation, we get + 17y (mod 17), which simplifies to = 0 (mod 17).
This is contradiction.
Therefore, must be less than The possible values for æ 0, 1. and Check each possible value of æ:
For 0: 2² +1= 2' 1 3, which is not divisible by 17.
For x = 2
+ = 2 + 5, which is not divisible by 17.
For 2: 2²a +1 2 1 16 1 17, which is divisible 17.
8. The only solution is (x. (2,
Thus, the solution is 1)
Figure 3. Example of posted problem, a user solution, and LLM-rewritten solution. The model fills in the trivial intermediate steps. reformats the response, and puts the final answer box.
enabling the detection of both the question and all relevant answers from the discussion.
overlap with the test sets of commonly used math bench- marks. Following the approach used in DeepSeek (Shao al., 2024), we employ 10-gram exact match de- contamination (Zhuo al., 2024) method ensure that dataset remains distinet from those benchmarks.
Step 3: Solution rewriting. Math solutions generated by users the AOPS forum are often concise, omitting details assumed to be common knowledge among the target audi- ence. For instance, a user might write (x + y2)/2 2 /ryz without explicitly mentioning the application of the AM- GM inequality to (x. yz). While such brevity is typical for expert-level discussions, I.IMS trained on these succinet solutions often struggle to maintain their chain-of-thought reasoning capabilities.
After following the steps described above, we have a total of 652K QA pairs, out of which 255,255 255 are before Jan We provide further
2024 and constitute the statisties of our dataset in Section 4. 1.1 and Figure
3.2. CuntaminationnResistaat Evaluation: LiveAoPSBench
Our experiments show that fine-tuning a model on these concise solutions significantly degrades its performance standard benchmarks (see Section 4.5 and Figure 5b). To address this issue, we utilize the Qwen 2.5 72B model (Yang al., 2024b) to rewrite all solutions into detailed, step- by-step explanations. This approach aligns with similar techniques used in prior work, such the Numina project (Li al., 2024), which also employed solution rewriting to improve response quality. An example of rewritten solution is provided in Figure 3, and the overall dataset curation process is illustrated in Figure 2.
Math LLMs are trained on large instructional corpora. common issue with current evaluation sets is the risk of con- tamination, where test samples may inadvertently overlap with training data. To create contamination resisstant bench- marks, we constructed our evaluation set by sorting the raw data based on the initial posting timestamp and including only the most recent entries. Our evaluation set, denoted as AoPSBeeh,, sourced from the AoPS forum, with posts strictly between January 2023 and September 2024. We utilize the same pre-processing pipeline, depicted in Figure 2, to extract QA pairs and have the raw solutions rewritten for consistency.
Step 4: Data Decontamination. After processing all the QA pairs, we apply data decontamination remove any
5

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination Evaluation
Filtering. The correctness of the solution is typically veri- fied by comparing the final answer to the human-annotated answer. Note that human auman-annotated answers may still con- tain errors, as we do not perform formal proofs or verifica- tion. When constructing an evaluation set, it is essential that each question has a concrete and definite answer, which is enclosed as ans format for ease of parsing, as illustrated in Figure 3. We start by applying a series of heuristic fil- ters to exclude proof-based questions and extract only those with explicit, boxed answers. To ensure that our test set does not contain problems included in widely used train- ing sets, we use an stricter 8-gram matching filter-stiee sttirtrtticter compared to the 10-gram filter used for training set decon- tamination. This helps eliminate any potential overlap with common training corpora (Hendrycks et al., 2021b; Cobbe al., 2021; Mitra al., 2024).
Table 2. 10-gram Overlap Statistics Across Time Periods
Time
23/01-04
23/05-08
23/09-12
24/01-04
24/05-08
Overlap (%)
13.24
11.65
12.82
9.92
6.88
Count
229
208
218
226
109
pipeline does not require human annotators, are able to continuously update our LiveAoPSBench in automated manner. This makes our benchmark an up-to-date and times- tamped evaluation set that is resistant to contamination, thereby providing a more reliable mathematical evaluation resource for the research community.
4. Experiments
4.1. Dataset Statistics
Cross-Chede by LLMs. A key challenge in building a fair evaluation set is ensuring the accuracy and validity of QA pairs. To automate this process, we employed two differ- ent models Llama3.117000 1--t:....17--Im- (Dubey et al., 2024) and Qwen2.5-72B-In (Yang et al., 2024a) to perform the rewrit- ing step twice for each question. Consequently, for each question Q. we obtain a triplet: (Aqwen, Allmaa Aorigina). ). If boxed answer is detected in Aoriginal. it is automatically accepted as a candidate answer for the question. Following this, we performed a cross-check between Aqwen and Allama removing all cases with inconsistent answers. This was done through string matching for text and value matching for num- bers, while a SymPy-based (Meurer et al., 2017) symbolic equivalence program was used for SymPy-parsable expres- sions. The final answers are obtained by deduplicating the candidate answers. Through this process, we constructed LivcAoPSBench, which contains 3863 examples, all of which are from posts in the year 2024. Further details can be found in Appendix A.
We provide a better overview of the AoPS dataset in Figure 4. As shown in Figure 4a, more than 60% of the questions have only one answer, while around 24% and 8% have two and three answers, respectively. Figure 4b shows the number of posts across each year, with cut-off of August 2024. We observe that each year at least 15K mathematical questions are posted to the forum. This translates to more than 1, 000 monthly questions, which shows the potential of the AoPS forum to be used as training, and especially evaluation set. Figure 4c shows a breakdown of the types of questions in our dataset. Proof questions and numerical questions with about 32% and 28% constitute the majority of the questions in our dataset.
Finally, Figure 4d shows the pairwise overlap between each pair of popular supervised fine-tuning datasets us- ing substring matching between the two datasets of each pair. Among the two Olympiad-level datasets (i.e., ours and Numina), our dataset has the least overlap with common datasets (with less than 14.1% overlap), which shows the number of new data points.
Quality Verification. We assess the quality of our dataset by having group of 10 graduate students annotate randomly selected 10% subset (386 cases) from our evaluation set. Fach human annotator verifies whether the final answer is correct based on the raw post, with each question annotated by two different individuals. We report the percentage of cases marked as correct by the human annotators to measure the correlation between human judgment and our method. Additionally, since Olympiad-level questions can be chal- lenging even for humans, also report the agreement to evaluate consistency between different groups of human annotators. More details can be found in Section 4.5.
4.2. Evaluation Set with Timestamps Mitigate Contamination
We analyze the relationship between timestamps and con- tamination using our evaluation set, tested against the Num- nia Training Set (released July 2024). As shown in Table 2, we report contamination levels based on 10-gram overlap rates across different timestamps. We observe a consistent decrease in contamination rate as the test data timestamps become more recent.
Evolving Evaluation with Up-to-date Data. Since our
4.3. Evaluating Open-Sourced Models
'The version of LiveAoPSBench we present in this paper corre- sponds to AveeAoSSBech-082 which contains problems posted between 01-2024 to 08-2024.
We evaluate the models' performance as a function of time window. As shown in Fig 1, we find that all the models experience a performance drop when evaluating 2024 ques-
6

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and CnttaminnationRessstaaa Evaluation
epochs). We explore three data mixtures for fine-tuning: (1) AoPS alone, (2) Numina alone, and (3) AoPS + Nu- mina. For each dataset, we use the full decontaminated dataset for fine-tuning (i.e., Numina with 824K QA pairs, and AoPS-Instruct with 647K QA pairs). After fine-tuning each model, we evaluate the performance of each model on the following standard competition-level benchmarks: (1) OlympiadBench (He et al., 2024). which is an Olympiad- level evaluation dataset. Following prior literature (Yang et al., 2024a), we take only the math questions which have final answers and do not contain images or figures. This leaves us with 675 samples from this dataset (2) Omni- MATH (Gao et al., 2024), which is a collection of 4428 problems from various mathematical olympiad competi- tions. (3) LivcAoPSBench set for the year 2024. The results are shown in Table 3. As shown by the table, fine-tuning with dataset consistently boost the performance.
Table 3. Performance comparison of different models fine-tuned on various datasets across multiple benchmarks. Bold values in the columns for No SFT, Numina, and AoPS- Ins represent the highest scores for individual datasets. Additionally, bold values for AoPS Ins indicate performance that matches or surpasses all other fine-tuning alternatives.
Olympiad Bench
Omni Math
Model
SFT Dataset
AoPS24
Math
No SFT
11.7
47.1
14.5
12.3
Doopsock-Math
Numina
16.3
55.5
22.7
17.0
Th-Ins
AaPS-Int Ins
19.0
58.8
24.3
17.8
Numina+ Numina-AoPS-las
19.7
58.8
25.6
18.0
SFT
15.40
56,30
21.20
15,90
Mathstral
Numina
16.60
54,60
23.40
17.10
7B
AoPS-Ins
23.60
60.80
27.10
19.90
Numinas Ins
24.90
59.60
29.60
21.10
No SFT
12.0
47.4
16.1
12.9
1.ama-3.2
Numina
12.9
49.5
19.3
14.4
3B-Ins
AoPS-Ins
16.7
54.6
19.6
16.4
Numina AoPS-Ins
17.4
55.6
22.8
17.2
4.5. Ablation Studies
SFT
5.30
28.80
4.70
7.00
Llama-3.2
Numina
8.00
32,70
6.40
9.70
In-Ins
AaPS Ins
10.00
34.70
11.10
11.00
Evaluation Quality Assessment. We assess the quality of our evaluation set in two ways: by measuring its corre- lation with well-established dataset and through manual evaluation over a subset of the data. First, He et al. (2024) compiled an Olympiad-level math evaluation set using man- ual assessment, which we leverage in our context to verify the quality of our method through the correlation between accuracies. Figure 5a, demonstrates that the evaluation on LiveAoPSBench is highly correlated with carefully estab- lished benchmarks such as OlympiadBench. This demon- strates that our automatically generated benchmark aligns closely with the quality of those created through extensive human effort. Next, we subsample 10% of our evaluation set and ask human annotators to verify the correctness of the final parsed answers by referring to the original post, Annotators are given three options: yes, no, and no-answer. "Yes" and "no" indicate whether the answer is deemed cor- rect, while "no-answer" is selected when concrete answer is not appropriate (e.g., abstract concept questions answered with conerete examples). As a result, we found that 92% of the annotations were marked as correct, while 5% were incorrect and 3% fell under the no-answer category.
Chmina-AoPS-Iin
11.20
36.60
12.00
11.70
tions compared to questions in 2023. This decline suggests that performance on earlier examples may not accurately reflect the true capabilities of LLMs, as the initial results could be inflated by inadvertent data overlap.
4.4. Instruction Fine-Tuning
We show that the collected training dataset is effective at improving the math reasoning capabilities of L.Mss. To this end, we choose 4 representative LLMs and fine-tune them on our dataset combined with the Numina (Li et al., 2024) dataset, and show that such a combination provides superior performance compared to training on either of the datasets alone.
We use the following set of diverse models for fine-tuning evaluation: (1) Mathstral-7B (Mistral, 2024): a math- specialized model derived from Mistral-7B (Jiang et al., 2023), (2) DeepSeekMath-7B (Shao et al., 2024): math- specialized model based on the DeepSeek family, and (3) Llama 3.2 3B (Dubey et al., 2024) and (4) Llama 3.2 IB (Dubey et al., 2024), two recent general state-of-the-art models. For each QA pair, only the question is used as the in- struction, with the rewritten solution serving as the response, formatted within the model's respective chat template. For instance, with Mathstral, we use the prompt: 
 
 question [/ INST] soluti i on for instruction tuning.
Rewritting's effect performance. We also ablate the effect of solution rewriting, which is an important part of our pipeline. As shown in Figure 5b, rewriting solutions into a step-by-step format substantially improves the test accuracy across all benchmarks. The Qwen-2.5 72B based rewriting performs favorably against Llama-3.1 70b based rewriting on competition
math benchmarks, while be- ing slightly worse easier grade-school math. Overall, we found Qwen to be a stronger model, providing a higher amount of details and being less verbose compared to Llama in rewriting solutions (see Figure 18 in the Appendix for a qualitative example). This suggests that rewriting solutions
Consistent with prior work, we train each model for three epochs (Shao et al., 2024; Yang et al., 2024b), as we observe additional epochs provide no further benefit (see Figure 10 in the Appendix for ablation studies on the number of
7

---

Leveraging Online Olympiad-Lonm Level Math Problems for LLMs Training and Iettamination-Resistaan Evaluation
1,000
0,003
0.270
0:200
6/727
Exuation
Ocier
Use
21000
12.76
Exprxntes
0R34 19
6.773
0,003
0,002
0,263
0.220
L.30
6.877
0.500
0.021
13000
0003
0.392
0.824
13000
3000
1800
(a) Number of answers per (b) Number of questions per year, based (c) Problem category distri-(d) Pairwise overlap between various on post date.
question.
bution.
datasets.
Figure 4, AoPS Dataset Statistics. The statistics are across all the datapoints in our dataset before split. In (d), the percentage at row and column shows the fraction of the training set of i-th dataset (hased exact substring match) present in the j-th dataset. Our dataset has the least overlap with others with less than 1 14.1% 1% overlap.
with stronger models can significantly improve performance on benchmarks.
5. Limitations
Absence of Visual Content. Our dataset currently focuses text-only problems, which may limit its effectiveness in certain areas, particularly geometry. Many geometry prob- lems rely heavily on diagrams to fully convey the problem statement. Incorporating relevant images and figures could significantly enhance the dataset's comprehensiveness and applicability, especially in visually-dependent mathematical domains.
Owenk Math lns
Spearman Correlation; 0,9890
45
Accuracy Olympaa Bench
40
Owerd 5-Miath7B-iin
Evaluation of Proof-based Questions. Our evaluation dataset focuses on QA pairs with clear, final answers, which is well-suited to a broad range of Olympiad-level problems. However, a significant portion of such types of problems involve more complex proof- based questions that require detailed logical reasoning and multiple steps. While we incorporate proof-based questions in our instruction-tuning pipeline, the current evaluation pipeline lacks the ability to evaluate such questions effectively.
Owen Nath 78-ire
Owen? SaMats) 1.44hx
35
Owerd 1atts--attt-11 1a
30
Deep Crddenenn
Intomiim22 Mathiprup
Gemmag 2754
Wathetre- 78.d
Numinaatt J..
DespSook M
Cammaz soiit
Accuracy LivetoPSBench (%6
(a) Correlation with the OlympiadBench dataset.
Quality Variability in Community- Community-Generated Content. The Community-driven content from the AoPS forum pro- vides a rich source of high-quality data. Nevertheless, as with any community gennrttte content, the quality of an- swers and solutions can vary. While our filtering and refine- ment processes have successfully mitigated much of this noise, incorporating more advanced techniques in future iterations could result in better consistency and precision.
80
74.5
78.2
Rownling
72
Liema-3 1-708
Ownnn-2 2 2-52B
02
57.5
Aocuracy (9g)
51.2
38.1
20
20.0
10.4
22.5
12
10.0
10.5
6. Conclusion and Future Work
16th
GSMBX
AMG2
Olarppod Banth
Datasets
In conclusion, this paper introduces the AnPS-Instruct dataset and LiveAoPSBench, leveraging community-diie driven content from the Art of Problem-Solving forum to address the challenges of limited training data and unreliable evalu- ation for LLMs solving Olympiad-level math problems. By developing a scalable and automated pipeline for extract- ing and refining question-answe pairs, this work presents dataset containing over 600, 000 QA pairs, along with an
(b) Ablation on Rewriting.
Figure 5. Ablations on LiveAoPSBench. (a) The performance of models on our benchmark is highly correlated with established datasets. (b) The effect of rewriting user solutions into step-by- step solution with two different models, on an carly version of our datasct. Rewriting solutions always improves accuracy, and using stronger models Icads to larger accuracy gains.
8

---

DeepSeck-AL. Deepscek-rl: Incentivizing reasoning ca- pability in Ilms via reinforcement learning, 2025. URL https: http///xxx.....abaaaa/ Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resistall Evaluation
up-to-date, contaminatiio--rsiitaant evaluation benchmark. Our experiments demonstrate significant performance im- provements across multiple standard benchmarks for mod- els tine-tuned on the AoPS- Instrucc, highlighting enhanced mathematical reasoning capabilities. Furthermore, the ob- served performance decline of various LLMs on LiveAoPS- Bench underscores the importance of continuously updating evaluation sets mitigate the risks of data contamination.
the Digital Research Alliance of Canada alliiance. ca, and companies sponsoring the Vector Institute www. vectorinstitute a #partners, and Advanced Research Computing at the University of British Columbia. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant. Sadegh Mahdavi and Muchen Li are supported by UBC Four Year Doctoral Fellowships.
For future work, there several promising directions to explore. First, while this paper focuses the AoPS forum, the pipeline developed is not limited to this domain. It is generalizable and can be applied to other online forums or different subject areas, enabling the creation of high-quality datasets for various fields, such as physics, computer sci- ence, or even non-technical disciplines. Expanding this pipeline to other knowledge-intent intensive communities could further improve the training and evaluation of LLM across disciplines. Additionally, the quality of the dataset can be significantly improved by incorporating more advanced LLMs into the pipeline. Leveraging state-of-the-ar mod- els for question extraction, answer detection, and solution rewriting would result in more accurate and detailed data, ultimately enhancing the effectiveness of the fine-tuned mod- els. Morcover, the problems is JoPS-Instrude could be used as a diverse set to distill larger models into smaller mod- els. Lastly, the current pipeline focuses on question-answer pairs with clear final answers, but a significant portion of Olympiad-level problems involves proof-based questions that require a deeper evaluation of logical reasoning, argu- ment structure, and intermediate steps. Future work could include adapting the pipeline to accommodate these proof- based problems, potentially using another advanced LLM as judge (Li et al., 2023), or incorporating formalization methods to better assess these complex solutions.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
AIMO. The aimo prize. sttpa://aimoprize.org November 2023.
AOPS. 2023 ame 12a. and 12b problems.
vin...niiiin com/
wikt//indxx.php/2222 AMC_ L______MMa_______1__11 cwmmwoooow...oo.. /iki/
http:://artooprrobeemsssl
index. andex.ppp/2033 Barp:02233_______0 AMC AMC_12B Problems, 2023.
AOPS.
2024 aime community page. https:
/ flartofprobleemolliii com/ htt////oomm/t/// 23370201_2022_ainne 2024.
Azerbayev. Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview .nt./ forum? wwwuuuiddd-Wggiiin 4Wgggn99111.5.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Impact Statement
This paper aims to advance the field of mathematical rea- soning. It is important to note that AoPS-Instruce and LiveAoPSBench contain no personal information. Addi- tionally, we have not identified any societal consequences that require special attention in this context.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint 0rXiv22100.14666 2021.
Acknowledgments
This work funded, in part, by the NSERC DG Grant (No. (GPIN-2022-046666) the Vector Institute for AI, Canada CIFAR AI Chair, NSERC Canada Research Chair (CRC), NSERC Discovery Grants, a Google Gift Fund, and the Govermentt of Canada's New Frontiers in Re- Resources used in
12948.
search Fund NFRFE-2023-00995 preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through
Ding, Y., Shi, X., Liang, X., Li, J., Zhu, Q., and Zhang, M. Unleashing reasoning capability of lims via seal- able question synthesis from scratch. arXiv preprint arXiv:2410. 18693, 2024.

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resittan Evaluation
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 qu q, 2024.
Information Processing Systems, 2022. URL https:
/ //operreviee.. fttt://oorrmmmm-tPeTEEEEEE
J., Beeching, E.,, Tunstall, I.., Lipkin, B., Soletskyi, R., Costa Huang. S., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L.,, Fleureau, Y., Lample, G., and Polu, S. Numinamath. https:// com/ projectnuuml a att.apprrrrrrr--sriieeeee blob/ / /w/i//aana/////aaaa///aa http///ain/mrr/////e/// http://maaaaaatt..aaaaaa pdf, 2024.
Gao, B., Song. F., Yang, Z., Cai, Z,, MiaO, Y., Dong, Q., Li, L.,, Ma, C., Chen, L., Xu, R., al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J.. Hu, J., Han, X., Huang, Y.. Zhang, Y., Liu, J., Qi, L., Liu. Z., and Sun, M. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Ku, I..W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics VVolume I: Long Pupers), pp. 3828-3850, Bangkok, Thailand, August 2024. Association for Computational Linguis- URL https:
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I. Guestrin, C., Liang, P., and Hashimoto, B. Alpacae- val: An automatic evaluator of instruction-folloinng models. https: / www.ttt.......a.aeeeeeetete a al paaca_evalll 2023.
Li, Y., Choi, D., Chung. J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling. J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alpha- code. Science, 37866624):1092-11009
tics. doi: 10. 10.186553w11222224dloooogg acl- loongnn 211.
2022.
Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, P., Bonazzi. F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel, A. R., Roučka, v., Saboo, A., Fernando, I., Kulal, S., Cimrman, R., and Scopatz, A. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 10.7717/peerj- 103. URL https: / //doi. orgg.oor//n 10. 7717 htt.:///...////rrrrrrrrr... peeri-cs.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Hendrycks, D.,, Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021b.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.0799 07974, 2024.
Miao, S.-Y., Liang, C.-C., and Su, K.-Y. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-9844 2020.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., et al. Mistral 7b. arXiv preprint 1rXiv:2310.068255 2023.
Mistral. Mathstral blog. https://mistral. news news/mat htt.///tattaaaaaaammma 2024.
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A.
Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402. 240.144000.. 2024.
R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. Mawps: A math word problem reposi- tory. In Proceedings of the 2016 conference of the north american chapter of the association for computational lin- guistics: human language technologies, pp. 1152-1157, 2016.
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathemat- ical web text. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //ppnrevveww.coo net contttfooummmt forum?ircii (KimjKHmmiiiaaa
Lewkowycz, A., Andreassen, A. J., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman- Solo, T., Wu, Y., Neyshabur, B., Gur- Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural
Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word prob- lems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp-
10

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Ressttann Evaluation
2080-2094, Online, June 2021. Association for Compu- tational Linguisties. doi: 10. 10.186653/11120211maacc 168. URL https:/ naacl meacl-main... 188.
Ying, H., Zhang, S., Li, L., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., al. Internim- math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06333. 2024.
Selsam, D., de Moura, L.,, Buzzard, K., Barton, R., Liang, P., Loos, S., and Wiedijk, F. Imo grand challenge. https: //imo //imo-grand grand- cha Saaallenger www.ggithhh...... io/, 2019.
Yu, L., Jiang, W. Shi, H., YU, Liu, Z., Zhang, Y., Kwok, J., Li, Z.,, Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openreview. net NENNiigNN.............
Shah, V. Yu, D., Lyu, K., Park, S., Yu, J., He, Y., Ke, N. R., Mozer, M., Bengio, Y., Arora, S., et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009, 2024.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mam- moth2: Scaling instructions from the web. arXiv preprint Sili:2405.035444 2024.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.03300 2024.
Zellers, R., Holtzman, A. Bisk, Y., Farhadi, A.. and Choi, Y. Hellaswag: Can machine re- ally finish your sentence?
In Annual Meet- ing of the Association for Computational Linguistics, 2019. URL https: /app.ap....//// semanticscholas org/ /ttp///popii:10000A...
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint 1rXiv:2408.001/6 2024.
Zhang, B., Zhou, K., Wei, X., Zhao, W. X., Sha, Wang, S., and Wen, J.-R. Evaluating and improving tool-augmented computation-intenas math reasoning. arXiv preprint SKIV:2336.024000 2023a.
Team, Q. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https: //wwenlm. github. wtt..oo//bllooo qwq 32b- bbpp---3b-prreeew//
Zhang, X., Li, C.,, Zong, Y., Ying, Z., He, L., and Qiu, X. Evaluating the performance of large language models on gaokao benchmark. 2023b.
Tong, Y. Zhang, X., Wang. R., Wu, R., and He. J. Dart- math: Difficulty-wware rejection tuning for mathematical problem-solving. Advances in Neural Information Pro- cexsing Systems, 37:7821-7846, 2024.
Zhu, Q., Guo, D., Shao, Z.,, Yang, D.,, Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 1931, 2024.
Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman, Openmathinstruct--t A 1.8 mil- lion math instruction tuning dataset. arXiv preprint arXiv:2402. 10176, 2024.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.1558 15877, 2024.
Trinh, T., Wu, Y., Le, Q. He, H., and Luong. T. Solv- ing olympiad geometry without human demonstrations. Nature, 2024. doi:
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li C., Li, C., Liu, D., Huang, F, et al. Qwen2 technical report. arXiv preprint arXiv: 10671, 2024a.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409. /2/22, 2024b.
Yang. S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint 2023.

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resittae Evaluation
A. More details on LiveAoPSBench
A.1. Evaluation pipeline statistics
To begin with, we have 14158 QA pairs with time stamps between Jan-2024 and Aug-2024. Decontamination with 8-gram matching is performed against Math and GSM&K training set (Hendrycks et al., 2021b; Cobbe al., 2021), which removes 664 Q-A pairs. After removing proof questions and non-boxed solutions, we are left with 7173 Q-A pairs over 5416 unique questions. Lastly, The LLM cross-check filters 1553 questions with inconsistent solutions and the resulting LiveAoPSBench contains 3863 questions. We apply the same pipeline described in Sec 3.2 to data with time stamp between Jan-2023 and Dec-2023 and get 5216 questions for the 2023 split result.
A.2. Human Annotation
As shown in Figure 6, we develop simple web interface for human annotators to verify the answers extracted by LLMs. Annotators compare the "Voted Answer", "Original Answers" and all posts in the original topic page identified by LLMs to verify if the "Voted Answer" matches the original posts' answers. The verification process provides four results: Positive ("Yes"), negative ("NoNNo Answer"), and neutral ("Not sure"). The "Not sure" option provided since verifying the answer sometimes requires certain mathematical foundation and significant amount of reading time. We also show highlight two examples of disagreement in Figure?
A.3. Derivation of Difficulty Levels
The difficulty levels in this dataset do not reflect the exact difficulty of the problems but rather approximate the general education background of the problem, e.g., this is "High School" level problem. However, a challenging high school problem may be more complex than an easy college-level problem. The classification is derived from the problem tag in the AOPS forum, where the categories correspond to "Middle School", "High School", "College", and "High School Olympiads". In addition, some problems originate from special forums, which do not fit into the above categories and are classified as "Others" in our dataset.
ATL Probirr Sciver
Crlli
Enust Acadery
AGPS Kaadery
Currentindax: 1 Toplc ID: 3331886 Quastion:
Online
SON.
PSCETTA
ONLINE
BOOKST
COMMUNITY
The cube labelted ss of the labels of the four vertices on face. Whet in the sRow numbrr of equal
Whymeth competionnl Jesunce Dctbbe
any labeling?
C SOAME MATH our MFMOO TM
Middle School Math
Post
Original Answer:
Pian by 1eg
Dube
answer is neaning that
curi sointe
This is exorpte
60 D0bbe High loys
sed suetv help on thi pribion
the cam the labekc of the four wett tie face. Whlt
Botton (2, 7, 4,
Left 18, 2,
11
6 Letter
monlle fpooed that all the wenn
lntuggled bi srying to other
anyol
mppie this diftre numbers excamppl
docy feoe addtt
Front (L. 4,
(3, 2,
ve staetyy
Voted Answerr:
myedee
Dottrm
Tep
Hight (6.
Yio
No
Book: (3 5.2.0)
Presious Toplc
Next Toplo
Open Second Window
Input
Jump to Topic
2
(mo cuthenpteeeen cereing faro seme mot thats now par
Prention
AuPSCriiee
Figure 6. Human Annotation Interface.
12

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resittee Evaluation
Example
Question: In a triangle, ABC, Angle BAC = 90"; AD the altitude from onto BC. Draw DE perpendicular to AC' and DF perpendicular to AB. Suppose AB 15 and BC 25. Then the length of EFF is?
Raw Post:
Because the triangle is right triangle, by the Pythagorean Theorem, the length of AC 252 = 20. The of ABC AC/ 15 * 20/ But can also be represented by 25 * AD 2. Putting them together we get 15 20 2 25 * 2.
So AD= 15 20 25 = 12. Because DE perpendicular to AC and DF perpendicular to AB, AEDF is a rectangle, which means
that =F= AD, EF 12. The answer is A.
Voted Answer: 12
Is the Voted answer consistent with answer raw post? Human Annotator 1:
Human Annotator 2: X
Example
Question: For positive integer k we write (L+=) = aotarrtazz"= t2..-t.......................... +.........aa fare" where at are the cnefficients of the polynomial. Find the sum of all the digits of smallest possible value of if ag + + ag .......k 1) divisihle by 2005.
Raw Post:
f(z) = (1 + (1 + 2x) ... (1 kar) = ao + ta +.... axæ
do + +... 4--1k-111 f(1) ak
on - 11-1-2-3... k k!
f(1) 203-.4..
-k 1)!
2005 (k+
2005 k!
2005 401
2 401
Voted Answer: 5
Is the Voted answer consistent with answer in raw post? Human Annotator 1: X
Human Annotator 2: /
Figure 7. We highlight two examples of annotation inconsistencies caused by human annotators: 1. Example 1: Annotator 2 failed to recognize that the answer explicitly stated in the raw post. Example The raw post does not directly provide final answer. Annotator was unable reason that 4 + 0 5 constitutes the correct solution.
B. Detailed Evaluation Results on LiveAoPSBench
B.1. LiveAoPSBeech-2022
We update the LiveAoPSBench with the complete 2024 data. This resulted in LiveAoPSBennhh2000 with 5328 questions. In figure 8 provide the latest result with the state-of-the-art reasoning models like deepseck-rl 2025) and QwQ(Team, 2024).
B.2. Evaluating open-sourced LLMSs
We have selected several mainstream open-source general LLMSs and math-specific L.Mss that demonstrate high performance on the previous math evaluation datasets. For math-specific LLMs, we choose DeepSeck-Math-7bbe (Shao et al., 2024), Mathstral-7B-vl (Mistral, 2024), 7b and 20b versions of Internm2-MMath NuminaMath- CoT (Li et al., 2024), 1.5B,7B,722 7,B version Qwen2-Mathh 1a122-Mat---nttruu Instruct ( (Yang et al., 2024a) and Qwen2.5-Man Instruct (Yang al., 2024b) as the representative of the math specific LLMss. Additionally, we include DeepSeek- SeepSek--Code--V C2
(Ying et al., 2024), 7B and 72B versions of
13

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resittae Evaluation
LiveAoPSBench-2024
Methods
70
DeepSeck R1-Distill Owen
DeepSeek- R1-Distil Dbstil-Qwen-70
Preview
DeepSeek Dlstill-Uama--e 00
Owen2 5 Mat-Matth-77---en
DeepSock-R1-DHie 1ir.ck-R--Ditlll-wee...33.
Phi-4
Qwen2.5
Owen2 5 Math-atth-5 3-8-ttttact
Pass81 Accuracy
DeepSeek-mmath mattaekk--tt--B----
Liama 1.nm--.1--80 3.1-58-Instruct
40
DeepSeek math instruct
Liame 22-30-instruut
30
20
10
2024-01
2024-02
2024-03
2024-04
2624-05
2024-06
2624-07
2024-08
2024-09
2024-10
2624-11
2624-12
Month
Figure 8. Accuracy trend time for LiveAoPSBench-2024 Livemathbench, github. ho/leaderboard. html.
For the most up-to-date results, please visit: https://
Lite-Instruct (Zhu et al., 2024), which is a code specialist model trained on both math and code corpus. For general purpose LLMss, We report performance on 1B, 3B and 8B versions of the Llama3 family models (Dubey et al., 2024) as well as 9B and 27B versions of Gemma-2-Instrrue (Team et al., 2024) model.
B.3. Detailed Results
The accuracy comparison for these mainstream open source LLMSs are shown in Tables 4, 5, 6 split by Month, Difficulty and Answer Type. The Month tables separately include evaluation results for 2023 and 2024. For the Difficulty and Answer Type tables, we use only the most recent evaluation results from 2024. Notably, the difficulty labels represent the general educational background of the problems rather than their exact difficulty. Over half the problems originate from educational backgrounds associated with High School High School Olympiads, and only around 7% are from Middle School, indicating our dataset's focus is more on the complex problems, Similarly, in the Answer Type Table, more than half of the problems categorized as numeric-int.
C. Training set details
C.1. Decontamination Details
We use 10-gram substring matching to decontaminate against test set for a comprehensive list of math evaluation datasets available. (Cobbe al., 2021; Hendrycks et al., 2021b; He al., 2024; AOPS, 2023; 2024; Zhang et al., 2023b; Lewkowyez al., 2022; Gao al., 2024; Miao al., 2020: Hendrycks et al., 2021a; Koncel-Kedziorski al., 2016; Patel al., 2021; Zhang al., 2023a). In Figure 9. We show the decontamination statistic for our dataset and Numina.
D. SFT Experiements
D.I. Ablation with controlled computation budget
As shown in Tab 3, Numina AoPS-Instruct performs favorably against using AoPS-Instruct or Numina alone. To show this gain is not simply achieved by doubling the computation available for fine-tuning. We control the total fine-tune budget the same for AoPS-Inntren only, numina only and AoPS-Instrues numina. This results in approximately
14

---

Leveraging Online Olympiad- Level Math Problems for LLMs Training and Cesiaminnatio-Ressittaa Evaluation
Model
2023
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Count
5216
483
388
444
415
472
412
396
505
381
409
404
507
DeepSeek-Codee V2-Lit---
22.45
23.40
19.33
22.97
22.17
22.67
21.12
23.99
25.74
22.83
23.72
22.28
18.93
DeepSeek-Math,
18.22
15.46
17.57
13.49
14.41
12.86
15.66
14.65
15.75
15.89
17.08
13.61
Deemmmm2-Mathpplus-220
18.23
18.84
16.75
17.79
18.31
20.55
17.72
16.41
18.42
17.06
21.27
18.81
16.57
Milemlm2-Math-plane
17.10
17.81
16.49
19.82
17.11
18.01
15.05
19.95
16.24
13.91
18.58
17.08
15.19
Mathstral-B--v0.1
15.91
17.18
14.95
15.54
14.22
16.31
16.75
16.67
18.22
13.91
14.18
18.32
14.20
NuminaMath-72B-CC
T
26.15
29.40
25.77
23.87
25.30
29.87
22.57
25.51
25.54
24.41
27.87
25.50
27.02
NuminaMath-7B-CoT
17.29
18.43
14.95
18.24
15,90
19.70
15.53
17.68
17.43
15.22
18.83
19,80
15.38
Dwen2-Math-1.525
29.06
31.47
26.55
31.31
31.08
28.81
26.21
26.77
29.90
27.82
29.58
32.43
26.43
Qwen2-Math-72B..
37.96
41.41
36.34
37.39
38.31
38.35
37.14
37.37
38.61
34,91
40.10
39.60
35.50
Math-7B-it Qwen2-Math-77Bit
33.07
33.13
35.31
33.33
31.33
36.86
28.64
35.10
33.27
30.45
32.52
36.39
30.57
Qwen2.5 1aann.55Math-.5BB---
34.72
36.02
32.47
35.81
34,94
37.50
32.04
31.31
36.04
33.86
36.67
35.15
33.73
Qwen2. 5 M-athh--2B-ii
42.04
44.31
38,
41.22
41.93
45.55
44.17
40.15
42.18
35.43
46.21
43.32
40.43
Owen2.5-Math-7B.
34.87
35.82
31.70
36.26
35.42
37.71
29.61
33.59
37.43
32.28
34.72
37.87
34,52
Llama-3 Llama-3.2-IB-it
6.75
6.83
5.93
4.73
5,54
9.53
5.10
8,08
6.73
8.66
7.09
7,92
5.13
1lama-3.2-3B.
13.77
14.49
13,40
12.16
13.98
14.19
13.11
16.92
12.08
14.44
15.16
13.61
12.43
Llama-3.1 3.1-8B-it
14.03
15.53
13.66
14.86
13.73
13.56
11.65
15.15
16.04
12.86
14.91
14.60
11.64
2-2ma-2-22h-ii
13.80
11.39
13.92
15.77
13.01
15.04
10.68
14.90
15.64
12.86
14,91
14,36
13.02
Gemma-2-966
12.42
10.97
10.05
13.51
12.53
13.35
11.17
13.38
15.05
10.76
13.45
13.61
10.85
Model
2024
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Count
3863
527
614
503
511
380
363
331
DeepSeek Coder- Coder-V2-Liees V2-Lite-it
20.86
18.14
22.77
25.41
22.47
18.00
17.89
21.21
19.64
SeepSeck-Math-7be.
13.64
14.04
17.08
15.31
15.71
10.37
10.53
13.50
9.97
Internim2- Math-plus-20b
16.93
14.51
22.01
17.59
17.50
14.48
14.74
18.73
15.71
Miermlm2-Muth-pluss
14.81
11.83
18.60
17.59
16.50
11.15
17.11
13.77
10.88
Mathstral-7B-v0.
14.29
11.04
17.65
17.43
16.50
10.96
12.11
12.95
15.11
NuminaMath- M2thnaa-att-72B-o 72B-CoT
24.95
22.08
28.65
28.66
23.46
23.48
22.37
27.55
22.36
NuminaMath-71B-Cot
16.13
13.56
20.87
17.10
19.68
13.89
13.42
16.25
12.69
Qwen2 M.5t--
26.84
25.24
29.79
30.46
27.24
22.90
25.79
28.37
23.56
Swen2-Math-72B-44
36.68
34.38
37.57
41.69
36.98
36.59
35.79
36.91
30.82
30.05
28.08
31.69
34.20
31.41
27.98
29.74
30.85
24.17
Owen2.5-Math-1155
32.57
28.71
33.59
39,25
31.21
31.70
30.00
34,99
29.61
40.56
40.54
41.56
45,11
40.16
37.96
40.53
42.42
33.23
Swwen.55-Math-P7
34.04
32.18
35.48
41.37
32.41
32.09
32.11
34,99
28.40
Llama-3.2- IB-it
5.80
3.63
7,40
6.84
6.16
4.70
5.53
7.16
5.44
Llama-3.2-3B-14
11.75
8.68
13.85
12.87
14.12
10.18
13.42
9.92
11.18
-lama-3.1-8B-ii
12.71
10.57
16.13
14.17
11.33
12.33
12.63
12.67
11.48
Gemma-2-2765
12.87
10.57
16.13
13.03
13.52
11.94
11.58
14.05
12.39
Gemma 2-
10.67
8.68
11.95
13.36
10.93
8.02
8.16
14.05
10.27
Table Accuracy per Month for Different Models
15

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resistaaa Evaluation
Table Accuracy per Difficulty for Different Models: The difficulty labels are for general education background of the problem and do not reflect the exact difficulty of the problem.
Model
Overall
Middle School
High School
College
High School Olympiads
Others
Count
3863
286
1349
314
889
1025
DeepSea--Coder----Licce
20.86
24.48
19.79
22.93
17.21
23.80
DeepSeek-Matth-7b-1
13.64
22.73
12.23
14.97
8.55
16.98
Intemlm2-MMathpluus-2200
16.93
24.83
15.64
19.11
12.71
19.41
Intemim2-Matthpluss777
14.81
20.63
13.94
17.20
9.67
18.05
Mathstral-7BB-vo.1
14.29
19.23
12.90
15.61
10.24
17.85
NuminaMath-722B-C
24.95
33.57
25.28
26.75
18.45
27.22
NuminaMatth-7B-CoT
16.13
18.18
15.12
16.24
12.60
19.90
Qwen2-Math-1.5B-i
26.84
32.17
26.17
28.98
22.61
29.27
Qwen2-Math-72B.
36.68
43.36
38.62
42.04
28.01
38.15
Qwen2-Math-77Bii
30.05
37.06
30.17
31.21
24.86
32.10
Owen2.5-Math-1.5BBBt
32.57
38.81
33.65
31.85
28.91
32.78
Qwen2.5-Math-72B-it
40.56
48.25
42.70
45.86
32.62
40.88
-wen2.5-Math-7Bit
34.04
42.66
34.84
36.62
27.11
35.80
Llama-3.2-1Biit
5,80
10.14
4.30
5.41
3.71
8.49
Llama-3.2-3B-4
11.75
18.53
10.23
9.55
8.10
15.71
Llama-3.1-8B-i 12.71
17.48
10.60
15.29
7.54
17.85
Gemma-2-27b-it
12.87
20.63
11.86
13.38
7.65
16.39
Gemma-2-9b-it
10.67
15.38
8.30
12.74
7.65
14.44
Table Accuracy per Answer Type for Different Models: As not all answers be casily verified, divide the answers into different types to facilitate more accurate comparison and more convenient observation of the structural distribution of the dataset.
Model
Overall
equation
expression
list
numeric-dec numeric-ing int
numeric-in irr
others
Count
3863
296
950
195
57
2114
176
75
DeepSeck-Codee Coree-Co-ore----------
20.86
18.24
16.00
20.00
15.79
24.36
11.93
21.33
MathScck-Math77b-ll
13.64
11.15
10.42
10.77
8.77
16.18
6.25
21.33
Interlln2-Mathpplus-220
16.93
14.53
12.74
13.33
14.04
20.20
8.52
18.67
Intemlmm-MMuh-plus-777
14.81
11.49
9.26
12.31
12.28
18.31
8.52
22.67
Mathstral-7B--o.)
14.29
14.53
10.74
6.67
12.28
16.65
10.80
21.33
NuminaMath-77B-CoTT
24.95
19.59
20.21
16.92
24.56
28.71
20.45
32.00
NuminaMath-7B-CoT
16.13
14.86
12.11
11.28
12.28
18.78
12.50
21.33
Qwen2-Matth-.5BB-t
26.84
23.31
22.11
28.72
15.79
29.52
25.57
32.00
Owen2-Math-72B-it
36.68
27.70
30.84
36.41
26.32
41.15
31.82
40.00
30.05
23.99
26.21
27.69
19.30
33.30
25,00
37.33
Owen2.5-Math- 1.5B-it
32.57
25.34
28.74
32.82
22.81
35.86
22.73
46.67
Qwen2.5-Math-
40.56
31.42
38.32
41.54
31.58
43.19
36.36
45.33
Qwwen2.5-Math-7IBiit
34.04
28.72
30.21
33.85
24.56
36.90
30.11
40.00
Llama-3.2-1B-1
5.80
2.70
3.79
2.56
5,26
7.66
3,41
5,33
Llama-3.2-3B-it
11.75
7.77
8.32
7.18
7.02
14.71
6.25
16.00
Llama-3.1-8B-it
12.71
4.05
9.58
10.77
17.54
15.33
9.66
21.33
Gemma-2-27bb.
12.87
7.77
10.11
9.74
14.04
15.28
9.09
16.00
Homma-2-9b-44
10.67
7.09
7.89
8.72
14.04
12.54
7.95
16.00
16

---

Leveraging Online Olympiad-Lewe Math Problems for LLMs Training and Contamination-Resittae Evaluation
Ours Raw (Total: 43992)
15487
14669
NuminaMath-CoT (Total: 40916)
Number of Matches (Exact/10 Gram)
10000
5386
5032
4680
3676
2136
1814
1000
934
635
539
math, questions
asdiv
mmlu,
aops_2024 olymplad bench
External Test Sets
Figure 9. Decontamination Statistics: We perform decontamination on the raw dataset to produce AoPS-Instruct, with the same method as the Numina- Math COT. Both datasets show considerable overlap with the MATH dataset. AoPS-Instrun exhibits more contamination within our 2024 split due to repeated questions, while Numina-Man Math-COT has higher contamination with other external datasets, reflecting its multi-source composition.
Table Performance comparison of original DeepSeck-Me Math,,, D0wen2.5-1.5B. and DeepScek-Men Mathh fine-tumed on solutions rewritten by Qwen2.5-1.5B The fine-tuned DeepSeek-Math significantly outperforms both the original model and the rewriting model, demonstrating that our dataset enhances reasoning capabilities beyond the limitations of its rewriting model.
Model
AIME24
AMC23
Olympiad Bench
Math
AoPS24
Omni Math
Bookseck-Matt-225
1/30
8/40
14.5
47.1
11.7
12.3
Qwen2.5-11.bb-l Deepseek-Mathh-bb-le (fine-tuned)
0/30
9/40
21.3
55.0
16.7
16.8
1/30
13/40
22.7
61.0
19.4
19,2
epoch on AoPS-Instruct or Numina or epoch of training on AoPS-Instruct + Numina. We show the curve of ACC on Math, OlympiadBench w.r.. training steps.
D.2. Rewriting model ablation
use Qwen 2.5 72B to rewrite the solutions, and then we fine-tune smaller models on our dataset. This may raise question of whether the effectiveness of our dataset would be limited by the capabilities of its rewriting model. To show the effectiveness of our dataset, use Qwen 2.5 1.5B rewrite the solutions and then fine-tune DeepSeek-Math 7B-instruct on the dataset. Table shows the performance of the original DeepSeek-Matt the performance of Qwen2.5-1.5B, and the performance of fine-tuned DeepSeek on Qwen2.5-11 1 B-rewritten solutions. As shown by the results, the fine-tuned version outperforms both models, which shows that our dataset can improve the reasoning capability beyond its rewriting model solution.
E. AIME and AMC results on the main models
Results of fine-tuning on AIME 2024 and AMC 2024 are present in table
17

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Ressttae Evaluation
LycAuPSRench
Nath
Modes
56
Model Numina
Numina
54
15
Accurarty
Accuracy
Accuracy
1.5
12
12
15
5000
8020
12000
12920
14003
19200
.0035
6003
9200
10002
12500
34000
36020
18000
6036
$000
15000
12920
14003
19200
1805
Training Steps
Training Steps
Training Steps
Figure 10. Ablation study on accuracy with respect training steps. Here, 18,000 steps approximately correspond to 6 epochs for AoPS-Instruct Numina, and epochs AnPS-Instruct + Numina. can see that LiveAoPSBench Numina consistenly improve training goes on.
Insturction:
You are given an online Math QA post. Your task is to identify whether the post asked is a concrete mathematical question. note that this means it shouldn't 't be abstract general question related math, and output the result (b)xed(0) for no and \hoxed(1} for yes. few examples provided below:
Few shot examples...
Now, your task is provide output for the following post:
Post: Post
Example Classify result:
(0/1)
Figure 11. Prompt for the Topic Filtering part in Fig 2.
F. Use of AoPS as a Data Source
Concurrent to our work, both Numina (Li et al., 2024) and Omni-math (Gao et al., 2024) also use AoPS as their data source. Different from us, Numina only includes data from the contest page with 30K questions 3, while we utilize the 1.07 available posts on this forum. Furthermore, Omni-math (Gao al., 2024) includes only 4428 evaluation questions from all timestamps, while include the most recent problems posted in 2024, well large-scale training set.
G. Prompts
We provide the Prompts used in our pipeline in Figures 11, 12, and 13.
H. Dataset Examples
We provide further examples of our dataset and its rewritten solutions in Figures 14, 15, 16, 17. 18, and 19.
18

---

Leveraging Online Olympiad-Lewe Math Problems for LLMs Training and DentaminnatinnRessstaee Evaluation
Insturction:
You given online Math QA forum where each post in each topic in the format "post i by user j: [post text]". Each user may reply to other users by quoting their post. Your task is to find all potential answers within the follow-up discussion, output them a structured json format.
Your output json must have one "answers" key containing the list of answers. Each answer must have two keys: "user" key identify the who posted the solution, "post number" identify which post number the answer originates from. not add any additional information to the question answers. case the dialogou does not contain any mathematical question, there are no valid answers, leave the "answer" key empty list. Before you output your JSON answer, provide short summary of the discussion. A few examples are proided below:
Few shot examples...
Now, your task provide ISON output for the following Topic:
post I by userl:
post 2 by user2:
Example Parse result:
"question": "Question from Post 1",
"answers": :
"user": "User2", "post number": 2
"use": "User4", "post number": 5
Figure 12. Prompt for the QA extraction part in Fig 2.
Insturction:
You are given solution to mathematical question. Your task is to re-write the solution into a step-by-step solution with itemized steps(l.... 2.... 3..... should re-write the solution formal clean way, without missing any details. Make sure include all the necessary steps and explanations. Do not include any irrelevant information in the Do add any additional information the solution. Always surround mathematical questions with $ symbols for LaTeX formatting. If the question contains final number, make sure to put the number in
boxed for LaTeX formatting.
Question: Question
Solution: Raw Answer
Now provide the formalized re-written answer. DO NOT include the question in your answer. the re-written solution.
Only respond with
Example Rewrite result: Rewritten solutions
Figure Prompt for the Solution Rewriting Part in Fig
19

---

Leveraging Online Olympiad-Lewe Math Problems for LLMs Training and ContaminationnResiitae Evaluation
A! Je 27 User Response 1:
Question: Find all pairs (k, () of positive integers such that
W.L.o.g we may assume k > I. k 2 ++2 then & divides the RHS, thus the LHS, thus k", but k, are coprime, thus this is only possible A: which would imply
contradiction.
So the cases k = 1, are left.
If & then we always have equality. 6t-1
If A
then the we need [k-1jt = kt~²= = (A 1)*.
Again A: 1 is supposed to divide power of k; thus we remain with k 2, so with 1, clearly heing solution.
User Response 2: If (k.l) solution, then (I.K) is solution too. Therefore can find solutions & 2 1. If 1=1 we have k = k! = k 1 or l = 2. I=2 have A²2 2 solution is k=2.
If 2 3 then < and 2 1, therefore solution is l=k.
All solutions (k.1) (1. (2, 1), (14, n), E
Qwen Rewritten Solution 1:
1. Without loss of generality, assume & 2 1. Consider the case l
The right-hand hand side (RHS) of the equation # kt is divisible by &
Therefore, the left-hand side (LHS) also be divisible by k
Since k and k 1 coprime, & 1 must divide k².
*
This is only possible if l: 2. which would imply 0, leading to contradiction.
Therefore, the only possible cases are k = 1 and & = I + 1.
4. If k =
The equation K' k! -T
simplifies to
* #t F x
which is always true.
l:
The equation becomes
(+11 PTI
(++1)
Simplifying, we get ji41
(14171 11+1
This further simplifies to 1)1-1 =
Again, and ++1 1 are coprime, so must divide power of + 1.
The only solution is 1, which implies 2.
Therefore, the pairs (Is, () that satisfy the equation are (k,
kc) and (k: (2, 1)
Qwen Rewritten Solution 2: (k:, I) is solution, then (1, k)) is also solution. Therefore, we can assume without loss of generality 2
Consider the case when = 1. The equation becomes I which simplifies to kd! The only positive integer solutions
to this equation are & = and & = 2.
3. Consider the case when / = 2. The equation hecomes 25 2
This simplifies to k² = - The only positive integer
solution to this equation is & 2.
x! 4. Consider the case when 1 2 3. We have
7V < and
1. For these two inequalities to hold simultaneously, it must be that
1F
k I.
Therefore, the solutions are (k. (1, 2). (2, 1), and (n, n) for any positive integer n.
Figure 14. An example illustrating two distinct rewritten solutions for a single posted problem, where each user has proposed a unique approach to solving it.
20

---

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Reesstae Evaluation
Table Performance comparison of different models on AIME24 and AMC23 henchmarks, Bold values indicate the highest scores for individual datasets.
Model
SFT Dataset
AIME24
AMC23
No SFT
1/30
8/40
Deepseek-Mail
Numina
0/30
12/40
7b-Ins
AoPS-Ins
1/30
15/40
Numina+AoPS-Ins
2/30
1 11/40
No SFT
0/30
16/40
Mathstral
Numina
0/30
15/40
7B
AoPS-Ins
0/30
12/40
Numina+AoPS-Ins
0/30
16/40
No SFT
2/30
11/40
Llama-3.2
Numina
1/30
6/40
3B-Ins
AoPS-Ins
2/30
11/40
Numina+AoPS-Ins
0/30
12/40
No SFT
0/30
5/40
Llama-3.2
Numina
0/30
6/40
1B-Ins
AoPS-Ins
0/30
6/40
NuminatAooPS-Me Ins
0/30
9/40
Table Comparison of fine-tuning DeepSeek-Maath-7---nsste on DART-Math-Hard (Tong et al., 2024) vs. AoPS-Instruct on various
benchmarks.
Dataset
Size
LiveAoPSBencch
Olympiad Bench
Omni Math
MATH
DART-MATH--Hrd
585K
14.4
21.8
15.4 17.8
52.5
JoPS-Instruce
647K
19.0
24.3
58.8
Question: ABC be an equilateral triangle. the plane (ABC) rise the perpendiculars AAA' and BB' same side the plane, so that AA'' = AB and BB' = AB. Determine the measure the angle hetween the planes (ABC) and (A'B'C"),
User Response:
Denote A' NA = D.Then BD BA = BC so LACD = 90". Thus the plane AA'C is perpendicular to CD and the angle between B'C ABC is equal to LA'AA 45a.
Qwen Rewritten Solution:
Denote the intersection of A'B' and AB as D.
2. Since BD BA BC, it follows that ABDA and ABDC are isosceles triangles.
Therefore, /ACD 90"
The plane A.'C is perpendicular CDD
The angle between the planes (A' B'C') and (ABC) is equal to LA'CA.
6.
Since AA' = AB and BB' AB, and given the symmetry and properties of the equilateral triangle, <4'CA 45".
Thus, the measure the angle between the planes (ABC) and (A'B'C") is 45
Figure An example rewritten geometry solution posted problem.
21

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resistaaa Evaluation
Question: Solve the equation in parameter p: 2² + pa² + (p 1 +
User Response: Note that p # Then & 1 p works, you can get other roots (when they exist) by quadratic formula.
Qwen Rewritten Solution:
1. First, observe that the parameter p must satisfy p * to avoid division by zero in the term
2. Next, consider the given equation:
pa + p
2+1=1
3. We test the value 1 to see if it is root of the equation. Substitute
1 p into the equation:
- p)2 p(1 p)2 + (p p (p-1+
p p-1 11-21 p) +1=
4. Simplify each term:
(1 - p)² = + 3p -p²
p(1 - p)² = p11
2p + D²) = 2p +
0-11 1+ (1- (p- 1)(1 p)
1 1-p
(p 1) - 1 -p? 2p 1 1= -p² + 2p
p-
5. Combine all the simplified terms:
1 1-3 - +
+ -1 -
Simplify the expression:
(1 1-33++3p -1--
1 1 3p
1 1
0=0
Since equation holds true, p indeed root the equation. To find the other roots, we can perform polynomial division of the original cubic polynomial by (æ - (1 p)) to obtain a quadratic polynomial.
9. The quadratic polynomial can then be solved using the quadratic formula:
+ / 02 4ac
2a
where a, b, and the coefficients of the quadratic polynomial obtained from the division. 10. The final roots equation are:
= 1-p and the roots of the quadratic polynomial obtained from the division.
11. The solution the equation is:
Figure 16. An example of an incomplete solution where the author left finding the other two roots of the equation to the reader. While the language model has adequately and correctly expanded the solution of the author, solutions of the equation not mentioned (neither by the author nor the LLM): p-p1- p-1+t/56ppt2
and 1-pt B p
2(p-1)
22

---

Leveraging Online Olympiad- Level Math Problems for LLMs Training and Contamination-Rosin Evaluation
Question: Find x 5 checkerboard with square removed that cannot be tiled using right triominoes(L. shaped). Prove that such tiling does not exist for this board.
User Response:
Alright, then colour
01010
11111
01010
11111
01010
The sum of the squares covered by each is at least 2, so if we place 8 L's we get a total least 16, meaning that every 1 must be covered. Hence removing makes the covering impossible.
Qwen Rewritten Solution:
1. Consider 5 checkerboard with the following coloring pattern:
1
0
1
1
1
0
1
0
0
1
0
2. Each right triomino (I-shaped) covers exactly 3 squares. When placed on the board, the sum of the numbers on the squares covered by each L-shaped triomino is least This is because each L-shaped triomino must cover least square with two squares with Os or Is.
If we place 8 L-shaped triominoes on the board, the total sum of the numbers on the squares covered by these 8 triominoes at least 8 x 2 = 16.
4. The total number of squares on the 5 board is 25. Removing square leaves 24 squares. Since each L-shaped triomino covers squares, L-shaped triominocs will cover exactly 24 squares. squares,
sum the numbers on these 24 squares must be at least 16. Given the coloring pattern, the sum of the numbers on all 25 squares is 13 (since there are 12 squares with 0 and 13 squares with 1). 6. If a square with a is removed, the sum of the numbers on the remaining 24 squares is 13 1 12.
the L-shaped triominoes to cover
Since the of the numbers the squares covered by the 8 L-shaped triominoes must be least 16, and the of the numbers the remaining squares only 12, impossible cover the board with L-shaped triominoes if a square with
1 removed.
Therefore, a 5 5 checkerboard with a square removed that cannot be tiled using right triominoes (L-shaped) is one where square with is removed. The tiling is impossible in this case.
Figure 17. A failure example for rewriting the solution. While the Qwen model gets the overall logic of the proof correct, it produces mistakes counting the number of "ones" on the checkboard and provides incorrect intermediate steps.
23

---

Leveraging Online Olympiad-Leve Math Problems for LLMs Training and Contamination-Resistaaa Evaluation
Question:
Let x be real number such that log, + log, 5 + logs 7 104,2 2/6 Find
User Response:
Turn everything into log2 of something, Here is algebra:
log, x + logza + logz3 Tx = logg1/2 //5/
log2 r + log, v5 logg VT = logg Ve5,
logg (z v5. (Tx) log, V/
5 VTu V
35v
Qwen Rewritten Solution:
1. Convert all logarithms to base 2: log, + logza 5 + logza 7a logzt/2 x 5/6
2. Simplify the logarithms using the change of base formula: log, + 1 log, 5 + 1/ log, 7x = logg x. 5/6
3. Simplify the right-hand side: log, + 1 log2 5 3 loga 7x 2 log, 9/G
4. Use the property of logarithms to combine the terms on the left- side: log, + log, 51/2 + log, (7z) = =log, x/3
Combine the logarithms the left-hand side: logz (z-51/2 (7x))//3 log
5/3
Since the logarithms equal, the arguments must be equal: 5/2 (7x) 1/3 = 5/5
7. Simplify the equation: x /5. 7/ = 5/3
Divide both sides by æ (assuming æ + 0): v5. VT 2/3
9. Raise both sides to the power of 3 to eliminate the cube root: (V5. Txxx> 22/23/33
10. Simplify the equation: 53/2 7x = x²
11. Simplify 33/2; 53/2 = 5v55
12. Substitute back into the equation: 5v5 7x =
13. Simplify the equation: 35v/5z x
14. Rearrange the equation form quadratic equation: x² -3555
15. Factor out :: æxx 35 = 0
16. Solve for z: = 0 or x 35
17. Since
is valid solution (as it would make the original logarithms undefined), the solution is:
35v
Llama Rewritten Solution:
To solve the given equation, we first need to express all logarithms in terms of logg- This can be achieved by applying the change of base formula for logarithms. Step 1: Express log, 5 and logs 7x in terms of logz. Using the change of base formula, we have: log, 5 loga
loga 5 Ioga 2
loga 5 2
loga v and logg 7x logz 7x
logz 7z Tong 2d
logg 7x
log, V7a.
Step 2: Express / in terms of logg-
Using the change base formula, we have: log,2 5/6
loga 2/8
loga 2/6 logg 2172
5//
loog 5/6 -logz (s. (x.5/092
logg v2
log, 5/3 = logg2 V 5
expressions from Steps and into the original equation. log, x + logg 55 + loga VTx logg 4/3 Step 4: Combine the logarithms on the left-hand side using the product rule for logarithms.
Step Substitute
log, (ar v5. 3/7x:) log, 5
Step Since logarithms are equal, the arguments the logarithms must be equal. Therefore, we have:
v5. 8/7x V
Step Solve x.
55. 7/
V
35
Figure Comparison Qwen 2.5 72B and Llama 3.1 70B on rewriting solutions. Overall, the Qwen model shows a higher level detail in expanding equations compared to Llama.
24

---

Leveraging Online Olympiad- Level Math Problems for LLMs Training and Contamination-Rcsan Evaluation
Question: Solve the equation in parameter p: 2 + pa² + (p 1 p-1 Ja =
Qwen Generated Solution: solve the equation x + px² (p-1+ 1+ + + 0 the parameter p. will follow
systematic approach. Let's denote the polynomial by ((:):
f(x) 2 + pa² 1p-1+ 1+ pー11 +
#### Step Check for Rational Roots We will the Rational Root Theorem check for possible rational roots. The Rational
Root Theorem states that any rational root, expressed in its lowest terms 7, must have p as a factor of constant term (1) and ? factor of the leading coefficient (1). Therefore, the possible rational roots are =1. the polynomial:
#i#li Step Test r Substitute x
f(1) = + p- 2 + (p 1+ )-11
=100)11-22 1+ #-1 + = -0 ++
For x = to be root, f(1) 0:
2p + 1 -
### Step Solve the Equation 2p + p-1
0 Multiply through by p to clear the fraction:
(2p + () 1) + 2p² 2p + p 0 2p =
Factor out p:
p(2p - 1)
This gives
solutions:
p=0 p=
### Step 4: Verify the Solutions We need to verify these values of p indeed make 1 root the original polynomial.
#### For p= = 0: Substitute = 0 into the polynomial:
f(x) 3 + (0- 10-1+ 1+ + +1
f(x) x² 2x +
Check (()) f(1) =
1= 2
So. = is a root when 0.
##### For
Subssitute p == into the polynomial:
f(x) = 3 + + + 1+ - x f(z) = + + ( 2) x f(x) = x² + x +1
Check f[]):
f(1) = + 12 3 11- 1+ So, is
+1) - = 11
whenp= 3.
#### Conclusion The only value of p which a root of polynomial is = 0.
Thus, the solution to the equation is:
Figure An example letting Qwen generate entire solution without the user input (from Figure 16). this case, model completely fails to respond correctly, misinterpreting the question and taking an incorrect approach to solving the problem. This shows the cruciality rewriting solutions (rather than generating entire solutions from stronger models).
25


Reviews:
[{"review_id": "RvZ279lGsA", "reviewer": "Reviewer_MpPL", "summary": "This paper introduces AoPS-Instruct, a dataset of 666K Olympiad-level math QA pairs, and LiveAoPSBench, a contamination-resistant benchmark, both sourced from the Art of Problem Solving (AoPS) forum. Using an automated pipeline, the authors extract and refine QA pairs, leveraging Qwen 2.5 72B to rewrite solutions into step-by-step explanations. Fine-tuning LLMs on AoPS-Instruct improves performance on benchmarks like OlympiadBench and Omni-MATH, while LiveAoPSBench reveals performance drops over time, indicating prior benchmark contamination and thus, highlights the importance of continually evolving datasets for fair LLM evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback.\n\n**Q1:** In Table 2, the size of the datasets on which the models have been finetuned have not been specified. It would be important to ensure that the amount of data on which the models are finetuned is same across different datasets to ensure fair comparison\n\n**A1:** We used the full Numina dataset, decontaminated against our evaluation benchmarks, resulting in 824K data points. In comparison, our dataset contains 647K data points. Despite being smaller, our dataset demonstrates a greater performance improvement than Numina, as highlighted in Table 2. We will add this information to our revised manuscript.\n\n**Q2:** For the comparison in Table 2, it would be useful to have a comparison with a stronger baseline. Several powerful synthetic datasets (for eg. ScaleQuest [1] and DART-Math [2] are plausible candidates).\n\n**A2:**\nScaleQuest: Thank you for bringing our attention to this very recent work. We will add a citation to this work as a concurrent work.\n\nDART-Math: Thank you for pointing this work out. Below, we take the DeepSeek model from the DART-MATH paper, and compare it with our DeepSeek-fine-tuned model:\n\n|   | AoPS24 | MATH | OlympiadBench | OmniMath |\n|-|--|--|---|--|\n| DART-Math   | 13.3   | 53.6   | 22.5         | 15.2     |\n| AoPS-Ins    | **19.0** | **58.8** | **24.3** | **17.8** |\n\nWe observe that our model outperforms the DART-MATH model on all benchmarks. We will add this table to our paper.\n\n**Q3:** The number of seeds over which the performances are computed has not been reported.\n\n**A3:** We measure the Pass@1 metric in all of our tables, and for non-R1 models, the common practice is to set a temperature of zero for performance measurement. Therefore, there is no stochasticity in the metric to repeat for multiple seeds. This is in line with other works in mathematical reasoning (e.g., see [2] and [3]) where there are no standard deviations to report for Pass@1.\n\n**Q4:**  It would be useful to report the performances of some state of the art models such as o1, DeepSeek-R1, DeepSeek-R1 distilled series, etc. on LiveAoPSBench. This would give a better idea of the difficulty level of the benchmark.\n\n**A4:** We have reported the performance of R1-distilled models in Figure 8 in the Appendix. However, assessing the full R1 and o1 models is both computationally and financially prohibitive due to their high operational costs, which prevents us from including their results.\n\n**Q5:** Are any measures taken to remove the 5% incorrect and 3% no-answer category questions from the final evaluation set? The presence of these amounts of degenerate questions could cause significantly wrong judgments about the performance of models.\n\n**A5:** LiveAoPSBench is a dynamically evolving benchmark, and we continuously work to refine the pipeline as we identify failure cases. We should note that current large language models perform well below this error-rate threshold (e.g., DeepSeek distilled models achieve only 52.2% accuracy on our benchmark). Moreover, even the human-labeled GSM8K dataset is known to contain a small percentage of label errors [1], yet it remained a highly discriminative benchmark until state-of-the-art models began to saturate it. Thus, while we are committed to enhancing LiveAoPSBench, we believe it remains valuable and informative in its current state.\n\n**Q6:** In Step 3 (solution rewriting), how is the solution that is to be re-written selected out of all the solutions provided by different users as extracted from the forum.\n\n**A6:** We input the whole topic discussion to the LLM, and the LLM must infer correct post indexes that contain correct solution. This is done by analyzing the forum discussion to detect correct answers. We use few-shot chain-of-thougt prompt to extract these (see Figure 12 for the exact prompt, and our code for few-shot examples).\n\n**Q7:** Why is Llama-3.1-70B used for Question-Answer Extraction as compared to a chat variant of the Qwen-2.5 series.\n\n**A7:** We apologize for the typo. In fact, we utilize Qwen-2.5 across the entire AoPS-instruct pipeline (as illustrated in Figure 1 and Table 1). We will correct this mistake in our revised submission.\n\n\n**Q8:**  Another work which attempts to tackle the problem of evaluation data contamination includes Shah et al., which uses an AI-Assisted approach for creating new and difficult mathematical questions.\n\n**A8:** Thank you for highlighting this relevant work. The work of Shah et al. aims to produce an evaluation dataset that tests compositional generalization by combining pairs of distinct mathematical skills extracted from the MATH dataset. This differs from our methodology, which emphasizes dynamically evolving, community-driven problem-solving data. We appreciate the suggestion and will include a citation to this work in our revised manuscript. \n\n[1] https://gradientscience.org/platinum-benchmarks\n\n[2] Yang et al., Qwen2.5-Math\n\n[3] Shao et al., DeepSeekMath", "claims_and_evidence": "The paper provides sufficient empirical evidence for all the claims made.", "methods_and_evaluation_criteria": "The description of methodology lacks details about the motivation behind certain design choices as well as other details. \n\n* Why is Llama-3.1-70B used for Question-Answer Extraction as compared to a chat variant of the Qwen-2.5 series.\n* In Step 3 (solution rewriting), how is the solution that is to be re-written selected out of all the solutions provided by different users as  extracted from the forum.\n* Are any measures taken to remove the 5% incorrect and 3% no-answer category questions from the final evaluation set? Presence of these amounts of degenerate questions could cause significantly wrong judgements about the performance of models.", "theoretical_claims": "The paper does not make any theoretical claims", "experimental_designs_or_analyses": "I have cetain questions and suggestions about the experiments reported in the paper.\n\n* In Table 2, the size of the datasets on which the models have been finetuned have not been specified. I believe it would be important to ensure that the amount of data on which the models are finetuned is same across different datasets in order to ensure fair comparison\n* For the comparison in Table 2, it would be useful to have a comparison with a stronger baseline. Several powerful synthetic datasets (for eg.  ScaleQuest [1] and DART-Math [2] are plausible candidates)\n* The number of seeds over which the performances are computed has not been reported. It is important to discuss the statistical significance of small differences (such as the difference in performance of models on AoPS-Ins and Numina+AoPS-Ins in Table 2.\nIt would be useful to report the performances of some state of the art models such as o1, DeepSeek-R1, DeepSeek-R1 distilled series, etc. on LiveAoPSBench. This would give a better idea of the difficulty level of the benchmark.", "relation_to_broader_scientific_literature": "This paper falls within the vast literature on developing novel benchmarks for evaluating mathematical reasoning abilities of LLMs. More specifically, this work focuses on developing contamination resistant evaluation benchmarks but continually collecting new olympiad level questions discussed on the AoPS forum. Another work which attempts to tackle the problem of evaluation data contamination includes [3] which uses an AI-Assisted approach for creating new and difficult mathematical questions.", "essential_references_not_discussed": "I believe the following papers are worth discussing:\n\nSynthetically Generated Training Datasets for Math Reasoning\n\n[1] Ding el al., 2024; Unleashing Reasoning Capability of LLMs via Scalable Question Generation from Scratch; https://arxiv.org/abs/2410.18693 \n[2] Tong et al., 2024; DART-Math: Difficult Aware Rejection Tuning for Mathematical Reasoning; https://arxiv.org/abs/2407.13690\n\nConnections with approaches for creating math evaluation benchmarks to address data contamination in existing benchmarks\n\n[3] Shah et al., 2024; AI-Assisted Generation of Difficult Math Questions; https://arxiv.org/abs/2407.21009", "other_strengths_and_weaknesses": "### Strengths\nThe paper is well motivated and presents systematic pipelines for creating olympiad level training datasets and contamination resistant evaluation datasets. The paper is generally well written and provides a good set of experiments and ablation studies to show the usefulness of the approach\n\n### Weaknesses\nPlease refer to *Methods and Evaluation Criteria* and *Experimental Design and Analyses* sections", "other_comments_or_suggestions": "I have mentioned all comments and suggestions that I have in previous sections. The paper attempts to tackle an important problem and proposes a pipeline which is more generally applicable. However, I believe it lacks certain important clarifications and details which I have highlighted previously.", "questions_for_authors": "Please see **Methods and Evaluation Criteria** and **Experimental Design and Analyses**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pMqPF7cy4", "reviewer": "Reviewer_Hg4s", "summary": "1) The paper constructs a dataset for Olympiad-level mathematical reasoning with a large scale and diverse problems, which is significant for LLMs’ development in mathematical problem-solving.\n2) Experiments demonstrate that the dataset can effectively improve LLM performance on benchmarks like MATH. Also, this dataset can serve as a benchmark with some resistance to contamination, effectively evaluating model performance", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below, we answer the main concerns raised by the reviewer:\n\n**Q1:** No conflict resolution protocol for handling discreprancies in community-provided answers (e.g., voting mechanisms).\n\n**A1:** For the LiveAoPSBench evaluation set, we only keep questions that have a closed-form answer, and we handle discreprancies between community-provided via LLM cross-check. \n\nIn the AoPS-Instruct training set, we include all recognized solutions based on forum discussions for two reasons: (1) Most of the Q-A pairs in training set either have no final answer (e.g., proof-based questions) or the answer is free form, making it hard to rely on voting mechanisms to resolve discrepency. (2) Prior works such as Alphacode[1], have successfully used partially correct solutions for training their models. Therefore, we choose not to apply majority voting filters, allowing users to customize the filtering according to their needs.\n\n[1] Li et al., Competition-Level Code Generation with AlphaCode, 2022\n\n**Q2:** Lack of validation for each step in the dataset construction pipeline. (whether LLM cross-cheching effectively removes incorrect answers.)\n\n**A2:** In Figure 5.b we show the effectiveness of LLM rewritting.\nThe effectiveness of LLM cross-checking has been qualitatively verified by human annotators. In Section 4.4, a human-annotated error rate of 8% was reported, whereas we empirically observed an error rate of over 20% before applying LLM cross-checking.\n\n\n**Q3:** The current approach detects duplicate data via substring matching but fails to identify semantically equivalent problems with different wording or multilingual translation issues (e.g., Chinese-to-English translations). Could semantic similarity be used instead?\n\n**A3:** N-gram decontamination is a standard practice in math data. We follow the DeepSeek-Math [2] paper, which uses 10-gram decontamination. Qwen-Math also applies decontamination based on 13-grams [3]. [4] Leverages the LLM for semantic based decontamination, but is not widely adpoted due to its inefficiecy when scaling up, and high false-positive rate.\n\n[2] Shao, et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024\n[3] Yang, et al., Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement, 2024\n[4] Yang, et al. \"Rethinking benchmark and contamination for language models with rephrased samples.\" arXiv preprint arXiv:2311.04850 (2023).\n\n\n**Q4** The paper does not effectively demonstrate that timestamps mitigate contamination. There is no quantitative analysis of new problem duplication rates, such as statistics on overlap between different time periods and previous datasets.\n\n**A4:** Thank you for the suggestion! We demonstrate the correlation of timestamps vs contamination by decontaminate against the massive Numnia Training set following the decontamination set up of Qwen2.5[3]. Below we report a change of contamination rate w.r.t to timestamp.\nWe report the statistics of overlap against Numina CoT Traing dataset (released July.2024) as below:\n\n|          Time      | Jan-Apr 2023 | May-Aug 2023 | Sep-Dec 2023 | Jan-Apr 2024 | May-Aug 2024 |\n|:-------------------|-------------:|-------------:|-------------:|-------------:|-------------:|\n| 10-gram Overlap Rate |        13.24%|        11.65%|        12.82%|         9.92%|         6.88%|\n| Overlapped Question| 229 | 208 | 218 | 226 | 109 |\n| Total Questions    |         1730 |         1785 |         1701 |         2278 |         1585 |\n\nAs we can see, the rate of potential contamination significantly decreases as timestamps increase. We will add this info to our revised manuscript.\n\n**Q5:** NuminaMath also sources data from AoPS and rewrites solutions using GPT-4o. How does your approach compare to theirs, and what advantages does it offer?\n\n**A5:** While Numina source some problems from AoPS contest pages, their problem-solution extraction mechanism is very simple, leading to very small number (30K QA pairs) of extracted solutions compared to our method. Their method only extracts problems from contest page (where the problems are formally written), and their solution extraction method takes the longest post which contains a black box at the end of the proof.\n\nInstead, we explore the entire AoPS forum discussions, where questions and solutions are shared in unstructured posts by community members. This requires extensive use of LLMs for data filtering and solution extraction, allowing us to extract 652K QA pairs (in contrast to Numina's 30K), resulting in a 21-fold increase in extracted data from AoPS. These improvements result in a stronger dataset, leading to higher performance gains as shown in Table 2 of our paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "1) Insufficient methodological details:\n1.1) No conflict resolution protocol for handling discreprancies in community-provided answers (e.g., voting mechanisms).\n1.2) Lack of validation for each step in the dataset construction pipeline (e.g. whether LLM cross-cheching effectively removes incorrect answers).\n\n2) The current approach detects duplicate data via substring matching but fails to identify semantically equivalent problems with different wording or multilingual translation issues (e.g., Chinese-to-English translations). Could semantic similarity be used instead?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1) The paper does not effectively demonstrate that timestamps mitigate contamination. There is no quantitative analysis of new problem duplication rates, such as statistics on overlap between different time periods and previous datasets.", "relation_to_broader_scientific_literature": "Similar work (such as the NuminaMath dataset) has been done on this topic.  The contribution of this paper is marginal.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Constructing a large-scale and high-quality dataset for mathematical reasoning is an important task and will promote the research on mathematical reasoning.\nHowever, the contribution of this paper is marginal. It is somewhat weak technically.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1) NuminaMath also sources data from AoPS and rewrites solutions using GPT-4o. How does your approach compare to theirs, and what advantages does it offer?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rsEiKfxPW", "reviewer": "Reviewer_tKXz", "summary": "The paper introduces a scalable pipeline that leverages the Art of Problem Solving (AoPS) forum to construct two key resources for advancing Olympiad-level mathematical reasoning with LLMs. (1) AoPS-Instruct is a large-scale instruction-tuning dataset containing over 600k QA pairs, extracted and rewritten from AoPS forum posts. (2) LiveAoPSBench is a continously evolving, timestamped evaluation set intended to minimize overlap (contamination) with existing training corpora. Through extensive experiments, the authors demonstrate that (1) fine-tuning on AoPS-Instruct improves model performance on various math benchmarks, including Olympiad-level problems, and (2) older test sets often lead to inflated accuracy due to potential data contamination.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below, we answer the main concerns raised by the reviewer:\n\n**Q1:** The evaluation is limited to the LLMs with fewer than 7B parameters. To strengthen their claims, the authors should extend their experiments to larger-scale LLMs (exceeding 7B parameters) and provide a more comprehensive assessment of the proposed method’s effectiveness.\n\n**A1:** We do not currently have the resources to fine-tune larger than 7B models, but we believe we provide substantial evidence that we hope motivates the community to push this further. To reiterate on some of the evidence we show in the paper:\n\n* Table 2 presents results for two 7B math-specialized models, alongside Llama 1B and 3B models, all consistently demonstrating improved accuracy across the board.\n* Table 6 in the appendix shows that performance gains transfer from a smaller 1.5B rewriting model to a larger 7B model.\n* For evaluation, we tested models up to 72B on our LiveAoPSBench, highlighting the benchmark’s difficulty and its correlation with uncontaminated benchmarks (Figure 5a).\n\n\n**Q2:** The evaluation set primarily consists of problems with concrete numeric or symbolic final answers. This excludes a large portion of Olympiad problems that require more open-ended proofs, which the pipeline and automated evaluations currently do not fully capture.\n\n**A2:** **Evaluating proof-based questions has remained a long-standing and persistent challenge in the mathematical reasoning community, with no widely accepted solution** outside formal languages like Lean—which fall beyond the scope of our work. A key limitation is the lack of large-scale training data for informal proof-based problems. As noted in our paper, approximately 30% of our training dataset consists of proof-based questions. We are optimistic that this dataset will support the development of new methods for evaluating proof-based reasoning.", "claims_and_evidence": "The authors claim that “Fine-tuning various LLMs on AoPS-Instruct lead to improved performance on standard benchmarks such as OlympiadBench, Omni-Math, and our LiveAoPSBench dataset, verifying the effectiveness of our dataset in enhancing math reasoning capabilities of LLMs.” However, their evaluation is limited to the LLMs with fewer than 7B parameters. To strengthen their claims, the authors should extend their experiments to larger-scale LLMs (exceeding 7B parameters) and provide a more comprehensive assessment of the proposed method’s effectiveness.", "methods_and_evaluation_criteria": "The evaluation criteria are somewhat limited due to the lack of larger-scale LLMs (exceeding 7B parameters) used.", "theoretical_claims": "This paper does not include the theoretical claims.", "experimental_designs_or_analyses": "The experimental analysis is valid. However, expanding the experiments to include a broader range of LLMs with more than 7B parameters would further strengthen the findings.", "relation_to_broader_scientific_literature": "This paper is not connected to the broader scientific literature.", "essential_references_not_discussed": "Regarding the mathematic benchmark for LLMs, the paper covers the necessary related works.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1.\tThe authors present a fully automated procedure—from extracting forum posts, filtering and rewriting answers, to building a “live” test set—showcasing a viable way to harness community-generated data for advanced math tasks.\n2.\tBy time-stamping problems and drawing from posts beyond a given cutoff date, LiveAoPSBench offers a valuable approach to reduce dataset leakage, giving a more faithful measure of actual LLM reasoning ability.\n3.\tThe authors provide a thorough set of experiments across multiple math benchmarks (Omni-Math, OlympiadBench, etc.) and demonstrate that training on AoPS-Instruct consistently boosts performance. They also show a strong correlation between LiveAoPSBench and a human-verified Olympiad dataset, indicating good dataset quality.\n\n**Weaknesses:**\n\nThe paper is well-structured and effectively conveys its main points. Improving mathematical reasoning capabilities of LLMs is an important problem in the research field, and the authors propose the corresponding method to address this problem. However, the experimental validation presented in this paper is weak.\n\n1.\tThe evaluation is limited to the LLMs with fewer than 7B parameters. To strengthen their claims, the authors should extend their experiments to larger-scale LLMs (exceeding 7B parameters) and provide a more comprehensive assessment of the proposed method’s effectiveness.\n2.\tThe evaluation set primarily consists of problems with concrete numeric or symbolic final answers. This excludes a large portion of Olympiad problems that require more open-ended proofs, which the pipeline and automated evaluations currently do not fully capture.", "other_comments_or_suggestions": "Figure 1 contains too many plotted lines, which may impact visual clarity and overall readability.", "questions_for_authors": "1.\tWould you explore evaluating the results on larger-scale LLMs (7B+ parameters)?  \n2.\tHave you considered adapting your pipeline to handle open-ended proof-based questions with no single “boxed” answer? What modifications might be needed for verifying correctness in such contexts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcEFtLfzgW", "reviewer": "Reviewer_m8Ms", "summary": "This paper created a new math dataset sourced from the Art of Problem Solving. It designed a pipeline that includes (1) raw data collection, (2) math question detection, (3) question-answer extraction, (4) solution rewriting, and (5) data decontamination. Steps 1 to 3 involve the use of LLMs in the processing. To provide an unbiased benchmark for evaluating the performance of LLMs on math, this paper collected discussions between January 2023 and September 2024 and processed them through a more complex procedure to ensure benchmark quality. The authors further used this dataset to evaluate the performance of open-source models. There is a clear decrease in performance concerning the time at which the discussions took place.\n\n\n## update after rebuttal\n\nI maintain my view that this paper is suitable for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. Below, we answer the main questions raised by the reviewer:\n\n**Q1:** For question with 2 or 3 answers, how do you handle them?\n\n**A1:** For the LiveAoPSBench evaluation set, we only retain questions with closed-form answers. If a solution provide multiple valid answers, they are summarized as a list of closed form answers. After this, we remove all questions that show discrepancies—i.e., different closed-form answers—between the community-provided solutions(see Section 3.2 LLM cross-checking). \n\nIn the AoPS-Instruct training set, we include all recognized solutions based on forum discussions for two reasons: (1) Most of the Q-A pairs in training set either have no final answer (e.g., proof-based questions) or the answer is free form, making it hard to rely on voting mechanisms to resolve discrepency. (2) Prior works such as Alphacode[1], have successfully used partially correct solutions for training their models. Therefore, we choose not to apply majority voting filters, allowing users to customize the filtering according to their needs.\n\n[1] Li et al., Competition-Level Code Generation with AlphaCode, 2022", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The design of the experiments is well-structured. The first part of the evaluation focuses on data quality. The authors randomly select 10% of the benchmark questions and have a group of graduate students evaluate their correctness. Additionally, they compare the correctness of models on the Olympiad benchmark and LiveAoPSBench, observing a clear positive correlation. An ablation study also confirms the necessity of solution rewriting.\n\nThe second part examines the effectiveness of the math instruction fine-tuning dataset. The results indicate that AoPS-Ins improves the model more effectively than Numina.\n\nThe third part evaluates the performance of open-source models on LiveAoPSBench. Dividing topics by time is a clever approach to avoiding contamination. The observed performance decline further indicates that benchmarking still suffers from data contamination to some extent.", "relation_to_broader_scientific_literature": "This paper provides a new Math Instruction Fine-tuning dataset and a math benchmark. This effort contributes to the community of AI for math.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "The short title at the top of each page exceeds the limit.", "questions_for_authors": "1. For question with 2 or 3 answers, how do you handle them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a benchmark based on art of problem solving forum. Compared to earlier efforts like NuminaMath, the benchmark is an order-of-magnitude larger. The most notable factor is the capability of creating a liveAOPSbench that can continually evaluate language models. 
Reviewers appreciated the benchmark contribution and evaluation on different models, especially on the LiveAOPSBench.
Some reviewers requested that bigger models be evaluated on the benchmark, but there are reasonable number of models evaluated on the live version. 
Overall, I'm very supportive of the paper's contribution on a live contamination-avoiding benchmark.
Decision: Accept (poster)