DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
Authors: Yuxuan Tong
Xiwen Zhang
Rui Wang
Ruidong Wu
Junxian He
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Mathematical Reasoning
Synthetic Data
Primary area: natural_language_processing
TL;DR: To achieve the best performance (for mathematical reasoning), more correct responses for difficult queries are crucial.

Abstract: Solving mathematical problems requires advanced reasoning abilities and presents notable challenges for large language models. Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries.
Hypothesizing that difficult queries are crucial to learning complex reasoning, we propose *Difficulty-Aware Rejection Tuning* (`DART`), a method that allocates difficult queries more trials during the synthesis phase, enabling more extensive training on difficult samples.
Utilizing `DART`, we have created new datasets for mathematical problem-solving that focus more on difficult queries and are substantially smaller than previous ones. Remarkably, our synthesis process solely relies on a 7B-sized open-weight model, without reliance on the commonly used proprietary GPT-4.
We fine-tune various base models on our datasets ranging from 7B to 70B in size, resulting in a series of strong models called `DART-Math`.
In comprehensive in-domain and out-of-domain evaluation on 6 mathematical benchmarks, `DART-Math` outperforms vanilla rejection tuning significantly, being superior or comparable to previous arts, despite using much smaller datasets and no proprietary models. Furthermore, our results position our synthetic datasets as the most effective and cost-efficient publicly available resources for advancing mathematical problem-solving. Our datasets, models and code are publicly available at https://github.com/hkust-nlp/dart-math.

G
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
Yuxuan Tong"), Xiwen Zhang", Rui Wang', Ruidong W Junxian He
TTinnghua University
Helixon Research
HKUST
tongyx21 hnngsi/saaaaiissttnnuuua.aa..
tsinghua. edu. junxianh@cse.un ust. hk
Abstract
Solving mathematical problems requires advanced reasoning abilities and presents notable challenges for large language models. Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results. However, our analysis of these datasets reveals severe biases towards easy queries, with frequent failures to generate any correct response for most challenging queries. Hypothesizing that difficult queries are crucial to learning complex reasoning, propose Difficulty-Aware Rejection Tun- ing (DART), a method that allocates difficult queries more trials during the synthesis phase, enabling more extensive training difficult samples. Utilizing DART, we have created new datasets for mathematical problem-solving that focus more on difficult queries and are substantially smaller than previous ones. Remarkably, synthesis process solely relies on a 7B-sized open-weight model, without reliance the commonly used proprictary GPT-4. We fine-tune various base models on our datasets ranging from 7B to 70B in size, resulting in series of strong models called DART In comprehensive in-domain and out-of-domain evaluation on 6 mathematical benchmarks. DART-Math outperforms vamilla rejection tuning significantly, being superior or comparable previous arts, despite using much smaller datasets and no proprictary models. Furthermore, our results position our synthetic datasets as the most effective and cost-efficient publicly available resources for advancing mathematical problem-solving
Introduction
Recent years have seen remarkable advancements in various tasks through the use of large language
models (LLMs) (Brown et al., 2020 Touvron et al., 2023 Chowdhery et al., 2023 Anthropic 2023
OpenAl al. (023). However, these models still struggle with complex reasoning (Hendrycks et al.
2021] et al. 2024 He et al. 2024 Lin et al. 2024). a cornerstone of human cognitive essential for tackling intricate tasks. Mathematical reasoning, in partieular, represents significant challenge
and stands as one of the most difficult categories of reasoning for state-of-the-ard LLMSs (Hendrycks)
al. 2021 Cobbe et al. 2021b Zheng al. 2022).
this work, we focus on mathematical problem-solving to explore enhancement of the mathematical reasoning abilities of pretrained LLMs. We investigate instruction tuning (Longpre et al. 2023 Wang
ct al. 2023). which is recognized as the most cost-effective method and achieves the ofate-of-the
art performance various mathematical benchmarks (Yu et al. 2024 Yue et al. 2024). Current SOTA instruction tuning methods for mathematical problem-solving are typically implemented as
Work done during visit to HKUST. Our datasets, models and code are publicly available aulatps:///gtthbb.com/kuuus
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MetaMath
MMIGC DART (Uniform)
VRT
VAT
DART-Math (Prop2Diff)
Uniform
42
Prop2Diff
150
39.7
38.8
38.7
Average Acc Benichnaato
37.4
# Responses rssessss
125
35.6
36.2
100
34.234.3
75
34
30.8
50
29
28.9
25
0
25
0.0k 2.5k 5.0k 7.5k 10.0k 12.5k 15.0K Query (Difficulty Descending)
Mistral-78
LIamaS-8B
Figure 1: Left: Average accuracy on six mathematical benchmarks. We compare with models fine-tuned on the best, public instruction tuning datasets for mathematical problem-solvin MetaMath (Yu et al. (0224) with 395k examples, MMIQC (Liu et al. 2024a) with 2.3 million examples, as well as vanilla rejection tuning (VRT) with 590k examples. Both DART-Math (Uniform) and DART-Math (Prop2Diff) use 590k training examples. Right: Number of responses for each query descending by difficulty across 3 synthesis strategies. Queries are from the MATH training split (Hendrycks et al. (021). VRT the baseline biased towards easy queries, while Uniform and Prop2Diff are proposed in this work to balance and bias towards difficult queries respectively. Points are slightly shifted and downsampled for clarity.
augmenting existing training datasets with synthetic data generated from proprietary models like GPT-4 (OpenAl et al., (023). A prevalent method of data augmentation is sample multiple responses to given queries from a strong model and filter out the incorrect ones. This method, known as rejection tuning, ensures the high quality of the augmented thought steps and yields competitive
performance (Yuan et al.) 2023 Yu et al. 2024 Singh et al. (023).
However, after careful examination of these SOTA synthetic datasets, find that they suffer from severe bias towards responses to easy queries and low coverage for hard queries. For example, as shown in Figure 2 (Left and Middle), while the original queries vary difficulty, the augmented samples in the MetaMathQA dataset (Yu et al. 2024) focus more on easier queries, with zero new responses generated for 51.1% of the most difficult training queries in the MATH training set (Hendrycks et al., 2021). This phenomenon commonly exists in rejection-sampinng based data synthesis which typically samples an equal number of raw responses for each query, disadvantaging difficult queries that are less likely to yield correct responses. We hypothesize that such biases hinder the learning mathematical problem-solving. since difficult examples are often deemed more crucial during training (Sorscher et al. 2022. Burns et al. 2023 Liu et al. 2024b).
To address this issue, we propose Difficulty-Aware Rejecting Tuning (DART), method that prioritizes more sampling trials for challenging queries, thereby generating synthetic datasets enriched with more responses for difficult questions compared to previous methods. Specifically, we develop two strategies to achieve this: Uniform which collects the same number of correct responses for all queries, and Prop2Difff which biases the data samples towards the difficult queries, contrasting with vanilla rejection tuning. These different strategies are summarized in Figure (Right), where the difficulty of query automatically assessed by sampling multiple responses and calculating the ratio of incorrect answers. Our difficulty-aware synthesis produces two synthetic datasets corresponding to Uniform and Prop2Diff strategies respectively, consisting of ~590k examples. Notably, while previous works mostly utilize GPT-4 to synthesize data, we only rely on the MethSeekMath-7B6 -B-RL model (Shao) cct al. 2024) to produce all the data, thereby eliminating dependence on proprictary models.
In our experiments, we evaluate DART based on Mistral-7B 7B (Jiang et al. 2023), DeepSeekMath 7B (Shao al. 2024). Llama3-8B, and Llama3-70B Meeta 2024). creating series of strong mathematical models that termed DART-Mate Across 6 nn-ommiin and challenging oft-of-domain benchmarks, -Mah significantly outperforms vanilla rejection tuning and the baselines trained on the previously established top public datasets as shown in Figure (Left), this is often achieved with smaller training data size. For example, MART-Math improves Llama3-8B from 21.2% to 46.6% on MATH (Hendrycks et al. (021). and from 51.0% to 82.5% on GSM8K (Cobbe et al. 2021a): Our results mark the DART-Math datasets as the state-of-the-an public resources of instruction tuning for mathematical problem-solving

---

MATH
100
MetaMachOAA ArsAuu
20
# Queries Covered
Responses (K)
15
18.7%
paNgg (N)
15.6%
10.5%
98.15
USMuth-78-tniiee
0SMaeh-78 AL (1 (10)
0SMath 78 RL 16))
10
107
207
10
MATH Difficulty Level
MATH Difficulty Level of Queries
Figure 2: Left: Number of queries in the MATH training set and the SetaMathQA-MATH-AAA difficulty levels annotated by humans. MetaMathQA MaA.aaathA----H----A-A- -AATH-ASATHT from the original training queries. We anmotate the query coverage ratio of MetaMathQA. While the most difficult queries (Level 5) are predominant the original set, synthetic examples bias towards easier queries, dropping over 50% of the most difficult queries. Middle: Total number of responses for queries across different difficulty levels in MetaMathQA M-A--AAT--AASSS
set across 5 generated through rejection sampling
The most difficult queries represent the smallest proportion, only accounting for 10.5% of all the samples. Right: pass accuracy of different DeepSecKMath (DSMath) models and temperatures (t) on MATH500 (Lightman et al. 2024). subset of MATH test set. With enough trials, models are actually able to sample out answer-correct responses to most (>99%) queries.
2 Biases in Rejection-Based Data Synthesis
In this section, we first introduce the background for rejection sampling and rejection tuning, and then present our examination on the biases of rejection-based data synthesis.
2.1 Background: Rejection Sampling and Rejection Tuning
We begin by formulating the data synthesis setting used for instruction tuning. For instruction tuning, the training dataset consists of (z, y) pairs, where x is the input query and y is the response. The process of data synthesis involves generating new (xz, y) pairs to augment the original training dataset, thereby enhancing performance. For each input query 24, it is typical to sample M responses from advanced models such GPT-4, forming the set { w/ (3) )
In the context of mathematical problem-solving, subsequent filtering step is often implemented to eliminate incorrect y,"), This elimination based on whether the final answer in the synthetic response aligns with the ground-truth answer This is crucial as mathematical reasoning poses a significant challenge for current LLMs,
and the generated y! may often be of poor quality. This method of response sampling is known as rejection sampling. and the subsequent fine-tuning process is referred to as rejection tuning, which is widely employed to enhance the mathematical problem-solving solving abilities of LLMs (Zelikman et al.. 2022 Yuan et al., 2023 Yu et al. 2024; (Siggh et al. 2023 Xu et al. 2024). In addition to response synthesis, the queries are typically kept constant (Singh et al. 2023 Hosseini al. [2044] Toshniwal et al. 2024) or altered in a controlled manner (Yu et al. [024] to ensure that ground-trure answers are readily available, which facilitates the implementation of rejection sampling. While some studies also synthesize queries without utilizing rejection tuning (Li et al., 2224 Tang et al.) (2024), our focus in this work is primarily on rejection tuning, a method prevalently used for advancing the mathematical skills of LLMs.
2.2 On the Imbalance of Rejection- Data Synthesis
Next, we examine representative synthetic dataset to identify the inherent biases present in rejection- based data synthesis as implemented in most existing works. Specifically, our analysis focuses on the AnsAug subset of the MetaMathQAAMMTI dataset that produces multiple responses for each query in the original training set of the MATH dataset (Hendrycks et al., 2022), through rejection sampling described in $2.1 MetaMathQA has been recognized as one of the most effective synthetic datasets for mathematical problem-solving.
dataset (Yu et al.) (002)), which is a synthetic
Sttrctly speaking, final answer correctness does not necessarily imply intermediate reasoning correctness. We do not make further distinction across this paper which is not our focus.
3

---

We concentrate on the MATH split because it is a notably challenging benchmark in mathematical reasoning, equipped with duman-annotann difficulty levels that aid in our analysis.
Rejection-based data synthesis biases towards easy queries: Across different difficulty levels, Figure 2 (Left) shows the original query distribution of the MATH training dataset as well as the new query distribution after synthesis in the MetaMathQA-Matt dataset. While the most difficult queries (Level 5) takes the largest proportion in the original query set, MetaMathQA changes the query distribution implicitly towards easier queries, dropping many hard problems. For instance, the proportion of Level 5 (the most difficult) queries notably decreases by 51.1% 1,, indicating that rejection sampling fails to generate any correct response for those queries. As result, as depicted in Figure [ (Middle), the responses to the most difficult queries only account for 10.5% of all the samples. Such a phenomenon generally exists in datasets synthesized through the conventional rejection sampling method outlined in 22.1 primarily because the same number of responses is sampled for each query, yet the likelihood of obtaining correct responses for difficult queries is significantly lower, sometimes even zero. We hypothesize that this bias towards easy queries could substantially undermine the effectiivness of instruction tuning. hard queries are often considered critical for instruction tuning (Lu et al. 2024; Liu et al. 2024b). We note that this bias towards easy queries is less pronounced on relatively simple datasets such as GSM8K (Cobbe et al. 2021a), where most queries are easier and it is not difficult to sample correct responses for most of the queries. However, the bias remains significant concern when tackling challenging tasks, which represent a more compelling and complex field of study for LLMs. Building on these findings, we will next introduce our method a potential remedy the limitations of vanilla rejection tuning.
3
DART - -ifficulty-Aware Rejection Tuning
3.1 Open-Weight Models Are Able to Generate Good Responses
Intuitively, we aim collect a sufficient number of responses for the difficult queries. To assess whether this goal is achievable, given that models might not generate correct responses for challenging queries despite extensive sampling, we explore the capabilities of DeepSeckMathh7BB-R (Shao et al. 2024), a strong model specifically trained for mathematical reasoning. Figure 2 (Right) demonstrates the pass@k accuracy on the queries in MATH500 (Lightman et al.] 2024 subset of MATH test set, indicating the proportion of queries that have at least one correct response when sampling k responses for each query. Notably, even though the synthesis model possesses only 7B parameters, a 90% passâtk accuracy can be achieved when sampling over 100 responses per query. These results are consistent with the findings from recent studies (Toshniwal et al., [024]|Shao [Shao et al., [2024] [Li et al.) 2024), which suggest that strong open-weight models are able to synthesize correct responses for most of the queries. This evidence supports the potential for effectively mitigating the insufficient coverage for difficult queries through strategic response sampling, which we introduce next.
3.2 DARS Difficult--Aware Rejection Sampling
Motivated by the observation above, we aim to collect more responses for harder queries. Specifically, introduce two strategies to increase the number of correct responses for difficult queries: (1) Uniform, which involves sampling responses for each query until each query accumulates k2 correct responses, and ki, is preset hyperparameter determined by the desired size of the synthetic dataset; (2) Prop2Diff, where we continue sampling responses until the number of correct responses for each query is (linearly) proportional to its difficulty score. The most challenging queries will receive kp responses and kip is a hyperparameter. This method introduces a deliberate bias in the opposite direction vanilla rejection sampling, towards more difficult queries. Prop2Diff is inspired by previous works that demonstrate difficult queries can be more effective to enhance model capabilities (Sorscher et al., 2022. Liu et al. 20246). Both the Uniform and Prop2Diff strategies prescribe specific number of correct response for each query, determined by k. or kp. Nevertheless, there are certain queries which we cannot sample out the designated number of correct responses even with extensive sampling efforts. To avoid endless running of the synthesis, we impose cap on the maximum allowable number of raw samples per query as Ammax - once this limit is reached for a particular query, we cease further sampling and retain any correct responses that have been gathered. The straightforward implementation of the Prff2Diff strategy risks generating no synthetic responses for casier queries if kp is set small. To mitigate this, guarantee at least one synthetic response for

---

Dataset
# Samples (k)
Synthesis Agent
Open-Source
WizardMath (uo et al.
2023
96
GPT-4
X
MetaMathQA Yu ct
2024
395
GPT-3.5
MMIQC (Luu ci al. 20244
2294
GPT-44GPT:3.5+Hmman
Orca-Math (Mitra ct al. [2024]
200
GPT-4
/
Swin-Math-VI..I (Li al.
2024
1440
GPT-4
X
KPMath-Plus (Huang
1576
GPT-4
X
MathScaleQA TTng of al.
2024
2021
GPT-3.5+Human
X
DeepSeekMatt-77B.
DART-Math-Hard
585
DeepSeekMatthhB-RR
Table I:
Comparison between our DART-Math datasets and previous mathematical instruction tuning datasets. Most of previous datasets constructed with ChatGPT, and many of them not open-source, especially for ones of the best performance.
each query when implementing Prop2Diff. While it might seem sufficient to rely on the original, real training dataset to ensure at least one human-annotated response per query, our findings highlight the importance of maintaining synthetic response coverage learn to solve easy problems, we will quantitatively shown in $4.3 partially because the human-annotated response is less detailed and not beneficial synthetic responses, demonstrated previously in Yu et al. (2024). For both Uniform and Prop2Diff strategies, we use the DeepSeckMath-78 ---R- model to synthesize responses. We refer the two sampling strategies as DARS Uniform and DARS- DARS-Prop2Ds Prop2Diff if respectively. Though most previous methods are aifficult--gnosticc, few methods try assigning more budget to more complex questions to boost coverage, such as ToRA (Gou al. 2024) and MARIO (Liao al., 2024). However, ToRA/MARIO mainly focus on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easy queries, while DARS explicitly controls the final distribution of the training dataset, completely eliminating the bias and also achieving higher coverage on the hardest queries. For more details about the comparison, we refer readers to requires assessing difficulties of queries, next we introduce an
Appendix A As DARS-Prop2Dis automatic approach to measure difficulties.
Evaluating Difficulty: Previous studies have used proprietary models like ChatGPT to assess the difficulty or complexity of data samples (Lu al. 2024; Liu al. 2024b). this work, we introduce new metric, rate - the proportion of incorrect responses when sampling na responses for given query as a proxy for difficulty. This metric aligns with the intuition that harder queries less frequently yield correct responses. We utilize DeepSeekMath-7B-RR as the sampling model to evaluate difficulty across all experiments the paper. Varying this sampling model to align with the generative model may further enhance performance, which we leave as future work. Notably, one of the benefits of fail rate is that allows to reuse the sampled responses during difficulty evaluation as synthetic responses for dataset construction. See implementation details in Appendix
B.2 B.2
3.3 The DARt-Math Datasets
We utilize DARS- Uni fnim form and DARS ProS-Pro to construct two datasets, Matt-Maihhoorrr respectively for instruction tuning. We use the original training queries of the GSM8K (Cobbe et al. 2021a) and MATH datasets synthesize responses. We maintain fixed queries to better isolate the effects of difficulty-aware rejection tuning, while techniques for query augmentation, as discussed prior studies (Yu et al., 2024), could be potentially incorporated to further improve the performance. The synthetic datasets are augmented with the original GSM8K and MATH training data to form the final datasets. We set kú in DARS-Unifore as 40 and k, in DARS-Prop2Dif
and -
192 to form both datasets of around 590k samples. Our data samples only involve natural language reasoning without using external tools such as code execution. Comparison of our datasets with previous datasets is illustrated in Table I Our datasets are generally smaller than most previous datasets, and in $4.2 we will empirically demonstrate that the DART datasets are the most cost-effective datasets publicly available. Remarkably, our approach solely utilizes DeepSeekMath-77 7B-RL to evaluate difficulty of queries and synthesize responses, without relying on ChatGPT that is commonly used in other studies.
Our approach typically requires more sampling trials than vanilla rejection sampling to generate a dataset of comparable size because difficult queries often need more samples to secure the required
5

---

number of correct responses. Despite this, it is crucial to point out that our overall training cost does not exceed that of vanilla instruction tuning. We emphasize that the data synthesis process is one-time effort. Once the synthetic dataset is created, it can be utilized for multiple training runs across various base models. Furthermore, this dataset will be publicly available, extending its urility to a wide range of users. From this perspective, the initial higher synthesis cost is effectively amortized over numerous training runs and the broad user base, rendering the synthesis cost virtually imperceptible to individual dataset users. We will discuss the synthesis cost further in $4.3
4
Experiments
4.1 General Setup
Below we summarize the key setup details, while we include more information in Appendix B
Data synthesis: We synthesize responses using the original training queries of the MATH and GSM8K datasets. As described in $3.2 we utilize DeepSeckMatth77BRR) to synthesize all the data. We use temperature sampling with adjusted temperature to sample answer-correct responses to difficult queries. We set the maximum number of output tokens 2048 and adopt top-p sampling with p = 0.95. We use chain-of-thought prompt (Wei et al. 2022) to synthesize. We use the vLLM library Kwon al., 2023) to accelerate the generation. In setting, sampling 35k samples on MATH / GSM8k queries takes about NVIDIA A100 GPU hour.
Training: We perform standard instruction tuning on our synthetic datasets DART-Math Uni and -ART-Math- Hard, based on several base models including Llama3-8B (Meta 2024), Mistral- 7B Jiang et al. 2023), and Llama3-70B as representatives of general models, and DeepSeckMath- 7B (Shao et al. )024) as the representative of math muth-speccillzed models. For simplicity, we keep most hyperparameters the same across different models and datasets, and tune only several key hyperparameters like learning rate and number of epochs, as detailed in Appendix B.I
Evaluation: For comprehensive assessment of mathematical reasoning of the models, we adopt benchmarks for both in-domain and out-of-domain (OOD) evaluation. Specifically, we use the GSM8K and MATH test set as the in-domain test. GSM8K consists of grade school arithmetic tasks and are considered much simper than MATH that contains challenging competition mathematical problems. For OOD test, we utilize the following four challenging benchmarks:
CollegeMath (Tang al. 2024): This test set contains 2818 college- level mathematical problems extracted from textbooks across 7 domains such as linear algebra and differential equations, testing generalization on complex mathematical reasoning in diverse domains.
DeepMinnd-Mathemattee (Saxton al. 2019): This test set contains 1000 problems from diverse range of problem types based on a national school mathematics curriculum (up to age 16), testing basic mathematical reasoning in diverse domains.
OlympiadBench-Matt (He al. 2024): This benchmark contains 675 Olympiad-level mathematical problems from competitions, which is a text-only English subset of Olympiad- Bench, testing generalization on the most complex mathematical reasoning.
TheoremQA (Chen et al.. 2023): This benchmark contains 800 problems focused on utilizing mathematical theorems to solve challenging problems in fields such as math, physics and engineering, testing generalization on theoretical reasoning in general STEM.
All results are from natural language reasoning without using external tools, through greedy decoding.
Baselines: We compare DART with the state-of-the-arr instruction-tume mathematical models such as MetaMath ( (Yu et al. 2024), MMIQC (Liu et al.) (2024a), KPMah-Plus (Huang et al. 2024). and Xwin-Math (Li et al. 2024p. We copy the results directly from the respective papers except for MetaMath and MMIQC, where we run our own training since their datasets are public. As shown in Table [] these SOTA datasets all rely on proprietary models for data synthesis. Another ablation baseline to DART is vanilla rejection tuning (VRT), where we synthesize dataset of the same size of 0.59M examples with DeepSeekMath RB-RL, using vanilla rejection sampling as described I We note that there are other strong models such as Yue et al. (2024); Gou et al. (2024) that are trained to solve mathematical problems utilizing code execution, we exclude them since this study focuses on reasoning without using tools.

---

Model
# Samples
MATH
In-Domain
GSM8K
College
DM
Out-of-Domain
Olympiad
Theorem
AVG
GPT-4-Turbo (24-04-09)
73.4
94.5
48.4
GPT-4 (0314)
52.6
94.7
24.4
Claude-3-Opus
60.1
95.0
Gemnni 1.5 Pru
67.7
70B
General Base Model
Momma--70--xwin-Mtttee
144
52.5
90.2
33.1
58.0
16.3
14,9
44.2
Llania-700B-CL
44.0
30.1
35.5
51.7
10.8
27.0
41.2
Llama3-708-Methhh
0.40M
44.9
31.9
53.2
11.6
21.9
41.9
Llama3-70B- 700m3-000-MMIIQC
2.3M
49.4
89.3
37.6
60.4
15.3
23,5
45.9
Llama3-70B-VRT
0.59M
53.1
90.3
36.8
62.8
19.3
48.5
MART-Wath-LLaam3--11 (Uniform)
0.59M
54.9
90.4 10.1
38.5 11.
61.3
19.1
27.4
10.6
DART DANT-Watt-L1amm2-008 (Pro22Dif))
0.59M
56.1 13.0
89.6 10.7
37.9 11.1
64.1
11.3
20.0
10.7
28.2 JO.4
49.3 10.8
7B Math-Speciallzzed Base Model
DeepSeekMath-TB TB-ICC
35.5
64.2
34.7
45.2
9.3
23.5
35.4
DeepSeekMath 7B Instruct
0.78M
46.9
82.7
37.1
52.2
14.2
28.1
43.5
DeepSee&Math7755 MMMMMIIQ
2.3M
45.3
79.0
35.3
52.9
13.0
23.4
41.5
DeepSeekMath 7B KPMuth Plus
1.6M
48.8
83.9
SappSeeMMtth77B-VV
0.59M
53.0
88.2
41.9
60.2
19.1
27.2
48.3
DART Math DSHath-78 (Unilorm)
0.59M
52.9
88.2
40.1 L1S
60.2
12.3
32.5 15.3
49.2 40.2
Math- MaMt--ath-DSSehha 3SNeth-T7
(Prop21iiI)
0.59M
53.6 10.6
86.8 L1.4
40.7
L12
61.6 11.4
21.7 12.6
32.2 15.0
49.4 11.1
7-8B General Base Model
1onn2-7--Xwin-Mmmm
1.4M
45.5
84.9
27.6
43.0
10.5
15.0
37.8
Mistral-78-ICL
16.5
45.9
17.9
23.5
3.7
20.3
Mistral-7B- 7aa-r---WaaaaMah--VV I (RL)
32.3
80.4
23.1
38.4
7.7
16.6
33.1
Mistral-7B-Mettakee
0.40M
29.8
76.5
19.3
28.0
5.9
28.9
Mistral-8-MMM00Q
2.3M
37.4
75.4
28.5
38.0
9.4
16.2
34.2
Mistral-7B TiathSs-B-Mathhculle
Mistral-7B.
Mearra-78-KPMatt-PPPP KPMMtt-Plals
1.6M
46.8
82.1
Mistral-7B.VV5
35.6
SarT-Math-iaatraat (Uniform)
0.59M
44.5
82.6 10.3
26.9 +2.7
16.0
16.4
13.2
16.4 10.2
37.4 13.1
MaRR--ath--Wattrl..
0.59M
81.1
15.4
45.1
19.5
Llama3-36-ICI
21.2
300
15.9
27.4
4.2"
15.8
23.5
Llama3-BB-MeeaMahh
0.40M
32.5
77.3
20.6
35.0
5.5
13.8
30.8
Mama3-8B-MMiIC
2.3M
39.5
77.6
29.5
41.0
9.6
16.2
35.6
VRT
23.9
DART-Math-
(Uniform)
0.59M
45.3
82.5 10.8
27.
13.2
48.2
16.5
13.6
14.3
15.4
10.5
38.7
(Pro2Dimm)
16.9
28.8
48.0
16.3
Table Main results mathematical benchmarks. College, DM, Olympiad, Theorem denote the MothegeMath. OlympiadBench-Mall TheoremQA benchmarks respectively. We annotate the absolute accuracy change compared the VRT baseline within the same base model. Bold means hest score within the respective base model. ICI,, MetaMath, MMTQC, and VRT haselines are from our own runs, while other numbers are copied from the respective papers or reports. For WizardMath and Kwin-Math, we take the public model checkpoints and evaluate ourselves using their official CoT prompt. f. For Xwin-Math, take the best public models that are based on Llama2 Touvron 2023). which is not a very fair comparison others.
DeepMind-Matheen
4.2
Main Results
Comparing with Vanilla Rejection Tuning: The main results are Table 2 DART-Math based on all four different base models outperforms the VRT baselines most benchmarks consistently. Focusing on performance with 7-8B general base models, DART-Math-LA ama3-8B (Uniform) sur- passes the VRT baseline across all 6 benchmarks by an average of 3.5 absolute points, while Mor--Math-Llama--- domain challenging MATH benchmark, DART-Math (Prop2Diff) enhances performance by nearly 7 absolute points for both Mistral-7B and Llama3-8B models. For OOD benchmarks, DART-Math (Prop2Diff) shows particularly notable gains more difficult benchmarks, with im- provements ranging from 5.2 to 9.5 absolute points on CollegeMath, DeepMind- Mathemmis, and OlympiadBench-Mainn This indicates effective generalization of our approach. These improvements over the VRT baselines demonstrate the effectiveness of the proposed difficulty-awan rejection sampling. We note that DART -Mtth does not greatly boost the relatively simple, in-domain GSM8K benchmark. This is expected, as explained in $2.2. because vanilla rejection tuning expected does not face severe bias issues like those seen in more challenging datasets. Thus, difficulty-wwan rejection sampling has limited impact easy datasets. Interestingly, on much stronger base models DeepSeekMath-78 and Llama3-70B, with about a l-point gain on average. We hypothesize that this is due to these models' extensive pretraining on mathematical content. This pretraining likely covers most skills that could learned from the GSM8K and MATH training queries, suggesting that the query set itsclf, rather than the
(Prop2Diff) achieves an average improvement of 4.5 points. On the in- VRT
improvement margin of -ARt over VRT narrows,
7

---

Mistral-7B
L-8ma3-88
DeepSeekMath-78
45
40
35
35
52
30
30
DART (Unifarm)
DART (Prop2DifT)
25
10
10*
10
10*
10¹
10*
10*
10¹
10f
Figure 3: Scaling curves of MATH test performance against number of training samples synthesized from MATH training queries, training is on three hase models.
responses, becomes the bottleneck. Thus augmenting the range of queries could be more effective strategy for future improvements.
Comparison with previous top-performing methods: DART-Math achieves superior or com- parable performance to previous best models. Specifically, when compared with MetaMath, DART-Math wins greatly all cases. Additionally, MART-Math-DSMath--a art results for models sized 7-8B on challenging benchmarks such as MATH, OlympiadBench-Me Math, and TheoremQA. On average, DART -Math -istral-7B (Prop2Diff) surpasses MBsttal--B-MMIQQ by 4.6 absolute points, despite using only a quarter of its training sample size. Compared with concurrent work KPMath- which relies on GPT-4 and has not released either the data or the model, our approach slightly underperforms on Mistral-7B for GSM8K and MATH. However, DART-Math excels against on DeepSeekMath-785 by significant margin, utilizing around one-third of its training data size. The Xwin-Math models perform well on the GSM8K benchmark but fall behind DMRt (Prop2Diff) on other challenging benchmarks overall, particularly with a more pronounced gap on 70B models although note that their models are based on Llama2 which is not very fair to compare with. Importantly, we fully open-source our datasets and models, designating both DART -Mth Sia---nifornn and MART-Math-Haal public instruction tuning datasets available for advancing mathematical problem-solving
achieves the state-of-the-
the pest-performing and most cost-effective
Additional results: For additional results, such as dosain-wise performance on MATH and comparison RL,, we refer readers to Appendix
4.3 Analysis
Scaling behaviors of different data synthesis methods: We study the scaling behaviors of our data synthesis approach and compare it to vanilla rejection sampling. As described in 2.2 our method is motivated to mitigate the bias towards easy queries that are only pronounced in challenging datasets. Therefore, in the scaling experiment we only synthesize responses for the training queries of the challenging MATH dataset and report the performance on the MATH test set. Figure 3 presents the results across three different base models as we scale the training data size from thousands to nearly 1 million samples. We observe a steady improvement in performance as the training data size increases exponentially. DART consistently outperforms VRT on general base models Mistral-7B and Lllama3-8B, achieving better scaling. On DeepSeckMath-7B 7B however, the performance differences between various approaches are minimal. Observing the absolute accuracy changes, DeepSeekMati 7B already achieves over 50% accuracy with just thousands of training samples, and scaling up to million samples leads to only a modest 3-point improvement. This is in stark contrast to the over 20- 20-point improvements seen on other models like Mistral-7B and Llama3-8B. As discussed in $4.2 we believe this phenomenon due to the MATH training queries not being particularly beneficial for DcepSeekMath-TBB which has undergone extensive math-specific continual pretraining. Consequently, for DeepSeekMath-TT the differences between these approaches are not significant, and the main bottleneck shifts to query coverage rather than the responses themselves.
Effect of one-response coverage: In $32 we describe that DARS-Prop2DifE iff can cause zero synthetic responses for easy queries, especially when the number of training samples is small. Therefore, we ensure that the easy queries have at least one correct response practically. Here we examine the impact of this one-response coverage by comparing the Prop2Diff strategy with and
8

---

3w7
66
3.4
2.2
62
2.0
Actieeingg Rarto
35
38
84
30
0.6
82
(+Coved
0.4
PoptBiff C-coorr)
Pros2 DaY
DART PProp20um
Unffom
10
4026
2045 Maax Raw Samples per Quarty
TTaining Samples
Triining Samples
Max. Raw Samplles per Quary
Figure 4: From Left to Right, (I) and (2): Scaling curves studying the effect of one-response coverage. "Prop2Diff (-Cover)" denotes MaRS-Prop2Dift without enforcing at least one synthetic response for each query, while "VRT (+Cover)" denotes vanilla rejection sampling enforeing at least one synthetic response for each query. (3) and (4): The total number of raw samples needed, and the actual ratio (r) of queries achieving the desiderata of the two DARS synthesis strategy for 585k-sized dataset curation respectively, when we vary the maximum allowable raw samples per query (Mmax).
without this coverage constraint, training data sizes increase. Figure (Left) displays the outcomes on the MATH and GSM8K benchmarks respectively. As anticipated, when the training data size is relatively small, the one-response coverage proves beneficial, particularly on the simpler GSM8K benchmark, improving accuracy by about points. This suggests that effective learning for easy problem-solving can be achieved with just one additional correct response. As we scale up the training data size, the natural increase in coverage for easy queries causes that the difference between the two approaches diminishes. Additionally, in vanilla rejection tuning to determine if adding one synthetic response for difficult queries could address its issue of low coverage for such queries. However, this modification does not significantly aid in learning difficult queries, as observed on the challenging MATH benchmark. This indicates that complex problems generally require a greater number of training samples for effective learning.
explore the implementation of one-response coverage
Synthesis cost: DART generally needs more sampling trials to synthesize the same size of dataset compared to vanilla rejection tuning, discussed in $3.3 It is important to underline that the synthesis cost, although initially higher, is a one-time expense. Once the dataset is synthesized, can be used by the community and to train numerous models, effectively amortizing the cost. To provide a quantitative understanding of the synthesis cost, we consider two main factors: 7'max the maximum allowable raw samples for each query, and the ratio of queries that achieve the designated number of responses. If Mmax is set too high, sampling may continue indefinitely for particularly difficult noisy queries, resulting in a high synthesis cost. Conversely, a small Ramax may result in many queries not gathering the sufficient number of correct responses, leading to a lower r. Figure (Right) illustrates the total number of raw samples required to synthesize 585k examples and the query achieving ratio r as we increase Tmmax- When 7mma reaches 2048, over 90% of the queries can collect the designated number of responses under DARS-Uniform, with a corresponding total number of samples around 5 million. To reach 90% achieving ratio for DARS-Pro Prop2Diff, Rmax needs to be least 8K, and the total number of raw samples exceeds 15 million. In our experiments, we achieved an over 95% ratio r, sampling approximately 150 million samples in total, which required running inference of DeepSeekMath-7IB-B
for about 160 NVIDIA A100 GPU days. Besides that synthesis is a one-time cost, we would like to emphasize the number of samples is not fair metric to compare synthesis cost between different works- our synthesis model of 7B size is relatively inexpensive and fast to run, compared to the much more costly and slower GPT-4 used in most previous studies. Moreover, achieving a query ratio as high as 95% may not be necessary to reach good performance. slightly lower ratio of 85% or 90% might not significantly impact performance but could substantially reduce the synthesis cost. We plan to explore this balance further in future work.
5 Discussion
In this paper, we focus on instruction tuning for mathematical problem solving, and discuss the impact of distribution and coverage of training queries across different difficulties. We identify the bias towards easy queries vanilla rejection tuning, and propose difficulty-awar rejection tuning, DART, remedy. Based on our approach, we create and open-source the best-performing and

---

the most cost-effective instruction tuning datasets for mathematical reasoning, without relying on proprictary models. Extensive experiments across various base models and benchmarks demonstrate the effectiveness of our approach.
Limitations: We utilize fail rate the difficulty metric, yet may be sub-optimal. Other metrics such as direct scoring (Liu et al. 2024b), Elo ratings, or the minimum pretraining compute to train model that can always answer correctly (Burns al. 2023 may be further explored. DART-Math is limited by natural language reasoning, while it is shown that generating and executing code helps solve mathematical problems significantly (Zhou al.
2024 Yue et al. 2024; Gou et al. [024] [iio
et al. [2024] Toshniwal et al. 2024)
we think the bias in vanilla rejection sampling also exists for code generation, and DART could be integrated to potentially improve code generation as well.
Acknowledgments
We thank Zhiyuan Zeng. Wei Xiong and Chenyang Zhao for helpful discussions. Yuxuan is partially supported by Tsinghua University Initiative Scientific Research Program (Student Academic Research Advancement Program).
References
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, al. Pytorch Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. pp. 929 947,99772222 2024.
Anthropic. Introducing claude, 2023.
URL tttpe::/ww..anbhrrp....omoo///
introducing- claude
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:2204.05866 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint a arXiv:2312.09399, 2023.
Yihan Cao, Yanbin Kang, Chi Wang. and Lichao Sun. Instruction mining: When data mining meets large language model finetuming. arXiv preprint arXiv:2307.06290 2023.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arx:v: www.::664.061111 06744, 2016.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
net/
FoorumiddWWm39977
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-133, 2023.
10

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021a
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.146,, 2021b.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024. URL https:// Voap
openreview
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksemia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.0899 2023.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. neurips. www.neii.c.c/pppprr.......ep. aatpapprrflllsppppprrrrrae URL /ttpp::://attaeetceooooaarrrreee iew.roceeedinngnnnuiiip.spp.pppe 221_:1/h//saaaeeedr////////22232
pdf
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint 21X4422402.06457, 2024.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2110.16110, 2022.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? The Twelfth International Conference on Learning Representations, 2024. URL httpps://pennrvvier net/ forum?ice VTTumUid-
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha. Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint
2019.
Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint uxXiv2107.02022 2021.

---

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXivv2403.04706, 2024.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting 11m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 12032, 2023a.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang. Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312 10302, 2023b.
Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output a reproducible pipeline, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL httpps:/ net/ fformn2id-v@LopNNEE01
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. International Conference on Learning Representations, 2018 URL
Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench:
Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402. 14809, 2024.
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing, 2024a.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2024b. URL /tttp:://ppeneviess Let/foorunL1-BBBKALLLL.... BTKAeLQLMw
Shayne Longpre, Le Hou, Tu Vu, Albert Wehson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22631 1 -2648. PMLR, 23-29 Jul 2023. URL htttp:://proceeeeee Longpre23a.hti html
mlr pal..press//2221
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.0955 2023.
Meta. Introducing meta llama 3: The most capable openly available Ilm date., 2024. URL https://ai. metaa.a..meeecccommmm www.ccommbbbl.. httpt/aa/cooo/oooeeeeeeeeeeeeeee aalog/meta-llama--
Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason Moore, Sartaj Singh, et al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017.
12

---

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh. and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL https //openreviev net//forum?dderrlgSSgg...
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402. 14830, 2024.
Radford M Neal. Slice sampling. The annals of statisties, 31(3):705-767, 2003.
Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B Blaschko, Guohao Dai, Huazhong Yang, and Yu Wang. Can Ilms learn by teaching? a preliminary study. arXiv preprint arXiv:2406. 14629, 2024.
NVIDIA. Tensorfloat-32 in the a100 gpu accelerates ai training, hpe up 20x, 2020. URL https://blogs. nvidia. com wttp:////////m/ooooooooo----/-// httpa/sss...ooo--epppppriiiio-oi
Josh OpenAI, Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference Learning Representations, 2019. URL http:tttss://pperevieie net//orrumm??h-HlggRRRRRRFY
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arviv preprint arXiv:2402.03300, 2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil. Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 2023.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances Neural Information Processing Systems, 35:19523-1999 2022.
Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followii following llama model. bictps://githbb
_alpacalpaaa_p 2023.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. Openmathinstrutt-:: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.101176 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
13

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023 3023.acl-long.755 754. URL haatpp::/acllattoooog..ooo
acl acl- -ong. 7544
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouge prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824 248377. Curran Associates, files https/ppapr/ee//////22/2
Inc., 2022. URL hattps: mewreeeddddnnnnnnnnnnneeeeeeeeee neurips wwpprppp..........p www.rpps.ccppppprrrrrr.. 1at000015524444ff1aao0770aa2a222
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.0433. 2024.
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math Improving math problem-solving in large language models with self-critique pipeline. arXiv preprint a0Xiv:2404.02893, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. https:
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmo'T: Building math generalist models through hybrid instruction tuning. The Twelfth International Conference on Learning Representations, 2024. URL httpp:://ppenreeieer net/
forum?id-yLC1ss777
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), pp. 1-2. IEEE, 2018.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu, minif2f: a cross-system benchmark for formal olympiad-level mathematics. In International Conference on Learning Representations, 2022. |ttpp wttp:///oroomddddd2/eeeeu.e
Aojun Zhou, Ke Wang, Zimu Lu. Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with bose- self-verification. In The Twelfth International Conference on Learning Representations, 2024. JRL.httpss://// hhttps://openrevier
14

---

A Comparison to Methods Based on Non-vanilla Rejection Sampling
Though both ToRA and MARIO have not released their datasets and focus on mathematical problem- solving using code in addition to natural language, which is out of our scope and thus not comparable, we try implement the natural- -anguall versions of their data synthesis strategies, which are comparable with DARS.
A.1 DARS produces distributions not biased towards easy queries
The most important difference between DARS and ToRA/MARIO is how responses are distributed across various queries while adjust the distribution either be uniform favor more difficult queries, rather than merely improving coverage, ToRA/MARIO mainly focus on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easy queries.
As shown in Table 3 though the absolute numbers of responses not directly comparable between different methods, distribution-wist we can see that ToRA/MARIO still produce fewer responses for difficult problems than the easy ones. This especially contrasts with DART-Math-Hare which produces, for example, 10x more responses for the MATH Level queries than for the GSMSK queries.
As demonstrated in Figure (Left), a high coverage rate (VRT+Cover) alone does not guarantee superior performance.
Size
RPQ in
RPQ in Level-wise MATH
MATH
Synthetic Dataset
(k)
GSM8K
Coverage
ToRA MARIO
72
5.03
5.01
4.99
4.95
4.77
3.84
93.4%
29
2.02
2.01
1.98
1.94
1.89
1.57
91.3%
MART-Math-Une Sn--Unnifonn
585
39.93
40.00
40.00
39.80
39.54
37.14
99.6%
DART-Mach Stath-Hard
590
8.49
14.28
33.52
54.94
79.59
107.06
99.6%
Table Comparison between datasets synthesized by methods based non-vanilla rejection sampling. "RPQ" means the average number of responses per query. The ToRA and MARIO datasets here are implemented by us according to their papers' descriptions, since the official implementations have not been open- open-sourced.
A.2 DARS achieves high coverage even the hardest queries
It is worth noting that a relatively high total coverage on MATH training set does not mean that the
dataset achieves 82.8% of coverage the MATH training set with evenly allocated budgets yet still admits missing significant portion of hard queries and biasing towards easy queries, as analyzed in Figure 2]
hard queries are well covered. For example, the BeetMMathQA-MATT-AAne
In Table 4 we show the coverage rate across all the difficulty levels by different methods. The ToRA- Corpus- 16k statisties show that it only covers 68% of the Level 5 MATH queries while DART-Math datasets cover 99.6%.
A.3 Details of Re-implementing Data Synthesis Strategies of ToRA and MARIO
Here we supplement more details on how we we replicate the ToRAMMARIO synthesis pipeline to conduct the analysis present in the general author rebuttal. Below we show the format "ToRA/MARIO' method -> how we adapt similar spirits for simpler replication" step by step (we use CoT format rather than tool-integrated reasoning for fairer comparison with our datasets):
ToRA:
15

---

MATH training set coverage
Total
Level 1
Level
Level
Level 4
Level
ToRA-Corpus- Saa--Corpu--1k--MAA 166-MTT
83.1%
97.7%
91.6%
86.5%
81.3%
68.0%
MetaMath-MATH-AAE
82.8%
98.1%
93.6%
86.7%
76.6%
48.9%
VRT Baseline
84.9%
99.6%
98.2%
95.2%
89.8%
62.9%
DART-Math-*
99.6 %
100.0% %
100.0%
99.9%
99.7 %
99.1 %
Table 4: MATH training coverage rates all the difficulty levels of different synthetic datasets. The numbers BORAA--Crpu--I---AATA are from their OpenReview page] The two the same coverage because of the "Cover" operation, which tries to ensure there is at least one correct response for each query.
datasets have
1. Once for each problem in MATH&GSM8K with GPT-4, keeping the correct responses. We follow this with GPT-4o min
2. 10 trials for each problem not correctly answered by greedy decoding with GPT-4 and keeping up correct responses per problem (to form ToRA-Corpus-I6k). -> We follow this with GPT-4o mini.
3. Training CodeLlama models on ToRA-Corpus 16k to perform rejection sampling next. DeepSeekMath-7BB... to
-> To avoid additional training for fairer comparison, replace the trained Codel models here to align with DART-Math.
(a) 64 trials for each problem in MATH&GSK8K with CodeLlama, getting 233k distinct correct responses. >> We follow this with DeepSeekMath- Baaiatthh----LL 7B-RL, getting 733k distinct correct responses.
(b) Correcting wrong responses by greedy decoding from the correct preceding portions (costing no more than 64 trials for each problem) with CodeLLaMA- getting 69k corrected responses. -> We simplify this by re-sampling another up 64 trials per problem for all the incorrect responses, getting 225k correct samples.
(c) Both ToRA and our adaptation: Randomly selecting up 4 correct responses per problem from steps (a) and (b).
4. Merge ToRA-Corpus- 6k and data from step 3 to form the final training dataset of 69k responses. -> We exactly follow this to form the final dataset of 72k responses.
MARIO:
1. Greedy decoding using GPT3.5 and GPT-4 each once for MATH&GSM8K, getting two responses for each query, only correct ones are kept -> We follow this but use GPT-4o mini sample responses for each query.
2. Sampling for 2 trials for each problem not correctly answered in step 1 using GPT-4, only correct ones are kept -> We follow this with GPT-4o mini.
3. Manually correcting responses for part of the remaining problems, then tuning Llemma- on it to obtain a synthesis agent for next steps -> this involves human annotation and is not comparable our approach. For simplicity, adopt DeepSeekMath-7B-- as the synthesis agent to align with the -ART datasets.
4. Sampling with 100 trials and keeping up 4 correct responses per problem for the remaining unanswered MATH queries, achieving 93.8% coverage on MATH -> we follow this and achieve 91. 1.% coverage on MATH.
5. Sampling with trial for problems introduced by MetaMath and keeping correct ones -> this step introduces new prompts and would only skew the distribution of responses, any, towards easy queries. We remove this step for simplicity, which would not affect our conclusion.
For GPT-4o mini, use the version of got-4o-mis milomminn--... 22024-07-18 by default.
16

---

B Experimental Setup
B.1 Training Setup
We train all the models using the Transformers library (Wolf et al. 2019).
Sequence Packing;
To efficiennly save computation wasted by padding tokens, we employ sequence packing (Krell et al. 202Tb. We shuffle all samples in each epoch before sequence packing, ensuring that the same semantic sequences are not always in the same computation sequence.
Batch Size: The computation sequence token length is set to 4096, considering that most sequences in the training datasets are shorter than this length. The batch size 64, though there are usually more than 64 samples in batch because computation sequence can pack multiple semantic sequences. We disable gradient accumulation (Lin et al. (018) by default, but when the memory not sufficient, increase the number of gradient accumulation steps and keep other settings unchanged. Specifically, we use 2 gradient accumulation steps when training Llama3-8B on 8 NVIDIA A100 GPUs under our setting.
We use the Adam optimizer (hhag 2018 with the weight decay as O. We linear warmup with warmup step ratio of 0.03 and cosine learning rate scheduler. The maximum learning rates are set as follows: Mistral-7B at lc-5, DeepSeckMath-75 and Llama3-8B at 5e-5, and Llama3-70B at 2e-5, We determine the values by searching through 1e-6, 5 1e-5, 2e-5,, 2 5, 1e-4 according to the MATH performance after training on MMIQC for 1 epoch.
Learning Rate:
The default number of epochs is 3. For MMIQC, we train for epoch following Liu et al. (2024a). For Llama3 models, we train for 1 epoch because preliminary experiments indicate that epoch consistently outperforms epochs.
"Training Epochs:
Prompt Template:
For the prompt template, we use the format following Taori et al. 2023
Prompt Template
Below is instruction that describes task. Write response that appropriately completes the request. Inttnnsttttt Anfquery>\nnan \nnn#= Reaponse: An
Other Details: For efficiency, We utilize various tools / librarres / techniques including:
the DeepSpeed distributed framework (Rasley et al. 2020 with ZeRO (Rajbhandari et al.
2020 stage
gradient checkpointing (Chen al.
2016
torch. compile (Ansel et al.
2024
miixei-preiision training (Micikevicius et al. 2018
BrainFloat16 (Kalamkar et al. 2019)
and TensorFloat32 (NVIDIA 2020
Hardware: For 7B or 8B models, we train on 8 NVIDIA A100 GPUs. For 70B models, we train on 32 NVIDIA A100 GPUs.
Training Time Cost The specific training time cost depends on too many factors to give a precise expression, such model architecture, model size, data content, training algorithm implementation, hardware environment, etc. Here we provide several data points under our setting for reference:
B.2 Synthesis Setup
2023 setting the maximum number of Generation: We utilize the vLLM library Kwon et al.
output tokens as 2048 and adopt top-p sampling with p 0.3 to 1.8 with step of 0.1 by using PeepSeekMath-7B-.o RL to sample answer-correct responses to queries MATH training set. observe the speeds to achieve specified correct answer coverage of different temperatures and find that, for DeepSeek Math-7B-a higher temperatures achieve faster, but 2 1.0 are quite similar and 1.7 cause the output to be nonsense. Besides, we find that higher temperatures produce more diverse responses by visualizing the embedings of response from
0.95. For temperature we search from
17

---

# Samples
Time
Dataset
Model
Hardware
(k)
(hour/epoch)
DART-Math-RRl Hard
585
DeepScckMath-7B
A100 GPUs
3
DART-Math-Hare
585
Mistral-7B
100 GPUs
DART-Math-Hard
585
Llama3-8B
A100 GPUs
3
DART-Math-Hard
585
Llama3-70B
32 A100 GPUs
6
Table 5: Examples of training time cost.
different temperatures to the same query using t-SNE (Van der Maaten & Hinton 2008). Finally, set the temperature specifically tailored for DART.
1.6. This choice should be fair since the temperature search is not
Grading: To judge whether the answers in responses correct not accurately possible, we implement an elaborate answer extraction and judgement pipeline based on regular expressions and SymPy (Meurer et al. 2017) symbolic calculation, which is able to correctly process most mathematical objects such as matrices (vectors), intervals, symbols besides numbers, as well as some special texts like bool expressions, dates and times.
Calculating Fail Rate: For efficiency, we merge DARS- -nns-unnform synthesis and calculating fail rates as mentioned in $3.2 Specifically, we set ki, 192 to synthesize our data pool, and based on all the responses sampled, we calculate fail rate for each query as
# all correct responses fail rate
# all raw responses
which would produce more accurate fail rate values but is not necessary for general algorithm implementations.
B.3 Evaluation Setup
Generation Like BB.2, we the vLLM library, setting the maximum number of output tokens as 2048 and adopting top-p sampling with 0.95. But we use greedy decoding (i.e. set temperature t= 0) for evaluation. Note that there might still be randomness from vLLM implementation despite using greedy decoding, so we run each evaluation 2 with at least 3 random seeds. When evaluating models trained by us, as shown in [.. All SFT & RL. models are evaluated with 0-shot, while base models with few- shot in-context learning (ICL): MATH (4-shot), GSMSK (4-shot), CollegeMath (4-shot), DeepMind Mathematics (4-shot), OlympiadBench-Math (4-shot), TheoremQA (5-shot). For baseline models, prompts in official implementations are used. Specially, the CoT version of Alpaca prompt template is used for WizardMath.
the Alpaca Taori et al. 2023) prompt template consistent with training
Grading We utilize the same pipeline $B.2 by default, except that, for Olympiad use the official implementation of answer correctness judgement component by He et al.) (2024), which utilizing the numerical error range information provided with query, but keep the answer extraction component of ours, because the official implementation fails to extract a non-negligible part of answers, especially for base model ICL.
C Additional Results
C.1 Domain-wise Performance on MATH
We test the domain-wise performance MATH for rejection-tuned models based on Mistral-7B and Llama3-8B. As shown in Table 6 both domain-wine and domain-macro-avvrgg. scores still show DART's significant improvement across all domains.
18

---

MATH Domains
Average
Model
Prob.
Prealg.
Num.
Interm. Alg.
Alg.
Precalc.
Gen.
Micro
Macro
LJama3-888 B-V
34.2
57.8
30.7
20.4
59.6
22.5
29.0
39.7
36.3
DART Metr-Math-LLLee -aath---amaa-888 (Uniform)
34.6
65.7
35.7
25.4
66.6
29.3
32.4
45.3
41.4
DART Math ------aaa----- 8B (Prop2Dift)
38.8
62.9
36.8
26.1
67.3
32.0
39.9
46.6
43,4
Mistral-BB-VE
32.1
56.3
29.6
19.0
58.4
22.2
30.7
38.7
35.5
(Unilorm)
33.8
59.8
35.2
24.4
64.1
28.8
34.2
43.5
40.0
Maath-Msstral (Prop2Didf)
36.1
35.4
45.5
42.3
Table MATH performance all the domains. Macro average assigns equal weights each domain, while micro average assigns equal weights to each query, which the same the phole-benchmar score. The names of the domains are Counting & Probability, Prealgebra, Number Theory, Intermediate Algebra, Algebra, Precalculus, Geometry, respectively. Bold means the best score within the respective base model.
C.2 DART achieves comparable performance with RL
DART is an SFT method, which is usually not comparable with RL method like GRPO used by DeepSeekMath-75
However, even considering comparison with DeepSeekMatt-7B-RRI, we find that sole SFT with DART can produce performance comparable with RI. on DeepSeekMath-7H. as shown by Table
Model
MATH
GSM8K
College
DM
Olympiad
Theorem
AVG
DeepSeekMath-7BB
53.1
88.4
41.3
58.3
18.7
35.9
49.3
MART-Math-DSMath (Uniform)
52.9
88.2
40.1
60.2
21.3
32.5
49.2
(Prop2Difl) 53.6
86.8
40.7
21.7
32.2
49.4
Table 7: Performance by DART and RI. on DeepSeekMath-71 College, DM, Olympiad, Theorem denote the CollegeMath, DeepMind-Matie Mathemattssss,ss, OlympiadBench-Math TheoremQA benchmarks respectively. Bold means the best score within the respective base model.
D Related Work
Data Synthesis: Rejection sampling (Neal. 2003 is a statistical approach used to generate samples from some target distribution that is not directly accessible (c.g., the distribution correct responses to all the queries). model training, this can be used for construting training data and usually implemented in some form of "sampling and filtering". Depending on the task, the supervision signal for filtering can be reward models, ground-trune -tuth answers, answer consistency, e.t.c. (Bai al. 2022 Zelikman et al. 2022 Huang et al.. 2022
Sitection-Sampling-Bee
Dong et al. 2023 Gulcehre et al., [2023] Yuan et al. 2023 Singh et al.] 2023). However, most of previous works sample the same number of candidates for each query, regardless of the query difficulty, unconsciously introducing a bias towards easy queries in the final training data distribution.
DART resolves this issue by explicitly controlling the final distribution with adaptive budget allocation of candidate samples.
Data Construction for Instruction Tuning Data have been seen one of the most critical factor for the performance of instruction tuning. Previous works construct metrics for data selection and construction in diverse ways, such training predictors (Cao et al. 2023 Lu al. 2024 Liu et al. 2024b), prompting LLMs (Chen et al. 2024), gradient-based metrics (Xia et al.] (2024) and heuristics (Li et al., 2023a Ning al.) 2024) But most of them do not consider the final distribution of training data. DART focus on the metric for difficulty and further controls the whole distribution, providing perspective for data selection and construction.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Main claims made in the abstract and introduction can accurately reflect the paper's s contributions to instruction tuning data construction and scope of mathematical reasoning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitationa are diseussed in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully disclose all the information needed to reproduce the main experimental results the paper to the extent that it affects the main claims and conclusions of the paper. See $ and Appendix B for details.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our datasets, models and code are publicly available at htttps://githmn com/hkust-nip/dee daatt
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ Soaede/se/ooeeSuumbsssiiiooiooi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeSbmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary understand the results. See Appendix B for details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments on LLMSs too expensive run for many times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information on the computer resources needed to reproduce the experiments. See Appendix B for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttts::///nnttt http///rrrr....pppuue//e/tttttta
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: We use data from common public mathematical datasets and synthesize data only about mathematics, with little impact on socicty. We do not observe any obviously negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use data from common public mathematical datasets and synthesize data only about mathematics, with low risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code, data and models are well documented and the documentation will be made publicly available alongside the assets following the review period.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "IKhTWGcONb", "reviewer": "Reviewer_dSs8", "summary": "The paper presents an approach to improving the performance of LLMs in mathematical problem-solving. The authors identify that current datasets synthesized using proprietary models like GPT-4, are biased towards easier queries. To address this, they introduce Difficulty-Aware Rejection Tuning (DART), which allocates more trials to difficult queries during data synthesis. This method generates datasets focusing on difficult queries using an open-weight model, DeepSeekMath-7B-RL, without relying on proprietary models. The authors demonstrate that models fine-tuned on DART-Math datasets significantly those fine-tuned on traditional datasets across various mathematical benchmarks, and beat the best baseline by average of roughly 3-4%", "questions": "1. The authors use \"fail rate\" as a metric to measure difficulty. However, has any analysis been performed to measure how good an estimate it is of actual model accuracy?\n2. In line 138, \"we sample till correct responses for each query is proportional to its difficulty score,\" does it mean linearly proportional?\n3. To the best of my knowledge, previous works usually use lower temperatures in the range of 0.7-1. However, the authors found 1.6 to be effective. Do the authors have a comparison of results between using a more standard temperature (e.g., 0.7 or 1) compared to 1.6?\n4. In Line 253, the authors state: \"We hypothesize that this is due to these models’ extensive pretraining on mathematical content.\" Do the authors have more points to substantiate this? For instance, it could be partly due to a slightly weaker or similar model being used to generate synthetic data. Further, the hypothesis: \"This pretraining likely covers most skills that could be learned from the GSM8K and MATH training queries\" may not be correct, since, at least for Llama2-70B, the model capacity should not be a bottleneck to achieving higher scores on MATH (e.g., Qwen models). Can the authors provide a more detailed reasoning behind this hypothesis?", "limitations": "Major Limitations are addressed in paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Technically solid paper with state-of-the-art results.\n- Mostly well-presented and easy to understand.\n- Comprehensive experiments and analysis.\n- Decent impact in improving mathematical capabilities of LLMs, with the authors publicly releasing their dataset.\n- By using an open-weight model, DeepSeekMath-7B-RL, the authors eliminate dependency on proprietary models like GPT-4, making the approach more accessible.", "weaknesses": "1. It is unclear how the hyperparameters of the baseline, VRT (vanilla rejection tuning), were tuned. For instance, as mentioned in Appendix A.2, sampling temperature is searched from 0.3 to 1.7 for DART. Was the same procedure used for VRT? Another caveat is the need for extensive hyperparameter tuning compared to baselines. Were similar extensive procedures for tuning performed for other baselines?\n2. It is unclear if the improved performance of the proposed method is due to difficulty or the topic of the problem. For instance, LEVEL 5 Math problems may have a higher number of geometry questions (or at least their fail rate is higher, resulting in fewer samples in VRT). An analysis of topic-wise performance comparing DART and baseline methods may clarify this.\n\n**Minor Weaknesses: **\n1. It is unclear how much advantage the method would provide in the case of other multi-iteration fine-tuning methods such as STaR and v-STaR. For instance, it is possible that after multiple iterations, VRT performs similarly to DART, since a higher number of samples will be collected from even the hard problems in second or further iterations.\n2. The data synthesis is only done using the DeepSeekMATH-7B model. It is unclear why this model was chosen. Previous methods using VRT-like methods typically use the same model for synthesis and generation. Thus, higher results in smaller models such as Llama-8B may partly be due to the use of stronger models' reasoning chains, making it similar to a distillation method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments! We address your concerns below. \n\n> **Q1**: It is unclear how the hyperparameters of the baseline, VRT, were tuned. For instance ……, sampling temperature is searched from 0.3 to 1.7 for DART.\n\n**A1**: We searched temperature from 0.3 to 1.7  according to accuracies by DeepSeekMath-7B-RL on the MATH training set, where $t=1.6$ is the highest temperature that does not suffer from a significant accuracy drop. $t=1.6$ is applied for both VRT and DART-Math. This choice should be fair since the parameter search is not specifically tailored for DART method.\n\n> **Q2**: It is unclear if the improved performance of the proposed method is due to difficulty or the topic of the problem. … An analysis of topic-wise performance comparing DART and baseline methods may clarify this.\n\n**A2**: Thanks for the advice! Difficulty and topic are naturally correlated. As shown in the following table for two models, both topic-wise and topic-macro-average scores (which assign equal weights to different topics) still show significant improvement on every topic by DART.\n\n| Model | Counting & Probability | Prealgebra | Number Theory | Intermediate Algebra | Algebra | Precalculus | Geometry | Micro Avg. | Macro Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Llama3-8B-VRT | 34.2 | 57.8 | 30.7 | 20.4 | 59.6 | 22.5 | 29.0 | 39.7 | 36.3 |\n| `DART-Math-Llama3-8B` (Uniform) | 34.6 | **65.7** | 35.7 | 25.4 | 66.6 | 29.3 | 32.4 | 45.3 | 41.4 |\n| `DART-Math-Llama3-8B` (Prop2Diff) | **38.8** | 62.9 | **36.8** | **26.1** | **67.3** | **32.0** | **39.9** | **46.6** | **43.4** |\n| Mistral-7B-VRT | 32.1 | 56.3 | 29.6 | 19.0 | 58.4 | 22.2 | 30.7 | 38.7 | 35.5 |\n| `DART-Math-Mistral-7B` (Uniform) | 33.8 | 59.8 | 35.2 | 24.4 | 64.1 | 28.8 | 34.2 | 43.5 | 40.0 |\n| `DART-Math-Mistral-7B` (Prop2Diff) | **36.1** | **61.3** | **35.4** | **26.0** | **65.7** | **31.1** | **40.5** | **45.5** | **42.3** |\n\n> **Q3**: Another caveat is the need for extensive hyperparameter tuning compared to baselines\n\n**A3**: To clarify, we apply the same training hyperparameters to all the experiments on the same base models, thus there is no additional hyperparameter tuning for DART compared to the baselines during training. For data synthesis, the only additional hyperparameters specifically chosen for DART are # responses per query (Line 174-175) and the sampling cap (Line 145), which is mainly decided by the desired final dataset size considering our resource constraints, we did not  “tune” these synthesis hyperparameters according to final performance. \n\n> **Q4**: It is unclear how much advantage the method would provide in the case of other multi-iteration fine-tuning methods\n\n**A4**: This is a good point. DART is compatible with multi-iteration fine-tuning as well and we can use the DART strategy to manipulate data synthesis for each iteration. We leave exploration on this as future work. \n\n> **Q5**: The data synthesis is only done using the DeepSeekMATH-7B model. It is unclear why this model was chosen. \n\n**A5**: The DeepSeekMath-7B-RL model is chosen as the strongest open-source model (though with only 7B size) for mathematical problem-solving. \n\n> **Q6**: Previous methods using VRT-like methods typically use the same model for synthesis and generation. Thus, higher results in smaller models such as Llama-8B may partly be due to the use of stronger models' reasoning chains, making it similar to a distillation method.\n\n**A6**: We agree that our approach is similar to a distillation method, and our goal is to create the best synthetic data for mathematical problem-solving. The mentioned works that use the same model for synthesis represent a different line of work that pursues\" self-improvement,\" while our paper can be viewed as pure data synthesis. This aligns with many existing works in this field that produced SOTA synthetic datasets using GPT-4, a stronger model.\n\n> **Q7**: In line 138, \"we sample till correct responses for each query is proportional to its difficulty score,\" does it mean linearly proportional?\n\n**A7**: Yes, it means linearly proportional. \n\n> **Q8**: Do the authors have a comparison of results between using a more standard temperature (e.g., 0.7 or 1) compared to 1.6?\n\n**A8**: We didn’t compare temperatures in effects on the final performance, and we only compare temperatures in a preliminary stage by evaluating DeepSeekMath-7B-RL, as explained in A1 above. \n\n> **Q9**: In Line 253, the authors state: \"We hypothesize that this is due to these models’ extensive pretraining on mathematical content.\" Do the authors have more points to substantiate this? \n\n**A9**: The hypothesis stems from the understanding that extended large-scale pretraining can reduce the need for meticulously curated SFT datasets. However, we did not rigorously verify this hypothesis, and we agree with the reviewer that \"a slightly weaker or similar model being used to generate synthetic data\" could be the reason as well.  \n\n> **Q10**: the hypothesis: \"This pretraining likely covers most skills that could be learned from the GSM8K and MATH training queries\" may not be correct, since, at least for Llama2-70B, the model capacity should not be a bottleneck to achieving higher scores\n\n**A10**: This is an interesting point. To clarify, we didn't mean model capacity is a bottleneck, instead, we meant the training queries are the bottleneck --  the queries in `DART-Math` are simply the original GSM8K & MATH training queries that limit the scope that the final dataset can generalize to. Therefore, the generalization accuracy could be bottlenecked by the training query scope no matter how we improve the synthetic responses. Our hypothesis is that math-specific pretraining may already cover most skills that could be learned from these queries, and further training with those queries could not provide significant values. A potential future direction could be synthesizing new queries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJoqm7jQlI", "reviewer": "Reviewer_neCQ", "summary": "The paper proposes a rejection sampling pipeline for automatically generating SFT data, emphasizing that harder data requires more trials. The difficulty is heuristically determined using the ratio of incorrect trials for each question. Experiments demonstrate that this method can outperform traditional rejection methods on various math benchmarks.", "questions": "Could you elaborate on how your approach differs from the rejection sampling strategy used in [1]?\n\n[1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The experiments are solid, showing significant improvements over traditional rejection methods.\n\n- The paper is clearly written and easy to follow.", "weaknesses": "The proposed Prop2Diff strategy lacks innovation. Assigning more budget to more complex questions in data synthesis is a common practice. For instance, in [1], which successfully annotated 83.1% of MATH questions, it is evident that harder problems were allocated more budget in rejection sampling. [1] also indicates that fewer and harder data can significantly and efficiently improve performance. The authors should discuss the differences between their approach and the one used in [1] more thoroughly.\n\n[1] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! We address your concerns below:\n\n> **Q1**: Assigning more budget to more complex questions in data synthesis is a common practice. For instance, in [1], which successfully annotated 83.1% of MATH questions\n\n**A1**: First, we note that most recent works in mathematical data synthesis, including nearly all commonly-used open-source datasets, allocate equal budgets to all questions regardless of their complexity [2,3,4,5,6]. While a few works such as the mentioned ToRA try assigning more budget to more complex questions to improve coverage, we highlight that ToRA still results in a bias towards easier queries. This is in stark contrast to our Uniform and Prop2diff strategies that focus on the distribution of responses, rather than merely improving coverage. Please see the Response to Q2 below for a detailed comparison with ToRA. \n\nAdditionally, we would like to point out that 83.1% coverage mentioned from ToRA is not an exceptionally high number -- for comparison, the MetaMathQA-MATH-AnsAug dataset achieves 82.8% of coverage on the MATH training set with evenly allocated budgets yet still admits bias towards easy queries as analyzed in Figure 2 of our submission. Below we show the coverage rate per difficulty level of different approaches. The ToRA-Corpus-16k statistics show that it only covers 68% of the Level 5 MATH queries while `DART-Math` datasets cover 99.6%.\n\n| MATH training set coverage | Total | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |\n| --- | --- | --- | --- | --- | --- | --- |\n| ToRA-Corpus-16k-MATH | 83.1% | 97.7% | 91.6% | 86.5% | 81.3% | 68.0% |\n| MetaMath-MATH-AnsAug | 82.8% | 98.1% | 93.6% | 86.7% | 76.6% | 48.9% |\n| VRT Baseline | 84.9% | 99.6% | 98.2% | 95.2% | 89.8% | 62.9% |\n| `DART-Math-*` | 99.6% | 100% | 100% | 99.9% | 99.7% | 99.1% |\n\n> **Q2**: Could you elaborate on how your approach differs from the rejection sampling strategy used in [1]?\n\n**A2**: Please check the general rebuttal for a summary of difference of our approach from ToRA. The most important difference between our approach and ToRA is how responses are distributed across various queries -- While we adjust the distribution either to be uniform or to favor more difficult queries, ToRA focuses mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries as shown in the results of the general author rebuttal. \n\nHere we supplement more details on how we replicate the ToRA synthesis pipeline to conduct the analysis present in the general author rebuttal. Below we show in the format as \"ToRA's method -> how we adapt similar spirits for a simpler replication\" step by step (we use CoT format rather than tool-integrated reasoning for a fairer comparison with our datasets):\n\n---\n---\n\n1. Greedy decoding once for each problem in MATH&GSM8K with GPT-4, keeping the correct responses -> we follow this with GPT-4o mini.\n2. Sampling for 10 trials for each problem not correctly answered by greedy decoding with GPT-4 and keeping up to 4 correct responses per problem (to form ToRA-Corpus-16k) -> we follow this with GPT-4o mini\n3. Training CodeLlama models on ToRA-Corpus-16K to perform rejection sampling next -> to avoid additional training for a fairer comparison, we use DeepSeekMath-7B-RL to replace the trained CodeLLama models here to align with DART-Math\n    1. Sampling with 64 trials for each problem in MATH&GSK8K with CodeLlama, getting 233k distinct correct responses -> we follow this with DeepSeekMath-7B-RL, getting 733k distinct correct responses \n    2. correcting wrong responses by greedy decoding from the correct preceding portions (costing no more than 64 trials for each problem) with CodeLLaMA-34B, getting 69k corrected responses -> we simplify this by re-sampling another up to 64 trials per problem for all the incorrect responses, getting 225k correct samples.\n    3. Randomly selecting up to 4 correct responses per problem from 3.1&3.2 -> we exactly follow this.\n4. Merge ToRA-Corpus-16k and data from step 3 to form the final training dataset of 69k responses -> we exactly follow this to form the final dataset of 72k responses.\n\n---\n---\n\nWe show the average numbers of responses per problem for different difficulty levels and coverages on the MATH training set in the table of the general rebuttal — distribution-wise we can see that the ToRA pipeline will still produce fewer responses for difficult problems than the easy ones, while **`DART-Math-Hard` produces, for example, 10x more # of responses for MATH Level 5 questions than the GSM8K questions.**\n\n[1] Gou, Zhibin, et al. \"Tora: A tool-integrated reasoning agent for mathematical problem solving.\" ICLR 2024. \n\n[2] Yu, Longhui, et al. \"MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.\" ICLR 2024.\n\n[3] Wang, Ke, et al. \"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning.\" ICLR 2024. \n\n[4] Liu, Haoxiong, et al. \"Augmenting math word problems via iterative question composing.\" Preprint 2024. \n\n[5] Huang, Yiming, et al. \"Key-point-driven data synthesis with its enhancement on mathematical reasoning.\" Preprint 2024. \n\n[6] Tang, Zhengyang, et al. \"MathScale: Scaling Instruction Tuning for Mathematical Reasoning.\" ICML 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSaUVq88kk", "reviewer": "Reviewer_M9p3", "summary": "The paper introduces Difficulty-Aware Rejection Tuning (DART), a novel approach for enhancing the mathematical problem-solving capabilities of large language models (LLMs). Traditional methods often produce datasets biased towards easier queries, limiting the models' ability to learn from challenging examples. DART addresses this by allocating more sampling trials to difficult queries during the data synthesis phase. The authors created two strategies, Uniform and Prop2Diff, to ensure a balanced representation of easy and difficult queries. Using only open-weight models, the authors generated new, smaller datasets that prioritize difficult queries.", "questions": "1. How to be aware of difficulty if not labelled\n2. How to choose $k_u$。\n3. The details of Prop2Diff is missing. How many samples were generated for each difficulty level? What is the equation for generating numbers?", "limitations": "The limitation is mentioned in the conclusion section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The DART method effectively addresses the bias towards easy queries in traditional rejection sampling, which is a significant contribution to the field.\n\n2. The paper provides a thorough analysis of the biases in existing datasets and clearly explains how DART mitigates these issues.\n\n3. The authors plan to make their datasets and models publicly available, contributing valuable resources to the research community.", "weaknesses": "1. The success of DART relies heavily on the ability of models to generate correct responses for difficult queries, which may not always be feasible for extremely challenging problems.\n\n2. While the focus on difficult queries is commendable, the quality of the generated responses for these queries needs to be high to truly benefit the training process. The paper does not provide a detailed analysis of the quality of these responses.\n\n3. The approach's reliance on extensive sampling for difficult queries might pose scalability issues, particularly for very large datasets or models with limited computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments! We address your concerns below. \n\n> **Q1**: The success of DART relies heavily on the ability of models to generate correct responses for difficult queries, which may not always be feasible for extremely challenging problems.\n\n**A1**: This is indeed a limitation for all rejection-sampling-based approaches, including both DART and the baselines. However, at least for the difficulty level of the MATH dataset, in Section 3.1 (Figure 2, Right) we have shown that DeepSeekMath-7B-RL is able to generate correct responses for nearly 100% of the queries in the MATH500 test set when sampling more.  \n\n> **Q2**: While the focus on difficult queries is commendable, the quality of the generated responses for these queries needs to be high to truly benefit the training process. The paper does not provide a detailed analysis of the quality of these responses.\n\nA2: This is a good point. In this work, we just use the final answer to filter responses as in most previous works, while evaluating quality of these responses is non-trivial yet potentially beneficial -- an additional reward model may be needed for assessing their quality as in the Llama3 paper. We leave exploration of this for future work. \n\n> **Q3**: The approach's reliance on extensive sampling for difficult queries might pose scalability issues, particularly for very large datasets or models with limited computational resources.\n\n**A3**: We have conducted quantitative analysis about the synthesis cost in Section 4.3, please check it for details. While our method indeed requires extensive sampling for difficult queries, we highlight that the synthesis cost is one-time and can be amortized by distributing the obtained datasets to a large open-source community to use. Additionally, our synthesis agent is only a 7B-size model, unlike most other works that use GPT-4. Thus, our synthesis cost per example is much cheaper compared to previous works, although we do need to synthesize more examples. \n\n> **Q4**: How to be aware of difficulty if not labelled\n\n**A4**: We sample multiple responses for each query with DeepSeekMath-7B-RL and use the proportion of incorrect responses (i.e., fail rate) as the difficulty score. We have described this in Line 158-166 of the submission.  \n\n> **Q5**: How to choose $k_u$\n\n**A5**:  Generally, according to the scaling curves in Figure 3,  larger $k_u$﻿ (or $k_p$﻿) correlates with higher accuracy. Thus, larger $k_u$﻿ (or $k_p$﻿) is overall preferred. However, the value of $k_u$﻿ (or $k_p$﻿) directly determines the final dataset size, so the choice of $k_u$﻿ (or $k_p$﻿) is primarily based on the desired final dataset size within resource constraints. In this paper, we choose $k_u=40$ and $k_p=192$ to ensure that both `DART-Math-Uniform` and `DART-Math-Hard` datasets end up with \\~590K examples.\n\n> **Q6**: The details of Prop2Diff is missing. How many samples were generated for each difficulty level? What is the equation for generating numbers?\n\n**A6**: As we described in Line 137-139, we compute a difficulty score for each query (see Line 158-166) and the number of correct responses per query is linearly proportional to its difficulty score, where the most difficult query receives $k_p=192$ responses — suppose the difficulty score is $r_i\\in[0,1]$, number of correct responses for a certain query $i$ is $k_p * r_i$. We keep sampling until the designated number of correct responses is collected or a max sampling cap $n_{\\max}$ is reached (Line 145).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uexL0jAiqC", "reviewer": "Reviewer_quxz", "summary": "This paper synthesizes a math reasoning dataset with a designed way of rejection sampling. Many base models show performance improvements on math reasoning tasks after instruction-tuning on this dataset. They promise to release the dataset and models.", "questions": "1. What’s the query coverage ratio on MATH training set constructed by Prop2Diff?\n2. Any figure or statistics to show the difficulty distribution of your DART dataset? \n3. Any case studies to show the generated responses of your DART dataset? How do you extract answers in the raw response? Responded texts from the LLM are quite likely not to follow your instruction as you apply such a high temperature in sampling process. It’s not likely that simple pipelines, such as regular expressions can achieve this.", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Their curated dataset achieves relatively good instructing-tuning performance with least data amount compared to other baselines. The dataset will be released.", "weaknesses": "1. The proposed sampling technique is trivial and incremental, when comparing with previous works, e.g., the uniform method is used in ToRA, and the prop2diff method is used in MARIO.\n2. There’s little improvement or even performance drop when tuning Mistral-7B and DeepSeekMath-7B \ncompared to other baselines. As mentioned in the analysis section, this dataset is somehow replaceable by math-specific continual pre-training + supervised fine-tuning (SFT).\n3. The major concern is that even the paper claims the proposed dataset is smaller, however, the LLM used to synthesize the smaller dataset is `DeepSeekMath-7B-RL`, which is trained on a larger SFT dataset. An alternative and reasonable response generation method should be leveraging the `DeepSeekMath-7B-Base` with proper prompting, as `DeepSeekMath-7B-Base` has not been supervised fine-tuned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! We address your concerns below. \n\n> **Q1**: The proposed sampling technique is trivial and incremental …… the uniform method is used in ToRA, and the prop2diff method is used in MARIO.\n\n**A1**: We respectfully disagree with the reviewer. Both ToRA and MARIO are distinct from our approach on how responses are distributed across various queries. While we adjust the distribution either to be uniform or to favor more difficult queries, ToRA&MARIO focus mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries as we show quantitatively in the general author rebuttal. Please check the general author rebuttal for a summary of our point on this concern. \n\nHere we supplement more details on how we replicate the MARIO synthesis pipeline to conduct the analysis present in the general author rebuttal (the ToRA synthesis pipeline is described in response A2 to Reviewer neCQ). Below we describe in the format as “how MARIO synthesizes data -> how we replicate a simplified version of it” step by step:\n\n---\n---\n\n1. Greedy decoding using GPT3.5 and GPT-4 each once for MATH&GSM8K, getting two responses for each query, only correct ones are kept -> We follow this but use GPT-4o mini to sample two responses for each query.\n2. Sampling for 2 trials for each problem not correctly answered in 1. using GPT-4, only correct ones are kept -> We follow this with GPT-4o mini.\n3. Manually correcting responses for part of the remaining problems, then tuning Llemma-34B on it to obtain a synthesis agent for next steps ->  this involves human annotation and is not comparable to our approach.  For simplicity, we adopt DeepSeekMath-7B-RL as the synthesis agent to align with DART-Math.\n4. Sampling with 100 trials and keeping up to 4 correct responses per problem for the remaining unanswered MATH queries, achieving 93.8% coverage on MATH -> we follow this and achieve 91.3% coverage on MATH.\n5. Sampling with 1 trial for new problems introduced by MetaMath and keeping correct ones -> this step introduces new prompts and would only skew the distribution of responses, if any, towards easy queries. We remove this step for simplicity, which would not affect our conclusion. \n\n---\n---\n\nWhile MARIO performs more sampling trials for difficult problems, it mainly uses coverage rate as the criterion, and a high coverage rate can be obtained while the distribution is still biased towards easy queries. \n\nWe show the average # of responses per problem for different difficulty levels in the table of the general rebuttal. Our replicated ToRA and MARIO datasets obtain a similar MATH coverage ratio and total size to their original ones — While the absolute # of responses is not directly comparable between different methods, distribution-wise we can see that ToRA/MARIO still produce fewer responses for difficult problems than the easy ones. **This contrasts sharply with `DART-MATH-Hard`, which produces, for example, 10x more responses for the MATH Level 5 queries than for the GSM8K queries.**\n\n> **Q2**: There’s little improvement or even performance drop when tuning DeepSeekMath-7B\n\n**A2**: While an improvement over the VRT baseline is smaller for Llama3-70B and DeepSeekMath-7B, we note that our results are already much better than the ones from the best open-source datasets (around 4-5 points better on average) on all 4 models. Thus the release of our datasets themselves is a significant contribution to the open-source community -- **`DART-Math` is the SotA, public CoT datasets for math problem solving and much better than existing datasets across all the assessed models.**   \n\n> **Q3**: this dataset is somehow replaceable by math-specific continual pre-training + SFT\n\n**A3**: This may be true, but we don't think it is a major issue. Strictly speaking, we suspect most carefully curated math SFT datasets are somehow replaceable by extensive continual pretraining + simpler SFT, yet this fact does not diminish the significance of these SFT datasets -- Continual pretraining typically occurs on a much larger scale, for instance, DeepSeekMath involves continual pretraining on 150B math-specific tokens, whereas `DART-Math` contains only \\~0.2-0.3B tokens. Given this scale, the efficiency provided by SFT datasets remains crucial and non-replaceable for most developers.\n\n> **Q4**: even the paper claims the proposed dataset is smaller, however, the LLM used to synthesize the smaller dataset is `DeepSeekMath-7B-RL`, which is trained on a larger SFT dataset\n\n**A4**: We don’t think using an instructed model is a major issue as we aim to create the best synthetic SFT dataset. This setting aligns with most related works that use GPT-4, an aligned model. When we claim the proposed dataset is smaller, we suggest the reviewer note that our baselines all use GPT-4/3.5 to generate data; these models are hypothetically trained on an even larger SFT&RL dataset than DeepSeekMath-7B-RL.\n\n> **Q5**: What’s the query coverage ratio on MATH training set constructed by Prop2Diff?\n\n**A5**: 99.6%\n\n> **Q6**: Any figure or statistics to show the difficulty distribution of your DART dataset?\n\n**A6**: Figure 1 (Right).\n\n> **Q7**: Any case studies to show the generated responses of your DART dataset? How do you extract answers in the raw response? \n\n**A7**: The answers are mostly extractable after we improve existing regular expression implementation to be more comprehensive. Please refer to Table 1 in the PDF of the general rebuttal for several cases of responses for the hardest MATH problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the insightful comments! While we address most concerns in the individual rebuttals, here In the general rebuttal we would like to clarify the difference between our approach and ToRA [1] / MARIO [2], a concern raised by Reviewer neCQ and Reviewer quxz. 

The most important difference between our approach and ToRA/MARIO is how responses are distributed across various queries -- While we adjust the distribution either to be uniform or to favor more difficult queries, **ToRA/MARIO focus mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries** (as we show below). As demonstrated in Figure 4 (Left) of our submission, a high coverage rate (VRT+Cover) alone does not guarantee superior performance. Since both ToRA and MARIO have not released their datasets, we replicate a version of their synthesis pipelines that is comparable with DART to illustrate that the resultant datasets still bias towards easier queries.

We detail our specific replication process for ToRA and MARIO in the individual rebuttals to Reviewer neCQ and Reviewer quxz respectively. Below we show the average number of responses per problem for different difficulty levels below of our replicated ToRA and MARIO datasets and the DART-Math datasets. Our replicated ToRA and MARIO end up with a similar MATH coverage ratio and total size to their original ones. While the absolute number of responses is not directly comparable between different methods, distribution-wise we can see that ToRA/MARIO still produce fewer responses for difficult problems than the easy ones. This contrasts sharply with `DART-MATH-Hard`, which produces, for example, 10x more responses for the MATH Level 5 queries than for the GSM8K queries. 

|  | GSM8K | MATH Level 1 | MATH Level 2 | MATH Level 3 | MATH Level 4 | MATH Level 5 | Our Size (Original Size in Their Papers) | MATH/Train Coverage |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ToRA | 5.03 | 5.01 | 4.99 | 4.95 | 4.77 | 3.84 | 72k (69k) | 93.4% |
| MARIO | 2.02 | 2.01 | 1.98 | 1.94 | 1.89 | 1.57 | 29k (29k) | 91.3%  |
| `DART-Math-Uniform` | 39.93 | 40.00 | 40.00 | 39.80 | 39.54 | 37.14 | 585k | 99.6% |
| `DART-Math-Hard` | 8.49 | 14.28 | 33.52 | 54.94 | 79.59 | 107.06 | 590k | 99.6% |

We hope the response above clarifies the difference between ToRA/MARIO and our approach.

[1] Gou, Zhibin, et al. "Tora: A tool-integrated reasoning agent for mathematical problem solving." ICLR 2024. 

[2] Liao, Minpeng, et al. "MARIO: MAth Reasoning with code Interpreter Output--A Reproducible Pipeline." arXiv 2024.

Meta-review: The paper is about mathematical reasoning. First, the authors find that there are synthetic datasets synthesized by general LLMs like GPT4 that result in a dataset biased towards easier questions. In this paper, the authors present a different approach based on DeepSeek-Math that results in a more balanced dataset with difficult questions well represented. 

The paper received a 3, 4, 4, 7, but the 3 increased to 4, 4 increased to 5, and 7 reduced the score to 6 based on other discussion. So, with a 4-4-5-6, the paper is quite borderline. There are two main criticisms of the paper. First, as one of the critical reviewers pointed out, and did make the highly positive reviewer go milder... concerns about a specific evlauation. Basically, many other papers used general models like GPT4 which are weaker in mathematical reasoning for dataset geneartion. This paper's methodology, when used over same models, also results in a weaker synthetic dataset. On the other hand, when DeepSeek-Math-RL is used to generate the dataset, the dataset indeed is much better than before, but it is only better than other other GPT4-based datasets. When comparing this dataset, the authors fine-tune DeepSeek-Math-"Base" model on this dataset, the performance is similar to (or sometimes worse than) the original DeepSeek-Math-RL. In other words, if we already had DeepSeek-Math-RL model at our disposal (since it was used for dataset generation), it is not clear why not use this model directly and unsupervised, and why fine-tune another model to reach this model's performance using their data.

The authors, in response, argue that "Our primary goal is not to create a math model for direct use, but to develop better data synthesis methods — the roles of data synthesis and the synthetic data are not replaceable by “directly use the open-sourced DeepSeek-RL”, for example, imagine someone wants to boost the math ability of Mistral-7B during post-training while still keeping it as a generic model, they can utilize our approach to synthesize data from another math-specific model, and incorporate the data together with other SFT data as commonly practiced nowadays, but directly using DeepSeek-RL does not fulfill the goal." 

As a meta-reviewer, I understand both points of view, and don't side with either. A better method for dataset generation is always a good idea. But, is the new dataset better because of a stronger LLM generating the data (DeepSeek-Math-RL) or a better algorithm is not clear. For a rigorous study, evaluation could have been done by using different LLMs to generate data, and different base models for fine tuning, creating a stronger understanding of values of various components. At the same time, creating a stronger dataset is indeed valuable for the reasons authors point out -- because it can help train all models, and not just rely on one model.

The other criticism of reviewers is wrt comparison of the algorithm vs MARIO and ToRA -- two other approaches for dataset creation. The reviewer believes that the technical novelty over these two is incremental. In response, authors point out that "ToRA&MARIO focus mainly on improving coverage without managing the distribution explicitly, leading to datasets that may still bias towards easier queries as we show quantitatively". The authors show with detailed experimental results that ToRA/MARIO produce fewer responses for difficult problems than the easy ones. I believe this experiment sufficiently addressed this concern.

Finally, there are some concerns about whether the model leads to "reward hacking", since the paper uses the final answer to filter responses as in most previous works. The reviewer considers it a super important phenomena strong enough to reject the paper, but I do not think so. This is true for all papers that use this kind of synthetic data generation, and as long as performance improves, this is still valuable to study. And the reviewer also pointed out the high pass@k accuracy for which I am comfortable with the authors answer.

To summarize, the only one criticism that continues to linger in my mind, is the use of different LLMs for data generation and training and comparing it to other datasets generated by a different LLM. They are consistently comparing their method with other open-source datasets created with a different model. A more rigorous experimental study would have made the paper a clearer accept. Still, the method does show improvements compared to vanilla rejection tuning, which is commonly used. With what the authors have shown in experiments, the scores are consistently better across settings. So, I will say that the paper is a borderline accept -- it can be taken in if there is space... but it would not be wrong, if the paper is rejected and the authors are asked to make the evaluation stronger.
Decision: Accept (poster)