AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
Authors: Woosung Koh
Wonbeen Oh
Jaein Jang
MinHyung Lee
Hyeongjin Kim
Ah Yeon Kim
Joonkee Kim
Junghyun Lee
Taehyeon Kim
Se-Young Yun
Venue: NeurIPS 2025 poster ()
Keywords: Language Models
Reasoning
Data Sampling
Training Efficiency
Self-Taught Reasoner
Post-training
Primary area: deep_learning
TL;DR: AdaSTaR enhances STaR by using adaptive sampling for diversity and curriculum to reduce training data imbalance, achieving best accuracy across six benchmarks while reducing training FLOPs by 58.6%.

Abstract: Self-Taught Reasoners (STaR), synonymously known as Rejection sampling Fine-Tuning (RFT), is an integral part of the training pipeline of self-improving reasoning Language Models (LMs). 
The self-improving mechanism often employs random observation (data) sampling. 
However, this results in trained observation imbalance; inefficiently over-training on solved examples while under-training on challenging ones. 
In response, we introduce Adaptive STaR (AdaSTaR), a novel algorithm that rectifies this by integrating two adaptive sampling principles: (1) Adaptive Sampling for Diversity: promoting balanced training across observations, and (2) Adaptive Sampling for Curriculum: dynamically adjusting data difficulty to match the model's evolving strength. Across six benchmarks, AdaSTaR achieves best test accuracy in all instances (6/6) and reduces training FLOPs by an average of 58.6\% against an extensive list of baselines. These improvements in performance and efficiency generalize to different pre-trained LMs and larger models, paving the way for more efficient and effective self-improving LMs.

AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
Woosung Koh!.3, Wonneen Oh Jaein Jang', MinHyung Lee², Hyeongjin Kim², Ah Yeon Kim Joonkee Kim², Junghyun Lee', Taehyeon Kim²; --uue- Yun
"KAIST AI, 2L.G AI Research, Yonnei University
freiss. freiss.koh,yee yunseyoun (kkasssstnnnkaiss........ @kkisst...... kr, krntaelyyonn11@@mail.coom
Abstract
Self-Taught Reasoners (STaR), synonymously known as Rejection sampling Fine- Tuning (RFT), is integral part of the training pipeline of self- imfroviig reasoning Language Models (LMs). The self-improving mechanism often employs random observation (data) sampling. However, this results in trained observation imbalance; inefficiently over-training on solved examples while under-training on challenging In response, we introduce Adaptive STaR (AdaSTaR), a novel algorithm that rectifies this by integrating two adaptive sampling principles: (1) Adaptive Sampling for Diversity: promoting balanced training across observations, and (2) Adaptive Sampling for Curriculum: dynamically adjusting data difficulty to match the model's evolving strength. Across six benchmarks, AdaSTaR achieves best test accuracy in all instances (6/6) and reduces training FLOPs by average of 58. 58.6% against an extensive list of baselines. These improvements in performance and efficiency generalize to different pre-trained LMs and larger models, paving the way for more efficient and effective self-improving LMs..
github.com/rriss-ohh//llllalaaa
Introduction
Language models (LMs) are demonstrating remark- able emergent abilities across diverse cognitive
Average Test Set Accuracy 1%.
tasks such as mathematical reasoning (Yao et al.,
4
4
A
2023; Chen and Li, 2024; Brown et al., 2024).
code generation (Sun al., 2024; Research, 2025),
and commonsense reasoning (Qwen Team, 2023;
Method
Google, 2023). Although LMs acquire founda- tional reasoning capabilities from large-scale pre- training and supervised finetuning (SFT), gener- ating high-quality, explicit reasoning steps, often called Chains-of-Thought (CoT) (Wei et al., 2022a, 2023, 2022b; Wang et al., 2023a), typically requires costly human annotation (Lightman al., 2024: Havrilla et al., 2024: Zelikman et al., 2024). Cre- ating such datasets expensive and scales poorly, presenting a critical bottleneck as tasks increase in complexity. This challenge motivates the de-
8-STAR" iZeng 2025
Reg (Singh al. 2024)
Base Model
55
Owen
15
15
Average Log
Figure Average test accuracy and FLOPs across six datasets for Llama 3.2 3B and three datasets for Qwen 2.5 5 Results consistently extend to Gemma 7B well. *We use out- come verification on B-STaR for fair compar- ison. Thus, the implementation with process verification may perform significantly better.
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Dataset
n-Shot CoT
LM
{(xi, yıl y;)
Query
Parameter Update
Observation i and CoT 6 Adaptive Sampling
{(x,, (xx)
Vi)
Verifier
CK SK Self-generated CoT answer
Win Statistic
Correct Dataset
Reward
Figure 2:
High-level schematic diagram of AdaSTaR. Other STaR-like approaches are equivalent to this diagram, excluding the win statistic w; computation and the Adaptive Sampling module.
velopment of methods that improve LM reasoning
without relying on extensive human annotation.
Self- relfoomproeeentt mechanisms, such Self-
Taught Reasoners (STaR; Zelikman al., 2022), also referred as ceeecctinnssapplee Fine-Tuning (RFT; Yuan et al., 2023; Singh et al., 2024), offer promising alternative. The core idea behind STaR to enable the LM iteratively improve itself: the model generates CoTs, verifies the final answer against ground truth, and fine-tunes on CoTs that yield correct answers. This iterative inference, verify, and train cycle allows L.Ss to generate their own training data, circumventing the need for human-annotated CoTss.
However, while reducing annotation costs, the standard STaR framework, which relies random data sampling, suffers from inefficiencies and learning challenges, The random sampling often leads to training data imbalance: the model wastes compute repeatedly re-training on examples can already solve, while potentially under- under-sampling more challenging examples where learning is most needed (Singh et al., 2024). This imbalance results in inefficient use of training compute and contributes to STaR's ' significantly slower convergence compared to standard SFT (see Fig. 5 in Appendix ŞA).
Furthermore, STaR's reliance outcome verification (checking only the final answer) means it can inadvertently train on flawed or suboptimal CoTss that happen to reach the correct answer (Kawabata and Sugawara, 2024; Lee al., 2025). Reinforcing these "false positives" can degrade the model's underlying reasoning capabilities. While Process Reward Models (PRM; Lightman et al., 2024; Zeng al., 2025) that assess the CoTs can mitigate this, PRMs require their own significant annotation and computational overhead (Lu et al., 2024; Setlur et al., 2025). We therefore view PRMs as an orthogonal approach. Consequently, a key challenge in STaR-based ceff-improvement is balancing the exposure to diverse problem difficulties with the need to maintain training data quality, as sampling harder examples more likely yield noisy or incorrect CoTs. This leads to a research question: How can STaR achieve efficient and effective seff-improvement by balancing diverse learning exposure while maintaining the quality of self self-generated CoTs?
Contribution. We propose Adaptive STaR (AdaSTaR), a novel method that integrates adaptive sampling into the STaR training loop. AdaSTaR implements two core intuitions: (1) Adaptive Sampling for Diversity: prioritizing under-trained examples to ensure balanced learning: and (2) Adaptive Sampling for Curriculum: regularizing the system to sample easier data when the model weaker early on. We empirically validate the effectiveness and efficiency of AdaSTaR through experiments across six reasoning datasets and an extensive list of baselines. AdaSTaR consistently improves both performance and computational efficiency. Remarkably, AdaSTaR not only achieves the highest test accuracy across all 6/6 benchmarks, but also simultaneously reduces the required training compute (FLOPs) by an average of 58.6 % compared to the strongest accuracy baseline (see Fig. l). These performance and efficiency gains generalize other pre-trained LMs and larger model size which we discuss further later.
Related Work. Although many works build on STaR, none, to our knowledge, target improving efficiency. Subsequent works improve performance at significant compute cost; AdaSTaR is comple- mentary, improving scalability and accessibility. V-STaR (Hosseini et al., 2024) adds a verifier LM to improve inference-time performance through best-of-N sampling (Snell et al., 2025). Iterative Reasoning Preference Optimization (Pang al., 2024) incorporates Direct Preference Optimization (Rafailov et al., 2023) term in its objective: to curate preference pairs, it increases CoT samples from

---

= in STaR to K = 30. B-STaR (Zeng et al., 2025) enhances LM exploration for more diverse reasoning, and trains a separate process reward model (Uesato et al., 2022; Lightman et al., 2024) for finer-grained verification, Lean-STaf (Lin et al., 2025) employs the Lean theorem prover (De Moura et al., 2015) and a frontier LM (GPT-4) to extend STaR to mathematical theorem proving.
Reinforcement Learning (RL) offers a parallel approach to enhance LM reasoning. also leveraging an iterative process. RL's reward-basee objective often yields long-CoTs (Shao et al., 2024; DeepSeek- AI, 2025; Liu et al., 2025b; Yu et al., 2025b; Sui et al., 2025; Kimi Team, 2025; Liu et al., 2025a; Yu et al., 2025a), unlike the short-CoT's (NVIDIA. 2025) typical of STaR-style SFT. While the significantly larger token generation size of RL-based Lon-CooTss result in top performers, integrating STaR's SFT remain a salient part of the training pipeline (Sui al., 2025). For instance, Kimi k1.5 (Kimi Team, 2025), a representative reasoning model, utilizes STaR to expand their primary SFT dataset. To address the difficult, mixed -axeedlanguage, and overly long CoT's, DeepSeek-RI (DeepSeek- AI, 2025) and Kimi k1.5 incorporate a STaR stage. Finally, DeepSeek-GRM (Liu et al., 2025b), a generalist reward model, also adopts modified STaR as its training's first stage. While these RL-based advancements are significant, our work concentrates enhancing the STaR stage.
2 Preliminary and Motivation
2.1 Preliminary: Self-Taught Reasoner (STaR) and its Variants
Let oi denote LM (Vaswani et al., 2017) parameterized by 0 at iteration We given supervised dataset D {(2), 3e) m Following Wei al. (2022b), each task is represented as (r, c, y). where x E X' is the query (input), E C is the CoT reasoning step(s), and y E y is the final answer. Since ground-trute CoTs C are unavailable, STaR aims to generate appropriate e to improve generalization. To achieve this, Ti generates (2z, (t) conditioned on fixed few-shot CoT exemplars e = {(Ie,, Ce, Ve) However, as no ground truth G; is available, we require sampling and verification. Given the supervised dataset, rule-based verifier defines reward signal r :== I(y: = ût), where I[(-) is the indicator function.
Let K E N denote the number of CoT traces sampled as follows (Fig. 2, blue). For the first E (1,2,... K). each observation is sampled once via (C,, Us) + T/ (e, xz). If = 1, it is accepted, and if r 0, it is resampled using rationalization (Zelikman et al., 2022): (e, @ v:), where the ground truth y is concatenated. In some extensions of STaR, K > samples are drawn without rationalization (Singh et al., 2024; Hosseini et al., 2024; Pang al., 2024; Zeng et al., 2025; Lin et al., 2025).
Correct samples D := ( é, M.l = út} are re-random-samen sampled down to match the per-iteration batch size 8 = o² B, then used for negative log- lilkeiehehodd (NLL) learning. Here, the step size o² is the number of parameter updates per iteration Here all superscript indicates iteration, not numerical exponent operation. Initial 3 = 40 8 = 320 as presented in the original implementation (Zelikman et al., 2022). 3 rises over time as we follow fe+11 = 1.2((3)) in the original implementation. However, alternative STaR-based approaches (Hosseini et al., 2024; Pang et al., 2024; Zeng et al., 2025; Lin et al., 2025; Peng al., 2025) remove this pre-determined 8t, and instead set 3r to |D,l-
Post gradient updates, Tá transitions to T 1+1 (Fig. 2, purple). Two inter-iteration strategies exist across STaR-based methods: (1) resetting: always retrain from the base model: T +- Train(m DJ (Zelikman et al., 2022; Hosseini et al., 2024; Singh et al., 2024); (2) accumulating: incrementally fine-tune from the previous model: x +- Train (T/j, D4)) (Pang et al., 2024; Zeng et al., 2025; Lin et al., 2025; Peng et al., 2025).
2.2 Motivation: Need for Adaptive Data Sampling
STaR's data sampling induces persistent inefficient imbalance in training data. A key finding is that STaR's sampling strategy leads to some observations being over-trained while others are under-trained. This training frequency imbalance is empirically illustrated in Fig. 3a. The pattern of variance in observation training frequency is persistent across all datasets examined (see Appendix for all visualizations). As the filtered set Dí, consists exclusively of observations for which the LM correctly produced ÿ., high variance naturally arises in how often each distinct observation i is
3

---

Percentage Wrong CoT (%)
55
53
Number of Observation:
350
50
STAR
AdaD
300
Under-Trained
Over-Trained
40-
37
250
-00
28
150
20-
100
20-
10 12
ARC.C
SVAMP
GSMBK
Accumulated Trained Frequency
Dataset
(a) Distribution of frequency trained of each observation & in iterations to 10; in ARC-C. Figure 3: Empirical motivation for the need for adaptive sampling of diverse observations (a), regularized with curriculum learning (b).
(b) Percentage of wrong CoT when the answer is correct
(2) û.) for AdaD and STaR-Acc.
trained. Consequently, more challenging observations (left tail of Fig. 3a) are included in D less frequently and become under-trained, whereas easier ones (right tail) are over-represented and thus over-trained. In this example, challenging observations are trained 1-2 times, while easier ones are trained 10-13 times. This results in inefficient compute usage, as resources are repeatedly spent on observations that the model TT already solves reliably. This situation mirrors the motivation for early stopping in LM training, which aims to avoid overfitting to already-solved data (Caruana et al., 2000; Kaplan et al., 2020; Hernandez al., 2021).
We further examine whether observations initially under- or over-trained remain in these respective regimes over iterations. Empirically, even after three iterations (t + 3), averaging across six datasets, 72.4% observations initially in the first quartile (Q1) training frequency remain in QI, and 91.2% of observations from the fourth quartile (Q4) remain in Q4. Detailed visualizations are available Appendix $ B. This suggests the phenomenon is chronic and does not self-alleviate without intervention.
Prioritizing harder examples for diversity elevates false positives, motivating curriculum- based regularization. However, encouraging training diversity by biasing sampling toward harder observations (left tail of Fig. 3a) can increase false positives. False positives are defined as cases where the predicted answer
correct but the generated CoT flawed (Singh et al., 2024; Kawabata and Sugawara, 2024; Lee et al., 2025). We empirically observe that sampling more challenging observations leads to poorer quality CoTs.
Following Wei et al. (2025) and Lee et al. (2025), we leverage the strongest available teacher model (Ho al., 2023) (GPT 40) to annotate false positives. We compare method encouraging diversity by sampling challenging observations (AdaD) against its baseline, STaR-ACK AdaD and STaR-Ace Acc are formally introduced
3 and 4.1. For each method, 100 observations are randomly sampled (without replacement) from D for CoT annotation. The precise iteration & for both methods is chosen by taking t = min(Bestlee (eBeetttr((Ada)) Bestleer(STaR-ACSS) stopped iteration. Further details and qualitative example are provided in Appendix C.
where Bestlter(-) is the early-
Fig. 3b illustrates that inducing increased training diversity degrade CoT quality, measured by the rate of false positives across four datasets. On average, sampling more diverse and challenging observations lead 9% increase in false positives. Hence, we propose to regularize for model strength to reduce the adverse effects of sampling diverse and challenging observations. To this end, our observation sampling algorithm adopts curriculum learning style approach (Xu et al., 2020; Wettig et al., 2024).
Method: AdaSTaR
This section presents AdaSTaR, an adaptive sampling algorithm designed address the problems highlighted in $ 2.2. Alg. presents the pseudocode, where lines unique to AdaSTaR are highlighted in green; the remaining lines follow standard STaR conventions. AdaSTaR incorporates two mechanisms: Adaptive Data Sampling for Diversity (AdaD) and Adaptive Data Sampling for Curriculum (AdaC).

---

3.1 Adaptive Data Sampling for Diversity
Diversity Statistic. We track two statistics for each observation i: the last iteration it was sampled, f, E No. and win statistic, w, E [0. 1]. Prioritizing observations with smaller t, values directly promotes sampling diversity. We use the last sampled iteration rather than the last trained iteration because prioritizing based training can cause the system to repeatedly attempt difficult examples cannot yet solve, particularly when the model is weak, early in training. Among observations with identical f, values, we prioritize those deemed more difficult. This approach is reminiscent of difficulty-awann aware methods successful in various machine learning scenarios, such as contrastive learning (Robinson et al., 2021), active learning (Xie et al., 2021), and dataset pruning (Zheng et al., 2023; Maharana et al.. 2024; Cho et al., 2025). A key contribution of AdaSTaR is its computationally efficient method for estimating observation difficulty within STaR systems.
We estimate difficulty using the win statistic wy, which is computed based on model perfor- mance at E, (the last iteration i was sampled):
Algorithm 1: AdaSTaR
Input: D, %g /* AdaD ($3.1; 1; lines 1-14)
w; = = : K E Ily, üs]. where ÿ: is from - a;). This represents the proportion of correct answers out of K CoT samples gen- erated at iteration &,. Next, we elaborate on why this is a sensible proxy for difficulty.
i + dict(i: ; 4 = 0} i=
w t diet(i: w; = 0] f=
init HieMinHeap(D,,... for iteration / 1,... do
D4, t 0, m 0
At each iteration 1, we want our model to max- imize P := P(y: Ús rx())) for all i's. As the model is fitted with likelihood maximiza- tion (Fisher, 1922), we expect p 2 p for any sampled at iteration t. It is therefore sensible to prioritize observations with the low- est pt values, as they require more sampling and can be interpreted as more difficult at iter-
w tmp + -icttfi w
tmp 0)
while
| i +- WHieMinHeapppeekkunee peek_neet m. 4- m 11
< 3r do
10
for sample k ... K do
11
(cs, (C) + T m x;):
12
W Imp + h tmp
+ = vıl:
ation t. It now remains to approximate p¹. A di- rect Monte Carlo estimate with K samples gives pf R p := & >x= I[y: = 2 < T (2:)]. However, computing this for every i at every iteration t requires K forward passes per ob- servation, which is computationally prohibitive. 15
if 9: = y; then
D + D U f(z C, K:);
/* AdaC (53.2; 2; lines 15-19) */ (+1 a, m <- Train (T). D()): for I,... |ma do
Instead, we reuse the most recent estimate of. 16
The win static computation I, induces no (run- 17
i HieMinHeap.pop:
time) compute overhead as the K samples are 18 an inherent part of the existing STaR system. Re- 19
f f t, w; w .mp
HieMinheeap,pushh(i L, w));
calling that & < refers to the last iteration in
which i was sampled, pf: the most recently available approximation to p.. Moreover, as we are priority-sorting with respect to i, we can expect that / - E, is not too large, i.e., f1 ~ p/.
Implementation. As input, AdaSTaR takes the original dataset DD base model T-1 and n-Shot CoT examplar e. For all observations, the statistics are initialized to 0 (lines 1, 2). In line 3, we utilize Cormen et al. (2022)'s Hierarchical Min Heap HieMinHeap to order the observations via the two statistics as follows: for two observations i,j E HieMinHeap(--
-i==iAu; wy)
i pooped/popped before j
sampled betiore
and j are last sampled at the same f. but is more difficult
For each iteration I, new empty Dt is initialized (line 5), which is used for the training at the end (line 15). We also initialize : 0. which counts the number of sampled observations (line 9), and tmp,, a dictionary of computed win-rates at iteration (line 12). The while loop sequentially samples i from HieMinHeap. then updates the win-rate w, tup over K samples of CoT-answer pairs (C., 0:) (lines 11-12) and adds (xia, C, fx) to Dt, if ÿj is correct (lines 13-14).
5

---

Remark 1 (Non-excessive sampling in line 7). The while loop terminates once [D+ 2 8 This
avoids overhead from exhaustively sampling all observations before pruning to 3", a practice in some prior STaR implementations (see Appendix D for further discussion).
3.2 Adaptive Data Sampling for Curriculum
To avoid over-sampling challenging observations (1 E.. + w.)) when the model is weak, we regularize AdaD using an adaptive curriculum. A natural approach is to incorporate curriculum learning (Hacohen and Weinshall, 2019; Kong et al., 2021) by mixing easier observations when the model is weak, then gradually reducing their ratio as it improves. This strategy aligns with curriculum learning for LM training (Pouransari et al., 2024; Li et al., 2024; Zhao al., 2025) and is supported by data selection literature showing that combining easy and hard samples yields better outcomes than selecting only hard samples (Zheng et al., 2023; Maharana et al., 2024; Cho et al., 2025).
We use the training accuracy a E [0, 1] from the current iteration / as a proxy for model strength (Alg. 1, line 15). When a is low (indicating a weaker model), a relatively easier mix of observations should be prioritized for subsequent sampling. This regularization is automatically phased out as a increases with training. Similar to tracking & and wi, using a introduces no additional computational overhead, as the training step (which yields a) is integral to the system. This explains our choice over, for instance validation set accuracy (not used in final evaluation); while potentially a more robust measures of generalization, these would require additional inference passes not intrinsic to the STaR loop.
Implementation. The curriculum component (Alg. lines 15-19) implements curriculum by adjusting statistic-uddate frequency based on model strength Of the m sampled observations per iteration, only the !ma?) highest-priority ones are popped: their statistics are updated (f, + t, uy < w (RP))) before reinsertion." ? Consequently, when a is low (model is weak), larger proportion of the m considered observations are not updated. These non-updated observations retain their existing statistics, increasing their re-selection likelihood in the subsequent iteration. This implicitly mixes easy observations when is low, avoiding the cost of explicitly identifying and mixing them.
4 Experiments
4.1 Experimental Protocol
Setup. We conduct our main experiments with Llama 3.2 3B (Llama Team, 2024). We also evalu- ate using Qwen 2.5 3B (Qwen Team, 2024) and Gemma 7B (Gemma Team, 2024) demonstrate the generality of our method across different model families. All base models are pre- pre-trained-oug models. For fairness, we optimize hyperparameters using the original STaR and apply them consistently across all methods. Further experimental details are provided in Appendix $ E.
Datasets. We attempt to get a wide coverage of reasoning tasks by using six well well-known datasets. We use the A12 Reasoning Challenge's Challenge set (ARC-C; Clark et al., 2018) for scientific reasoning, CommonsenseQA (CQA; Talmor et al., 2019) for commonsense reasoning, and CLadder 1.5 (Jin al., 2023) for causal reasoning. For natural language inference reasoning we use Adversarial NLI (ANLI; Nie et al., 2020). For mathematical reasoning we use GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021). For the mathematical reasoning datasets, we disable rationalization (i.e., providing hints) as it meaningfully degrades performance. Moreover, we unavoidably use Qwen 2.5 3B for GSM8K, as all STaR-based methods fail to self-improve with Llama 3. 2 3B as the base model. We discuss this further in Appendix $ F.
Evaluation. We use two evaluation metrics: Test Set Accuracy (Acc.) and Floating Point Operations (FLOPs). The corresponding early-stopped (Caruana et al., 2000) epoch (e) and iteration (it) for vanilla SFT and STaR- STaR-based approaches, respectively are reported. All methods are given an equal and large compute budget to ensure that the peak value is obtained via carly-stopping. For reprodueibility. we evaluate accuracy using zero-shot greedy decoding unless stated otherwise. We use FLOPss as our
"The choice of f(a) = a² is a hyperparameter. It allows more repetition of easy observations when the model is weak, and rapidly phases out this regularization effect the model strengthens.

---

Table 1: Empirical results where Test Set Accuracy (%, 1) is reported under zero-shot greedy decoding, excluding the 5-SC evaluation. Total training costs are reported Peta FLOPs (PFLOPs, 4). Best Acc. and PFLOPs is bolded, and second best is underlined in each section (excluding SFT). In (red) we quantify percent PFLOPs reduction against the highest accuracy baseline.
Evaluation
ARC-C
CQA
CLadder 1.5
Metric
(*)
(
(1)
(I)
PFLOPs (J)
SFT
61.4
1.0c
7.0
71.8
c.0
21.0
31.0
7.0
382.3
SFT 8-COT
59.0
1.5
10.5
71.6
2.5
60.1
13.6
3.0
163.9
SFT 5-SC
63.8
4.5c
31.6
76.4
2.5
60.1
45.2
8.0
137.0
STaR
71.6
13
351.4
72.2
25 it
2877.8
53.4
25 it
8427.3
STaR-Ful1
69.8
27
739,4
72.2
12
1502.7
53.8
19
6523.7
STaR-Ace
73.2
9663.0
STaR-koo-Full
71.8
5
135.8
76.0
10 1158.3
94.2
15 it
4465.4
STaR-Acc-Pull1-K
71.4
1760.9
RM
70.8
637.1
72.8
it
1548.4
53.4
10498.3
B-STaR
67.8
2
222.8
68.4
it
800,9
52.8
4
3937.3
AdaSTaB (ours)
73,8
10
174.4 (1 72.7%)
78.0
20
779,3 32.7%)
95.6
23 it
3610,0 19.2%)
Evaluation
ANLI
GSMSK
SVAMP
Metric
Acc.
PFLOPs (U)
Ace.
PFLOPs (J)
Acc.
PFLOPs
SFT
64.2
e
262.9
61.0
2.5
177.3
57.0
5.5c
21.7
328.7
S-SC
49.2
2
131.5
67.2
2.5
177.3
61.5
5.5
21.7
"STAR"
61.0
"4195.3"
26.0
Tii
20922
71.0
20'R
313.88
STaR-Full
57.6
13 260416
72.6
it 684.8
57.5
37 IL
348.5
STaR- ABB--Acc-Flll
64.6
5
986.0
74.6
it 333.0
74.0
18
167.3
STaB-Acc-Full-E
58.8
2528.4
1456,5
7
229,3
Ra
63.0
9
10938.5
77.0
it 2229.1
75.0
4
247.8
B-STaR
AdaSTaR (ours)
66.8
21
13409 (1 62.0%)
77.0
1L
19.3 93.7%%
75.5
at
65.7 () 71.3%)
computational cost metric memory usage remains approximately constant across methods. FLOPs are computed empirically following the method used by Kaplan al. (2020), Sardana et (2024).
Baselines. We categorize our baselines into two groups: (1) Vanilla SFT methods: Regular SFT, SFT with 8-shot chain-of-thought prompting (SFT 8-CoT; Wei et al., 2022b), and SFT with 5-sample self-consistency decoding (SFT 5-SC; Wang et al., 2023b) with temperature 0.7.
First, STaR (Zelikman et al., 2022), and STaR-Acc where the model is accumu- lated instead of being reset every iteration t. Most works that build on STaR choose accumulate the model over iterations. We incorporate AdaSTaR on STaR-Ace, as STaR consistently performs empirically worse. Next, STaR-Full and MAAA--AccFFlll which an alternative approach to eliminating the CoT sampling inefficiency described in Remark 1. In -Full, the predetermined 3 replaced with the total number of correct samples, i.e., |D' Therefore, no adaptive observation sampling scheme can be used when implementing -Full. Peng et al. (2025)'s underlying algorithm can viewed as TTTRR-ccC-Ful1 Additionally, we include STaR-Acc-Pull-& where -K denotes larger CoT generation sample size K. The majority of STaR-based methods (Hosseini et al., 2024; Pang et al., 2024; Zeng et al., 2025; Lin et al., 2025) adopt -Full-K as their core strategy. our ex- periments we set K : as larger K did not meaningfully improve performance, while dramatically raising compute cost. Furthermore, for -K, we omit rationalization (i.c., providing ground truth as hint), prior works in this setting do not employ it.
(2) STaR variants:
We include EM (Singh al., 2024), an improvement over RFT (Yuan et al., 2023) mentions the under- and over-training imbalance we discuss in 2.2. RM utilizes cut-off threshold per observation to ensure training diversity. Finally, we include B-STaR (Zeng et al., 2025) with outcome verification for insight. B-STaR is the only method that builds on STaR with open- opurnsource code, allowing for faithful replication. Although Lean-STaR (Lin et al., 2025) is open-source, tailored to mathematical theorem proving and thus incompatible with our benchmarks.
4.2 Results
We first briefly discuss the baselines' performance. As organized in Tab. 1, although STaR-based approaches often outperform SFT accuracy, they incur substantially compute costs (measured in FLOPs). Aligned with the existing literature's tendency model accumulation (-Acc), see that
7

---

STaR
STaR-Full
Acc
STaR.Acc--F Acc--Ful Full
SToR Acc-Ful--ll- Full-K
- Rest
B-STaR
AdaSTaR (ours)
Test Accuracy
0.9
6712
0.9
2820.2
0.8
0.6
0.8
0.7
0.7-
WW
0.4
0.6
0.6
100
200
300
150
200
250
500
1000
1250
PFLOPS
PFLOPS
PFLOPs
Figure 4: Visualizing the entire learning curve for SVAMP on Llama 3.2 3B (left), Qwen 2.5 3B (center), and Gemma 7B (right). Each method's 's curve is charted up to its best (early-stopped) iteration. The highest test accuracy is marked as a star, and second best as a diamond. As some methods converge only after significant amount of PFLOPs, for legibility of shorter curves, use dashed lines, and annotate the precise PFLOPs cost on the chart.
no model accumulation in the case of STaR, Full- and ReST commonly performs poorly. However, contrary to existing approaches" large K, we do not necessarily observe performance improving as we scale K. ResTEM uses K = 11, CARR-AcC-Full- and B-STaR uses K 5, and all remaining approaches use K 2.
Comparing our AdaSTaR to baselines, AdaSTaR performs best in terms of accuracy in 6 of bench- marks relative 10 baselines, all while reducing training FLOPS by mean of 58.6% (minimum of 19.2% to maximum of 93.7%) relative to the strongest accurac--achieving baseline. If there are numerous tied best baselines, we use the lowest PFLOPs to be conservative. Finally, for an intuitive visual understanding of our HieMinHeap, we provide empirical visualizations in Appendix $ G.
To further evaluate generality, test AdaSTaR datasets that perform relatively weakly Llama. 3.2 3B using different base models and sizes. Therefore, on Qwen 2.5 3B, well known to be strong mathematical reasoning, we experiment experiment on ARC-C, ANLI, and SVAMP, as we observe that all methods perform significantly worse on GSM8K, relative Qwen 2.5 3B. Among these five datasets (GSM8K is excluded this is in the main text), AdaSTaR achieves best test accuracy of times, while demonstrating similar levels of training cost (FLOPs) reduction. Comprehensive results are presented in Appendix $ H (Qwen 2.5 3B) and $ (Gemma 7B).
ARC-C, GSM8K, and SVAMP. On Gemma 7B
For intuitive visualization across different base models, visualize the entire learning curve trained on SVAMP for Llama 3.2 3B, Qwen 2.5 3B, and Gemma 7B in Fig. 4. Notably, across all three base models, AdaSTaR achieves faster gains in test accuracy under equal compute budgets. This aligns with the findings of Singh et al. (2024), which empirically demonstrate that performance gains from STaR-based approaches transfer well to larger-scale base models.
4.3 Ablation Study: Role of Diversity and Curriculum Design Choices
Set-up. To gain more granular understanding of the adaptive sampling mechanism, we evaluate three ablation variants AdaSTaR and analyze the standard deviation (SD) of observation training frequencies to assess whether the under- and over-training patterns observed in Fig. 3a are mitigated. The first version is AdaSTaR without (wo.) AdaC, which is synonymous to AdaD. Secondly, AdaSTaR Wi, which changes the HieMinHeap to regular MinHeap, only considering the last sampled iteration & for priority. Finally, we experiment with a shority-flippel version (AdaSTaR-PP) which prioritizes w; first and 1, second.
Results. We provide empirical results Tab. including STaR-Ace as AdaSTaR is mounted on top of STaR-Acc. Aligned with the described theory in $ 3, AdaD (AdaSTaR wo. AdaC) most effectively reduces under- and over-training on average (4 SD). However, contrary the intuitive expectation that increased diversity (L SD) would improve test accuracy, we observe sharp decline. We see that including AdaC solves this problem effectively while simultaneously maintaining high levels of trained observation diversity (1 SD).

---

Table 2: Ablation empirical results with Accuracy (1), and Standard Deviation (SD). SD of obser- vations' trained frequency distribution is computed from iterations to 2. to 10, or 1 to 20 for benchmarks that converge very quickly (GSM8K), quickly (ARC-C, SVAMP), or slowly (CQA, CLadder 1.5, ANLI), respectívely. Largest Acc. and lowest SD is bolded, and second is underlined.
Evaluation
ARC-C
CQA
CLadder 1,5
ANLI
GSM8K
SVAMP
Average
Metrie
Acc.
SD
Ace.
SD
Acc.
SD
Acc.
SD
Acc.
SD
Acc.
SD
Acc.
SD
Acc
73.2
1.50
74.6
1.11
94.2
1.36
64.8
1.07
77.0
0.47
71.5
4.78
75.9
1.72
AdaSTaR wo. AdaC
72.0
1.14
74.4
0.90
52.4
1.13
65.8
0.88
75.4
0.00
70.0
4.61
68.3
1.44
AdaSTaR wo.
73.6
1.39
74.6
0.92
93.4
1.13
64.0
0.92
76.8
0.33
73.0
5.19
75.9
1.65
AdaSTaR-PF
72.4
1.82
74.8
1.00
67.8
1.24
64.4
0.98
77.0
0.32
72.0
5.02
71.4
1.73
AdaSTaR (ours)
73.8
1.26
78.0
0.99
95.6
1.12
66.8
1.04
77.0
0.32
75.5
3.98
77.8
1.45
AdaSTaR wo. wi does indeed, on average, reduce SD, but fails to meaningfully improve test accuracy. Therefore, we can conclude that leveraging w, induce sampling more challenging observations within tied & is a salient part of AdaSTaR. We can decompose the rise in training diversity by quantifying the fall in SD throughout STaR-Acc - AdaSTaR wo. w;; + AdaSTaR: 1.72 - 1.65
1.45. AdaSTaR-PF fails to reduce SD, as it aggressively samples challenging observations (L w;), resulting frequent resampling of difficult examples. also results worsened test accuracy, likely due to poorer CoT quality (see 3.2).
5 Discussion and Additional Empirical Takeaways
We first discuss salient aspects of our adaptive sampling mechanism in AdaSTaR (1, 2), then present additional empirical insights drawn from extensive experiments with datasets and basclines under the STaR framework (3, 4).
(1) Near Zero Compute Cost Statistics. Notably, AdaSTaR's observation sampling algorithm adapts based on three statistics: L, w;, and which costs virtually no overhead run-time to compute. While the HieMinHeap does incur some run-time compute, it is negligibly minor. Our empirical tests indicate that run-time overhead is near zero relative to the (inference) sampling and training stage. The same can be said for the minimal memory footprint. Therefore, AdaSTaR is a lightweight extension that measures and leverages statistics extractable within the existing STaR system.
(2) Balancing Diversity and Difficulty through A daptive Sampling. Our key finding is that promoting observation diversity SD) while regularizing for model strength consistently improves performance and reduces training compute cost (Tab. 1. 5, 6). Our ablation study (Tab. 2) confirms that only encouraging inference diversity without difficulty measure (AdaSTaR wo. w;) does not lead to performance improvement. However, we also observe that failing to regularize for difficulty when the model weaker (AdaSTaR wo. AdaC) leads to even worse outcomes. Thus, adaptively sampling more challenging observations becomes increasingly effective as model strength improves.
As seen in Tab. (and also supported by Tab. 5, 6), using model accumulation (-Acc) consistently leads to improved performance. Across all experiments in the main text and Appendix, transitioning from STaR to STaR Acc, and from STaR-Full to STaB-Acc-Full. leads to average accuracy improvements: 73.6% - 79.0% and 67.8% - 78.8%, respectively, along with a corresponding average reduction in FLOPs of 16.4 % and 37.9%. This result is particularly noteworthy given that recent literature is divided on the use of -Acc, with some adopting it (Pang et al., 2024; Zeng et al., 2025; Lin et al., 2025; Peng et al., 2025), while others omit it (Zelikman al., 2022; Hosseini al., 2024; Singh et al., 2024).
(3) Model Accumulation is Generally Better.
(4) Cold Starting with STaR Does Not Always Work. We empirically find that the viability of self-improvement via STaR depends on the difficulty of the task relative to the strength of the base model. Therefore, as discussed in $ 4.1 and Appendix F, while STaR-based approaches fail to self- improve on Llama 3.2 3B, self-improvemmnt can be realized on the better pre-trained Qwen 2. 3B. This potentially explains why Peng et al. (2025) uses instruction-tunet base model instead of cold starting from a pre-trained-only model. Similarly, Hosseini et al. (2024) and Zeng et al. (2025) includes an SFT stage prior to the self-immpovemment stage. Aligned with recent large reasoning

---

model training (DeepSeck-A) 2025; Kimi Team, 2025; Liu et al., 2025b), the key takeaway is that STaR-based algorithm is part of a larger training pipeline. Precisely which stage within the training pipeline it should be integrated into is an open problem.
6 Limitation and Future Work
We discuss relevant limitations, to the best of our knowledge, and avenues for future research. First, natural direction for future work is to explore combinations of AdaSTaR with other advanced STaR-based methods. For instance, investigating the performance of a combined AdaSTaR and an inference-time verifier, such that in V-STaR (Hosseini et al., 2024), presents promising research avenue. Such explorations are beyond the scope of the current study. Second, while our experiments demonstrate AdaSTaR's efficacy, a larger computational budget would have permitted evaluation even larger-scale models. Nevertheless, our empirical study provides robust evidence of AdaSTaR's effectiveness across three distinct models: Llama 3.2 3B, Qwen 2.5 3B, and Gemma 7B. Moreover, existing work (Singh et al., 2024) suggests that gains from STaR-based training on smaller models often amplify on larger scales, implying our findings may well extend or even strengthen with increased model size. Furthermore, the model sizes used in our study (up to 7B parameters) are comparable to those in related STaR literature (Zelikman al., 2022, 2024; Zeng et al., 2025) that uses 6 to 7B base models. Third, similar to other adaptive methods such as Adam (Kingma and Ba, 2015) and AdaGrad (Duchi al., 2011), AdaSTaR introduces a new hyperparameter f(a) == a². more granular tuning is deferred to future work. It is anticipated that such tuning could lead to further enhancements AdaSTaR's performance and efficiency. Finally, building upon our discussion ($ 5), salient direction for future work is to investigate the optimal integration of various STaR-based methods within the end-to-end training pipeline incorporating RL-style long CoT generation. This investigation is particularly pertinent given the current divergence in methodologies: the STaR stage is implemented either prior to RL (Kimi Team, 2025; Liu et al., 2025b) or subsequent to (DeepSeck- AI, 2025). Furthermore, critical open question is whether, and to what extent, enhancements achieved during the STaR phase directly propagate to performance gains in the subsequent RL stage.
Lastly, we discuss broader impact in Appendix $ J.
Acknowledgments and Disclosure of Funding
This work was improved by collaborating with researchers at LG AI Research. J. Lee and S.-Y. Yun were supported by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea gSVErA (No. RS-2022-11220311, Development of Goal- Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects, No. RS 2044-7888,4,88,, Al Research Hub Project, and No. RS-2019-I111906 Artificial Intelligence Graduate School Program (KAIST)).
10

---

References
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. www.:/ooggab//a/////// 21787.
arXiv preprint arXiv:2407.2 2024. URL
Rich Caruana, Steve Lawrence, and C. Giles. Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping. In Advances Neural Information Processing Systems, volume 13, pages 381-387 MIT Press, 2000. URL https::/pprcceeedteeeee hewpeeeedddnnnnneneeeeeeeeeeeeee neurips..... ctt.ccccppp/ htppppppr_flleee pdf.
Sijia Chen and Baochun Li. Toward Adaptive Reasoning in Large Language Models with Thought Rollback. In Proceedings of the s1 International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7033-7056. PMLR, 21-27 Jul 2024. URL
https:/ ftttps://proceedingg mlr. press/ /v25// chen24y html.
Yeseul Cho, Baekrok Shin, Changmin Kang, and Chulhee Yun. Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty. arXiv preprint arXiv:2502.06905 2025. URL pesttss:/arrtie org/ abs 2abbs/2500 2502. 06905.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. arXiv preprint arXiv: 1803.000 2018. URL https:://aaatt 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:21 1468, 2021. URL
http://xxxvv..gg/bbaa// htrggabs/2111 14168.
Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob Raumer. The Lean Theorem Prover (System Description). In Automated Deduction-CADE.S 25th Interna- tional Conference on Automated Deduction, Berlin, Germany, August 1-7, 2015, Proceedings 25. pages 378-388. Springer, 2015. URL attps::/lite springer com/ httpchtt//111100///7.
DeepSeek-AL. DeepSeek-RR R1: Incentivizing Reasoning Capability LLMs via Reinforcement Learn- ing. arXiv preprint htt::2501.112445 2025, URL https://arxii: arggabbs/5001.22//
John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12(61)2212 2011. URL
http://jmlr org/pari wwppps/ppvvvvi /ver/s//v///ee
duchi http//111///chhh11
html.
Ronald Fisher. the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character,
222(594-644 1922. URL
/ httpp:://ooyalloodeeppll.l.iiiis http:ppwwwwiiiinoooo/oooeoe 1098//
rsta. 1922. 0009.
Gemma Team. Gemma: Open Models Based Gemini Research and Technology. arXiv preprint arXiv:2403.08295, 2024. URL https:///axta http:///wxxxvv../aabaaase/
wttp./.wor///a//20000000066 08295.
Gemini Team Google. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312./1880 2023. URL https:///a/ae /ttp/v/.vvv/////////////////aaaa www.riiw.w.aaaaaaeeeee aas/aas///1 11805.
Guy Hacohen and Daphna Weinshall. On The Power of Curriculum Learning in Training Deep Networks. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2535-2544, PMIRR, 09-15 Jun 2019. URL /ectp::/pproceee
mll. www..press/vvie http://////aahhet/eeaa http//hhhhhh......m html.
Alexander Havrilla, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements. In Forty-first International Conference on Machine Learning, 2024. URL https: http///ppwweeeeeeee//eee http://ppprrrreeeeeeeeerrrrrmmmm htt.://eeeeooo///../LL.RO......

---

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling Laws for Transfer. arXiv preprint arXiv:2102.01293, 2021. URL htttpp///arrr..ooo//////////2 01293.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large Language Models Are Reasoning Teachers. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 14882, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: acl-long.830) 830//
URL htttps://aclanthologyyy org/2023.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training Verifiers for Self-Taught Reasoners. In First Conference on Language Modeling, 2024. URL https://openrevier
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations, 2022. URL https: http://pppereeee.eeeee met/fooummic. NZeVKeeFYf9.
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fer- nando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLadder: Assessing Causal Reasoning in Language Models. In Advances in Neural Informa- tion Processing Systems, volume 36, pages 3 31038-31000 Curran Associates, Inc., 2023. URL wtwee///oormiidddettoo0o0
attts::/oppnreet
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08360 2020. URL https: //axxiv. org/abs/////aaooooo/goooee 2001. 08361.
Akira Kawabata and Saku Sugawara. Rationale-Aware Answer Verification by Pairwise Self- Evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11118-161999 96, Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi: 10.18653/112222.emmllp maiallp-mmin..055 905. org/2024. emnnp-main. 905/-
URL ptattp::/aallantholog./o
Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with L.LMs. arXiv preprint arXiv:2501. 12599, 2025. URL https:// httpe////aaxvvvoooaaaaaeaaae /wwpg//as/500..... 12599.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: http///axxvvveie http//awwww.o.raaaraaaaaaaaa/aat hbs///bb///// http//2....111.../ 1412.6980
Yajing Kong, Liu Liu, Jun Wang, and Dacheng Tao. Adaptive Curriculum Learning. In Proceed- ings of the IEEE/CV International Conference Computer Vision (ICCV), pages 5067-5076, October 2021. URL httptttp///opeaaccesssss thecvf. com/ cotateett////o22//////t///////// 2021 _acpellpppprre_ html.
Jaehyeok Lee, Keisuke Sakaguchi, and Jin Yeong Bak. Self-Training Meets Consistency: Improving LLMs' Reasoning with Deisiistene-Drriven Rationale Evaluation. In Proceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10519-10539, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-1888 189-6. URL https: / h/t.taholaaatntogo..ooooo org/ 2025. naacl-long. 1111--1on5522 558//
Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura. Active Learning for Abstractive Text Summarization via LLM--Dterming Curriculum and Certainty Gain Maximization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8959-8977 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.186//11/2224 win.224....niinnnnn..22222. URL htt.httts//acaaatthh/ooo 2024. emnl.
org/
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations, 2024. URL https: :tttpp:://ppereviieeee net/ forum/id==88LOpNEE11
12

---

Haohan Lin, Zhiqing Sun, Sean Welleck, and Yiming Yang. Lean-STaR: Learning to Interleave Thinking and Proving. In The Thirteenth International Conference on Learning Representations. 2025. URL https: http:::tttppeeeeeeeeeeee http///ppppreeeeeeeee/ee wwwptorum//eeeeeeeee htw.s///r
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding RI-Zero-Like Training: A Critical Perspective. arXiv preprint arXiv:2503.20783, 2025a. URL https:///arat org/ arg/abs//a a/////b//55033 20783.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-Time Scaling for Generalist Reward Modeling. arXiv preprint arXiv:2504.02995, 2025b. URL https:: /ttp///avvvv/.//////////////aav/ .rgg hwggabbs/2504 024955.
Llama Team. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.2178. 2024. URL
https: //ar:v/.////vv//// /ww.iii.o/.aai.eeee.oeee hatg//b//22/0 2207. .17833.
Jianqiao Lu, Zhiyang Dou, Hongru WANG, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. AutoPSV: Automated Process-Suppe Supeevssed Verifier. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:///ppeeevee.ettttoo wtw.///wwwwee///m/iiieeeee eDAPWWOGs9.
Adyasha Maharana, Prateek Yadav, and Mohit Bansal. ID² Pruning: Message Passing for Balancing Diversity & Difficulty in Data Pruning. In The Twelfth International Conference on Learning Representations, 2024. URL https:/
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 4885-4901, Online, July 2020. Association for Computational Linguistics. doi: 10. 111.655/111/222...mmmmmm...44 URL attps://aclanthoooggy wwg/o020..o......ooo.. mell-mmin.444
NVIDIA. Nemotron-H: A Family of Accurate and Efficient Hybrid Humba-Transformer Models, 2025. URL battps://aaattt org/ abs 0abs//5000 2504. 03624.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Ja- son Weston. Iterative Reasoning Preference Optimization. Advances Neural Information Processing Systems, volume 37, pages 11661--16666 Curran Associates, Inc., 2024. URL https://openreviier http:://pppppaeeeeeeeeeeeeeeeaap fit.forrumm..--XKKWWWWW
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online, June 2021, Association for Computational Linguistics. doi: 10. 10..1883/vv1// 12255////22221111 maacl-main.16 URL ptttps://aclantolllo
org
/021 www.aaaallaaaaama main. 168/.
Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing. Regenesis: LLMs can grow into reasoning generalists via self-improvement. In The Thirleenth International Conference on Learning Representations, 2025. URL htttpts///ppereeeteeee net/ Dorum?id:YUYDJsH0F33.
Hadi Pouransari, Chun- -iang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, and Oncel Tuzel. Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum. In Advances in Neural Information Processing Systems, volume 37, pages 36121-36147. Curran Associates, Inc., 2024. URL https::/ wtw.pparrrriiieeeeeffo.rrmmm net/forum? id=rSM9SIYMDi.
Qwen Team. Qwen Technical Report. arXiv preprint arXiv:2309.1660 2023. URL https: //arxiv. org/ abs/a3/9.,....//////////////9/// 16609.
Qwen Team. Qwen2.5 Technical Report. arXiv preprint arXiv:2412. 151 2024. URL. https:
// //arxiv..o org/ ab///ab///222// 15115.
13

---

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model Secretly a Reward Model. In Advances in Neural Information Processing Systems, volume 36, pages 33728-53744 Curran Associates, Inc., 2023. URL httph:ttp///m httttp:://opnnreveeee http://viw....eeeerrmmmreeeeaaaa
LG AT Research. EXAONE Deep: Reasoning Enhanced Language Models, arXiv preprint arXiv:2503. /2524, 2025. URL httts:t/////iiii http:///arxiiv..r/abbssss/ wttp/oog////aaeaeeeooeee 12524.
Joshua David Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka, Contrastive Learning with Hard Negative Samples. In International Conference on Learning Representations, 2021.
URL https
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond Chinchill-Oppimmll Ac- counting for Inference in Language Model Scaling Laws. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33445-43460 PMLR, 21-27 Jul 2024. URL https::/ppocceeeeee sardana24a html.
mlr pprsss/v35..
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL ptttps::/openreeee
wwwnteetoouuue Ad= AqlzLH.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint aXXiv:2402.03300, 2024. URL https://arxi: org/ abs/ /202 03300.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison. Jachoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Ellsaydd, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL httph::////oot //penreviei net f wormmoommmmm 1NAyN-1 Expert Certification.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Test- Test-Time Compute Optimally Can be More Effective than Scaling LLM Parameters. In The Thirteenth International Conference on Learning Representations, 2025. URL https: https://t////eeee http////ppe/eeieeeeeeeeeeeeeeeee net/ fost/fooo forum?id= 4FWAwZtd2n.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop Overthinking: A Survey Efficient Reasoning for Large Language Models. arXiv preprint arXiv:2503. 194,, 2025. URL https://arxiv: org/ 2503. 16419.
Zhihong Sun, Chen Lyu, Bolun Li, Yao Wan. Hongyu Zhang, Ge Li, and Zhi Jin. Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Torino, Italia, May 2024. motemmai..503
Resources and Evaluation (LREC- COLINC 2024), pages ELRA and ICCL. URL https: htt///alaanttt aalaaaaolannoogocooca org/2024 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@AA A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Pupers), pages 4149-41 4155, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.1 8653/v1/N19 1421. URL https: // a/aclamm aaaclantholoog org/ org/N19- N191111911
14

---

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.44275 2022. URL https://arxivv .rrg abs/2211. 14275.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30, pages 6000-6010. Curran Associates, Inc., 2017. URL.
iew.rroeeeediinnnnene.......... neurips. wwwpcc/pppe..... files/llleeeeeee www.seewaeeeeeaea..aaa patpe//pppprr///////eeee (05oe222555eeee1000000cccccaaaaa Papeeer..aaaaap..ppp....
Boshi Wang. Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards Understanding -fain-of-Thoughe Prompting: An Empirical Study of What Matters. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2717-2739, Toronto, Canada, July 2023a. Association for Computational 153. URL. https: taattss:/accaaree http///aaaaaooooooorrrg./r org/2033.
Linguistics. doi: 10.18653/v1/2023.aad acl-long. 153/.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning Lan- guage Models. In The Eleventh International Conference on Learning Representations, 2023b.
URL
Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong. Jingyang Lin, and Mei Han. Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates. ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025.
URL htttps://pperreeee
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Lan- guage Models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL stttp:://oeenreveew.nett wwtp://nne//ooummmmeee ttme://formm//ddsskSSUUdddl Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc., 2022b. URL https: //tppr/iie///eeeiiieeeeeeeeee/ee nww/foeuutteaieeeee http://eeereeeeeeeeeeeerr//////a _dd___jQMMeeB_B_
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu,
Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently. arXiv preprint arXiv:2303 0008466, 2023. URL https: :/tpttps:////iiiv wwwpaaawii..oooaraaaaaaaaaa httgg/ab//3333 03846.
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting
Data for Training Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52915-52971. PMLR, 21-27 Jul 2024. URL https: btttps:/pprocet httppp/////oocceggs..oo wtt.:emww..presssemmmm https//////wwttttttt/ html.
Shuai Xie, Zunlei Feng, Ying Chen, Songtao Sun, Chao Ma, and Mingli Song. DEAL: Difficulty- Aware Active Learning for Semantic Segmentation. In Computer Vision - ACCV 2020, pages 672-688, Cham, 2021. Springer International Publishing. ISBN 978-3-030-69525-5.
URL
https://link.
08-030-6880 000----0000-09555-.02 httpe////iinnssspinnneeeeeeee sprinpriieee wwwiingrr.comcee/ htt.eccoooccccceee hamm/chapter/10.. 0atp//////...00/7778-------0---0
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Curriculum Learning for Natural Language Understanding. In Proceedingy of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655//11/2000aall m42n. URL https:
//aclanthology www.//20......ooo/o/
2020. .0120.acl-00 acl- m42n.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, volume 36, pages 1 11809-11822. Curran Associates, Inc., 2023. URL https: httpa//wwweee..eeeooooooaaoeeeaa
15

---

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin. Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open- Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503. 2025a. org/ ab0/ 14476.
URL
Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient Test-time Scaling with Code. arXiv preprint arXiv:2504.00810, 2025b. URL aattts://arxiaa org/ abb/
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv preprint arXiv:2308. 01825, 2023. URL https: ptatpp://arxiv... org/ awaaaaa///// 01825.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping Reasoning With Rea- soning. In Advances in Neural Information Processing Systems, volume 35, pages 15476-15488. Curran Associates, Inc., 2022. URL
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. In First Confer- ence on Language Modeling, 2024. URL
net orum iidiiddooi ORXPiSOHH..
Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-STaR: Moni- toring and Balancing Exploration and Exploitation in Self-Taught Reasoners. In The Thirteenth International Conference on Learning Representations, 2025. URL htwph:tt/////ett btttp:://openrevieet net/ forun?id-P6dw2Jp..6
Zirui Zhao, Hanze Dong. Amrita Saha, Caiming Xiong, and Doyen Sahoo. Automatic Curriculum Expert Iteration for Reliable LLM Reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http::///ppnnae
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric Coreset Selection for High Pruning Rates. In The Eleventh International Conference on Learning Representations, 2023, URL https:/ /tpp::///ppeeeeieer net orrmm ind 1xV.6.
16

---

A Total Training Time Comparison
Fig. 5 is conducted using full fine-tuning Llama (Llama Team, 2024). The training
run-time is set to the early-stopped epoch (iteration) (Caruana et al., 2000).
ANUI
GSMBK
STaR
SFT B-shot CoT
C0A
ARC-C
100
Training (Hours)
Training (Hours)
Figure 5: Total training run-time on 4x 4XRTX 3090 24G, across three common reasoning datasets CLadder 1.5, ANLI, CQA, SVAMP, GSM8K, and ARC-C. STaR, SFT, and SFT 8-shot Chain- of-Thought is evaluated under zero-shot greedy decoding. Training times across SFT 5-sample Self-Consistency, SFT 8-shot Chain-of- of-Though, and SFT differ their best early-stop epoch differs.
17

---

B Observation Distribution Visualizations Across All Datasets
Number @bservation
350-
300
Under-Trained
Over-Tralned
250-
[+3
200
150
[04] [+1
100
2+
-01
-02
[+3
02
+01
Accumulated Trained Frequency
10 12
Percent of [Qi] or [Q4] Quartiles
(a) Distribution of frequency trained of each observation in iterations I to 10 in ARC-C.
(b) Initial Quartile [Q1] and 4 [Q4] are computed based on iterations to 10.
3000
Number of Observations
[01]
(+2
2000
t+3
3500
[04]
500
2
Accumulated Trained Frequency
Percent of [Q1] [Q4] in Quartiles
(c) Distribution frequency trained of cach observation in iterations 1 to 10 in CQA.
(d) Initial Quartile [Q1] and [Q4] are computed based on iterations to 10.
Number of Observations
2000
[Q1] 1
2+2
1500
-01
8+3
-03
-02
-04
3000
[Q4]0+1
10 12
Accumulated Trained Frequency
Percent of [QL] [Q4] in Quartites
(c) Distribution frequency trained of each observation in iterations to 10 in CLadder 1.5.
(f) Initial Quartile [Q1] and [Q4] are computed based on iterations to 10.
Number of Observations
3000
[Q1] C+
2500
c+2
2000
t+3
2500
[04]
1000
500
Accumulated Trained Frequency
Percent of [Q1] or [Q4] in Quartiles
(g) Distribution of frequency trained each observation v in iterations to 10 in ANLI.
(h) Initial Quartile [Q1] and 4 [Q4] are computed
based on iterations to 10.
Figure 6: ARC-C (a. b), CQA (c. d). CLadder 1.5 (c, f), and ANLI (g. h) datasets illustrate persistent relative under- and over-training across observations.
18

---

[11
Number of Observation
100
2+2
t1.3
[04]
t+3
Accumulated Trained Fraquency
Percent of [Q1] [Q4] in Quartiles
(a) Distribution of frequency trained of each observation in iterations I to 10 in GSM8K.
(b) Initial Quartile [Q1] and [Q4] are computed based on iterations to 10.
Number of Observations
(+3
[04]
t+2
[+3
Accumulated Trained Frequency
2
Percent of [Qi] [Q4) Quartiles
(c) Distribution of frequency trained of each observation in iterations 10 SVAMP.
(d) Initial Quartile [Q1] and 4 [Q4] are computed based iterations to 10.
Figure 7: GSM8K (a, b) and SVAMP (c, d) datasets also show consistent patterns of over- and under-training across sample quartiles.
C AdaD Induces Poor Quality CoT
We use the following prompt.
Prompt for GPT 4o Annotator
System: You are an expert QA/CoT reviewer. For each entry, check whether the chain-of- thought (CoT) logically supports the answer. Respond ONLY with an array of the indices (O-based) of entries whose CoT is WRONG, Please output like: [1, 6], and explain why
User: [ question chain of thought: answer: I
questtin chain of thought: answer: }]
We now provide a qualitative example. Consider this example question.
Question (z;) Example
There are 6 houses on a block. a mailman has to give 24 pieces of junk mail each block, how many pieces of junk mail should he give in each house?
Using this question, we examine the CoT's generated by AdaD (red container) and STaR-Ace (green container).
19

---

Incorrect Co'I' (AdaD)
Given CoT:
6 blocks, and 24 pieces of junk mail for each block.
So, 24 pieces for blocks.
24
Then calculates:
4 pieces per house.
6
Error Explanation:
Misinterpreted "6 houses on a block" as "6 blocks."
Incorrectly calculated division using blocks instead of houses.
Correct CoT (STaR-Aec)
Given CoT:
6 houses per block, total of 24 pieces per block.
24
Thus, each house gets: 4 pieces.
6
Correct Steps:
Correctly identified number of houses per block.
Correctly distributed mail equally each house.
D Excessive CoT Sampling Inefficiency
D.1 Problem: Excessive Sampling, then Filtering
salient observation we make is that an unnecessarily large amount of CoT samples are unused in
training. Remember that, STaR inferences the entire dataset D {(11, ê1, 01), (ZN ČN 0N }. then filters down to correct samples D = {Ir,, & Ui) [l( = }:}}. We denote size [D M Next, it throws away or re-uses parts of D 4 to the predetermined per iteration batch size 3 As mentioned in 2.1, 3r = a² x 3, where of is the number gradient update steps per iteration and
is the batch size for each gradient update step.
In the case that M > 8r, some (a,,, G, (i), discarded. Such discarded samples can not be cached and use in the next iteration t + because the fundamental idea of iterations is that an improved model Th+ is used to generate new samples. The compute and memory wastage, especially in earlier iterations, is significant. For concrete understanding. visualize this sampling inefficiency empirically across the datasets in Fig. 8.
D.2 Existing Solution
However, mentioned in $ 2.1, all methods that resolve this excessive sampling (M*-39) problem of STaR (Hosseini et al., 2024; Pang et al., 2024; Zeng et al., 2025; Lin et al., 2025; Peng et al., 2025) simply removes this pre- pre-determined S and instead set O to DD This approach can viewed as bringing the blue curve up the red curve; i..., 8t + | We name this approach as -Full in our experiments ($ 4). B-STaR (Zeng et al., 2025) also embodies this approach. ReSTEM (Singh et al., 2024) does not resolve this problem they keep the filtering mechanism, as it is useful in their algorithm.
"This can known from line Alg. STaR paper. This the case their open-source code. "Excluding some edge scenarios M
8*
20

---

1200
Data Samples
Data Sample
26000
Samples
1000
800
Iteration
Iteration
17 Itaration'
(a) ARC-C
(h) CQA
(c) CLadder 1.5
8000
500
8020
Date Samples
Data Samplee les
Samples
550
5000
4020
2000
1000
Iteration
Reration
ittrationn t (f) SVAMP
(d) ANLI
(e) GSM8K
Figure 8: Visualizing the CoT sampling inefficiencies in STaR across numerous datasets. Initial
g 40 presented in the original implementation. 3 rises over time follow the +20 of gradient update steps every iteration in the original implementation. That is, f == 1.2( If
M there is inference sampling inefficiency M' data samples are used.
D.3 Our Approach
Alternatively, as we aim to reach peak performance as computationally efficiently as possible, we keep STaR's original 5e curve, and instead, sample CoTs (Ira, 0₁ 9t) T/ (z) up till [D 8 is filled. This approach be viewed bringing the red curve down to the blue curve; i... [Di B
E Further Details on Experimental Configuration and Setting
Common Configuration. We primarily conduct experiments numerous nodes with 8xRTX 3090 24G, with equivalent hardware specifications across nodes. For a few compute heavy experi- ments we use nodes with 8 8 A00 40G. All training is done the same arbitrary seed value of 10. This value has never been changed. Hyperparameters are organized in Tab. 3.
Parameters
ARC-C
CQA
CLadder 1.5
ANLI GSMSK
SVAMP
Batch size
8
8
8
Learning
10
10
10
10
10
Weight decay
0.01
0.01
0.01
0.01
0.01
Warm steps
100
100
100
100
100
100
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Model precision
bfl6
bfl6
bf16
bfl6
bf16
bf16
Samples for self consistency
5
5
5
Inference decoding temperature
1.0
1.0
1.0
1.0
1.0
1.0
Evaluation decoding temperature
0
0
0
0
Rationalization (default)
Truc
Truc
True
Truc
False
Falsc
Table 3: Hyperparameters across datasets.
21

---

Dataset Configuration. For ARC-C, we combined the train and validation dataset for training. The ANLI dataset is comprised of RI, R2, and R3 versions. For our experiment, we used R1, and random sampled (without replacement) 10,000 samples for efficient evaluation. In GSM8K, high quality ground truth e is already available in the SFT dataset. To compare whether STaR able to improve on the SFT case where high quality ground truth e is unavailable, we do not include the in the SFT dataset. That is, we only train on (z, y) as all STaR-like approaches are not given access to c. Dataset and evaluation sizes are provided in Tab. 4.
Dataset
Train set
Test set
ARC-C
1,418
1,172
CQA
9,741
1,140
CLadder 1.5
8,089
2,023
ANLI (R1)
10,000
1,000
GSM8K
7,473
1,319
SVAMP
800
300
Table 4: Train and test set sizes for each dataset
ReSTEM Configuration. We follow the original implementation' 's ReST configuration (Singh et al., 2024) as close as possible. The only change we make reducing := 32 and cut-off threshold value of 10 to K 11 and cut-off threshold value to 3. This is done larger K and cut-off threshold values resulted worsened performance with dramatic rise in compute cost. We kept the ratio of K cut-off threshold close the paper's implementation.
For instance, when sampled = 11, an easy observation i may result correct samples, while more challenging ones may result in 2. In this case, if the threshold is set to 3, the observation with 8 correct (I,, &, 9r) will be reduced to a maximum of 3, shrinking the imbalance from 8:2 to 3:2.
B-STaR Configuration. We follow the original implementation's 's B-STaR configuration presented their paper (Zeng et al., 2025) as close as possible. For any implementation that not explicitly specificed in the paper, use their official open-source implementation. We set the range of temperature search space as [0.4, 1.1] in increments of 0.1 as the paper. We set := 5 as in the paper. We set their balancing hyperparameter n* == in the paper. The only change their training queries (M) per iteration. We first experimented by setting M == 2627 as they did for their experiments that did not include SFT stage, pre-STaR training. However, this resulted in poor performance. In response, we set M to the entire original dataset size, which helped performance.
make is
Ll Alama 3.2 3B Fails to Self-Improve on GSM8K
STaR-based methods fail self-improve on GSM8K using Llama 3.2 as the base model (Fig. 9). Therefore, we use Qwen 2.5 3B instead in the main text.
STaR-Acc AcC--Fl
0.34
0.32
Accuracy
0.30
0.28
0.26
10
Iteration
Figure 9: Visualizing the learning curve for STaR-Acc and ITaR-Acc-Ful1 for GSM8K using
Ll ama 3.2 3B as the base model.
22

---

G Visualizing Empirical Heaps
Last Sampled Iteration E
11
13
15
17
19
21
23
25
27
29
6
8
10
12
14
16
18
20
22
24
26
28
18
17
15
14
13
Iteration
11
10
7
6
5
1
20
60 Percentage (%)
100
20
40
80
100
Percentage (%)
(a) ARC-C
(b) CQA
22
21
25
76
20
72
17
21
15
Iteration
14
Iteration
13
16
10
8
20
60
80
100
20
40
80
100
Percentage (%)
Percentage (%)
(c) CLadder 1.5
(d) ANLI
16
14.
Iteration
Iteration
9
20
40
80
100
20
40
80
100
Percentage (%)
Percentage (%)
(c) GSM8K
(f) SVAMP
23

---

Qwen 2.5 3B Base Model Empirical Results
as the base model. Experiment settings are equivalent to the main experiments. GSM8K in Tab. 5 is equivalent to that of Tab. 1, as the main text's GSM8K is Qwen 2.5 based. We describe why Tab. is Qwen 2.5 3B based in 4.1.
Refer Tab. for empirical results using Qwen
Table 5: Qwen 2.5 3B empirical results where Test Set Accuracy (%, 1) is reported under zero-shot greedy decoding, excluding the 5-SC evaluation. Total training costs are reported in Peta FLOPs ()). Best Acc. and PFLOPs is bolded, and second best is underlined in each section (excluding SFT). In (red) we quantify percent PFLOPs reduction against the highest accuracy baseline.
Evaluation
ARC-C
GSM8K
SVAMP
Metric
PFLOPS (J)
Acc.
PELOPS (1)
Acc. (f)
PFLOPs
SFT SFT + COT
33.8
43.9
61.0
2.5
177.3
68.5
1.89
76.2
7.5
50.6
68.0
70.9
86.5
4.0
15.2
SFT + 5-SC 3TaR
67.4
40.5
67.2
2.5
177.3
73.5
1.89
80.4
2t
"S25.9
76.0
1
4099.2
92.5
8
96..2
STaR- Pull
83.2
22
606.2
72.6
it
684.8
91.5
16 at
196.2
STaR-Ace
264.1
77.0
305.2
95.0
129.6
STaR Acc Acc-Full
84.6
4
110.8
74.6
2
333.0
93.5
6
73.0
STaR-Ace-Pull-A
82.2
2
225.
77.0
2
1456.5
92.0
2
105.3
ReST
81.0
ST4.6
77.0
2229.1
92.0
677.2
B-STaR
83.2
583.3
72.6
2
1185.7
91.0
3
150.4
AdaSTaR (ours)
85.0
12
239.9 (1 0%)
77.0
it
19.3 () 73%
94.5
8 it
83.9 (L 35.3%)
I Gemma 7B Base Model Empirical Results
Refer to Tab. 6 for empirical results using Gemma 7B the base model. We use Low-Rank Adaptation (LoRA; Hu et al., 2022) fine-tuni ane-tuning set to rank 32. All other settings are equivalent to the main experiments.
Table 6: Gemma 7B empirical results where Test Set Accuracy (%, 1) is reported under zero-shot greedy decoding, excluding the 5-SC evaluation. Total training costs are reported in Peta FLOOPs (L). Best Acc. and PFLOPs is bolded, and second best is underlined in each section (excluding SFT). In (red) we quantify percent PFLOPs reduction against the highest accuracy baseline.
Evaluation
ARC-C
ANLI
SVAMP
Metric
Acc.
PELOPs (1)
PFLOPS ())
PELOPS (1)
SET
49.2
e.5
0.01
66.0
5
1.1
61.0
Je
D.D4
4e
SET S.SSC
66.0
0.5
0.01
67.6
Te
1.6
61.0
3e
0.04
STaR
$2.0
17'
5300.6
62.1
it "19100
15 HI
'332.3"
STaR-Ful1
76.2
it
93.5
13.8
12 it
3536.6
67.5
36
832.9
STaR-Ace
85.4
it 383,0
62.0
20 it
92288,8
87.0
13 it
281.3
STAB-Acc-Full
81.6
it 533,0
61.8
8
3751.0
87.5
21 510.6
STaR-Ace-Pull-K
85.0
Ht 13344
65.0
12 it
20743.3
87.5
G 485.7
ReSTEM
$1.6
5
1221.0
62.2
17 it
43060.8
82.5
1245.6
B-STAR
84.8
it 1936.0
63.8
22 it
27315.1
84.0
it 2610.2
AdaSTaR (ours)
85.4
32 16.2%)
65.2
20
3055.4 85.3%)
89.5
14
207.0
(1..4%))
24

---

J Broader Impact
The development of AdaSTaR presents notable positive societal benefits stemming from its ability to achieve strong performance with significantly reduced PFLOPs.
Environmental Sustainability: By lowering the computational requirements (FLOPs) for training effective models, AdaSTaR contributes to more environmentally sustainable AI practices. This reduction directly translates lower energy consumption and diminished carbon footprint associated with model development and deployment.
Economic Value and Accessibility: The substantial computational savings unlock eco- nomic advantages. These include reduced operational costs for training and inference, making advanced AI technologies more accessible to broader spectrum of users. Aca- demic institutions, startups, and researchers with limited computational budgets can benefit, potentially accelerating innovation and democratizing access to state-of-the-ar model devel- opment.
Accelerated Research and Development: Efficiency gains shorten model development cycles, allowing for faster iteration and exploration of new architectures and applications.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are justified by extensive empirical results (see 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There a clear limitations section in the Appendix. This part of the appendix is referenced in the main text well.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper's findings are not theoretical in nature.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper can be quickly and easily replicated by readers. We have all relevant experimental design and settings available in the main text and appendix (see 4 and E). Moreover, we have open sourced the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code and data is open sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wttpccppppppceeeeeeee..ec...ceee gguidde/Codeeaee
ssiiiiolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All details are provided in the main text and appendix (see $ and E).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments evaluated using greedy decoding. We do not claim that method statistically significant. Nevertheless, we have extensive experimental validation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have details in Appendix $ E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp::////uiis....e. http:////t...ccccccccciiiieeeeee
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our re- search complies with its principles. Our work does not involve human subjects, sensitive proprietary data, potentially harmful applications. We have ensured transparency, reprodueibility, and responsible use compute resources, and have carefully preserved anonymity in accordance with double-blind blind review guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, see Appendix $
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any models, datasets, tools that pose high risk for misuse. Therefore, safeguards for responsible release are not applicable in this case.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All credit has been appropriately given in text.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our github repository is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Not applicable our study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable to our study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We clearly and transparently indicate all parts where LLMs have been used (see 2.2).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httpaa://eeupp...eeee http::pp..../oooofeeeeeee/////// for what should or should not be described.
32


Reviews:
[{"review_id": "gFSw3JnOTZ", "reviewer": "Reviewer_suRc", "summary": "This work proposes an adaptive finetuning method that prioritizes training data base on a so-called win statistic. This improves training efficiency over random sampling, as not every chain of thought is equally useful in downstream tasks, and some are even overly complicated yet lead to the correct outcome.\n\nThe authors show that STaR typically re-balances the training data, possibly inadvertently, and that the increased diversity can degrade quality of training CoTs.\n\nThe approach therefore proposes the following:\n- tracking win statistics: how often we got the correct answer out of K CoT samples the last time this item was sampled\n- using these statistics to sort a heap of datapoints by difficulty,\n- training on the samples that were last sampled at more recent iterations, or deemed more difficult\n\nThis results in considerable training time improvements and/or an increase in performance over variations of STaR.", "questions": "1. When are some of the heuristic approximations a good choice, and when might they pose a risk? \n2. How brittle or robust is this procedure compared to its (often simpler) alternatives? More insight would help convince me that this work is a useful procedure, and not just a well-tuned modification of STaR.\n3. Iiuc the downside of the difficulty criterion is that it may still oversample too-difficult examples. This is currently countered by using alpha as guide for sampling. Can this have inadvertent effects, e.g. sampling near-impossible datapoints?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Ability to cut training time in half yet get the same performance is quite compelling. This result is consistent, shown (with varying success but success nevertheless) on multiple datasets.\n- Pseudocode makes it fairly easy to re-implement this method, if one understands the heuristic choices in section 3.1 well.\n- Clear takeaways based on observations that are useful for practitioners using StaR or related methods.\n- The empirical heap visualizations in the Appendix show the training procedure and evolution of priorities very well.\n\nWeaknesses\n- Some choices are heuristics, for example using the most recent estimate of the K-sample estimate of p_i^t relies n the assumption that the difference between the timesteps at which data was last sampled is not too large. This may not always be true in practice. \n- The difficulty estimate similarly relies on an intuition that making a wrong prediction corresponds to difficulty of the sample. In some cases, datapoints may be impossible to solve for current model capacity, yet one correct verification means these will often be sampled as well iiuc. Do you consider this a problem?\n- Combining model accumulation and adaptive data sampling through alpha makes the procedure potentially more sensitive to noise/outliers. Do you have any studies on variance across runs, or insights on when the system breaks?", "was_revised": true, "final_justification": "Authors adequately addressed my concerns and promised several improvements.\n\nI believe the paper is technically solid, shows good improvement over the previous methods. It could be more clearly written, especially on what sets it apart from previous work. I therefore maintain my rating.", "rebuttal": "Thank you for your thoughtful and highly constructive comments. We are especially encouraged to see that you recognize:\n\n-   **Clear Takeaways, and Impact**\n\t-   “Clear takeaways … useful for practitioners” (`suRc`)\n-   **Consistently Strong Performance**\n\t-   “performance is quite compelling … result is consistent” (`suRc`)\n-   **Consistently Strong Efficiency Gains**\n\t-   “considerable training time improvements” (`suRc`)\n-   **Straight-forward Implementation**\n\t-   “fairly easy to re-implement” (`suRc`)\n-   **Clear Communication**\n\t-   “visualizations … show the training procedure and evolution of priorities very well” (`suRc`)\n   \nWe happily address the weaknesses **[W]** and questions **[Q]** you bring-forth in detail below. The following discussion has helped us improve parts of the updated version of our paper. Please feel free to share any further questions or concerns, as we welcome continued discussion for the broader research community.\n\n### **[W1] $t - \\tilde{t}_i$ May Be Large**\n\nWe acknowledge that, in cases where the labeled train set $\\mathcal{D}$ is large, the last sampled iteration may be large. However, we believe AdaSTaR remains effective even with larger $t - \\tilde{t}_i$ for two reasons:\n\n1.  Large post-training datasets typically comprise multiple sub-datasets [1, 2, 3, 4, 5] in practice, allowing AdaSTaR to be applied to each subset independently, keeping $t - \\tilde{t}_i$ manageable.\n    \n2.  Our empirical evaluation (**Appendix E**, **Table 4**) across training sets of 800 to 10,000 observations shows no degradation or efficiency deterioration as dataset size increases. Contrarily, we see the opposite: consider our largest train set, ANLI. Across Llama 3.2 3B and Gemma 7B as pre-trained models, our results show consistent 1st place accuracy, and PFLOPs reduction of $\\downarrow 62\\%$ and $\\downarrow 85.3\\%$, respectively (**Table 1, 6**). This is a greater improvement than the average of $\\downarrow 58.6\\%$.\n\nSetting $t - \\tilde{t}_i = 0$ would require significantly more inference compute, contradicting AdaSTaR's efficiency objective. $t - \\tilde{t}_i = 0$ entails sampling each $i$, every iteration $t$. Nevertheless, we acknowledge that future works that choose to focus on maximizing peak accuracy over efficiency have the option to obtain statistics at $t$.\n\n\n### **[W2] Win Rate = Difficulty?**\n\nWe use win rate $w_i$ (**line 179**)​ as a difficulty proxy because it directly reflects how challenging each sample is for the specific pre-trained model being fine-tuned.\n\n> In some cases, datapoints may be impossible to solve for current model capacity, yet one correct verification means these will often be sampled as well iiuc. Do you consider this a problem?\n\nRegarding samples that are impossible for current model capacity: this concern motivates our design choice to prioritize the last sampled iteration $\\tilde{t}$ in the HieMinHeap (**Equation 1**). By ordering samples primarily by $\\tilde{t}$ the algorithm naturally cycles through different observations regardless of their difficulty, preventing it from getting stuck on any single challenging example.\n\nFurthermore, going one step further, to ensure that the algorithm does not get stuck on a near-impossible observation we make the following design choice in **lines 168 - 170**:\n> We use the last sampled iteration rather than the last trained iteration because prioritizing based on training can cause the system to repeatedly attempt difficult examples it cannot yet solve, particularly when the model is weak, early in training.\n\nIn practice, we find that this design prevents the pathological behavior you describe while still appropriately prioritizing challenging samples that drive learning progress.\n \n\n### **[W3] Robustness and Stability**\n\nDue to the high compute (on-policy reinforcement learning-like) nature of STaR approaches, having numerous runs was out of our computational budget. While multiple runs would be ideal, our experimental design prioritizes reproducibility and includes sufficient evidence of stability:\n\n1.  As mentioned in **Appendix E**:\n> All training is done on the same arbitrary seed value of 10. This value has never been changed.\n2.  Evaluation is done using greedy decoding (**Table 3**).\n3.  In the paper, no instability or catastrophic failures can be observed across 12 experimental settings spanning three pre-trained model families (**Table 1, 5, 6**; **Figure 4**)\n\nWe suspect that baselines and related works do not include numerous runs for variance or confidence interval as there is not a meaningful level of training instability. Specifically, [6, 7, 8, 9, 10, 11] do not report variance or confidence intervals, so we follow their evaluation protocol for fair comparison\n  \n### **[Q1] Heuristic Choice**\n\nWe believe that our main heuristic choice comes from $f(\\alpha) := \\alpha^2$ where we discuss its underlying design choice in **lines 232 - 238**, and the footnote on **Page 6**. We acknowledge that this is a heuristic choice as we were unable to do an extensive empirical search on potential $f(\\alpha)$.\n\nNevertheless, we have put our best effort to point this out and discuss this in the paper. Consider **Appendix J**, **lines 780 - 783**:\n\n> Third, similar to other adaptive methods such as Adam (Kingma and Ba, 2015) and AdaGrad (Duchi et al., 2011), AdaSTaR introduces a new hyperparameter $f(\\alpha) := \\alpha^2$. A more granular tuning is deferred to future work. It is anticipated that such tuning could lead to further enhancements in AdaSTaR’s performance and efficiency.\n\n**When is $f(\\alpha) := \\alpha^2$ a good choice?** Our results show consistent improvements across 7 baselines and 3 model families despite minimal hyperparameter search, suggesting this choice is robust as a starting point. The quadratic form provides smooth curriculum transitions with monotonically increasing behavior ($f'(\\alpha) = 2\\alpha \\geq 0$ for $\\alpha \\geq 0$) while maintaining stability across different architectures.\n \n\n**Potential risks.** The main risk is suboptimality; a more thorough search over $f(\\alpha)$ could yield better performance. However, we observed stable training dynamics across all experiments, indicating low risk of training instability.\n\nFinally, we acknowledge that including an intuitive explanation for future researchers and practitioners’ $f(\\alpha)$ choice is important. We are planning to add the following part to the paper:\n\n> $f(\\alpha)$ should be set such that the range is $f(\\alpha) \\in [0, 1]$. A value of 0 corresponds to encouraging easy observations, while 1 corresponds to completely turning off the curriculum mechanism.\n\n### **[Q2] Well Tuned STaR?; and Robustness**\n\nWe respectfully disagree that AdaSTaR is a well-tuned version of STaR for the following three **(1, 2, 3)** reasons.\n\n**(1)** First, we note in **lines 244 - 245**:\n> For fairness, we optimize hyperparameters using the original STaR and apply them consistently across all methods\n\nThat is, we have already given a common hyperparameter search advantage to the original STaR baseline.\n\n**(2) Algorithmic novelty.** AdaSTaR introduces fundamentally new components: (i) online difficulty statistics ($\\tilde{t}$, $w$) computed with negligible overhead, and (ii) adaptive curriculum scheduling via $f(\\alpha)$. These are not hyperparameter tuning choices but algorithmic innovations absent from prior work.\n\n**(3)** Lastly, as mentioned in our response to **[Q1]**, our single hyperparameter $f(\\alpha) := \\alpha^2$ was chosen without extensive search, yet consistently outperforms 7 baselines across 3 model families. This suggests robustness rather than cherry-picking.\n\nWe believe that the robustness part of this question is intimately related to our response to **[W3]**. We would like to further note that the aforementioned [7, 8, 9, 10, 11] in **[W3]** are significantly more complex versions of STaR.\n\n### **[Q3] Sampling Near-impossible Observations?**\n\nPlease refer to our response to **[W2]**.\n\n### **Reference**\n[1] Teknium. “OpenHermes-2.5”, HuggingFace 2023.\n\n[2] Hugging Face Smol Models Research. “SmolTalk v2”, HuggingFace 2025.\n\n[3] Beijing Academy of Artificial Intelligence. “Infinity-Instruct”, HuggingFace 2024.\n\n[4] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel. \"Crosslingual generalization through multitask finetuning.\", ACL 2023.\n\n[5] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le. “Finetuned Language Models are Zero-Shot Learners”, ICLR 2022 (Oral).\n\n[6] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. “STaR: Bootstrapping Reasoning With Reasoning”, NeurIPS 2022.\n\n[7] Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing. “ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement”, ICLR 2025 (Oral).\n\n[8] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishab Agarwal. “V-STaR: Training Verifiers for Self-Taught Reasoners”, COLM 2024.\n\n[9] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. “Iterative Reasoning Preference Optimization”, NeurIPS 2024.\n\n[10] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. “B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners”, ICLR 2025.\n\n[11] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. “Scaling Relationship on Learning Mathematical Reasoning with Large Language Models”, Arxiv 2023."}, {"review_id": "MSNIpdph22", "reviewer": "Reviewer_2JDr", "summary": "AdaSTaR tweaks the classic Self-Taught Reasoner loop by picking training examples that have been neglected for a while and are still hard for the model, mixing in a few easier ones early on. The sampler is light-weight and drops straight into existing STaR code. Experiments on a broad set of reasoning tasks show better accuracy with a lot less training compute, and the gains hold across several small language  models.", "questions": "1. Will AdaSTaR work on training on a mixture of data from different datasets?\n2. How do AdaSTaR-trained model generalize to unseen test data? \n3. How do you calculate training FLOPs in detail?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper clearly spots a waste pattern in vanilla STaR.\n2. Implementation is simple: two counters per example and a min-heap, making it a seamless plug in to existing STaR code.\n3. Tests cover diverse datasets, base models, and baselines; results are consistent and the efficiency win is compelling.\n\nWeaknesses\n1. To my best understanding, Table 1 reports accuracy when training and evaluation share the *same* dataset, leaving open whether AdaSTaR helps models generalise to new tasks; transfer experiments would strengthen the claim.\n2. (Minor) The training FLOP reported in Table 1 reflect early-stopped checkpoints, not full-budget training. In scenarios where practitioners still spend the entire budget, the adaptive sampler’s faster convergence might not translate into actual cost savings, even though the speed-up is valuable in its own right.", "was_revised": true, "final_justification": "The additional experiments strengthen the paper. But I still think AdaSTaR would need to pair with some ways of prediction on when to early-stop, in order to demonstrate its actual speed-up in a real-world setting. I decide to maintain my score.", "rebuttal": "Thank you for your feedback and contribution to our research community. We are especially encouraged to see that you recognize:\n\n-   **Clear Takeaway**\n\t-   “clearly spots a waste pattern” (`2JDr`)\n-   **Consistently Strong Performance**\n\t-   across several small language models” (`2JDr`)\n-   **Consistently Strong Efficiency Gains**\n\t-   “a lot less training compute” (`2JDr`)\n\t-   “diverse datasets, base models, and baselines; results are consistent and the efficiency win is compelling” (`2JDr`)\n-   **Straight-forward Implementation**\n\t-   “seamless plug in” (`2JDr`)\n\nWe are happy to address the weaknesses **[W]** and questions **[Q]** you bring-forth in detail below. Please feel free to share any further questions or concerns, as we welcome continued discussion for the broader research community.\n\n### **[W1 & Q1] Transfer Generalization and Mixture of Datasets**\n  \n**Transfer Generalization.** You are correct that our study is done out-of-sample (held out test set) but for the same (train, test) dataset. Task transfer evaluation, while valuable, is not the focus of existing STaR methods [2, 3, 4, 5, 6, 7, 10], as these approaches are typically designed for task-specific improvement. Additionally, transfer evaluation is particularly challenging for the smaller pre-train-only models we focus on (Llama 3.2 3B, Qwen 2.5 3B, and Gemma 7B), which have limited cross-task capabilities compared to larger models.\n\n\n**Mixture of Datasets.** Following the standard protocol of related works [2, 3, 4, 5, 6, 7, 10], we do not provide any mixture of dataset experiments. While we believe this is an excellent future direction, we believe it is fair to assume that AdaSTaR or any other STaR-based approaches can be applied to each training set. Large post-training datasets typically comprise multiple sub-datasets [11, 12, 13, 14, 15] in practice, allowing AdaSTaR to be applied to each subset independently.\n\n**Additional Experiments.** Nevertheless, to provide some experimental results during the rebuttal phase, we run simple experiments (due to computational and time constraints) inspired by [10] to show that AdaSTaR’s results remain consistent in transfer and mixture of datasets.\n\nThe experiment protocol here is:\n\n**1.**  Train: GSM8K → Test: SVAMP\n\n**2.**  Train: GSM8K + SVAMP → Test: GSMK + SVAMP\n\n**3.**  Train: SVAMP + ARC-C → Test: GSM8K + CQA\n\nThe base model is Qwen 2.5 3B, and the train set is randomly shuffled. All other settings remain consistent with the original paper. We choose STaR-Acc as the baseline as it is the baseline with the best average performance and efficiency in our work’s experimental results.\n\n| **1.** | SFT | STaR-Acc | AdaSTaR |\n|---|---|---|---|\n| Accuracy ($\\uparrow$) | 80.0 | 88.0 | 89.0 |\n| PFLOPs ($\\downarrow$) | 248.2 | 509.1 | 65.2 |\n\n| **2.** | SFT | STaR-Acc | AdaSTaR |\n|---|---|---|---|\n| Accuracy ($\\uparrow$) | 71.2 | 84.1 | 84.3 |\n| PFLOPs ($\\downarrow$) | 196.4 | 223.7 | 25.25 |\n\n| **3.** | SFT | STaR-Acc | AdaSTaR |\n|---|---|---|---|\n| Accuracy ($\\uparrow$) | 42.8 | 68.2 | 68.4 |\n| PFLOPs ($\\downarrow$) | 10.55 | 186.15 | 97.4 |\n\nThis suggests that AdaSTaR's adaptive sampling benefits persist even in generalization and data mixture set-ups.\n\n\n### **[W2] Early Stopping**\n\nThis is a natural training behaviour of deep learning systems. The early-stopped iterations are very much akin to early-stopping epochs for SFT, in the sense that after that point the model overfits and accuracy tends to decrease. Practitioners can apply standard early stopping techniques [1] to capture these efficiency gains in practice.\n\nFor example, training SVAMP with Gemma 7B shows clear overfitting after iteration 14 (peak accuracy 89.5%), with subsequent performance declining to 80.0% by iteration 21:\n\n| iter 1 | iter 2 | iter 3 | iter 4 | iter 5 | iter 6 | iter 7 | iter 8 | iter 9 | iter 10 | iter 11 | iter 12 | iter 13 | iter 14 | iter 15 | iter 16 | iter 17 | iter 18 | iter 19 | iter 20 | iter 21 |\n|--------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| 0.565  | 0.625  | 0.695  | 0.685  | 0.74   | 0.755  | 0.815  | 0.79   | 0.82   | 0.845   | 0.845   | 0.855   | 0.89    | 0.895   | 0.88    | 0.86    | 0.875   | 0.865   | 0.835   | 0.755   | 0.8     |\n\nFor instance, \"stop after two non-improving iterations\" [1] would terminate at iteration 16 (14+2) with a final 89.5% accuracy, capturing efficiency gains. While AdaSTaR does not offer perfect prediction on when to early-stop, we do not believe that this is a weakness relative to related works: [2, 3, 4, 5, 6, 7, 10].\n\n\n### **[Q2] Generalization to Unseen Test Data**\n\nWe would like to clarify that AdaSTaR splits the training and test set to ensure that the test set is not seen during the training stage. You can feel free to view **Appendix E Further Details on Experimental Configuration and Setting**’s **Table 4**, which explicitly shows this.\n\n### **[Q3] How are FLOPs Computed?**\n\nWe will expand upon lines **259 - 261**:\n\n> We use FLOPs as our computational cost metric as memory usage remains approximately constant across methods. FLOPs are computed empirically following Kaplan et al. (2020), Sardana et al. (2024).\n\nWe follow Kaplan et al. (2020) [8], Sardana et al. (2024) [9] to compute total flops as:\n\n$FLOPs := 6ND_{tr} + 2ND_{inf}$, where $N$ is the parameter size, and $D_{tr}$, and $D_{inf}$ are the total token count for training and inference respectively. Factor of 6 accounts for forward pass ($2N$) + backward pass ($4N$) in training. Factor of 2 accounts for forward pass only during inference\n  \n\n### **Reference**\n\n[1] Bengio, Yoshua. \"Practical recommendations for gradient-based training of deep architectures.\" Neural networks 2012.\n\n[2] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. “STaR: Bootstrapping Reasoning With Reasoning”, NeurIPS 2022.\n\n[3] Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing. “ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement”, ICLR 2025 (Oral).\n\n[4] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishab Agarwal. “V-STaR: Training Verifiers for Self-Taught Reasoners”, COLM 2024.\n\n[5] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. “Iterative Reasoning Preference Optimization”, NeurIPS 2024.\n\n[6] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. “B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners”, ICLR 2025.\n\n[7] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. “Scaling Relationship on Learning Mathematical Reasoning with Large Language Models”, Arxiv 2023.\n\n[8] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. “Scaling Laws for Neural Language Models”, Arxiv 2020.\n\n[9] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws.” ICML 2024.\n\n[10] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models”, TMLR 2024.\n\n[11] Teknium. “OpenHermes-2.5”, HuggingFace 2023.\n\n[12] Hugging Face Smol Models Research. “SmolTalk v2”, HuggingFace 2025.\n\n[13] Beijing Academy of Artificial Intelligence. “Infinity-Instruct”, HuggingFace 2024.\n\n[14] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel. \"Crosslingual generalization through multitask finetuning.\", ACL 2023.\n\n[15] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le. “Finetuned Language Models are Zero-Shot Learners”, ICLR 2022 (Oral)."}, {"review_id": "tWISlPFmeR", "reviewer": "Reviewer_NZVF", "summary": "This paper proposes to modify STaR, an algorithm used in improving the training of many language models (LMs), so that its sampling remains both diverse and iteratively moves towards more challenging observations (curriculum learning). The authors point out several existing limitations with research involving STaR and address these issues with their proposed AdaSTaR. They test it on 6 benchmarks with a few LMs, and their proposed work achieves greater test accuracy.", "questions": "I quite enjoyed this paper, and am convinced by the paper's overall goal. I may consider changing my score to a Strong Accept if you can clarify some of my questions below. In particular, I am most interested in understanding the CoT exemplar notation better (my questions about lines 100 and 104) and the last question at the bottom about why use the formula chosen for capping how many updates to be made (see my questions/comments for line 234).\n\nLine 100: The symbol \"e\" is used both on the left of the equation and as an index to the indexed set. The math appears wrong to me. Did you mean something like: \"CoT exemplars $\\mathcal{E} = \\{ \\langle x_{e}, c_{e}, y_{e}  \\rangle \\}\\_{e = 1}^{E}$.\"? Then, shortly after, there is a reference to $c_{i}$: \"However, as no ground truth $c_{i}$ is available, ...\" Please confirm my understanding that the training dataset has CoT for each query, but we are not guaranteed it is the ideal CoT. \n\nLine 102 and/or line 104/108: What is the significance of the \\hat? Is it to denote that it was obtained/sampled from an LM? Edit: seems the \\hat is clarified on lines 179-180, but the authors should consider moving similar language earlier to its first appearance.\n\nLine 103: Is $k \\in [K]$ saying that $k$ is an element of the equivalence class of $K$? I am unsure this was the intended meaning here, as $[ \\cdot ]$ has a special but somewhat standard mathematical meaning, and $K$ was previously defined as $K \\in \\mathbb{N}$. \n\nLine 104: Is $e$ referring to a specific exemplar, its associated index, or the entire set of fixed few-shot CoT exemplars? Same issue with lines 105-106. Also, for as little as RL has been mentioned in this work thus far, I assume that \"If $r = 1$, it is accepted, and if $r = 0$, ...\" is referring to a reward as $r$? \n\nAlgorithm 1: Why are the variables on lines 1 and 2 being defined as dictionaries when they could just be vectors like $\\bar{t} = \\mathbf{0}$ and $w = \\mathbf{0}$? After all, they are indexed by integers, and all entries are initialized to zero (line 206). Seems unnecessarily verbose, unless you wanted the explicit size mentioned. Also, do we actually need the temporary variable defined on line 6? It is used on line 12\n\nLine 234: Why is this formula used for determining the highest priority? Why square the model strength? Why multiply it by $m$, and why apply the floor function? Additionally, the footnote uses $f$ to denote a function, but this does not match the previous text. I am also unconvinced by this existing justification in the footnote, and cannot see the relationship between $\\lfloor m \\alpha^{2} \\rfloor$ and why that should be used for popping the highest-priority observations. It does not seem to be very sophisticated or nuanced, but rather seems random or from ad hoc experimentation for a cutoff threshold that got decent results. Why not do something more nuanced, like dynamically determining the cutoff from the HieMinHeap?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Impressive contribution focusing on efficiency. \n+ Related work is thorough; I particularly appreciate the rigorous literature supporting the RL claim yielding long CoTs.\n+ The paper is written very well overall. Most of the material is explained nicely with professional and helpful visuals. \n+ The authors' proposed contribution is easy to follow, as they explained it. Their work seems to be very important to advancing LM research by improving the efficiency of STAR. \n+ Experiments are well done and diverse. \n\nWeaknesses:\n- Lines 111 to 113 are poorly written/justified. The authors simply state details about $\\beta^{t = 1}$, and how it follows the original implementation, but do not explain why they chose that despite them mentioning alternative STaR-based approaches remove the pre-determined $\\beta^{t}$ and instead set it to the size of the correct samples.\n\nVarious suggestions for edits, some minor, others major:\n\nLine 67: Should this be FLOPs instead of PFLOPs? Earlier in the abstract, it was the former. I believe this is a typo, but later on, in the Table 1 caption, you use Peta FLOPs. \n\nLine 75: Change \"... objective. To curate preference pairs, ...\" to use a semicolon instead of the period. The sentence that follows is directly related to the previous in finishing its thought, so it might be best to make that more explicit in the writing.\n\nLine 103 does not start the paragraph off well. Write: \"Let $K \\in \\mathbb{N}$ denote the number of CoT traces sampled as follows...\"\n\nLine 204: No need to use shorthand notation of \"w.r.t.\" - plenty of space to write it out completely. If used later in text, then \"with respect to (w.r.t.)\", but the undefined shorthand could confuse some readers.\n\nLine 207: Consider rephrasing to: \"... we utilize \\citeauthor{cormen_hie_min_heap}'s Hierarchical Min Heap (\\texttt{HieMinHeap})...\" - it looks strange with the shortened version in-line with the longer name.\n\nLine 229 - 230: The previous justification (lines 228 - 229) for why training accuracy is used over validation or test set accuracy is reasonable, but test set accuracy shouldn't even be an option by any means, as you are likely aware. I'd almost suggest removing its mention from the text in these lines, as using the test set accuracy to improve a model would be wrong on many different levels. \n\nLine 261: update so it says: \"... following techniques used by Kaplan et al. (2020) and Sardana et al. (2024).\" instead (or similar wording)\n\nTable 1 caption: Probably need to fully define PFLOPs before using it.\n\nAvoid using the word \"average\". Change \"average\" on line 293 to make use of \"mean\". For instance, \"... reducing training FLOPs by a mean of 58.6% (minimum of ...)\"", "was_revised": true, "final_justification": "Recommendation is unchanged except that a reduction of clarity from 3 (good) to 2 (fair) has been applied. I find some of the mathematics to be ambiguous and a bit sloppy, but if the authors are committed to addressing these matters, I shouldn't have further concerns.", "rebuttal": "Thank you for your thoughtful feedback and valuable contribution to our research community. We are especially encouraged to see that you recognize:\n\n-   **Clear Takeaways, and Impact**\n\t-   “very important to advancing LM research” (`NZVF`)\n-   **Consistently Strong Performance**\n\t-   “Experiments are well done and diverse” (`NZVF`)\n-   **Consistently Strong Efficiency Gains**\n\t-   “Impressive contribution … on efficiency” (`NZVF`)\n-   **Straight-forward Implementation**\n\t-   “proposed contribution is easy to follow” (`NZVF`)\n-   **Clear Communication**\n\t-   “Related work is thorough” (`NZVF`)\n\t-   “written very well overall … explained nicely with professional and helpful visuals” (`NZVF`)\n\nWe address the weaknesses **[W]** and questions **[C]** you bring-forth in detail below. Your contribution has helped us improve parts of the updated version of our paper. Please feel free to share any further questions or concerns, as we welcome continued discussion for the broader research community.\n\n### **[W1] Justification on Pre-determined $\\beta^t$**\n\nYou raise an important point. As we address this in detail (with visualizations) in the paper we would like to direct you to **Remark 1** (**lines 214 - 216**) and **Appendix D**. To briefly summarize, we stick with the original STaR’s direction as this was logically more sample (compute) efficient.\n\nFurthermore, the alternative approach of using the “Full” correct set is embodied in numerous baselines (STaR-Full, STaR-Acc-Full, STaR-Acc-Full-K, B-STaR, and ReST$^{EM}$). We empirically validate our discussion in **Remark 1** (**lines 214 - 216**) and **Appendix D**, as seen in **lines 269 - 271**, **Table 1, 5 , 6**, and **Figure 4**. In short, our approach that chooses not to use the alternative approach (“Full”) realizes both performance and efficiency gains\n\n### **[W2] Writing Improvements**\n\nWe appreciate your thorough suggestions, and have accordingly made the following revisions:\n\n**(1)** Line 67: PFLOPs → FLOPs\n\n**(2)** Line 75:\n\n> ... objective. To curate preference pairs, …\n\n→\n\n> \"... objective: to curate preference pairs, ...\"\n\n**(3)** Line 103:\n\n> K \\in \\mathbb{N} number of CoT traces are sampled as follows…\n\n\n→\n\n> Let K \\in \\mathbb{N} denote the number of CoT traces sampled as follows…\n\n**(4)** Line 204: w.r.t. → with respect to\n\n  \n\n**(5)** Line 207:\n\n> ... we utilize the Hierarchical Min Heap $\\texttt{HieMinHeap}$ (Cormen et al., 2022) …\n\n→\n\n> ... we utilize Cormen et al. (2022)'s Hierarchical Min Heap $(\\texttt{HieMinHeap}$) ...\n\n**(6)** Line 330:\n\n> ... for instance, validation or test set accuracy …\n\n→\n\n> ... for instance validation set (not used in final evaluation) accuracy …\n\nWe wholly agree with your statement on not using test accuracy. To our best knowledge, it is common practice to tune models using the train + validation set, then finally evaluating on the test set [1, 2, 3]. Please let us know if this judgement seems incorrect; we will remove this sentence entirely as it is not a part of our core contribution.\n  \n**(7)** Line 261:\n\n> FLOPs are computed empirically following Kaplan et al. (2020), Sardana et al. (2024).\n\n→\n\n> FLOPs are computed empirically following the method used by Kaplan et al. (2020) and Sardana et al. (2024).\n\n\n**(8)** Table 1 Caption:\n\n> Peta FLOPs (\\downarrow)\n\n→\n\n> Peta FLOPs (PFLOPs, \\downarrow)\n\n\n**(9)** Line 293:\n\n> 58.6% on average\n\n→\n> by a mean of 58.6%  \n\n### **[Q1] CoT Exemplar Notation**\nThanks for pointing this out; $e$ should not have been used on both LHS and RHS. We will modify the RHS: $e$ → $\\epsilon$, then correspondingly update the remaining text to follow this notation.\n  \nWe would like to further clarify on your questions:\n\n> Please confirm my understanding that the training dataset has CoT for each query, but we are not guaranteed it is the ideal CoT.\n\n> Is e referring to a specific exemplar, its associated index, or the entire set of fixed few-shot CoT exemplars?\n\nFor each given data set $\\mathcal{D}$, a few shot $e$ ($E$ in size) is commonly available as CoT exemplar [4]. Therefore, there is no query-specific $e$, rather there is a single $e$ for all queries (given the same $\\mathcal{D}$). This is the *de facto* CoT setting [5, 6], used equivalently across all related works.\n\n### **[Q2] \\hat Notation**\n\n> Is it to denote that it was obtained/sampled from an LM?\n\nYes, this is correct. Akin to **lines 179 - 180**, we would like to direct you to **line 99**, where we describe this:\n\n> To achieve this, $\\pi_\\theta^t$ generates $\\langle \\hat{c}_i, \\hat{y}_i \\rangle$ ...\n\n### **[Q3] [K] Notation**\n\nWe apologize for the confusion. We acknowledge that we have not explicitly defined the notation $[K]$ as $\\{1, 2, \\cdots, K\\}$, which is quite standard in computer science (e.g., 1.1.1 Example in [7]).\n\nTo avoid confusion, we will replace $[K]$ → $\\{1, 2, \\cdots, K\\}$.\n\n### **[Q4] r = reward?**\nYes, we would like to direct you to **lines 101 - 102** where we mention this:\n\n> Given the supervised dataset, a rule-based verifier defines a reward signal $r := \\mathbb{I}(y_i = \\hat{y}_i)$, where $\\mathbb{I}(\\cdot)$ is the indicator function.\n\nFor a more detailed discussion on the relationship between RL and STaR we would like to refer you to the following journal paper [8].\n\n### **[Q5] Algorithm 1’s Notation**\n\n**(1)** As mentioned in **line 162**, **Algorithm 1** represents a pseudocode. Therefore, our goal was to explain the algorithm in the clearest manner possible. We chose to use dictionaries as it intuitively provides a mapping for each query $i$.\n\nNevertheless we acknowledge that in the actual implementation a list (vector) will guarantee O(1) look-up when the index is known, thus, we are open to updating Algorithm 1 to vector-based. Please let us know if you believe that using vectors in the pseudocode is as intuitively straight-forward as the dictionary notation.\n\n**(2)**\n> Also, do we actually need the temporary variable defined on line 6? It is used on line 12\n\nWe realize that this was an error on our part; this will be removed in the updated version. Our original intent was to indicate that if a query $i$ has a small win rate $w_i$, it should not be accidentally re-sampled (.peek_next) in the same iteration $t$. However, like our code implementation, this is not necessary, as peeked $i$ are not pushed back in during the same iteration $t$.\n\n### **[Q6] Popping Mechanism for Curriculum**\n\n> Why multiply it by $m$?\n\nRecalling that $m$ is the total number of sampled observations, an implementation with no curriculum would be:\n> for $1, \\cdots, m$ do\n\nHowever, we are attempting to regularize the algorithm through a curriculum-style mechanism where we leave some easier samples to be re-sampled next iteration (see **lines 221 - 224**). Therefore, instead of popping and pushing $m$ observations we want it to pop and push $n \\leq m$ number of samples where $n \\in [0, m]$. To achieve this we multiply $m$ by $f(\\cdot) \\in [0, 1]$. As we want $f(\\cdot)$ to reflect model strength, we let $f(\\alpha)$ be a function of $\\alpha$.\n\nThe reasoning behind the choice of $f(\\alpha) := \\alpha^2$ is as described in **Page 6**’s Footnote:\n\n> The choice of $f(\\alpha) := \\alpha^2$ is a hyperparameter. It allows more repetition of easy observations when the model is weak, and $\\textit{rapidly}$ phases out this regularization effect as the model strengthens.\n\nConsidering $f(\\alpha) \\in [0, 1], m(1 - f(\\alpha))$ is the number of easy samples that are going to be mixed into the next iteration. Therefore, $\\alpha^2$'s sharper curve will be a faster phase-out of this easy observation mix, as $\\alpha$ increases.\n\n> why apply the floor function?\n\nThis is required as we require an integer value. It is not possible to pop a non-integer real value number of observations.\n\nWe further discuss the potential limitation of this design choice in **Appendix E**’s **lines 780 - 783**:\n\n> Third, similar to other adaptive methods such as Adam (Kingma and Ba, 2015) and AdaGrad (Duchi et al., 2011), AdaSTaR introduces a new hyperparameter $f(\\alpha) := \\alpha^2$. A more granular tuning is deferred to future work. It is anticipated that such tuning could lead to further enhancements in AdaSTaR’s performance and efficiency.\n\n> Why not do something more nuanced, like dynamically determining the cutoff from the HieMinHeap?\n\nAs observed in other adaptive methods (**Appendix E**; **lines 780 - 783**), it is challenging to introduce no new hyperparameter to the system. It is unclear if there is an obvious approach to set $f(\\alpha)$ directly from the HieMinHeap. We do not believe an introduction of a hyperparameter takes away from the core contribution.\n\nConsidering that we have done a minimal search for $f(\\alpha)$, yet achieved consistently best test accuracy and FLOPs reduction against 7 baselines and across 3 model families suggests that there is more room for improvement. We are optimistic that future empirical searches will further drive performance and efficiency gains.\n\n### **Reference**\n\n[1] Mitchell Wortsman, et al, “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time”, ICML 2022 (Oral).\n\n[2] Sonia Meyer, et al, “A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case”, Arxiv 2024.\n\n[3] Zhenqian Shen, et al, “Efficient Hyper-parameter Optimization with Cubic Regularization”, NeurIPS 2023.\n\n[4] Gao, Leo et al. “The Language Model Evaluation Harness”, Zenodo 2024.\n\n[5] Jason Wei, et al, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, NeurIPS 2022.\n\n[6] Xuezhi Wang, et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, ICLR 2023.\n\n[7] Stanley, Richard P. \"Enumerative combinatorics volume 1 second edition.\" Cambridge studies in advanced mathematics (2011).\n\n[8] Avi Singh, et al. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models”, TMLR 2024."}, {"review_id": "pwOEdEKtv8", "reviewer": "Reviewer_5Fqt", "summary": "The paper introduces AdaSTaR, an enhanced version of the self‑taught reasoners training loop that replaces random data sampling with two adaptive strategies: one that up‑weights examples the model has rarely seen to maintain coverage, and another that shifts the mix of easy and hard problems to match the model’s improving skill. This balanced, curriculum‑like sampling lets the model focus on challenging cases without wasting compute on items it already solves. Tested on six reasoning benchmarks, AdaSTaR attains the best accuracy in every setting  and the gains carry over to larger and different pre‑trained language models.", "questions": "Please refer to the Strengths And Weaknesses.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- This paper tackles a clear pain point in self-taught reasoners: the random sampling loop wastes compute on easy cases and neglects hard ones. \n\n- The authors proposed an adaptive sampler that tracks how often each example is seen and how often the model answers it correctly, then chooses future batches to balance coverage and follow a curriculum that matches the model’s current skill. \n\n- The experimental section is broad, covering six public reasoning datasets that span science, commonsense, causal, natural-language inference, and mathematics, plus checks on three different base models.\n\n**Weaknesses**\n\n- Some design choices are not well explained. For instance, the sampler first sorts items with the oldest sampling time and, within that set, prefers the hardest cases, which puts difficult problems early. It will be beneficial if the authors clarify why this order is better than the usual easy‑to‑hard order that often helps learning.  \n\n- The technical depth is not large.\n\n- The proposed method keeps outcome verification, it may reinforces chains that reach the right answer for the wrong reason.\n\n- The approach still requires ground-truth final answers to decide which chains are correct, so it cannot help in settings where only unlabeled questions exist, and the paper does not discuss how to relax that need.", "was_revised": true, "final_justification": "I am still not fully convinced by the explanation for the specific design choice of prioritizing the hardest cases within the oldest sampling time group. While I understand the motivation to address over-training of easy examples, the authors have not provided sufficient theoretical or empirical justification for why this particular ordering (oldest-first, then hardest-first) is optimal compared to alternative curriculum strategies. Additionally, this work primarily combines existing models without proposing novel theoretical contributions or formal analysis of the sampling strategy's convergence properties. Considering these limitations, I will keep my score unchanged.", "rebuttal": "Thank you for your feedback and sharing your time contributing to the community. We are especially encouraged to see that you recognize:\n\n-   **Clear Takeaway**\n\t-   “tackles a clear pain point in self-taught reasoners” (`5Fqt`)\n-   **Consistently Strong Performance**\n\t-   “best accuracy in every setting … gains carry over to larger and different pre‑trained language models” (`5Fqt`)\n-   **Consistently Strong Efficiency Gains**\n\t-   “experimental section is broad … plus checks on three different base models” (`5Fqt`)\n  \nWe address the weaknesses **[W]** you bring-forth in detail below. Please feel free to share any further questions or concerns, as we welcome continued discussion for the broader research community.\n\n### **[W1] Usual Easy-to-Hard**\n\nWhile easy-to-hard approaches are commonly used by researchers and practitioners, our approach, while related, is meaningfully different. More precisely, our main motivation for the need for easy-to-hard scheduling is different, and it is done automatically.\n\nAs described in **2.2 Motivation: Need for Adaptive Data Sampling**’s **lines 122 - 140**, and visualized in **Figure 3**, we find that the random observation sampling mechanism in STaR causes a sub-optimal over-training of easy and under-training hard observations. Therefore, in our framework, we attempt to encourage sampling more challenging observations. However, up-sampling more challenging observations raises false positives, and thus, we require a curriculum-style implementation (**lines 141 - 159**).\n\nMoreover, the easy-to-hard is done wholly automatically without researcher’s manual data selection. This is an important contribution as the observation order selection is done without human labor.\n\n###  **[W2] Technical Depth**\n\nWe respectfully disagree with the assertion that our work lacks technical depth. AdaSTaR appears straightforward because we invested significant effort in making the algorithm transparent and easy to reproduce, but the underlying solution is not shallow. In fact, our contribution requires several non-trivial points that, to our knowledge, are absent from prior STaR literature:\n\n**(1)** We clearly and rigorously identify a problem in existing STaR algorithms (**Figure 3 (a), 6, 7, 8**).\n\n**(2)** We discuss, then empirically analyze why a naive solution is problematic (**Figure 3 (b)**, **Appendix C**).\n\n**(3)** We set-up a comprehensive and thorough experimental study from the perspective of performance and efficiency (**Table 1, 5, 6, Figure 4**). Moreover, we agree with Reviewer `NZVF`’s comment that the “Related work is thorough”.\n\n**(4)** We include an ablation study (**Table 2**) to thoroughly understand the components of our method, and also how the method affects the distribution of the trained observations.\n\n  \n\nFurthermore, we believe that our approach’s straight-forward nature is a strength as this means that the method has been communicated clearly, and reduces replication challenges. This has been listed as a strength on the other three remaining reviews (`suRc`, `2JDr`, `NZVF`).\n\n  \n\n### **[W3] Outcome Verification**\n\nWe believe that this weakness is out-of-scope as this is an orthogonal direction. The most common and *de facto* setting is where only the final answer label ($y$) is provided, making our approach most wide-reaching. Similarly, related works [1, 2, 3, 4, 5, 6] use outcome verification. Our work and others alike, achieve strong performance gains even without process verification.\n\n\nWhile Process Reward Models (PRMs; [7]) are an important orthogonal line of work, they require significant human annotation, stronger models, or significantly more compute to make possible.\n\n\n### **[W4] Label-free?**\n\nWe believe that this is an excellent opportunity for future work [9]; but out-of-scope for this paper. Our method and other STaR based methods [1, 2, 3, 4, 5, 6, 8] all assume the most common setting where the final answer label ($y$) is available. Training base models with labeled datasets (using e.g., AdaSTaR) is an integral and irreplaceable part of the training pipeline of LMs [10, 11].\n\nWe believe that **[W3]** and **[W4]** are great opportunities for future developments, but it is challenging to cover every possible direction in a single self-contained paper.\n\n\n### **Reference**\n\n[1] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. “STaR: Bootstrapping Reasoning With Reasoning”, NeurIPS 2022.\n\n[2] Xiangyu Peng, Congying Xia, Xinyi Yang, Caiming Xiong, Chien-Sheng Wu, and Chen Xing.\n\n“ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement”, ICLR 2025 (Oral).\n\n[3] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishab Agarwal. “V-STaR: Training Verifiers for Self-Taught Reasoners”, COLM 2024.\n\n[4] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. “Iterative Reasoning Preference Optimization”, NeurIPS 2024.\n\n[5] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. “Scaling Relationship on Learning Mathematical Reasoning with Large Language Models”, Arxiv 2023.\n\n[6] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models”, TMLR 2024.\n\n[7] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe. “Let's Verify Step by Step”, ICLR 2024.\n\n[8] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. “B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners”, ICLR 2025.\n\n[9] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, Bowen Zhou. “TTRL: Test-Time Reinforcement Learning”, Arxiv 2025.\n\n[10] OLMo Team. “2 OLMo 2 Furious”, Arxiv 2025.\n\n[11] Qwen Team. “Qwen2.5 Technical Report”, Arxiv 2025."}]

Author rebuttal:

Meta-review: This paper introduces AdaSTaR, an adaptive data sampling method to improve the efficiency of STaR. The core contribution is replacing STaR's inefficient random sampling, which often over-trains on easy examples while under-training on difficult ones, with two adaptive sampling strategies: one that promotes diversity by prioritizing less recently sampled data points, and another that creates a curriculum by dynamically adjusting data difficulty to match the model's current capabilities. 

The primary strength of this work is its compelling and consistent empirical result; it successfully addresses a well-known "pain point" in STaR by substantially improving training efficiency and model performance across a broad range of datasets and models. The proposed method is also simple and lightweight, making it easy to implement within existing STaR frameworks. The main weaknesses were the heuristic nature of some key design choices and the practical realization of the efficiency gains is dependent on an effective early-stopping strategy, which the paper does not provide.

During the rebuttal period, the authors were highly responsive and effectively addressed most of the reviewers' concerns. New experiments added also addressed the concern about the merits of Adastar in transferability and generalizability. Overall, the paper is sound, and the demonstrated improvements in training efficiency are significant. The problem of inefficiently sampling easy problems is a known limitation of STaR, and this paper provides a practical and effective solution to this pain point. Initial concerns about the method's generalizability were well-addressed during the rebuttal. Of the remaining weaknesses, for the heuristic nature of the curriculum, the AC believes that the substantial practical gains appear to outweigh the lack of a theoretical proof of optimality. The dependency on early stopping is a valid point, and while it is a general problem in the field, the authors are encouraged to discuss this limitation more explicitly. The merits of this work ultimately outweigh its flaws.
Decision: Accept (poster)