Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
Authors: Jiarui Yao
Yifan HAO
Hanning Zhang
Hanze Dong
Wei Xiong
Nan Jiang
Tong Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models; Reinforcement Learning; Math Reasoning; Gradient Variance Minimization
Primary area: reinforcement_learning

Abstract: Chain-of-thought (CoT) reasoning in large language models (LLMs) can be formalized as a latent variable problem, where the model needs to generate intermediate reasoning steps. While prior approaches such as iterative reward-ranked fine-tuning (RAFT) have relied on such formulations, they typically apply uniform inference budgets across prompts, which fails to account for variability in difficulty and convergence behavior. This work identifies the main bottleneck in CoT training as inefficient stochastic gradient estimation due to static sampling strategies. We propose GVM-RAFT, a prompt-specific Dynamic Sample Allocation Strategy designed to minimize stochastic gradient variance under a computational budget constraint. The method dynamically allocates computational resources by monitoring prompt acceptance rates and stochastic gradient norms, ensuring that the resulting gradient variance is minimized. Our theoretical analysis shows that the proposed dynamic sampling strategy leads to accelerated convergence guarantees under suitable conditions. Experiments on mathematical reasoning show that GVM-RAFT achieves a 2-4x speedup and considerable accuracy improvements over vanilla RAFT. The proposed dynamic sampling strategy is general and can be incorporated into other reinforcement learning algorithms, such as GRPO, leading to similar improvements in convergence and test accuracy.

Optimizing Chain-of-Thought Reasoners via Gradient
Variance Minimization in Rejection Sampling and RL
Jiarui Yao!-
Yifan Hao!
Hanning Zhang'
Hanze Dong²
Wei Xiong'
Nan Jiang'
Tong Zhang'
'University of Illinois Urbana-Champaign
Champaign
MMicrosoft Research
Abstract
Chain-of-thought (CoT) reasoning in large language models (LLMs) can be formal- ized latent variable problem, where the model needs to generate intermediate reasoning steps. While prior approaches such as iterative reward-ranked fine-tuning (RAFT) have relied on such formulations. they typically apply uniform inference budgets across prompts, which fails to account for variability in difficulty and convergence behavior. This work identifies the main bottleneck in CoT training as inefficient stochastic gradient estimation due to static sampling strategies. We propose GVM-RAFT, prompt-speciffe Dynamic Sample Allocation Strategy designed to minimize stochastic gradient variance under a computational budget constraint. The method dynamically allocates computational resources by moni- toring prompt acceptance rates and stochastic gradient norms, ensuring that the resulting gradient variance minimized. Our theoretical analysis shows that the proposed dynamic sampling strategy leads to accelerated convergence guarantees under suitable conditions. Experiments on mathematical reasoning show that GVM-RAFT achieves 2-4x speedup and considerable accuracy improvements over vanilla RAFT. The proposed dynamic sampling strategy is general and can be incorporated into other reinforcement learning algorithms, such as GRPO, leading to similar improvements convergence and test accuracy.
Introduction
We consider mathematical reasoning with large language models (LLMs)): given prompt x E X. and aims to produce a correct final answer 2 Z. A prevalent approach in this area is the chain-of- thought (CoT) reasoning (Wei et al., 2022), which the model generates step-by-step rationale y before outputting the final answer 2. In practice, we are typically given pre-trained and instruction fine-tuned LLM parameterized by 0g and training samples {(x,, } prompt-answer pairs. Additionally, we assume access to verifier r*, + {0. 1} that indicates whether predicted answer correct or not. This is standard in practice and is particularly popular by the recent DeepSeek-R1 project (DeepSeek-Al et al., 2025), which suggests using only symbolic verifier rather than training a neural reward model. The goal is to improve model performance by training to generate high-quality CoT rationales that improve the final answer accuracy unseen prompts.
We formalize CoT reasoning latent variable problem, treating the rationale y as hidden. From this perspective, we propose new algorithmic framework based on the algorithm, which we formalize in Section Prior works such as Sordoni et al. (2023); Singh et al.
(EM)
"The first two authors contributed equally with random author order, detailed contributions deferred Ap- pendix A. Emails: {jiarui14, yifanh12, hanning5, wx13, nanjiang, Gozhang)@illing edu, hanzedong@miccrosen wrosoft....ooooooooooooooooooo.. com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(2023) have shown that this EM framework can be implemented as variant of iterative eoward-ranke fine-tuning (RAFT) (Dong et al., 2023: Touvron et al., 2023). also known as rejection sampling fine-tuning in the literature. Specifically, RAFT iteratively alternates between the E and M steps:
1. E-step: Prompt LLMs to generate n responses per prompt, and keep responses with the highest reward only (with the correct final answers). This process can be thought to be approximating the posterior distribution of the latent variable and the evidence lower bound (ELBO). See Section 2 for formal presentation.
2. M-step: Fine-tune the LLMs the selected responses from the E-step. The fine-tuned model is used for the next-iteration E-step.
(Final Policy)
GVM Iterations
fpr O o Accept
[GJi
Lipschitz Coefficients
T times
Stage 2
Budget Assignment
Stage Estimation
1T-1
2
(SFT Model)
Figure 1: The demonstration of the whole pipeline for GVM. Starting from To, which a supervised fine-tuning (SFT) model, GVM will iteratively conduct the estimation and budget assignment process for T times according to the description in Algorithm 1. Each iteration could be decomposed into two stages, and the final policy model will be returned after those iterations.
Although RAFT and its variants have been widely applied tasks in the post-training stage of LLMs, they are primarily motivated by the idea of imitating the best-of-n sampling rather than variance reduction. As result, theoretical understanding is often lacking. For instance, these RAFT-style approaches typically adopt a uniform sampling strategy that treats all prompts equally, ignoring differences in sample efficiency or convergence behavior
In this work, we revisit CoT reasoning under the EM framework and provide a deeper theoretical analysis. We identify the key bottleneck as the Monte Carlo estimation of the ELBO gradient during the E-step, which arises due to the intractability of going over all possible CoT rationales. Previous works mainly use the best-of-n sampling to allocate the inference budget uniformly (Sordoni et al., 2023; Singh et al., 2023), which overlook the differences the convergence rate under different prompts. To address this, propose dynamic sample budget allocation strategy that adaptively assigns computational resources across prompts based on theoretical insights. This leads more efficient Monte Carlo estimation of the ELBO gradient. Our resulting algorithm, a refined RAFT variant with dynamic inference budget scheduling through Gradient Variance Minimization (GVM-RAFT), achieves substantial performance improvements and even sometimes outperforms state-of-the-arr deep RL methods such as GRPO (Shao et al., 2024) in our Qwen-based case studies. In particular, the sample budget allocation schedule itself can be of independent interests and we also extend our techniques to the RL algorithm, where it also brings notable improvements. We summarize our contributions follows:
1. We revisit the EM framework and RAFT in the context of CoT reasoning, and identify that major limitation of current approaches lies in inefficient stochastic gradient estimation caused by uniform and static sampling strategies (i.e., best-of-n sampling). which fail to account for prompt-specific difficulty and convergence behavior.
2. Motivated by the goal of minimizing the variance of stochastic gradient, we propose dynamic sampling strategy that adaptively allocates computational resourees based

---

prompt hardness and gradient norms. Our approach provides both intuitive theoretical insight and rigorous convergence guarantees, establishing principled framework for efficient on- policy sampling under computational budget constraints.
3. We apply GVM to both RAFT++ and GRPO algorithms with real-world experiments on mathematical reasoning tasks. Our results demonstrate that the proposed approach achieves
x speedup in convergence rate, with considerable improvement of the final test accuracy.
Problem Formulation and EM Framework
In this section, we formally define the problem, review existing approaches, and motivate our algorithm designs. We consider the chain-of-thought (CoT) reasoning process as:
h y y x E X E ) E Z,
where x is prompt, y is the intermediate CoT and = is the final predicted answer. We fit the data [z, y, 2| within the following distribution function class:
E (y 2/2 0) Pyy| 0) P(2ly. 0) R").
(I)
Our target is to learn a good model M (0) E. which minimize the negative log- bogeliieldhood of predicting the correct answer:
(()) Ee In (= 0),
where do prompt distribution and PP(-10) denotes the distribution induced by the model with parameters 0. While many math datasets include reference CoT rationales, we do not include these reference rationales y here. This because recent practice typically does not fine-tune L.LMs on these CoT rationales after the pre-training/SFT stages (DeepSeek-Al) et al., 2025; Team et al., 2025).
The process from to z can be complex, making difficult to directly estimate the distribution P(z r) from the observed data [zr, z]. However, by introducing a latent intermediate variable the conditional distributions Pyy x)) and P( y) are often easier to estimate, thereby simplifying the problem:
(()) EEodo
Σ ()Pl2,0 0, VEV
Introducing the intermediate CoT y naturally motivates the of the ampectationmmaximizaain (EM) algorithm, which iteratively approximates the posterior over latent variables.
Derivation of the EM algorithm. We consider a training set B {(z, ) drawn from de with 2; being the labeled ground-truth truth answer to illustrate the idea. Following the standard derivation of EM algorithm, can bound the L(I) follows:
m
m
Plu. 24 2, () EQ.()) (1(v)
((0)
- E In
DEY
Ply, zi(20,0) 0) Q:(1)
<
Qn(v) In
(3)
1=1 PEY
E Ey-Qr(n In Ply, 2, 21 ( Σ H(Qi()))
(o(
where Qi(-) is probability distribution over y so that Q:(y) 2 0 and Eyey Qi(v) = 1. The last inequality is from the convexity of - In(.) and Jensen's 's inequality (Lemma 3) and H(p) -E In ((t) is the entropy.
In the E-step, our goal is find (v) to get sharper upper bound for L(0). In particular, the equality is reached with the posterior distribution of y: P(ule,, 0) (P(
PVv 2,(201
P(2)(,,,)) Eyey P(yla, == PPyl,,, -P(s)yy 0) Z(z,,
Q:(v) P(y 2),0)
(4)
3

---

where Z(x;, 2s. 8) denotes the normalization constant and the second equality uses Equation (1).
Here- -Jp(0) th (y): is referred to the evidence lower bound (ELBO)" in the litera- ture (Bishop and Nasrabadi, 2006; Kingma et al., 2013), and P(y 24, 0) is the induced posterior distribution, which is not the equivalent to the autoregressive distribution of CoT when we present the LLMs with prompts and answers.
In the M-step, to minimize [(0), we can fix Qv( in Equation (4) and indirectly minimize To (0).
Note that we do not include the entropy loss because it is a constant in the M-step since we will fix Q:(y). To summarize, the EM algorithm will altemnae between the following two steps: iteration t,
1. E-step: Update the posterior distribution of the latent CoT Q' according to Equation (4) and obtain the Jq. defined in Equation (3).
2. M-step: The goal of the M-step is to update LLMs (6,) to minimize Jqu:
m
E ((() In Ply, 24 22 0).
The updated model (f is used for the next-iteration E-step.
To apply the EM algorithm, we need sample y (y) approximate the objective. central challenge is that computing (v) and Tor (0) requires summing over all possible latent CoT reasoning y E y as we need to get the normalization constant in Equation (4). This process usually computationally intractable. Therefore, the objective Jor (f) must be approximated via sampling.
To approximate the posterior Q:(y), one standard approach is rejection sampling (Neal, 2003). We remark that we refer rejection sampling to the one in statistics, which is used to approximate a target distribution (v(v) by proposal distribution P(y) sr, 0), which
sample from. In the literature of RLHF or LLM, rejection sampling is often referred to as the best-of-n sampling (Bai
al., 2022; Ouyang al., 2022). Specifically, to approximate Qy)) by Pzyl 0), the rejection
sampling proceeds as follows:
1. Sample y P(yla,, 0);
2. Draw u ~ Uniform([0, 1)):
3. Accept y if S Flyl where M satisfies Q,(v) S M M PPyl,,, 0) for all y E y
Q:(u)
We notice that a valid choice of M is 1/Z(z;, 2;, 0), leading to an accept probability of P(z.ly, 0). In mathematical reasoning, given the CoT rationale, the final predicted answer is typically of low randomness. Then, the rejection sampling in statistics aligns well with the best-of-n sampling, where we only keep all the responses with the correct answer. This eventually leads to the RAFT- type algorithms. We remark that this connection between the EM framework and the RAFT-type algorithms has been previously observed in Singh al. (2023); Sordoni et al. (2023).
However, we argue that the current best-of-n sampling implementation is overly coarse and insuffi ciently faithful the true E-step. Specifically, if a prompt x; is very difficult for current M(.)), the density of Fly [r; 0) will concentrate on the wrong CoT rationales so that (P(2; [y, 0) is close to zero can accept one valid In contrast, if our current model (0,)) can already output correct CoT rationale y in most of the time, we will accept most of the generated responses. Eventually, with a fixed inference budget, this uniform allocation (n responses per prompt) tends to bias the accepted samples toward easier prompts with higher acceptance rates.
In this case, the accept probability will be very low and need many samples before
Notation
The true marginal negative log-likelihood is denoted as L(0). The negative ELBO at iteration is denoted as L. which is equal to
m
(² = Jq: Σ H(Q.(w))
Our derivation in the next section will focus on Jor (0) since entropy term is considered to be constant in M-step when fixing Q We also present a notation table in Table to improve readability.
We consider the negative log likelihood here it becomes an upper bound

---

Algorithm Mcta Algorithm: GVM-EM
1: Input: Initial parameters Oo. training samples D = [(r, zi) number of epochs T, initial posterior
Q" = P(- (0).
2: for t 0,....,
EEsteep (Expectation):
Sample set samples B, {mi, 24 ) Update the posterior distribution over latent CoT rationales
('(+) using Equation (4).
5: For each prompt 7;, compute the required number of samples nł according (1) Theoretical Proposition
or (2) Practical Algorithm
Perform rejection sampling to obtain accepted responses y. Collect corresponding (æ,, 24, y) D
such that ~ (.).)
7: > M-step (Maximization):
8:
Update model parameters via gradient descent using:
1 VoM((O)) = m18
Σ kyE
Vo log P(ys. az
9:
end for
10:
Output: Final model (fr).
3 Gradient Variance Minimization by Dynamic Sample Allocation
To address the limitations of best-of-n sampling, propose a dynamic inference budget allocation strategy that adapts to the acceptance rates of rejection sampling for each prompt z;. The overall meta-algorithm is presented in Algorithm 1, and in what follows, describe the budget allocation mechanism in detail.
3.1 Dynamic Inference Budget Scheduling to Minimize Gradient Variance
Unbiased gradient estimation. We begin by formulating the true gradient at iteration t under the EM objective Jo::
m Σ
m Σ E By V In Ply, 24 zz, 0),
VJg: (0)
Σ WEY
V
(5)
where (1( = P(yl,,,,,, 2,, 0(-1) is the posterior distribution of y. However, this distribution is intractable to compute exactly. Therefore, we approximate (y(y) via rejection sampling by drawing ni times from current LLMs P(y|,, 05-1). This leads the following unbiased estimator for Equation (Detailld proof deferred to Appendix E.)
Lemma 1 (Unbiased Gradient Estimator). In the iteration t, denoting D as the of accepted samples on related to (Iz; 21), we have the following unbiased gradient estimator for Jqu:
m
Σ
In 0r-1),
(6)
BEDI
where pt (y-P((la..)) P(zily 0) is the average accept rate of rejection sampling.
Variance-aware sampling allocation. While the estimator above is unbiased, its variance can vary significantly across prompts. Prompts with low acceptance rates introduce high variance due to the small number of accepted samples. Reducing variance is crucial for efficient training with stochastic gradient, as also emphasized in prior works statistics and optimization, including Roux et al. (2012); Johnson and Zhang (2013); Defazio et al. (2014); Chen et al. (2018). To design more efficient sampling strategy, we analyze the variance of the gradient estimator and optimize the allocation.
Lemma (Upper Bound of Variance Gradient Estimator )
Σ
v (In (ln 2i |,, 211,,
E Eynnr || V (In Ply, 2, 2,,0,, 0)
JEED;
G²
5

---

The proof is deferred to Section E. Given a fixed total sampling budget N, we seek to allocate {nā} to minimize this upper bound: min
G²
S.L. S. m N.
In practice, prompts totally beyond the ability of the current LLMs or cannot be evaluated by the verifier (e.g., due to some label error). This leads to extremely low acceptance rates and unstable gradient estimations. To mitigate this, introduce regularization term that penalizes sampling such prompts. The revised objective becomes:
G a/(P)) DAN
Σ 8=1
Σ = 1=1
min
S.T.
where a > 0, 2 2 are hyperparameters that control the regularization strength. For example, as the accept rate of - O, the weight (1 a/ (/ + O, which leads to the fact that sample size nf + and prevents excessive sampling uninformative prompts. Recent studies such as Xiong et al. (2025a) have also demonstrated the critical role plays for stable training of online RL algorithms.
Solving the regularized optimization problem in Equation (7) yields the following closed-form solution to the optimal sampling allocation:
Proposition 1. The optimal number of samples allocated to each prompt is:
Gi/ V
+ op
G
n N. Ei G1/ D + T.
Vi I...., m.
pt
(m:)a-
The proof's is deferred to Appendix E.
Remark 1. Accepted sample size has lower bound
(/)(0(-1)))100)
Σ
Σ
m
G
Di (Dt wtto-1 G.
-1/2
With Remark 1, our proposed budget scheduling method is not only efficient but also guarantees sufficient number of accepted samples during the training process, even in the presence of informative prompts. Algorithm 2 shows a practical implementation of the GVM algorithm.
3.2 Theoretical Result
In this section, we present the theoretical guarantee loss convergence. Without loss of general- ization, we assume that each E-step is followed by le M-steps. For the t-th E-step, let Dy denote the batch size for prompt x; at the r-th M-step, where kt k < r S kt, and the corresponding sample batch is denoted by B, The upper bound loss function we construct is denoted by (0):
(0) == - By In P(y, 2)2,
We take the notations below for simplification:
T k-1
T
Ak( =
Ak(
VL (0 (1(( 0,
T
2(k,T) = Σ Am/ 0
EV (get-k+r) 0,
where V(ght-44)) mild smoothness conditions, we can derive the following result.
m アー Ey V9 (InP(y,, 2i 0kc-)+))) Under
me
Theorem (Decreasing rate with smoothness condition.). Suppose - In Pny, 2|22, 0) is / 1moothooth with respect to 0. If < 7 < 7, then the proposed algorithm satisfies that
2 ((K,T))
B [[C(0<T) [())) (((0"))-[[[e" [C(0o) - -
27
"The expectation is taken over all sources of randomness, including the sampled data {D}} and the selected batches }B }.

---

In Theorem 1, with sufficiently large sample size, Rikk, T) will be small enough, which ensures that the right-hant side of the inequality negative. This guarantees the loss function decreases each iteration. Furthermore, if the loss function exhibits convexity, we can derive the following result:
Theorem 2 (Decreasing rate with smooth and convex condition.). Suppose In P(y. 2|2. 8) is 1/m-smooth and corvex with respect to 0. If < n S 7/2, then the proposed algorithm satisfies that
E [C(0w))) [()')] - E (C(06) [())) -
1) (2, T) 2k
A (K, 4k
17
7 ()kkT). 4k
In Theorem 2, the right-hand side includes additional negative term, - (k, T); which indicates a faster rate of decrease in the loss function. Specifically, during each E-step, the gradients across M-steps vary only slightly, this implies that:
E
(L.(0m=<++))11 ~ (2b
E Vi (00-- A2th)) ka
which further induces that
1/2 ((k,T)
" (k) +
" (k;
--
" 1 (k,
7 -1 T),
" T(K,T)
2k
4k
4k
2
4k
27
as well as
2 (k, T)
7 4k
Ak, +
"
Ω(k T) < < 7 - T)
+ " 2(k, T).
2k
4k
27
All of the proofs deferred to Appendix E.
Guaranteed Decrease in Our Proposed Method. From the theorems above, we know that with a sufficiently large sample size, Si(, 7') becomes small, and the upper bound for E (C(0kz) - [((*)] E [C(()) - [(t') } is strictly negative. This implies that our proposed method consistently decreases the objective loss, thereby ensuring an effective training process.
Reduction of the variance term Okk 7). Okk, T) leads to more efficient training. On one hand, increasing the sample size reliably reduces this term. On the other hand, under finite budget in practice, an effective budget scheduling strategy can also reduce Q(k: T) and thereby improve training efficiency. Our algorithm achieves this via the scheduling scheme defined in Algorithm 2.
According to the upper bounds derived above, a smaller
Trade-off in Budget Scheduling Update Frequency. In our algorithm, the budget scheduling is updated every i optimization steps. Updating more frequently (i.c., using a smaller k) improves training efficiency but increases computational overhead. Conversely, updating less frequently reduces computation, but as optimization proceeds, the loss gradient norm C tends to become small, which can slow down training. Therefore, choosing an appropriate value of & requires balancing this trade-off between efficiency and computational cost.
3.3 Practical Implementation
Practical implementation of GYM. In this subsection, we describe how to implement the inference budget allocation strategy in practice, summarized in Proposition 1, and the implementation is referred to GVM Gradient Variance Minimization. Although the optimal sample sizes are of closed form, the expression involves the average accept rate p and the gradient norm Gt,,, both of which depend on the posterior (-(-) and are therefore not available directly. Specifically, suppose that we are given the training set (2), 2 and current LLM M (0). We can write:
ps En
G.
=Ey-Q |IV In Ply, 2,/2,,0) 0)]
We also use sampling to estimate these quantities. We can first generate N' samples per prompt to Ji N'
get (x,, vi. 2 13 1- Then, we can compute the following empirical estimators:
Σ 1( 24)
p
=
Σ
1 N'.
Io In Ir, 0)12.
N
ISIEN
7

---

Then, we simply plug these empirical estimators into Proposition to get the sample sizes. The entire procedure is summarized in Algorithm 2.
Average accuracy: GVM-RAFT++ RAFT
8, Bn
W3..22.N -3.
0.450
Average Accuracy
GVM-RAFT++. We implement GVM and the meta EM Algorithm in highly online fashion, building on the RAFT+ framework. In each iteration t, we draw set of prompts {æ,, } ~ dn and use the current model ootd to collect D' U as the replay buffer where the inference budget allocation is determined via Algorithm 2. RAFT++ then uses these samples to compute stochastic gradient estimator of the objective JQr. To accelerate training, we perform multiple gradient steps per iteration in a mini-batee way. This causes the model distribution shift away from the distribution used to generate the data. To address this mismatch, RAFT++ incorporates importance sampling and clipping strategies from PPO (Schulman al., 2017) into the original RAFT, arriving at the following loss function on the prompt-response pair (xr a):
0.375
0.350
0.325
GVI RAFT++
GVM RAFT++
0.300
20
40
60
80
100
120
10
15
20
25
30
35
Steps
Steps
Figure 2: Average accuracy of (GVM-) HEVM-RRAF+++ with sample sizes 8 and 32, respectively, on Math500, Minerva Math, and Olympiad Bench, with base model 1wwe2.55MMath-1
jal
(R) =
min (s(())ccip((((0)11---11
I(r(x, = argmaxr(z, a:)), (8)
lx and where (0) Te (i.
and a, is the t-th token of a. Here, the indicator ensures that we only train accepted responses: those approximating the posterior via rejection sampling.
Extension to the RL algorithms. While we focus primarily on RAFT-like algorithms, the proposed GVM strategy be readily adapted other RL-style fine-tuning algorithms. We focus the GRPO, which receives significant attention recently due to its successful application to training DeepSeek-RI. Specifically, for each prompt x, GRPO will sample > 1 responses and compute the following advantage for the t-th token of the i-th response: A₃(x, a2) 5s mrenn(r..""".. where ry std(r, ... .Fm) denotes the final reward of the i-th response. This leads to the following loss function for GRPO,
m
To Ve, Ascs
[O) Ex (o (olx)
m
11
THaa (yii Vi,ce)
3DD [TO [rett] }}
clip(
e, +c)
Te (yi, x vi
4 Experiments and Results
4.1 Experiments Setup
To validate the effectiveness of the proposed methods, we conduct experiments with Qwen2.5-Mathh 1.5B and Qwen2.5-Math-78 (Yang et al., 2024b). We focus on the mathematical reasoning task and use Math Verify as the verifier. For detailed experiments setup such as hyperparameters and compute resources, please refer to Appendix C.
4.2 Main Results
In this section, summarize the results from integrating GVM into both RAFT+ and GRPO algorithms as a sample budget rebalancing strategy. The performance is measured by Average @ 8, which means randomly sample 8 instances from the model with non-zero temperature, and take
patttp:://giee httpg//www..o/gggggggg/////t/tte Verify

---

the average accuracy as the final result. For Dwen2.5-Mathh 1.5B, we use a temperature of 1.0 in evaluation, while for Qwen2.5-Math-7H we use temperature of 0.7 as the entropy loss is higher after integrating the clip higher trick, which leads to more diverse outputs.
GVM Improves Efficiency with Comparable Performance From Table we could conclude that GVM could improve the final performance of both RAFT++ and GRPO when applied on Qwen2.5- Math-1.5B. For Qwen2.5-Math, convergence rate. This verifies the effectiveness of GVM using both the accept rates (standing for the hardness) and the Lipschitz coefficients (standing for the gradients) of different prompts.
the final performance comparable to baselines, under a faster
Table 1: Performance of different algorithms across five benchmarks including Math500 (Hendrycks al., 2021), Minerva Math (Lewkowyez et al., 2022), Olympiad Bench (He al., 2024), AIME24, and AMC23. From the results, we could observe that after reweighting the sample size of prompts, GVM-RAFT++ and GVM-GRPO could outperform both vanilla RAFT++ and GRPO.
Method
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
5 Average
Base
56.35
17.00
25.22
3.33
37.81
27.94
Qwen2.5
GRPO
70.78
29.00
33.56
10.00
47.19
38.11
Math-1.5B
RAFT+
69.02
27.71
31.74
9.58
44.06
36.42
GVM-GRPO
73.92
29.96
36.26
12.92
49.06
40.42
GVM-RAFT+
72.90
29.04
36.20
9.17
Base
42,00
12.82
19.20
12.92
30.00
23.39
Qwen2.5
GRPO
81.20
36.03
44.15
20.83
63.
49.07
Math-7B
RAFT+
$1.68
35.85
43.83
20.83
63.12
49.06
GVM-GRPO
81.55
36.26
43.56
22.92
65.00
49.86
GVM-RAFT+
81.00
36.67
43.48
22.92
61.56
49.13
Base
24.28
4.37
4.80
0.83
9,06
8.67
GRPO
33.35
6.99
7.65
1.67
12.19
12.37
Llama-3.2
IB-Instruct
RAFT+
30.10
6.62
6.46
0.83
13.44
11.38
GVM-GRPO
32.02
6.76
7.22
2.08
15.31
12.68
GVM-RAFT+
30.08
6.07
6.48
0.83
13.44
11.38
Base
35.62
9.83
9.09
2.92
15.00
14.49
Llama-3.2
20.67
8.33
27.50
25.81
38 Instructt
RAFT+
47.38
17.69
17.20
8.33
29.06
23.93
GVM-GRPO
51.75
17.74
19.48
7.50
33.44
25.98
GVM-RAFT+
49.05
20.04
17.87
6.25
29.06
23.93
In Figure we display the step-wise performance of both RAFT++ and GVM-RAFT++ based on Qwen2.5-Man -0t255-- with sample sizes per prompt of 8 and 32, respectively, GVM could enhance the convergence rate evidently, with about 2 speedup for N' 8, 8n and 4x speedup for N' = 32, N = 32n measured in update steps compared to the baselines. Besides, upon convergence, GVM-RAFT++ could achieve around 1. 1.25% and 5% performance gain for both configurations.
GVM Could Be Generalized to RL Algorithms Though our derivation and theoretical proofs are based on EM algorithm, the sampling strategy itself could disentangled from the pipeline and utilized solely. Take the GRPO algorithm proposed in Shao et al. (2024) as example. Figure demonstrates that with the same sample budget rebalancing strategy as in RAFT++, GVM-GRPO could perform similarly to GVM-RAFT++: This further verifies the effectiveness of GVM single sampling strategy compared to being deployed the EM pipeline. Zhong et al. (2025) makes extra assumptions (Example 3.5) on the reward structure, which enables their framework's 's natural generalization to RL from a theoretical perspective. The similar ideas shed light on the generalization of GVM RL algorithms, and verified the empirical effectiveness of GVM in experiments, indicating the success of budget reweighting through gradient variance minimization.
Conclusion, Discussion and Limitations
In summary, we propose a novel dynamic sampling and updating strategy GVM, which could assign prompt-specific sampling budget in fin-grained manner. The algorithm consists of two stages: a pre-sampling phase that estimates the difficulty of each instance and computes prompt- specific Lipschitz coefficients, followed by an update phase that performs parameter optimization. Our experiments have demonstrated the effectivvness of GVM, achieving faster convergence and even superior final performance under suitable settings compared to other baselines. GVM could

---

improve the convergence for both rejection sampling backed pipelines and then be generalized to RL algorithms like GRPO, which demonstrates the significant potential for adaptively reweighting the sampling and update budget. Finally, we also provide rigorous theoretical analysis and establish performance guarantees for this class of two-stage algorithms.
The experiments are conducted with Qwen series base models, while the effectiveness of GVM still awaits a broader verification on other base models. Besides, we believe GVM could generalize to other RL algorithms like PPO and Reinforce, while more experiments need to be performed to support the hypothesis. These could serve the directions for possible further explorations.
Acknowledgment
This work is partially supported by NSF grant No. 2416897 and ONR grant No. N000142512318 18. This research used the DeltaAl (NSF OAC-2320345) and Delta (NSF OAC-2005572) advanced computing and data resourees, supported by the National Science Foundation and the State of Illinois.
References
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. (2024). Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXhv:2402.14740
AI Meta (2024). Llama model card.
Anthony, T., Tian, Z., and Barber, D. (2017). Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M. and Munos, R. (2023). A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. /2036.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D.,, Fort, S., Ganguli, D.,. Henighan, T., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 055662.....
Bishop, C. M. and Nasrabadi, N. M. (2006). Pattern recognition and machine learning, volume Springer.
Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345.
Chen, J., Zhu, J. Teh, Y. W., and Zhang, T. (2018). Stochastic expectation maximization with variance reduction. Advances in Neural Information Processing Systems, 31.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng. C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F,, Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Wang. J., Chen, J., Yuan, J., Qiu, J., J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, I.,, Zhang. I.,, Zhao, I.., Wang, I.., Zhang, I.., Xu, I.., Xia, I.., Zhang, M., Zhang, M., Tang, M., Li. M., Wang. M., Li. M., Tian, N., Huang. P., Zhang, P., Wang. Q.. Chen, Q. Du, Q.. Ge, R.. Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L., Chen, R., Lu. S., Zhou, S., Chen, S., Ye, S.. Wang, S., Yu, S., Zhou, S., Pan, S., Li. S. S., Zhou, S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng. W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W. Zhang, W., Xiao, W. L., An, W., Liu. X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li. X. Q. Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X. Zhou, X., Wang, X., Shan, X., Li, K., Wang, Q. Wei, Y. X., Zhang, Y. Xu, Y., Li. Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang. Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y.,
Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y.
Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma. Y., Tang, Y., Zha, Y., Yan, Y. Ren,
10

---

Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma. Z., Yan, Z., Wu, Z., Gu, Z.. Zhu. Z., Liu, Z., Li. Z., Xie. Z., Song. Z., Pan, Z., Huang, Z., Xu, Z.. Zhang, Z., and Zhang, Z. (2025). Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning.
Defazio, A., Bach, F,, and Lacoste-Julien, S. (2014). Saga: fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang. T. (2023). RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions Machine Learning Research.
Dong, H., Xiong, W. Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. (2024). Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863.
Gulcehre, C.,, Paine, T. L.., Srinivasan, S., Konyushkova, K., Weerts, L.., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. (2023). Reinforced self-training (rest) for language modeling.
arXiv preprint arXiv:2308.006 08998.
He, C., Luo, R., Bai, Y., Hu, S., Thai, L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 10l1:2442.140000
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874
Hoffman, M. D., Phan, D., Dohan, D., Douglas, S., Le, T. A., Parisi, A., Sountsov, P., Sutton, C., Vikram, S., and Saurous, R. A. (2023). Training chain-of-thought via variale inference. In NeurIPS.
Hu, J. (2025). Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprini arXiv:2501.03262.
Hu, X., Zhang, F., Chen, S., and Yang, Z. (2024). Unveiling the statistical foundations of chain-of- thought prompting methods. arXiv preprint 1455
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El- Ei-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel,
A., Carney, A., et al. (2024). Openai ol system card. arXiv preprint arXiv:2412.16720.
Johnson, R. and Zhang, (2013). Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26.
Kingma, D. P., Welling, M., et al. (2013). Auto-encoding variational bayes.
Kool, W., Hoof, H., and Welling, M. (2019). Buy 4 reinforce samples, get baseline for free!
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A.. Anil, C., Schlag, I., Gutman-Solo, T., al. (2022). Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857.
LI, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A.. Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. (2024). Nu- minamath. https:/ ht///gtthbb cwwccrooooo///e com/project.o. numina/ aamo-proggess /ttp://oioggeeeeeeeee//////oo/// prize/blob/main report/
pittor/nnmiiaaa numina dataset. d.f.
Li, 7.,, Xu, T., Zhang, Y., Yu, Y., Sun, R., and Luo, Z.-Q. (2023). Remax: simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv e-prints, pages arXiv-2310.
Neal, R. M. (2003). Slice sampling. The annals of statistics, 31(3)7005-767.
11

---

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Owen, and Zhou, (2000). Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135-143.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Di- rect preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305.18299.
Roux, N., Schmidt, M., and Bach, F. (2012). A stochastic gradient method with exponential convergence rate for finite training sets. Advances in neural information processing systems, 25.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344
Shao, Z., Wang. P. Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. (2024). Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXiv:2402.033: 03300.
Sheng. G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. (2024). Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256.
Shi, T., Wu, Y., Song, I.., Zhou, T., and Zhao, J. (2025). Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint ar arXiv:5504.05200.
Singh, A., Co-Reyes, D., Agarwal, R., Anand, A., Patil, P., Liu, J., Harrison, J., Lee, J., Xu, K., Parisi, A., al. (2023). Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585
Sordoni, A., Yuan, E., Côté, M.-A., Pereira, M., Trischler, A., Xiao, Z., Hosseini, A., Niedtner, F., and Le Roux, N. (2023). Joint prompt optimization of stacked 1lms using variational inference. Advances in Neural Information Processing Systems, 36:58128 15-51111-
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M.. Pires, B. Á., and Piot, B. (2024). Generalized preference optimization: A unified approach to offline alignment. arXiv preprini 4rX1v:2402.05749
Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. (2025). Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint 125559,
Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. (2024). Dart-math: Difficulty-wware rejection tuning for mathematical problem-solving. arXiv preprint arXiv:2407.13666 33690
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.. Bhosale, S., al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228
Wei, J., Wang, X.. Schuurmans, D.,, Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. (2022). Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2483.
Williams, R. J. and Peng, J. (1991). Function optimization using connectionist reinforccmmnnnn learning algorithms. Connection Science, 3(3):241-268.
Xiong, W., Dong, H., Ye, C., Wang, Z.,, Zhong, H., Ji, H., Jiang, N., and Zhang, T. (2023). Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.
Xiong, W. Shi, C., Shen, J., Rosenberg, A., Qin, Z.,, Calandriello, D., Khalman, M., Joshi, R., Piot, B., Saleh, M., et (2024). Building math agents with multi-turn iterative preference learning. arXiv preprint ar a09.:2249..22392.
12

---

Xiong, W., Yao, J.. Xu, Y., Pang, B.. Wang, L., Sahoo, D., Li, J.. Jiang, N., Zhang, T., Xiong, C., et al. (2025a). A minimalist approach Ilm reasoning: from rejection sampling reinforce. arXiv preprint arXiv:2504. 1343.
Xiong, W. Zhang, H., Ye, C., Chen, L., Jiang, N., and Zhang, T. (2025b). Self-rewarding correction for mathematical reasoning. arXiv preprint wrXiv:2502.19133
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C.,, Liu, D., Tu, J., Zhou, J., Lin, J., et al. (2024a). Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 24X1:2209.12122..
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. (2024b). Qwen2.5-math technical report: Toward mathematical expert model via self ielrorippoeeett, arXiv preprint arXiv:2409.12111
Yu, Q. Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Fan, T., Liu, G., Liu, I.., Liu, X., et al. (2025). Dapo: open-source IIm reinforcement learning system at scale. arXiv preprint MKKY:25033.447700
Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. (2023). Scaling relationship learning mathematical reasoning with large language models. arXiv preprint arXiv:2308. 01825.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. (2022). Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15576-
Zhang, H., Yao, J., Ye, C., Xiong, W. and Zhang, (2025).
Online-dpo-rl: Un-
locking effective reasoning without the ppo overhead. https://effiiientnun unicorn- s.t..notion.ie.o wttee/tliieeeeeeeOeO DPO-RI-Unlocking Siffooocing-fffcciieeee.. Reavonnna Haassoing-Withoutthhee Without-the-PPO PPO-OvereOadddd 190804477e7758000000344444442111
Notion Blog.
Zhang, T. (2023). Mathematical analysis of machine learning algorithms. Cambridge University Press.
Zhao, P. and Zhang, T. (2014). Accelerating minibatch stochastic gradient descent using stratified sampling. arXiv preprint arXiv: 1405.3880.
Zhao, Y., Joshi, R., Liu, T. Khalman, M., Saleh, M., and Liu, P. (2023). Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1042
Zhong, H., Feng, G., Xiong, W., Zhao, L., He, D., Bian, J., and Wang, L. (2024). Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.188 18922.
Zhong, H., Yin, Y., Zhang, S., Xu, X., Liu, Y., Zuo, Y., Liu, Z., Liu, B., Zheng, S., Guo, H., et al. (2025). Brite: Bootstrapping reinforced thinking process to enhance language model reasoning. arXiv preprint arXhv:2501.18858
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The pipeline of our workflow is demonstrated by Figure 1, and details reflected in Section and Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss some limitations of the current work in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The main theoretical results are provided in Section 3, and Section E and Section offer complete proofs and technical lemmas.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The settings and configurations of our experiments are summarized in Section 4 and Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use existing open-source datasets as training and test datasets, and will release our code repo for public access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpiccccceecee hepollcc/giieeee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings and details are summarized in Section 4 and Section
C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The test accuracy is evaluated through the average multiple rollouts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details about compute resources for reproducing the experiments are summarized in Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eurrpp..p// htc/public//eie Meriiic/Etiiiettee hete/thhiiiiiieeeeeeeeeeeeeeeeee wethhiisssieeneeneeeeeeeee dnes?
Answer: [Yes]
Justification: requirements.
We have checked the Code of Ethics and spared effort to achieve the
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work mainly focuses the mathematical reasoning ability of LLMs. which has little effect on society except the relevant research field.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
The paper does not pose such risks, as we train small open-source models on limited amount data, and the mathematical reasoning ability for LLMs has little potential to be misused in other fields.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The data and code we use are all open-sourced and cited in our paper. We also mention the resource and license of the code in our code repo
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The processed datasets and code will be released in public along with docu- mentation introducing how to use them.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are only used for polishing the writing and checking any spelling or grammatical errors in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (attps://neun) http:/pp...c/oooofeeeeee//////50 for what should or should not be described.
20

---

A Author Contributions
This work stems from all authors' valuable contributions and close collaborations.
JY implements the workflow based on the algorithm pipeline; conducts preliminary experiments and verifies the effectiveness of GVM in the early stage; scales up the experiments; mainly writes the experiments-relevan sections of this paper.
YH initializes the project; contributes to the algorithm proposal and the clarification of the pipeline procedure; derives the theoretical analysis for GVM and establishes corresponding bounds for the algorithm; mainly writes the sections on methodology and theoretical analysis.
HZ devotes to the conduct of experiments; contributes to the ablation studies of the tradeoff between entropy loss and model performance; reviews related works on the techniques mitigate fast entropy loss in the early stage of RL training and provides insightful suggestions on how to balance the tradeoff.
HD implements baselines and provides practical implementation suggestions; conducts a series of experiments, including baselines comparisons, GVM iterations in both RAFT++ and RL (GRPO) algorithmic settings; provides discussions and suggestions about experiment design; contributes to writing.
WX provides insightful advice through the project timeline on theoretical analysis, experiment design, and paper writing; contributes to the initial codebase and baseline training; mainly writes the introduction, related works, and problem formulation sections; and revises the draft version of this paper.
NJ and TZ support the work and provide computational resources, guidance, and suggestions for theoretical analysis, experiment design and paper writing.
B Related Work
Iterative rejection sampling fine-tuning, line of works studies the RAFT-type algorithms (iterative rejection sampling fine-tuning) (Dong et al., 2023; Touvron et al., 2023; Gulcehre et al., 2023; Yuan et al., 2023). These algorithms iteratively generate responses per prompt, discard those with incorrect answers, and fine-tune the LLM on the remaining accepted samples. Another closely related method STaR (Zelikman et al., 2022), which also involves iterative training on self-generated CoT responses. In comparison, STaR uses greedy decoding rather than sampling, which deviates from the EM principle and suffers from severe bias across prompts of different difficulty levels (see Section 2 for details). STaR also proposes to provide the answer in the prompt to generate CoT responses for harder problems. However, our preliminary experiments suggest that LLMs often generate wrong reasoning path and output the correct answer regardless of the context. Additionally. STaR retrains from the original pre-trained model in each iteration, rather than bootstrapping from the updated model. These algorithms have been widely applied in the post-training stage of LLMs since then (Touvron et al., 2023; Xiong et al., 2025b; A Meta, 2024; Yang et al., 2024a). We also mention in passing that this algorithm framework can further date back to the expert iteration in the RI. literature Anthony et al. (2017). Lastly, Tong et al. (2024) study inference budget allocation in the context of synthetic data construction. Their approach uses fixed model to repeatedly sample responses until m correct CoT rationales are obtained per prompt (m is a parr-paraeeter). While this setup also studies budget-aware sampling, they only study the offline setting and their method lacks dynamic scheduling mechanism so cannot be applied to more general iterative online algorithms.
Chain-of of-uhuughttt reasoning under EM framework. Our work is also closely related with the line of work studying EM framework in CoT reasoning. Singh et al. (2023) connects RAFT with the EM algorithm under the CoT reasoning framework. We will cover such a connection in Section for completeness and also discuss the limitation of the vanilla RAFT algorithm, and motivate our algorithm designs. Another related work Sordoni et al. (2023). which proposes a gradient- EM-based approach for prompt selection, emphasizing different direction. Our work is also closely related to Hoffman et al. (2023), which proposes TRICE that optimizes an EL.BO-inspired objective using MCMC to approximate the posterior over the latent CoT rationales. TRICE maintains table of
21

---

prompt CoT pairs and, in each iteration, samples new CoT y per prompt, and replace the old CoT record yotd with y if y leads to correct final answer. They also propose to add mean-zent baseline to reduce the variance of the gradient estimator (Owen and Zhou, 2000). In comparison, we design dynamic inference budget allocation scheduling for a more effective estimation of the ELBO gradient by rejection sampling so the algorithm framework and techniques diverge significantly. Moreover, we implement our algorithm in a highly online and on-policy way (see Section 3.3 for details) to achieve the best performance. The lazy update of TRICE in maintaining the table of prompt-CoT pairs may not fit in this scenario. Beyond algorithmic differences, we also provide a theoretical analysis under standard conditions such as smoothness, which is missing in all these prior works. Hu et al. (2024) offers a more detailed statistical perspective on CoT reasoning as latent variable modeling. However, they mainly focus on in in-context learning rather than iterative fine-tuning Lastly, recent work Zhong et al. (2025) proposes to use a neural network to approximate the posterior distribution of latent CoT rationales and train the network by PPO. They then use the network to generate responses for the SFT or DPO algorithms, whose techniques are completely different from ours. They also provide a theoretical convergence analysis of EM algorithm under the KL-regularized Markov Decision Process (MDP) framework using the techniques from RL theory literature (Zhong et al., 2024). This type of analysis typically ignores the optimization complexity and requires to find the exact minimizer of E-step and M-step at each iteration, thus differing from our techniques and results.
RL algorithms for LLM post training. Reinforcement learning has played a central role in the post-training of LLMs. The most prominent example is reinforcement learning from human feedback (RLHF), which brings the first generation of Chat-GPT3.5 (Ouyang et al., 2022; Bai et al., 2022). These methods typically optimize learned Boaleyy-Terry reward model (Bradley and Terry. 1952) using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017). To simplify the RLHF, long line of works has proposed simplified contrastive learning objectives based on pairwise comparisons, such as Slic-HF (Zhao et al., 2023), DPO (Rafailov et al., 2023), IPO (Azar et al., 2023), GPO (Tang et al., 2024), and their online variants (Xiong et al., 2023; Dong et al., 2024; Zhang et al., 2025). After the release of OpenAT's O1 (Jaech et al., 2024) and DeepSeek's ' RI (DeepSeek-A) et al., 2025), RL methods based on verifier scores, which evaluate the correctness of the final answer, have attracted significant attention in building reasoning models (Yu et al., 2025; Shi et al., 2025). This project also follows the recent convention and falls into this framework. In particular, to avoid the critic in PPO, both industry and academic researchers have revisited REINFORCE-StyL algorithms (Williams and Peng, 1991), including ReMax (Li al., 2023), RLOO (Ahmadian et al., 2024; Kool et al., 2019), GRPO (Shao et al., 2024), Reinforce++ (Hu, 2025), and Reinforce-rej (Xiong et al., 2025a). We also mention in passing that the RAFT-type algorithms variant of Reinforce-style algorithms, where reward is either or O. Throughout the rest of this paper, we primarily present our methods with RAFT, since it aligns most naturally with the EM objective. However, the presented dynamic inference budget scheduling can be of independent interests beyond the RAFT algorithm. We will show that integrating it into other RL-style methods, such PPO, REINFORCE, or GRPO, can also bring notable improvements.
also be viewed as a special
C Experiment Details
Sample size accepte rate
350
Iteration 2
300
Iteration 3
Iteration
250
Sample Size
250
200
300
150
100
100
50
50
0-
0-
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accepted Rate
Accepted Rate
Accepted Rate
Figure 3: The assigned sample budget in GVM-RAFT+ with N' = 32, N = -32n for three iterations.
22

---

Visualization We present the relationship between the final sample budget and the accept rates of different prompts for GVM-RAFT++ Figure 3. From the trend, it could be inferred that the sample budget allocated to prompts increases smoothly as the accept rates decrease. However, other affecting factors together determine the trend of the sample budget besides the hardness, where the Lipschitz coefficients measuring the contribution of the samples by the gradients play a role. If we do not take the impact of Lipschitz coefficients {Gr} into account, then we could have the allocated sample budget to prompts with different levels of hardness measured by accept rates as shown in Figure 4.
Hyperparameters We utilize verl (Sheng et al., 2024) as the training framework, and implement the RAFT++ as Xiong et al. (2025a) show that the additional importance sampling and clipping improve over the vanilla RAFT (Dong et al., 2023). For each iteration, we use prompt batch size of 1024, and use mini-batch size 256 for gradient update. The max prompt length is set to be 1024, and the models are allowed to generate most 3072 tokens so that they do not exceed the context window of 4096 tokens. There is no warmup stage and the learning rate is chosen to be constant le- 6. For the training dataset, we use the Numina-Math (LI et al., 2024). For simplicity of implementation, we update the posterior distribution (i.e., the GVM in Algorithm 2) every 10 steps. In other words, we will compute the inference buget allocation for the next 10 M-steps and take 10 M-steps before entering another E-step. The 10 M-steps and the E-step are together referred to as one iteration of GVM-RAFT++ We also experiment with the clip higher (Yu et al., 2025) to use an asymmetric clip threshold design. Specifically,
clip the importance sampling ratio Te/ /e/Te... Te.. to the range (1 - flow + hhggh), ), where klow = 0.2, and thigh = 0.28, 0.4 then 0.35 for RAFT++ and GVM-RAFT respectively. We omit this technique in GRPO training as observe it leads to worse performance in our experiments.
To understand how to choose the hyperparameters and 3, we first simplify the question and omit the impacts of gradients here. Then the sample weight of function f(x) V
with different
choices of a and / is visualized as the figure below. From Figure 4, we could see that in order to
assign more sample weights on problems with lower accept rates, we should choose both a and as low as possible within a reasonable region. Therefore, in our experiments, we fix a 0.001 and 2.0. For all the hyperparameters, please refer Table 2.
Table 2: Full hyperparameters.
Parameter
Value
le - 3 2
B batch size mini batch size max prompt length
1024
256
1024
max response length
3072
learning rate KL loss coefficient
1c - 6
0.001
As for the choices of & and |8 is GVM stage two sample budget assignment, we visualize the effects of both parameters in Figure 4. From the figure, we could see that when 5 is fixed, the smaller a is. the more sample budget will be allocated to harder prompts. Similarly, when or is fixed, the smaller B is, the more sample budget will be allocated to harder prompts.
To analyze the impacts of which layers are used in the gradients calculation and the reduction methods of gradients, we conduct series of experiments using Owen2-math-77 I-Innrruct (Yang et al., 2024a). For the layers from which we retrieve the gradients, we choose the top one, Im m__hed, and the bottom one, embed_tokenn tokens, as the activated layers for loss backpropagation and gradients calculation. For gradients reduction, we compare both summation and average by sequence length. From the results shown in Figure 5, we could see that different kinds of layers for gradient calculation do not make a huge difference on the final sample sizes, while the summation of gradients biases the sample sizes to problems with lower accept rates, which we speculate the correct answers for problems with lower accept rates could steer the model more than easier problems. Therefore, in our experiments, we choose summation as the method for gradients reduction. According to the relationship between the sample sizes and accept rates. The relationship between gradients and accept rates, it could be
23

---

Effects of a and B on f(x) = 1/v x ax-
a=1.0e.04, 8=
a=1.0e-04, 8=3.5
a=1.0e:04, 8=5.0
8=1.0e-33, p=2.0
a=1.0e-33, 8=3.5
a=1.0e-03, $=5.0
a=1.0e-02, 8=2.0
4
a=1.0e-02, B-3.5
f(x)
a=1.0e-02, 8=5.0
3
2
1
0.0
0.2
0.4
0.6
0.8
1.0
Figure 4: Visualization of different choices and 3 for f(x) 1/Vx axl-3
inferred that in a high level, the gradients are larger for prompts with lower accept rates if we use the average reduction. At the same time, is not always the case, prompts with higher accept rates may also have large gradients.
In the second stage of GVM, to make the total sample budget exactly N. we sort the prompts according to their accept rates in descending order, and add more samples to the prompts with lower accept rates. In our experiments, for GVM, each iteration consists of nine steps of update as the amount of training data per iteration is slightly less than 10 x 1024 = 10240, which is the amount of data needed for ten steps of update with train batch size 1024.
For the implementation by fixed size (for example, 4 in our experiments), and copy the prompts multiple times by n:/4. Then we set the rollout number to during the training procedure.
EVM--RPO, we divide the sample budget calculated the second stage
For compute resources, we mainly conduct the experiments on NVIDIA RTX A6000 and H100 GPUs, and each iteration in GVM typically takes 90 minutes with sample sizes N' 8, N 8n a 4 HH00 GPU server.
C.1 Ablation Study
GVM Balances the Sampling Budget in a Stable Way Figure 9 displays the performance trend concerning the training steps. with each training step corresponding to exactly 1024 x samples, which means that the definition of "step" here may not be mapped to the actual training step. Take the case N' 16, N 16n for example, under this setting, each iteration optimized over 1024 prompts, with total sampling budget 1024 16 4 x (1024 x 4), therefore corresponding to 4 steps the figure. Though increasing the number of samples in the first stage, N', could make more prompts be accepted at least once, thus leading to more useful prompts during the training procedure, convergence rate does not change evidently. This means GVM could possibly estimate the difficulties of the prompts with a relatively small sampling budget to sufficient threshold, and we could choose small N" and N realistic applications.
Existing RL training paradigms with Sywen-2.5-Math- suffers from distribution collapse and worse pass@n. In Figure 6, we observe that GVM consistently improves the convergence rate for both RAFT++ and GRPO, and across both the 0wwn-2.5-Matth-1.. and 7B models. However, also find that (1) GVM yields larger gains with the 1.5B model than with the 7B model; and (2)

---

GVM is more effective when combined with GRPO than with RAFT++. To better understand these observations- as well as the dynamics of GVM-w we conduct further ablation studies. Specifically, one can interpret vanilla RAFT++ as imitating a fixed best-of-n policy. In contrast, GVM-RAFT++ adaptively selects a best-of-n, policy for each prompt Zi, based on its difficulty and gradient signal. GRPO REEINFORCE-baSee
further incorporates unlearning from negative samples. Intuitively, all these methods are influenced by the quality the underlying best-of-n; policy. To examine this effect, we plot the pass @n curves of different checkpoints below.
We plot the pass curves of GVM-RAFT++ using both the Qwen-2.5-Maite 1.5B and 7B models in Figure 7. As shown in the figures, the pass@n curve (for n 2 4) remains relatively stable across different training iterations for the 1.5B model. In contrast, the performance consistently degrades with the 7B model. Notably. the GVM-RAFT++ 7B model at iteration 10 underperforms even the 1.5B model when n 2 40. Since the imitation target gets worse and worse during the training, the GVM-RAFT++ does not improve the final test performance with the 7B model. We also mention in passing that is expected that the current RL training paradigm cannot improve the pass @ at large n, trend also observed in prior work (Shao et al., 2024; Xiong et al., 2024). However, our results model.
show that the pass@ @n actually gets worse with the Qwen-2.5-Matte
We note that the observed performance difference essentially arises from existing RAFT++ or GRPO training paradigm, rather than the proposed computational resource allocation strategy (i.e., GVM). In Figure 8, we plot the pass @n curves for both the vanilla RAFT++ and vanilla GRPO. As shown in the first two plots, RAFT++ without GVM already exhibits the same trend: the 1.5B model maintains stable pass @ass@n n curve, whereas the 7B model shows clear degradation as training progresses. The middle plot further indicates that the "clip higher" technique helps mitigate this decline hence improving the final test performance. Finally, GVM still improves GRPO performance with the 7B model, likely because its pass@n n curve deteriorates more slowly than RAFT++ This is due to the use of negative samples in GRPO (Xiong et al., 2025a).
In addition, we observe that for Qwen2.5-Math- RAFT++ are higher than RAFT++, and when N' = 16, the training rewards of GVM-RAFT+ are approximately the same as RAFT++, while when N' 32, the training rewards are lower than RAFT++. This implies that using a larger N' for GVM will bias the sample budget to harder prompts, which may be because with a larger N', harder problems are more likely to be solved least once compared to when N' is smaller.
when N' = 8, the training rewards of GVM-
D Notation Table
All of the notations are listed in Table 3.
Table 3: The table of notations used in this paper.
Notation
Description
x. y,
Prompt, CoT rationale, and predicted answer the space of prompt, CoT rationale, and predicted answer the number of training samples (Ir, 24) the posterior distribution P(y](,, zi, 0) on the 1-th E-step the collected sample set for (a,, 24, y) on the t-th E-step the size of scheduling budget for prompt 24 on the t-th E-step the average accept rate for prompt T; on the t-th E-step the whole size of sampling budget
X,Y, z
D
n
P
N
(a. B)
tuning parameters in budget scheduling
B
the sample batch corresponding to Iz on the r-th M-step the batch size of B required number of samples for prompt 2; on the 1-th E-step the number of M-step after each E-step
n
k
0st-k4r
the model parameter on the r-th M-step within the t-th E-step learning rate upper bound loss function on the t-th E-step
(,)
25

---

E
More Theoretical Results and Missing Proofs
Proof of Lemma 1.
Taking the expectation of Equation (6), we obtain:
Pln 2/22, (0
E
p
MyED;
E yy P(120
(acceppt y3) ( V In P(ys, 2/,2, 0r-))
p
T By
V In Plys, 2, 1c-1)
p
m
By
In In 2, Jr, 1)-
which is the same Equation (5).
Proof of Lemma 2.
m
Σ
V ((nF(W),2(15;00 0))
nlp!
MyED!
m
My (((1,,,0)1)) (1aaceept y (In P(y,, 2: 0)))
By PP.yP [[accept V (In P(uy, 2/,2,, 0)
(ntpt) 2
m
p E リの Oi V (In 0))
j=1
V (In P(y, x 2/,0, 0))1
P=QQ
If we consider further strongly convex loss function, there will be:
Theorem 3 (Decreasing rate with smooth and strongly convex condition.). Suppose - IInP((. 2|2, 0) 1/7-smooth and H- H-strongly convex with respect to 0. If0 < 1 S min(7/2, 1/(4kl1)) then the proposed algorithm satisfies that
Hn
E [[(Owr) - -(0') - 2(00))
(k(
(k:
8k ((,TT),
2k
4k
where
T k
Aa( =
VL (Osz x+m)1 0.
1=
m==
Proof of Proposition 1. Based on the objective function in Equation (7), we can define the Lagrange objective function as
m
G
m
(((n),)) = Σ
1=1 1 a
allaa//l (v!)
p.
n
N).
26

---

Taking derivative on the variables, we could obtain that
ag
2
G
Ant,
ie [m],
1+ ap p!(n))2
89
N -
Σ n
ØX
To achieve the quations as we can obtain the result as
ag
0g
0,
0,
On!
0x
G./ V 1 + (p/)a1
G,
n = N
e [m].
1= G1/ Vet + Tof)o-
M + (pp)-T
which finishes the proof.
Proof for Theorem
To simplify the analysis, we denote
m
gkt-k+r
Σ
Vo (InP(yj 2 (2 0kc-
m
My MEB
With the fact that In P(y, 2 2/,,00 0e) are 1/ smooth, we can obtain
n1² E ght k+rl
(0st L. (Okz k+)) ( 112
24
for any r k on iter-t. Summing the above inequality over all r. and with the fact that EL((k)) S (0xe) S ELe a) = EL((ee=)),
we have
Co k-1
" Blggtt ++rl?
E [C((as)) ((0kI-x)]
E
27
n/²
Σ Bl|V k+r
(EV Ell
27
n² EV (git-
5
11
B|| V Cr (0st )1
24
where the last inequality is due n/y < 1. Further summing t over 1,..., T. can obtain that
T k-1
T k-1 1
"
E [C((wr) C( -
E (
+
EV (ghr-k4r),
2
2>
p=
Proof for Theorem 2. The proof techniques are mainly from Zhao and Zhang (2014). To be specific, for each iteration t, we denote that
1 m
8kt- 0kt
Σ
Vo (In 2,/24,, Out k+r)) Okc-ktT Oxc-k
m
m
1
Σ
(ln 21/20,, O Okt-ktr) - In Plyj, 26/20,, 0k)-k))
kt
ktr
7
27

---

which implies that
Eòki-ktr ((V(<(Oke-ktr)) Oxc-Rtr 0k(-k) (Lc(Omt ( 2 0.
Based on these notations, we can obtain that
|10 k
0
110ke k+r+1
0 2(ngka-ktr: Oke-ktr 0k(-k) 11ngke
21
2n
E
(In 25/23, Oxe- - In (M) 2 21/24, Oke- |1 ktrll?
== My W
taking expectation of both sides, we have
Bl - 0 - 0 0 2 21/ (OL
Summing over r -= 0.1,...,k 1, we have
(CK(01())+)) (0ke x))
El 0x
+
17
Ellght -
EE
27
2
Also, considering the fact obtained in previous analysis as:
1-1
k-
E ((G((0== - (000
(o +
Σ
17
E k+rll?
27
P=0
r=0
r=0
can combine the two inequalities above, and obtain that
E [C(()))) - -(0k=- x)] S ((C,(0n)) (0ke S
Σ
E
(CC,(0n- - Le (e(0ne
r=0
1
k-1 E V (0me )1
11²
E 1ik 0lc-a|1
Σ
+
Σ
E strl1
2nt
2
27
r=0
7
k-1
(0m(0km
17
11²
k-1
V Le 1kk (bxr)))) 11
+
172 k-1
EV (gkc-
2k
h
2
27
2 r=0
r=0
1-1
k-1
k-1 Σ
<
7
E
VL.
12
Σ
EV (ghe
2k
4k
4h
P=0
r=0
where the last inequality is from n/7 < 1/2.
Summing over f 1, T. we have
T
T k-1
E [(OwT) C(bb)] x
V (0 (0
2k
4h
(=1
r=0
r=0
T
RV
(okr
4k
t=1
Proof for Theorem 3. For each iteration I, we denote that
m
dst - k++
Σ
Vo Plyj, P1 25/05; 0ve- O2 Ose-n
KtT
m
B
1
Σ
(In 2/2,, 0kI-b4r) - In P(ys, 24 (x,, 0kc-x))
m
H
10:
0
112.
2
28

---

which implies that
H - 0.
15tt--4r (V Our 04r- (0 (t
0
Based on these notations, we can obtain that
|00t-k+7 - x - | -
11 1
2 O k+r 0kt - |
-1j0kt
21
E
(In zi 2i, O2e- - In P(uj, zi 2,, Ome-k)) + 2
l 1 xxl?
KTT
ByEB
|[nght 11? taking expectation of both sides, we have
H 110ke-btr - 0k
B 0 E El - 0xI- EL, -
- (Oke-bh) +
2
Summing over r 0, ..,,...,k 1, have
E (C,(((er - (Okt
11 0 xll
E Bl| ktr
Out 111
E
E k 11?
2n
2
Also, considering the fact obtained in previous analysis as:
1-
1-
E|C -,
Σ
Bl
17
E 11hh--krrl
2y
r=0
-=0
can combine the two inequalities above, and obtain that
E [f())))) - L(0kz- x)] S S (Oae(( - (x(0 k)) S
E -+-0+)++++)))) - (0st
==0
1
11
<
Oxt k1l2
Bll0ne-k4r Out 11
E [o
2np
2k
r=
1
1/2
E 11gge-
k
27
7
E
k-1
V C. (0 14r))
1(11
172
k-1
VoLc (k:: x+r) 212
172 k-1
EV (gkc-847)
2k
27
27k 1=0
P=0
Hn
Hn²
B1|
EV(gk-k4m)
2k
2k
m=0
m=0
Hn²
E
V La
(1 k+r)|
2
E
V (B
2k
2k
4k
r=0
1=1
m=U
11 EV (gkt-ktT),
Σ
8k
where the last inequality is from n/7 S 1/2.
Summing over i. 1,..., T. have
E [[((0)T) - [(()))
T
T k-
E (C(0)) - [(0')]
Σ
x4m)
(Oke
2k
4k
37
r=0
T
BV (gkr-k47)
8k
t=1
0=0
29

---

E.1 Discussion on sample size
To be more specific, achieve negative upper bound, the sample size on each batch should satisfy that:
by j=1 Ve In P(yy, 2, 0)1
2 " Σ Th
||
Ve In P(y,, 0)1
Technical Lemmas
Lemma 3 (Jensen's Inequality). Suppose that (ww) is convex function Ω. Consider W1, Wm E Ωt and non-negative numbers on, Om E R so that Eis1 0g = 1. Then,
S aso(w;)
1=1
1=1
More generally, let p be a probability measure St then of ( S (w)) In particular, since is convex (by triangle inequality of the norm), we know that
|2|| S E||2]].
Proof. See Proposition A.9 of Zhang (2023) for a proof.
G More on GVM Algorithms
Here we present a practical version of GVM in Algorithm 2.
Algorithm 2 GVM: Practical Implementation
I: Input: Model parameter 0, training samples (wi, 2, " total inference budget N, parameter estimation sample size N penalty parameter (a, B).
2: for i 1, ... do
3: For each prompt æ,, sample N' times to {a., vú, 4 1
4: Estimate the accept rate p and Lipschitz bounds G, on each prompt x, as
Σ
1( = 2.)
N
G
Σ
Pon. 0)1
N'p
5: Calculate sample size {n&} as:
Gi/ / م
nu N.
D Gi/ 1p: i
end
7: Output: (n.]
general algorithm pipeline for rejection sampling is displayed in Algorithm 3, while in practice, we directly use the outcome-based reward signal to decide whether prompt will be chosen not.
30

---

Algorithm 3 Rejection sampling
1: Input: probability with current model (1 0--1), P(zi/u, 0e-1)), auxiliary probability qlyles, 2.), sample size ni, accept rate pi, empty set D
2: for .,..., do
Obtain sample point Y, from the distribution g(y) x,,
4: Sample random point u, ~ (((0,) 1).
5: Calculate (3(5) as:
(-10) 0t-1)
Qi(0)) Ey P(yl,,, 0: 0t-1// Eg E. P(ul=., (c-1))((),,)))e))) 0c-1)
6: if S x,Q:(,,)
ol3, [. then
7: Enlarge D = D {15).
end
9: end for
10:
Output: set 27.
Average accuracy for different sample sizes of GVM-RAFT4
0.46
0.44
Average Accuracy
0.42
0.40
0.38
Setting =
0.36
N'=6,N881
0.34
16=16,N:66
32, 32n
Step
Figure 9: The average accuracy GVM-RAFT++ with different sample sizes for both stage one and stage two on Math500, Minerva Math and Olympiad Bench. From the trend, though the number of sample sizes in both stage one and stage two increases, the convergence rate stays almost the same.
31

---

embed tokens gradients mean
0.8
200
0.6
Sample Size
Gradients
0.4
100
50
0-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
Accept Rate
Accept Rate
embed tokens gradients
300
200
250
Sample Size
Gradients
150
100
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accept Rate
Accept Rate
Im head gradients mean
200
5
Sample Size
150
Gradients
100
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accept Rate
Accept Rate
Im head gradients sum
200
3000
2500
Sample Size
Gradients
2000
100
1500
1000
50
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accept Rate
Accept Rate
Figure 5: Comparison among different choices of the layers for gradient calculation and the methods of gradients reduction on Qwen2.5-Mathh
32

---

Qwen2.55 0wen225-Math-1.55
Qwen2.5-Mathh
0.46
Average Accuracy
0.44
0.42
GVM-GAPO
GVM-GRPO
GVM-RAFT-+
0.46
GVM-RAFT++
0.36
0:44
60
80
120
140
40
60
100 120
Steps
Steps
Figure 6: The average accuracy of RAFT++, GVM-RAFT++ and GRPO, GVM-GRPO with N' N 8n respectively Math500, Minerva Math and Olympiad Bench. Applying the GVM sample strategy to RL algorithms like GRPO achieves similar results to GVM-RAFT++ compared to vanilla GRPO. For vanilla RAFT+ and GRPO, the rollout number per prompt is set to 8 as well.
0.80
0.75
0.75
0.70
0.70
0.65
1.58 pass@n
GVM-RAFT+ 155 Per pass@n
0.65
CVM-RAFF- 7B itet pass@n
GVM RAFT ++ 1.58 iter pass@in
GV1 RAI ++ 7B iter pass@n
RAFT 1.58 paceai
RAFT iter
0.60
0.55
30
40
50
60
10
20
30
40
50
60
Figure 7: The pass @n @a temperature is 1.0.
curve of GVM-RAFT++ on Qwen-2.5- 1,5B and 7B. The evaluation
0.800
0.755
0.50
0.57
0.72
60.25
0.700
2,77
No cli "igher With dip 7igho
0.72
42f7.4 10 30503
0190 cisa 100 (HISED
0.00
0.72
Figure 8: The pass @n curve of various algorithms. Left: RAFT+ Qwen-2.5-Mathh 1.5B. Middle: RAFT++ on Qwen-2.5-Math-77 with and without clip higher technique. Right: GRPO on Qwen- 2.5-Math-7P
33

---

H
Case Study
Here we display several instances from several benchmarks we used respectively. these cases, the model trained by vanilla RAFT++ did not generate correct answer within a fixed budget of attempts, while models trained by GVM-RAFT++ or GVM-GRPO managed to find the correct answers.
Math5(0) Example
The expression 2 3 4- 5 + is equal to 121, since multiplication is carried out before addition. However, we can obtain values other than 121 for this expression if we allowed to change it by inserting parentheses. For example, we can obtain 144 by writing
Problem
(2 (3-4)) + = 144.
In total, how many values can be obtained from the expression 3 parentheses? (Note that rearranging terms is not allowed, only inserting parentheses).
3- 5-54--4
by inserting
Answer
get different results by inserting parentheses various ways. Let's denote the expression with parentheses as E. The expression 2-3 3- + without parentheses is evaluated as:
RAFT+ Output Given the expression
5.45--40 5 1,
2 3 4 5 + 120 121.
Next, let's consider the possible ways to insert parentheses: (2 3) 4
2. 4) (5-555+ (5 3. (2 3 4) 5 + 14. 2 (3 (4: 5) +1) 5.
3 1-.4-33 5) +
However, not all of the above are valid expressions because we need evaluate the expression
step by step. For simplicity, let's evaluate each possible corrected expression:
(2 4 (5
(5 + 6 4
144 2 (3 4) (5 +
2 12 144
121
54 1) = 2 20 + 1)
(60 + 1) 2 122 5. 3. + 120 +1 121
From the above evaluations, see that we get the values 144, 144, 121, 122, and 121, which means the distinct values are 121, 122, and 144.
So the total number of distinct values we can obtain is 3. The answer is:
GVM-RAFT++ Output Given the expression 2 3 4 + 1, we need to determine how many distinct values can be obtained by inserting parentheses in different ways. First, let's ' note that without parentheses, the expression evaluates to 2-3 2-3.4-554 5-1+1 = 120 121.
When we insert parentheses, the expression can take different forms based on where the parentheses are placed. Let's evaluate all possible ways of inserting parentheses:
1. (2 - 4) (5 + 1) (24) 144 2. 2. (3 4) + = 60 + 1 120 + = 121
3
2.3-
126 (2 3) 5
(6) 21 126
2. 5 = 21 122
By evaluating these different forms, we see that the possible values are 121, 122, 126, and 144.
So the distinct values that can be obtained are 121, 122, 126, and 144. Therefore, the total number of distinct values is 4.
The final answer is
34

---

Minerva Math Example
Problem Preamble: A spectrum taken of single star (i.e., one in binary). Among the observed spectral lines is one from oxygen whose rest wavelength is 5007. The Doppler shifted oxygen line from this is observed be at a wavelength of 5012. The star is also observed to have proper motion, μ4, of are second per year (which corresponds to distance of 60pc from the Earth.
1.5 x 10 radians per second of time). It located Take the speed of light to be 3 x 10 meters per second.
What is the component of the star's velocity parallel its vector the Earth (in kilometers second)?
Answer To find this longitudinal velocity component, we use the Doppler shift, finding
V,
300 km/s.
5000
RAFT++ Answer To solve this problem, we need to follow these steps:
1. Calculate the radial velocity of the star using the Doppler effect. 2. Use the radial velocity and the proper motion to find the component of the star's velocity parallel its vector the Earth.
Step Calculate the radial velocity The radial velocity, v, of the star can be found using the
formula for the Doppler shift of photon:
Aobs
Arest
Avest
where Aods is the observed wavelength and Arest is the rest wavelength.
Given: Arest 5007 Á Aobs 5012 A
10 m/s
We first need to convert the wavelengths from to meters:
1 10 m
So:
A 5007 x 10 10
Aabs 5012 x 10 10-10 m
Now can calculate u:
5012 10-10 5007 10-
e 3x 10
5007 x 10-10
10-10
V = 3 x 10
5007 x 10-10
1 = 3 X 108
5007
15 x 10
29956 m/s
5007
~ 29.96 km/s
Step 2: Calculate the parallel velocity component The proper motion μ is given as 1.5 x 10 rad/s, and the distance the is 60 pc. We convert pc 3.086 10
distance = 60 x 3.086 10 m 1.8516 x 10 m
The radial velocity v is the component of the star's velocity parallel to its vector the Earth plus the effect of the proper motion:
e - + x
where l is the component of the star's velocity parallel to its vector to the Earth, is the proper motion, and d the distance. Since calculated 29.96 km/s 29960 m/s, can solve for fnl
= e p x d
35

---

29960 (1.5 x 10 13 x 1.8516 x 10
29960 2777.4
6 ~ 27182.6 m/s
2I 27.18 km/s
So the component of the star's velocity parallel to its vector the Earth approximately 27.18 km/s.
The final answer is:
27.18
GVM-GRPO Answer To solve this problem, we need to determine the component of the star's velocity parallel to its vector to the Earth. We can break this into two parts: 1. "Ress-wavelength shift using the Doppler effect:** The Doppler effect tells us that the observed wavelength Aoos of spectral line with rest wavelength Ares and velocity e relative to the observer is given by:
Aobs
Aresi 1 +
C
Here, Ashs- = 5012 A, Arest 5007 A, and c = 3 x 10th m/s. We can rearrange the equation solve for
Aobs
1 +
Arest
5012
V
= 1 +
5007
5012
5007
C
5012 5007
5007
5 5007
x 3 x 10th m/s
v
5007
Converting the velocity from m/s to km/s:
5
x 3 x 10a
x
3
x 10 X
km/s
km/s R 299.5 km/s
5007
1000
5007
2. Proper motion effect: ** The proper motion ls of the star is given as are second per year. This be converted to angular velocity using the formula:
arc second
1 year
3 1 year
X 3.154 x 10 seconds
3.154 x 10? are seconds per second
The proper motion μ in seconds per year, so need convert it radians per second: radian
1 year
=1 are second/year x 206265 are seconds 3.154 x 10 seconds
1
206265 x 3. 3.154 x 107 radians/second ~ 1.5 x 10-13 radians/second
The component of the star's velocity parallel to its vector to the Earth (vi) is given by:
U| = μ x d
where d is the distance to the star in meters (60 60
10 = 1.8516x 18 m):
1.5 10 x 1.8516 1028 m/s 277.74 74 km/s
Since the rest- wavelength shift caleulation gives velocity of approximately 299.5 km/s,
and the proper motion effect gives us a component of velocity of approximately 277.74 km/s, conclude that the component of the star's velocity parallel to its vector the Earth is dominated by the raveenght shift effect. Therefore, the final answer is:
300 km/s
36

---

Olympiad Bench Example
Problem A cubic sequence a sequence of integers given by an = n + bn + cn + d, where b, c and d are integer constants and n. ranges over all integers, including negative integers. Determine the possible values of 02015 42016 for a cubic sequence satisfying the condition in part (a). Answer
RAFT+ Output First, let's understand the cubic sequence on = n²² + bn + en + d. We need find the possible values of 02015 02016- Given that on is cubic sequence, it can be expressed in the form On = n dn + en + d. Therefore,
02015 2015 + 2015 + 2015 + d
02016 2016 b 2016 + 2016 +
The product of these two terms be expressed
02015 42016 = (2015 + 2015 + c. 2015 + d) (2016 + 2016 c 2016 d)
Since the sequence terms given polynomial, and the polynomial's coefficients take any integer value, the product 02015 42016 could theoretically take any integer value depending the integers and d. Hence, the possible values of a2015 a2016 span all integers L.
Thus, the set of possible values of 02015 02016 is all integers Z. The final answer is:
GVM-GRPO Output The general form of cubic sequence is given by n2 + bn en + d. We need find the possible values of 02015 02016- First, let's write down the expressions for
a2015 and a2016:
2015 b 2015² + 2015 +
02015
02016 2016 + b. 2016 + c 2016 +
To find 02015 02016, let's look at the difference between consecutive terms of the sequence:
an+1 On = [
|n + bn + cn + d)
= + [[
- cn]
[n + 1
n²)] + ic(n + n2)
=I1- 2n + 2n n²)l |d
= + +1
= 3n 3 2bm 1+
3m + (3 +1+
Since c are constants, the difference a,+1 - a, is quadratic function n. This means a, itself must be cubic function, which confirms the form a, = n + bn + en + d. Q2016:
Now, let's look at the product a2015
02015 2015 20152 + 2015 + d
02016 2016 b- 2016 + 2016 + d
The product 02015 02016 can be simplified by noting that:
02015 02016 (20153 + 2015 c 2015 + d) (20163 + b 2016 c 2016 d)
37

---

This can be simplified further by noting that:
02016 02015 + (2016 20153) 6(2016 - 2015") c(2016 - 2015)
-22015 + (20162 + 2016 2015 + 2015 2016 + 2015) 6(206 2015) +
Since the terms are cubic, the exact values of ag015 and a2016 depend on the specific values of b,c and d, but the general form shows that 42015 and d2u16 are very close together. Specifically, since 2016 is very large compared 2015 the terms involving b, c, and may cancel out in the product.
Thus. the product 02015 02016 essentially the square of 02015 plus some constant terms. This suggests that the only possible values for 02015 02016 are those that can be expressed as the square of an integer. Therefore, the possible values of 02015 02016 are the squares the product of integers.
The final answer is:
This is because the only integers 02015 and 02016 can satisfy are given the cubic nature of the sequence and the sequence behavior around n 2015 and n 2016.
AIME24 Example
Problem A list of positive integers has the following properties:
The sum of the items in the list is 30.
The unique mode the list is 9.
The median of the list is positive integer that does not appear in the list itself.
Find the sum of the squares of all the items in the list. Answer 236
RAFT++ Output Given the conditions of the problem, we can determine the structure of the list step by step. Let's denote the list of positive integers as 01, 02, ......... On-
1. The sum of the items in the list is 30. 2. The unique mode of the list is 9. Since 9 is the unique mode, must be the most frequently occurring number the list. 3. The median of the list is positive integer that does not appear in the list itself.
Let's start by considering the minimum possible values for n. Given that the mode is and must appear more frequently than any other number, let's consider small values for In = 5, the list would have to include 9 three or more times to be the mode. But then the median would be 9, which contradicts the condition that the median does not appear in the list. So n 2 6.
If n 6, the list could potentially have 9 twice and the other numbers distributed such that the median is the third number the sorted list and does not appear in the list. the median 6, the list [6, 9, 9, a, b, c] would satisfy the conditions, where a, b, S 6 and a, of 9. The sum of the numbers should be 30:
a+b+c=6
Since b, c. < 6 and they are distinct integers, the only possible values for b, are 1,2, Thus, the list [6, 9, 9, 1, 2, 3] or any permutation that maintains the same properties. The sum of the squares of the numbers is:
6 + + 9 + 12 2 + 3 36 + + 81
212
We now verify that this is correct by checking the conditions. The is 30, the mode is 9. and the median is 6, which does not appear in the list. The final answer is:
212
38

---

GVM-GRPO Output To solve this problem, we need to carefully consider the properties of the list of positive integers:
The sum of the items in the list is 30. 2. The unique mode of the list is 9. This means that appears more frequently than any other number the list. The median of the list is positive integer that does not appear in the list itself.
Let's denote the list of integers [01, ........ an) where a1 5 On Since 9 the unique mode, it must appear at least twice, and more frequently than any other number. The median the middle value when the list is sorted. Since the median is not in the list, n must be odd. Let's assume To find the sum of the squares of all the items in the list, we need to determine the possible distribution of the numbers. Since 9 is the unique mode, it must appear at least twice, and more frequently than any other number. The list must include other numbers, but must be the most frequent number. The sum of the numbers is 30, and the median must be positive integer that does not appear in the list.
2k + 1. The median is then 0++1-
Let's start by assuming the simplest case where 9 appears exactly twice. This mcans a1 a2 = 9. The list then looks like [9, 9, az, ag...., ., a2k+1]. The sum of these numbers is 30, so
9 03 04 ... 02k++1 30. This simplifies 18 as 04 + ... 02 30,
or d3 + 04 +... + 02k-1 = 12.
then = 3, and the list is [9, 9, 0z). The median is 9, which is not a valid solution since must appear more frequently than any other number. If A: 2, then is [9, 9, 03, d4, as). The median is a3. Since 9 appears twice, 03 must be different from 9. The sum of the other three numbers is 12. The possible combinations are (1, 1, 10), (1, 9). (1, 8), (1, 4, 7). (1, 5, 6). (2, 2. 8). (2, 3, 7). (2, 4, 6). (3, 3, 6). (3, 4, 5). However, since 9 must be the mode, a3 cannot be Also, a3 must be a positive integer that does not appear in the list. The only valid combination is (1, 2, 9). but 9 is not a valid median. The next valid combination is (1, 8). where the median 03
The median 0k+1 must be positive integer that does not appear in the list. If
and the list
This means the list [9, 9, 3, a4, as), and the remaining sum is 12 3 = 9. The only way to split 9 into two positive integers that are or 3 is and So the list is 19.
8.. The sum of the squares of these numbers
9a + 92 3 + 1 + 8 81 81 + + 64 236.
Therefore, the sum of the squares of all the items the list is 236
AMC23 Example
Problem Rows 1, 2, 3, 4, and 5 of triangular array of integers are shown below.
1
11
13 1
1551
171171
Each row after the first row is formed by placing a 1 at each end of the row, and each interior entry is I greater than the sum of the two numbers diagonally above it in the previous row. What is the units digits of the sum of the 2023 numbers in the 2023rd row?
Answer
RAFT++ Output We need to determine the units digit of the sum of the 2023rd row in the triangular array.
Observing the Behavior of Units Digits 1. #*First Row:** (units digit is 1) 2. **Second
Row:** ** (units digit is 1 + 1 = 2) 3. **Third Row:** 1 (units digit is 3) 4. **Fourth
6) 5. **Fifth Row: ** 1 7 11 7 (units digit is 7 = 18, units digit is 8) 6. ** **xixt Row: (7+19) (units digit is + = 18, units digit is If we continue observing this pattern, we notice that the units digits of the sums of the diagonals in each follow specific pattern which repeats every 2 rows: First row: 1 Second row: Third row: Fourth row: 6 Fifth row: 8 Sixth row: 6
Row:* 1 5 1 (units digit is 1 +
39

---

We notice that the units digits of the sums for rows 3, 5, 7,9..... are 1, 8, 1, 8, and for rows
2, 4, 6, 8,... are 2, 6, 6, 6,
Since the 2023rd row is odd row, its units digit of the sum will be the same as that of the 3rd, 5th, rows, which is 1. Therefore, the units digit of the sum of the 2023rd row
GVM-RAFT++ Output To solve this problem, need analyze the triangular array and find a pattern. Let's start by examining the structure the array.
1. The first row is: 1 2. The second row is: 1 1
3. The third row is: 1 3 4. The fourth
row is: 1 5
1 5. The fifth row is: 1 7
7 1
The pattern for each interior entry is that it is greater than the sum of the two numbers diagonally above it in the previous row. To find the units digit of the sum of the numbers in the 2023rd row, can focus the units digits of the numbers in each row.
Notice that each row starts and ends with 1. The units digits of the interior numbers follow specific pattern. Let's calculate the units digits the first few rows to identify pattern:
Row 1: (units digit: 1) Row 2: 1 1 (units digits: 1, 1) Row 3: 3 1 (umits digits: 1,
1) Row 4: 5 5 1 (units digits: 5, 5, 1) Row 5: 1 7
7 1 (units digits:
1.7,1
We can see that the units digits repeat a pattern. Let's look the units digits of the interior
numbers more closely:
Row (units digit: 1) Row 2:
(units digits: 1) Row 3 1 (units digits:
3. 111) Row 4: 5 5 1 (units digits: 5, 1) Row 5: 1 7 11 7 1 (units digits:
1,7,1,7,1
It seems that the units digits of the interior numbers in each row follow pattern where the units digit of each interior number determined by the sum of the two units digits diagonally above it, incremented by 1. For example, in row 5, the units digit of the middle number 11 is I because + 5 == 11, and the units digit of Given this pattern, the units digits of the sums of the numbers in each row will also follow pattern. Specifically, the units digits the sums of the numbers the rows will 2, 4, 6, 8, O, and then repeat (since the sum of the units digits in each row is always more than the sum of the previous row's 's units digits).
The units digits of the sums of the numbers in the
follows: Row 1: 1 (units
digit of sum is 1) Row 2: + = 2 (units digit of sum 2) Row 3: + 3 + = (units digit of
is 5) Row 4: 1 + 5
12 (units digit of sum is 2) Row 5:
1+7 + 11 7 = 27 (units digit of sum is 7)
We can see that the units digits the sums repeat every 4 rows: 1, 5, 2. Therefore, the units digit of the sum of the numbers in the 2023rd row will be the same as the units digit of the sum of the numbers in the 3rd row (since 2023 mod 3).
So, the units digit of the sum of the numbers in the 2023rd row is 5.
Let's write the final answer:
40


Reviews:
[{"review_id": "2BwuHfNOGu", "reviewer": "Reviewer_juWC", "summary": "This paper presents an improved version of the iterative reward-ranked fine-tuning (RAFT) method.\n\nRAFT is a large language model fine-tuning method in which, at each step of fine-tuning, a reward model is used to select only the high-reward samples (i.e., prompts and their corresponding responses). This approach has been shown to effectively improve model fine-tuning and reduce noise in the training data.\n\nThis work argues that the RAFT method has limitations due to its static rejection of samples and its neglect of the coverage behavior across different prompts. To address this, the authors propose a dynamic rejection sampling method that dynamically allocates the sampling budget and rejection decisions per prompt. The paper provides both theoretical and empirical analyses of the method.", "questions": "1. Why are the experiments only conducted on small-scale LLMs and methods?\n   Adding more justification for this choice, and including experiments on other variants or larger-scale models, would significantly strengthen the paper and increase its impact.\n\n2. When introducing theoretical concepts, can you provide more intuition behind them?\n   For example, when discussing the optimal budget allocated for each prompt, it would be helpful to explain why the given equation makes sense. While a complete theoretical analysis is valuable, offering some high-level intuition would make the paper more accessible to a broader audience.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. The problem addressed in this paper is an important real-world challenge. Post-training of large language models (LLMs) requires effective fine-tuning strategies. Efficiently rejecting low-quality samples while reducing variance is critical for improving fine-tuning outcomes. Existing methods have not explored dynamic approaches in this area, and this paper offers a valuable incremental advancement in that direction.\n\n2. The paper provides both empirical and theoretical analyses of the proposed dynamic sampling allocation method, which enhances the quality and clarity of the work.\n\n### Weaknesses\n\n1. The overall originality of the work is incremental, as the core idea of intelligent rejection sampling has already been explored in previous methods such as RAFT. This paper builds on existing approaches rather than introducing a fundamentally new concept.\n\n2. As noted in the conclusion, the method is only evaluated on a limited set of baselines and models—specifically the Qwen and PPO series. It would be valuable to see this method extended and tested on a broader range of LLMs and training setups.", "was_revised": false, "final_justification": "", "rebuttal": "**Q 1**. The overall originality of the work is incremental, as the core idea of intelligent rejection sampling has already been explored in previous methods such as RAFT. This paper builds on existing approaches rather than introducing a fundamentally new concept.\n\n**Response:** Thanks for the opinion. The previous works like RAFT are essentially RL algorithms, which focus how to utilize existing data to optimize the policy model. In contrast, our method focuses on improving the sampling strategy of training data, instead of proposing a novel training algorithm. Take RAFT as an example, for a given prompt $x$, it will sample n responses ${y_1,\\cdots,y_n}$, and then call a reward model to obtain the rewards for each response ${r_1,\\cdots,r_n}$ and select the response with the highest reward to form a training pair $(x,y_{\\arg\\max_i r_i})$. It assign each prompt $x$ with equal sampling budget $n$, without differentiating different samples based on their characteristics like difficulty and contribution to the model (measured by the gradient). Our GVM algorithm considers both the sample difficulty and gradients, and assign different sampling budget based on them. Then we could feed the samples to different RL algorithms like RAFT or GRPO, etc. It is a two-stage pipeline instead of substituting other algorithms.\n    \n**Q 2**. As noted in the conclusion, the method is only evaluated on a limited set of baselines and models—specifically the Qwen and PPO series. It would be valuable to see this method extended and tested on a broader range of LLMs and training setups.\n\n**Response:** We select Qwen model series as the base models following the common practice in the line of LLMs math reasoning research, where Qwen serves as one of the strongest and most commonly used open-source base model. As mentioned in question 1, our GVM algorithm serves as a pre-stage of the vanilla RL optimization stage, therefore it could be combined with any RL algorithms. For demonstration purpose, we choose two commonly adopted ones - RAFT and GRPO. To make our conclusion more generalizable, we conduct more experiments on Llama series base models. Due to computation limitation, we first conduct experiments with models at 1B and 3B scale, and will continue to do experiments on larger scales like 7B. We report the metric average @ 8 as follows. One iteration in GVM experiments here corresponds to 9 steps of optimization in vanilla experiments. It could be seen GVM achieves a nearly 2 times acceleration for comparable performance.\n| Model                     | MATH500 | Minerva Math | Olympiad Bench | AIME24 | AMC23 | Average|\n|---------------------------|---------|--------------|----------------|--------|-------|--------|\n| Llama-3.2-1B-Instruct     | 24.28   | 4.37         | 4.80           | 0.83   | 9.06  | 8.67  |\n| RAFT++ step 120           | 30.10   | 6.62         | 6.46           | 0.00   | 12.50 | 11.14 |\n| GVM-RAFT++ iter 6         | 30.08   | 6.07         | 6.48           | 0.83   | 13.44 | **11.38** |\n| GRPO step 120             | 33.35   | 6.99         | 7.65           | 1.67   | 12.19 | 12.37 |\n| GVM-GRPO iter 7           | 32.02   | 6.76         | 7.22           | 2.08   | 15.31 | **12.68** |\n| Llama-3.2-3B-Instruct     | 35.62   | 9.83         | 9.09           | 2.92   | 15.00 | 14.492  |\n| RAFT++ step 120           | 47.38   | 17.69        | 17.20          | 8.33   | 29.06 | 23.932  |\n| GVM-RAFT++ iter 5         | 49.05   | 20.04        | 17.87          | 6.25   | 29.06 | **24.454**  |\n| GRPO step 120             | 53.40   | 19.16        | 20.67          | 8.33   | 27.50 | 25.812  |\n| GVM-GRPO iter 5           | 49.25   | 18.61        | 17.81          | 8.75   | 31.88 | 25.260  |\n| GVM-GRPO iter 8           | 51.75   | 17.74        | 19.48          | 7.50   | 33.44 | **25.982**  |\n\n**Q 3**. Why are the experiments only conducted on small-scale LLMs and methods? Adding more justification for this choice, and including experiments on other variants or larger-scale models, would significantly strengthen the paper and increase its impact.\n\n**Response:** We conduct our main experiments on Qwen 1.5B and 7B base models due to the computation source limitation, and conduct additional experiments on Llama 1B and 3B series during the rebuttal period with an 8*A100 GPU instance. For experiments on larger model scale, we hope to generalize GVM when more computation resources are available.\n\n**Q 4**. When introducing theoretical concepts, can you provide more intuition behind them? For example, when discussing the optimal budget allocated for each prompt, it would be helpful to explain why the given equation makes sense. While a complete theoretical analysis is valuable, offering some high-level intuition would make the paper more accessible to a broader audience.  \n\n**Response:** In our algorithm, we allocate the budget for each prompt according to Proposition 1, which takes into account both problem difficulty and model efficiency. Specifically, to address problem difficulty, we allocate more budget to harder problems, which associated with larger gradients and lower accept rates during sampling, as this can improve model performance. At the same time, to ensure model efficiency, we avoid allocating excessive budget to extremely difficult problems with near-zero acceptance rates, as this would lead to inefficiency."}, {"review_id": "YgKYOXJf6z", "reviewer": "Reviewer_V851", "summary": "This study identifies inefficient stochastic gradient estimation caused by static sampling strategies as the primary bottleneck in Chain-of-Thought (CoT) training. To address this challenge, the authors propose GVM-RAFT - a Prompt-Specific Dynamic Sample Allocation Strategy that optimizes stochastic gradient variance under computational budget constraints. The proposed algorithm is supported by theoretical guarantees, and comprehensive experiments demonstrate its consistent performance superiority over existing methods.", "questions": "See the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This work identifies the main bottleneck in CoT training as inefficient stochastic gradient estimation due to static sampling strategies, and proposes a new algorithm to address this issue. I believe this is an important work.\n2. The authors provide theoretical guarantees for the proposed algorithm, and conduct experiments to show the outperformance over existing methods.\n\nWeakness:\n1.  For the theoretical guarantees, with a sufficiently large sample size, Ω(k,T) becomes small, and the upper bound for $E[L(θ_{KT}) − L(θ_∗)]−E[L(θ_0) −L(θ_∗)]$ is strictly negative. How about the case when the sample size is not large enough? The authors do not provide the details about how large the sample size needs to be to make the RHS strictly negative.\n2. For the GVM-GRPO, the authors have not provided the details of how to generalize the design to GVM-GRPO. Is it directly using the Dynamic Sample Allocation Strategy to generate different numbers of answers for different prompts in GRPO?", "was_revised": true, "final_justification": "I believe this is a good work with solid contributions.", "rebuttal": "**Q 1**. For the theoretical guarantees, with a sufficiently large sample size, Ω(k,T) becomes small, and the upper bound for E[L(θKT)−L(θ∗)]−E[L(θ0)−L(θ∗)] is strictly negative. How about the case when the sample size is not large enough? The authors do not provide the details about how large the sample size needs to be to make the RHS strictly negative.\n    \n**Response:** Thanks for pointing this out. o ensure that the right-hand side (RHS) of the bound is strictly negative, we require the following condition to hold for each prompt $x_i$: \n$$\\sum_{i=1}^{n} b_i^{-1} \\left\\lVert E_y \\nabla_{\\theta} ln P(y, z_i | x_i, \\theta) \\right\\rVert_{2}^{2} \\le  \\left\\lVert \\sum_{i=1}^{n} E_y \\nabla_{\\theta} \\ln P(y, z_i \\mid x_i, \\theta) \\right\\rVert_2^2 $$\n, where $b_i$ is the batch size for $x_i$, and $n_i \\ge b_i$ is the total allocated budget for $x_i$. In practice, this condition is not stringent and is often satisfied, since a proper $b_i$ can  help reduce the variance and norm of the gradient on the left-hand side. However, in scenarios with extremely small sample sizes, we cannot guarantee the stability or performance of the EM algorithms, as the estimation becomes highly noisy. Nevertheless, compared to the standard EM algorithm, GVM-EM still provides theoretical advantages by dynamically allocating samples to reduce the term $\\Omega(k, T)$, which contributes to better convergence behavior even under limited data.\n    \n**Q 2**.  For the GVM-GRPO, the authors have not provided the details of how to generalize the design to GVM-GRPO. Is it directly using the Dynamic Sample Allocation Strategy to generate different numbers of answers for different prompts in GRPO?\n    \n**Response:** The sampling allocation strategy is shared when applying to different optimization algorithms like RAFT or GRPO. It is disentangled from the underlying RL training algorithm, and thus could be applied to different algorithms after allocating the sample budgets. In our experiments GVM-RAFT++ and GVM-GRPO, the sampling stage for GVM is exactly the same."}, {"review_id": "Voru8HX7ZV", "reviewer": "Reviewer_FHJP", "summary": "This paper makes an important contribution to optimizing CoT inference training for large language models (LLMs). The core innovation is to introduce dynamic resource allocation into the EM framework, and to address the bottleneck of static sampling strategies in existing methods (such as RAFT) by minimizing the gradient variance.", "questions": "Please see the Weaknesses part.", "limitations": "No", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\n\n1） The GVM algorithm (Algorithm 2) is proposed to dynamically allocate the sample budget by monitoring the prompt acceptance rate and gradient norm to minimize the gradient variance (Equation 7). This solves the static sampling limitations of methods such as RAFT and significantly improves resource efficiency.\n\n2)  The paper provides a complete convergence analysis, proving that under the conditions of smoothness and convexity, GVM can ensure the monotonic decrease of the objective function. This fills the gap in the lack of theoretical support for existing RAFT work.\n\n3) The superiority of GVM is verified on multiple mathematical datasets.\n\n\nWeaknesses:\nAlthough the paper makes significant contributions, this paper have some issues:\n\n1) ​​Experimental generalization is insufficient: The experiment is based only on the Qwen model series (Qwen2.5-Math-1.5B/7B), and the effect on other mainstream LLMs (such as Llama or GPT architectures) is not verified. This limits the generalizability of the conclusions.\n\n2） Computational overhead is not quantified: The pre-sampling stage of GVM (Algorithm 2) requires additional samples to estimate, but the paper does not report the computational cost (such as GPU hours or total number of samples).\n\n3) The comparative benchmark is not comprehensive: Table 1 shows that GVM outperforms RAFT++ and GRPO, but it is not compared with the latest RLHF methods (such as online DPO or PPO variants), which may underestimate competing methods.\n\n4) ​​Limitations of theoretical assumptions: Theorems 1-2 rely on smoothness (1/γ-smooth) and convexity assumptions, but the LLM loss function is often non-convex. The paper does not discuss convergence guarantees in non-convex scenarios, which may overestimate the robustness of the theory.\n\n5) ​​Fuzzy implementation details: The GVM-RAFT++ loss function (Equation 8) in Section 3.3 introduces importance sampling and clipping, but does not explain the choice of hyperparameters (e.g., ϵ value).", "was_revised": false, "final_justification": "", "rebuttal": "**Q 1**. Experimental generalization is insufficient: The experiment is based only on the Qwen model series (Qwen2.5-Math-1.5B/7B), and the effect on other mainstream LLMs (such as Llama or GPT architectures) is not verified. This limits the generalizability of the conclusions.\n    \n**Response:** Thanks for the suggestion for adding more generalizability validation experiments. We conducted more experiments with Llama series models, on both RAFT++ and GRPO algorithms. Due to computation limitation, we first conduct experiments with models at 1B and 3B scale, and will continue to do experiments on larger scales like 7B. We report the metric average @ 8 as follows. One iteration in GVM experiments here corresponds to 9 steps of optimization in vanilla experiments. It could be seen GVM achieves a nearly 2 times acceleration for comparable performance.\n| Model                     | MATH500 | Minerva Math | Olympiad Bench | AIME24 | AMC23 | Average|\n|---------------------------|---------|--------------|----------------|--------|-------|--------|\n| Llama-3.2-1B-Instruct     | 24.28   | 4.37         | 4.80           | 0.83   | 9.06  | 8.67  |\n| RAFT++ step 120           | 30.10   | 6.62         | 6.46           | 0.00   | 12.50 | 11.14 |\n| GVM-RAFT++ iter 6         | 30.08   | 6.07         | 6.48           | 0.83   | 13.44 | **11.38** |\n| GRPO step 120             | 33.35   | 6.99         | 7.65           | 1.67   | 12.19 | 12.37 |\n| GVM-GRPO iter 7           | 32.02   | 6.76         | 7.22           | 2.08   | 15.31 | **12.68** |\n| Llama-3.2-3B-Instruct     | 35.62   | 9.83         | 9.09           | 2.92   | 15.00 | 14.492  |\n| RAFT++ step 120           | 47.38   | 17.69        | 17.20          | 8.33   | 29.06 | 23.932  |\n| GVM-RAFT++ iter 5         | 49.05   | 20.04        | 17.87          | 6.25   | 29.06 | **24.454**  |\n| GRPO step 120             | 53.40   | 19.16        | 20.67          | 8.33   | 27.50 | 25.812  |\n| GVM-GRPO iter 5           | 49.25   | 18.61        | 17.81          | 8.75   | 31.88 | 25.260  |\n| GVM-GRPO iter 8           | 51.75   | 17.74        | 19.48          | 7.50   | 33.44 | **25.982**  |\n\n**Q 2**. Computational overhead is not quantified: The pre-sampling stage of GVM (Algorithm 2) requires additional samples to estimate, but the paper does not report the computational cost (such as GPU hours or total number of samples).\n    \n**Response:** We’d like to mention that there is a tradeoff for the sampling budget allocation estimation - if the samples used to estimate the difficulty of the problems is large, it takes more time to generate while would return a more accurate estimation; and if the samples used in the estimation stage is small, the estimation would be less accurate but more efficient. In our main experiments, we choose $N^\\prime=8$ as the number of responses per prompt in our stage 1 difficulty estimation. On an 8*A100 GPU instance, it takes about 20 minutes additional computation for the GVM stage, which may slightly increase as the training goes due to the response length increasing. Compared to the RL stage which takes about 100 minutes per iteration, the additional overhead introduces 0.2 times the original training time. If we set $N=c\\cdot N$ where $c$ is the average samples per prompt in the RL training stage, then larger $c$ will correspond to less additional overhead introduced by the GVM stage.\n    \n**Q 3**. The comparative benchmark is not comprehensive: Table 1 shows that GVM outperforms RAFT++ and GRPO, but it is not compared with the latest RLHF methods (such as online DPO or PPO variants), which may underestimate competing methods.\n    \n**Response:** Our algorithm serves as a plug in algorithm for sampling budget allocation and should be combined with other RL optimization algorithms together for policy model update. GVM is not a new standalone RL algorithm itself, thus we choose the commonly used PPO and GRPO as the underlying RL optimization algorithms. The whole training pipeline could be divided into two stages - with the first one corresponding to GVM sampling budget estimation, and the second one corresponding to the vanilla RL optimization.\n    \n**Q 4**. Limitations of theoretical assumptions: Theorems 1-2 rely on smoothness (1/γ-smooth) and convexity assumptions, but the LLM loss function is often non-convex. The paper does not discuss convergence guarantees in non-convex scenarios, which may overestimate the robustness of the theory.\n    \n**Response:** Thank you for the insightful comment. In fact, Theorem 1 relies only on the smoothness assumption and does not require convexity, so it remains applicable in non-convex settings. We agree that this point should be clarified more explicitly in the main text, and we will revise the manuscript accordingly to avoid any confusion.\n    \n**Q 5**. Fuzzy implementation details: The GVM-RAFT++ loss function (Equation 8) in Section 3.3 introduces importance sampling and clipping, but does not explain the choice of hyperparameters (e.g., ϵ value).\n \n**Response:** The clipping parameter used in the objective of RAFT++ is $0.2$ in general, while we adopt the clip-higher trick from DAPO [1] with $\\epsilon_{\\rm high}=0.3$ for experiments like GVM-RAFT++ to mitigate the rapid entropy loss drop. The parameters should be adjusted according to the training dynamic like entropy loss curve. For other parameters, please refer to Table 2 in Appendix B.\n    \n[1] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" *arXiv preprint arXiv:2503.14476* (2025)."}]

Author rebuttal:

Meta-review: Summary: This paper introduces GVM-RAFT, a dynamic sample allocation strategy for fine-tuning reasoning models that minimizes stochastic gradient variance by adaptively assigning computational resources based on prompt difficulty and gradient norms.

Strengths: The reviewers all agree that the paper is well-motivated and develops a theoretically-grounded dynamic sampling method, which addresses a key inefficiency in existing fine-tuning approaches and is supported by strong empirical results showing a 2-4x convergence speedup.

Weaknesses: The main initial concern shared by reviewers was that the experimental validation was limited to a single model family, which raised questions about the generalizability of the results.

Reasons for Decision: The decision to accept this paper is based on the clear and significant contribution of its dynamic sampling strategy, which offers substantial and well-documented efficiency gains for an important training paradigm. The strengths of this novel approach, as recognized by all reviewers, outweigh the initial limitations.

Discussion Process: The authors effectively addressed the reviewers' primary concern regarding generalizability by providing a new set of experiments on the Llama model series, demonstrating that the method's benefits are not confined to a single architecture.
Decision: Accept (poster)