Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
Authors: Chao Wang
Tao Yang
Hongtao Tian
Yunsheng Shi
Qiyao Ma
XiaotaoLiu
Ting Yao
Wenbo Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Policy Optimization
Down-Sampling
Primary area: reinforcement learning
TL;DR: We propose the D$^3$S framework, an approach by dynamically selecting only a small subset of the most informative samples and tokens during training to achieve faster convergence and SOTA performance with less computation.

Abstract: Critic-free methods like GRPO reduce memory demands by estimating advantages from multiple rollouts but tend to converge slowly, as critical learning signals are diluted by an abundance of uninformative samples and tokens. To tackle this challenge, we propose the **Dynamic Dual-Level Down-Sampling (D$^3$S)** framework that prioritizes the most informative samples and tokens across groups to improve the efficiency of policy optimization. D$^3$S operates along two levels: (1) the sample-level, which selects a subset of rollouts to maximize advantage variance ($\text{Var}(A)$). We theoretically proved that this selection is positively correlated with the upper bound of the policy gradient norms, yielding higher policy gradients. (2) the token-level, which prioritizes tokens with a high product of advantage magnitude and policy entropy ($|A_{i,t}|\times H_{i,t}$), focusing updates on tokens where the policy is both uncertain and impactful. Moreover, to prevent overfitting to high-signal data, D$^3$S employs a dynamic down-sampling schedule inspired by curriculum learning. This schedule starts with aggressive down-sampling to accelerate early learning and gradually relaxes to promote robust generalization. Extensive experiments on Qwen2.5 and Llama3.1 demonstrate that integrating D$^3$S into advanced RL algorithms achieves state-of-the-art performance with generalization while requiring fewer samples and tokens across diverse reasoning benchmarks.

Published as a conference paper at ICLR 2026
LEARNING MORE WITH LESS: A DYNAMIC DUAL- LEVEL DOWN-SAMPLING FRAMEWORK FOR EFFI- CIENT POLICY OPTIMIZATION
Chao Wang
Tao Yang²* 24
Hongtao Tian
Yunsheng Shi?
Qiyao Ma
Xiaotao Liu
Ting Yao?
Wenbo Ding!
1
Tsinghua University
WeChat, Tencent
University of California, Davis
ABSTRACT
Critic-free methods like GRPO reduce memory demands by estimating advan- tages from multiple rollouts but tend to converge slowly, as critical learning signals are diluted by abundance of uninformative samples and tokens. To tackle this challenge, we propose the Dynamic Dual-Level Down-Sampling (D'S) frame- work that prioritizes the most informative samples and tokens across groups improve the efficiency of policy optimization. D operates along two levels: (1) the sample-level, which selects subset of rollouts maximize advantage variance (Var(A)) We theoretically proved that this selection is positively corre- lated with the upper bound of the policy gradient norms, yielding higher policy gradients. (2) the token-level, which prioritizes tokens with high product of ad- vantage magnitude and policy entropy (|Ai,ll x H4.)). focusing updates on tokens where the policy is both uncertain and impactful. Moreover, to prevent overfitting high-signal data, S employs dynamic down- wwwn-samlinng schedule inspired by curriculum learning. This schedule starts with aggressive down-sampling to accel- erate early learning and gradually relaxes to promote robust generalization. Exten- sive experiments Qwen2.5 and Llama3. demonstrate that integrating D into advanced RL algorithms achieves state-of-the-an performance with generalization while requiring fewer samples and tokens across diverse reasoning benchmarks.
INTRODUCTION
Reinforcement Learning (RL) has become instrumental in aligning Large Language Models (LLMs)
with human values and preferences (Ouyang al. 2022), leading to the emergence of various alignment algorithms. Among these, critic-free methods like Group Relative Policy Optimization
(GRPO) (Shao al. (004)) and Group Sequence Policy Optimization (GSPO) (Zheng et al. 2025) have marked a crucial step towards greater memory efficiency. These methods estimate advan- tages using relative rewards from a group of sampled responses, thereby eliminating the need for separate critic network (Shao et al. 2024). However, while the memory bottleneck is alleviated, efficiency challenges remain. The precision estimation of advantages utilized in training depends on the quality of the sampled groups. Larger groups risk diluting critical learning signals from a few key samples and tokens, as these signals can be overshadowed by the averaging effect numerous undifferentiated samples (e.g., groups dominated by mostly correct or incorrect samples in mathe- matical reasoning tasks). Conversely, smaller groups may struggle to yield diverse samples due to insufficient sampling. This trade-off constrains the optimization efficiency of critic-free algorithms.
To tackle this issue, Razin et al. 2024 2025) reveals that raising the variance of reward signals accelerate convergence, higher reward variance (Var(F)) creates steeper optimization land-
"Contributed equally.
Work done while interning at Tencent.
1 'Corresoonding Author. Tuckytyann@@tenceee
ding..wwnbbb.. wenbo@ebbbla.... tai waginnggaa.... edu.cc

---

Published as a conference paper at ICLR 2026
scape. However, for typically critic-free methods (e.g., GRPO and GSPO), advantages are computed by normalizing the selected subset, resulting in a fixed advantage variance of 1. Our theoretical analysis in Section 2.1 demonstrates that maximizing Var in critic-free methods imposes a fixed upper bound on the policy gradient norm, whereas maximizing advantage variance (Var(A) (4) intro- duces variable upper bound positively correlated with the advantage variance. This indicates that maximizing Var(AA has the potential to yield higher policy gradients within the core subset, thereby accelerating policy convergence toward the optimal trajectory.
b
0.35
Consumed Tolentt%)
100
GRPO
D
GRPO
D
80
Gradient Norm
0.30
GR
20
Pass011
0.25
D
40-
0.20
20
0-
0.10
0
Epoch
Epoch
Epoch
Figure 1: Comparison of training dynamics between D'S and the GRPO: (a) token consumption ratio, (b) gradient norms, and (c) Pass@1 1 scores. Compared to the original GRPO, the integration of reduces token usage, accelerates policy convergence, and delivers superior performance.
Building on this insight, we propose Dynamic Dual-Level Down- Nown-Sampling (D'S) framework, which operates two levels. First, at the sample-level, instead of maximizing Var(I)). D selects sam- ples by first estimating the group-relative advantage across the entire batch and then maximizing Var(A) to identify the subset for optimization. Second, at the token-level, we further consider both advantage and entropy metrics, proposing the product of advantage magnitude and entropy (|A.,rl Hi,)) as a measure of token importance. The advantage magnitude reflects the relative sig- nificance of tokens, while entropy quantifies uncertainty, an essential factor for guiding exploratory reasoning in reasoning tasks (Wang et al. 2025). Moreover, to prevent the policy from overfitting to a limited set of high-signal data and compromising its generalization, we introduce dynamic down-san wwwn-samplnng schedule. Inspired by curriculum learning (Bengio al.) 2009). which progresses from simple to complex, the dynamic down- sampling schedule begins by prioritizing a smaller sub- set of high-signal samples and tokens to accelerate early-stage learning. As training progresses, the data scale gradually expands, incorporating more samples and tokens to improve generalization over the thorough data distribution beyond narrow high-signal subset.
To verify the effectiveness of D²S, we conduct extensive experiments across various RI. settings (i.e., GRPO and GSPO) on challenging mathematical reasoning tasks. The results demonstrate that incorporating DÒS into GRPO and GSPO consistently improves performance while reducing sam-
ple and token requirements. A direct comparison of training dynamics, as illustrated in Figure T (Yang et al.] 2024) as the backbone and is trained AIME24 (MAA 2024]. Compared to the original GRPO, D optimizes fewer than 20% of the tokens (Figure Tap
uses Owen
while achieving higher policy gradients (Figure lb leading to significantly faster convergence and superior Pass @1 scores on the test set (Figure Ic). Specifically, when using Qwen2.5-Math- 7B as the backbone, GRPO with D'S achieves average improvements of 4.5 in Pass and 3.7
in Pass@8 across seven datasets, compared to the original GRPO. Similarly, with Llama3.1-81
Instruct (Grattafiori et al. 2024) as the backbone, GRPO with D$S outperforms the original by 3.3
in Pass @ and 7.8 in Pass @8. Moreover, our analysis highlights the following key findings: (1) Both sample-level and token-level down-sampling effectively eliminate undifferentiated signals in the early training stages, accelerating policy convergence.
(2) In the later training stages, the dy- namic down-sampling schedule plays a crucial role in enhancing the generalization of D''s S. (3) DåS better manages entropy fluctuations, reflecting more stable policy training.
2 THEORETICAL ANALYSIS
2.1 PRELIMINARIES
Formally, let x represent an input query from the dataset D. A large language model with parameters 0 is defined policy no. For each query, the policy no generates multiple responses, with group
2

---

Published as a conference paper at ICLR 2026
size of G. For the i-th response Vs. the number of tokens is denoted as (3a). GRPO (Shao al. 2024
removes the need for a standalone value network by estimating group-relative advantages directly from G. The optimization objective is given
JORro(0 ()
(. "r (-lx)
(I)
1 1
Ua Σ
min(w: ( (0,,, +,) -DKL
G
12.30.el2.31..c) where the importance ratio is w;,( new (Vi. (m. which will be clipped by hapeeeearameter
regulates the constraint on the KI--dieegeecce DEL. The group-normalized advantage is derived by standardizing the reward signal R within each group:
4 R(x, vi) à EE R(r,ys)
std (M(, W.)
Building upon GRPO, Zheng et al. 2025 proposes GSPO, which utilizes the full sequence as con- text for next-token prediction. While adopting the same token-level group-relative advantage signal defined in Equation [] GSPO further incorporates a novel importance ratio based on sequence like- lihood. The optimization objective of GSPO denoted as:
Jasro (0)
- (12)
(3)
Bel
min(s ( (0),
+, 1,
- - (To Fraf))
where the importance ratio is s (0)
( IT lar) Tesu(o;
no e (r.
with sg[:] denoting
sg mea (Vt. [=,))
stopping gradient. The group-normalizer advantage is same as Equation 2
2.2 UPPER BOUNDS ON GRADIENT NORMS
We begin by analyzing the upper bound the gradient norm in GRPO (also applicable to GSPO), as it determines the scale of policy updates and directly impacts training efficiency.
Proposition 1. The gradient of JGRPo satisfies:
I/V JoRro S (1;
(4)
where (x: 0) denotes a static parameter related input x and model parameters
Proof. See Appendix B.I
Proposition T highlights key property of group-based methods: the gradient norm is capped by fixed upper bound, regardless of the explicit reward variance Var(R). We then extend Proposition to scenarios where optimization is performed on a subset sampled from the group G.
Proposition 2. The gradient of JeRPO which selects a subset from original rollout group with size G satisfies:
(VoJarro S 2 ((x; 11)-1) (Var(A'))
(5)
where Var(A') is the advantage variance of the selected subset from rollouts.
Proof. See Appendix B.2
Lemma 1. Let A be set of M elements that is standardized as B[A] = O, Var(A) = For any integer N such that < N S -M, there exists a subset A' C with [A' = satisfying Var(A') 1.
Proof. See Appendix B.3
From Proposition 2 we can observe that under the down-sampling perspective, the gradient norm
of GRPO has a variable upper bound, which is positively correlated with the advantage variance Var( A'r( of the subset. Consequently, the strategy of maximizing Va)(R) (Razin et al.) 2024 2025 Xu et al. 2025) in group- methods encounters two major limitations. First, even if Var (ar(R) R maximized within the selected subset, the advantages still computed under the constraint of
3

---

Published as a conference paper at ICLR 2026
fixed Var(A') and cannot change the gradient norm upper bound. Second, the advantages are estimated within a smaller, biased subset of the original group, leading to unstable advantage estimation. This naturally motivates an alternative approach: compute normalized advantages using all samples from the original group, and then select a subset that maximizes the variance these normalized advantages. Lemma proves that subset with variance of least 1 can be extracted from a set of normalized advantages. This indicates that such a subset leads to a higher upper bound on the gradient norm. The trends in Figure lb provide empirical evidence supporting this property.
3 METHOD
In this study, we introduce the Dynamic Dual--evel Down-Sampling (D*S) framework, which im- proves training efficiency through a two-tier approach: a sample-level down-sampling strategy and a token-level selection mechanism.
3.1 SAMPLE-LEVEL CROSS- CROSS- ADVANTAGE- BASED
Building on the insights from Section 2.2 rather than selecting subsets of rollouts based on maximiz- ing Var(Rt) (Razin et 2024 2025 Xu et al. 2025), we propose a refined method that identifies core subset of samples based on their group-relative estimated advantages. This approach prioritizes maximizing the variance of advantage signals within the selected subset. Formally, given a query and rollouts Squery = ((r, y/, A,) : E [1,]], the selected subset Squery is defined follows:
Squery arg max Var(Ag)
SCS 18|=Ns
where N3 is the number of selected samples within the group Squery. and Ag A2,
A2. A
denotes the advantage set of Squary. Each advantage A; is defined as the mean of the token-level advantages Ai, over the sequence. Xu et al. (2025) shows that the subset maximizing variance can be efficiently selected from the (N) possible combinations, where Ng- N + n.eg In our scenario, adopt this implementation, where Ng. N.p refers the samples with the highest positive advantages, while Ng. S neg corresponds to those with the lowest negative advantages.
Moreover, considering that gradient updates performed in batches and there are significant ad- vantage disparities across different groups (e.g.. groups with all correct or incorrect predictions may result in uninformative zero advantages), we introduce a cross-group operation to select high- variance subset from the entire batch. Formally, let Spatch {Squery b E [1, B}} be a batch of rollouts, where B is the batch size and N = B N₂ is the total number of selected samples. The high-variance subset can then be obtained as:
Spatch arg max Var(Ag)
SC Shan
The group operation retains the original distributional properties, the advantages pre- normalized within each group, and no additional normalization applied.
3.2 TOKEN-LEVEL: ENTROPY-ADVANTIN WEIGHTED SELECTION
Intuitively, responses often consist of combination of easy tokens (where the model is both con- fident and accurate), neutral tokens (with minimal impact on outcomes), and critical tokens (where the model is uncertain and decisions significantly influence rewards). Policy entropy serves measure the model's uncertainty (Cui et al.) 2025) and as an indicator of potential performance gains. Wang al. 2025) demonstrates that the top 20% of tokens with the highest entropy dominate the policy gradient. Treating all tokens equally during updates dilutes the gradient signal, akin to averaging out meaningful information with noise.
To this end, we propose a token-level selection mechanism that integrates generation entropy and its advantage into unified importance metric for ranking tokens across all selected samples. Specifi-
4

---

Published as a conference paper at ICLR 2026
cally, high -eepoimpottacce tokens are identified follows:
Hi, = Σ (token; Ti, Mi,c)) log No (tokenj) Ii Vi,ct)
(8)
T = top K% (Hi,r, Mi.t E S, key = |A,, x Hi,e)
where H1 denotes the entropy of t-th token in i-th response, and K indicates the proportion of selected tokens. High entropy H indicates higher uncertainty in the model's decision at that token position, fostering exploration and enhancing policy diversity. Advantage A,, quantifies the impact of token on policy improvement, where a larger [A reflecting greater potential-posttivee or neg- ative-for optimízation. During each update, only the top K% of tokens contribute to the gradient of 0. By selecting tokens with high entropy and advantage. computation is focused on the most informative decision points, encouraging the policy to resolve uncertainty reward-critical regions.
3.3 DYNAMIC DOWN-SAMPLING SCHEDULE
The sample-level and token-level strategies discussed in Sections 3.1 and 3.2 refine policy updates by prioritizing high-signal samples and tokens. Although this approach accelerates convergence by leveraging stronger gradients, also heightens the risk of reward hacking or overfitting. The model might over-exploit limited of trajectories that seem highly informative in the early stages, but struggle to generalize as optimization progresses. To address this, we introduce dynamic down- sampling schedule that progressively reduces the intensity of down-sampling training advances.
Specifically, we employ a linear schedule to interpolate between the initial aggressive configuration (Nint, iink) and the final milder configuration (Nimal: Kina)) based the training progress p [0, 1]. We use N to regulate the number of retained samples per query, while K(P) determines the proportion of retained tokens within each sample:
(N(P) R()) (1 p) [Niis, Kinll + p [Ninal, Kinal]
At the beginning of training (p- 0), the model prioritizes the most informative rollouts and tokens to accelerate learning. As training advances (p - 1), its focus broadens, incorporating more diverse signals enrich learning and expand its scope. As empirical results show in Figure 3 variance- based selection enhances early performance but loses effectiveness over time and risks overfitting. In contrast, the dynamic schedule sustains both fast convergence rate and consistent improvements. Our dynamic scheduling mechanism draws inspiration from curriculum learning. This dynamic curriculum ensures that the model can "generalize" to the entire data distribution while utilizing high-signal data, thus maintaining continuous performance improvement while avoiding overfitting.
3.4 S OPTIMIZATION OBJECTIVE
By integrating the sampling strategies outlined above, the objective function of D is expressed in Equation TO
(
E E D., (vr
1 iES le
[min (u, clip(we.. E, 1 + A., -BDKL (Tell (10)
S
EXPERIMENT
4.1 CONFIGURATION
Datasets and Evaluation We train each model using the DeepScaleR (Luo et al. 2025) dataset, which includes AIME problems from 1984 to 2023, AMC problems prior to 2023, and questions from other sources. . During training, the AIME24 (MAA] (024) is used as a validation set to monitor the oft-of-domain performance of policy in real time. Evaluation is conducted on diverse set of
benchmarks, including AIME25 (MAA (025). AIME24 (MAA 2024). AMC23 (MAA) 2023).
GSM8K (Cobbe et al. 2021), MATH (Hendrycks et al. 2021). MinervaMath (Lewkowycz al.
5

---

Published as a conference paper at ICLR 2026
2022) and OlympiadBench (He al. 2024). For each question in these benchmarks, generate 32 parallel outputs and compute Pass metrics. Rewards are assigned to the entire sequence based on the correctness of the answer, verified using math verify tool (Kydliček] 2025).
Models We employ various models to systematically evaluate the proposed DÒS framework. Qwen2.5-Math-75 (Yang et al., 2024), pre-aligned model, specifically optimized for mathe- matical tasks. Llama3. 1-rmaa..--B----. InSB-unttruct (Grattafiori et al.] (2024) serves as general-purpose baseline model, while OpenMath2-LLanea -laamm2--- 1-8B (Toshniwal et al. [2024), a fine-tuned variant of Llama3.1- 8B-Instruct is included for comparison. Besides, a smaller model, 1ren2.5-Math-J-. Yang et al.) 2024), utilized to assess adaptability across varying model scales. Each model is configured using its officially recommended settings as shown in Table
Baselines To assess the effectiveness of our approach, we integrate D into two popular RL algorithms: GRPO (Shao et al. 2024) and its variant, GSPO (Zheng et al. 2025), which enhances GRPO by improving sequence- level advantage estimation. Additionally, we compare our method with PODS (Xu et al. (2025), down-sampling strategy via maximizing reward variance. To further analyze the impact of different stages of D²S, we compare several variants: (1) D'S, which applies sample-level down-sampling by maximizing advantage variance as defined in Equation 6 (2) D w/Cross, which incorporates gross operation as described in Equation 7 and (3) D'S, which combines sample-level and token-level down-sampling but excludes the use of the dynamic down- sampling schedule, as defined in Equation g Parameters are listed in Table 6
4.2 MAIN RESULTS
Table 1: Experimental results on various mathematical reasoning benchmarks using different model backbones. We report Pass 1/Pass @8 scores, computed from 32 parallel runs. The best results are highlighted in bold, while the second best are underlined.
Model AIME24 ATME25 AMC23 GSM8K MATH Minerva Olympiad Average
Qwen2,5-Muth-7B
Base
8,9/33.2
2.3/13.4
22.8/70.4
30.1/83.2 27 27.9/64.6 8.4/33.7
4.1/14.6
14.9/44.7
GRPO
13.2/37.6
5.5/21.6
47.0/83.5
64.9/94.3
48.5/70.2
19.8/45.0
9.7/19.8
29.8/53.1
+PODS
16.1/40.5
7.8/24.5
$2.8/8155
73.M95.0
53.071.1
24.6/47.5
11.0/20.7
34.1/54.4
+D'S
20.3/48.2
7.9/25.8
54.4/87.1
71.3/9577
52.2/71.5
25.0/48.2
10.7/20.8
34.3/6.8
GSPO
15.8/42.4
6.7/25.3
50.8/81.2
72.095.2
52.1/7100
24.2/47.3
10.8/20.7
33.2/54.7
+PODS
15.4/40.9
6.5/22.9
51.9/8166
72.9/95.3
52.9/7111
25.0/47.6
10.9/20.9
33.6/54.3
+D'S
18.3/3.3
8.3/26.9
53.2/83.8
76.0/96.1 54.9/71.4
28.4/51.1
11.5/21.11
35.8/56.2
Swen2.5-Math-11
Base
4.7/23.7
2.1/13.2
21.3/60.8
23.7/75.8
18.6/56.3
7.5/30.0
5.0/15.8
11.8/39.4
GRPO
10.0/28.3
6.1/19.9
46.2/77.0
27.3/94.4
53. 1/.9/
20.8/43.7
10.2/18.9
32.0/50.2
+PODS
12.2/30.6
5.9/22.3
47.4/5..1
77.0/94.2
53.2/69.3
21.8/440
10.3/18.3
32.5/50.5
+D'S
11.2/32.2
6.9/24.0
48.6/79.7
77.5/94.1
53.7/69.5
23.5/44.5
10.6/18.4
33.1/51.8
GSPO
11.1/29.9
6.9/23.0
49.7/79.8
78.1/94.3
53.5/69.4
23.0/44.1
10.5/19.4
33.3/51.4
+PODS
12.3/32.9
6.9/24.0
47.8/77.1
77.4/94.1
53.4/69.4
22.5/43.3
10.2/18.7
32.9/51.4
+D*S
11.4/2.88
8.2/25.2
48.4/79.1
78.0/94.1
54.0669.6
22.9/43.2
10.5/19.0
33.3/5.9
IJama3.
1-8-
Base 1.7/10.9
0.4/2.8
15.0/47.3
57.7/92.8
29.3/55.9
14.7/38.5
2.2/8.2
17.3/36.6
GRPO
2.0/5.0
0.0/0.0
13.7/33.4
78.6/93.5
31.5/52.0
15.9/35.6
2.1/7.2
20.5/32.4
+PODS
2.8/9.9
0.3/2.5
14.5/38.1
77.1/93.3
31.5/52.6 16.3/38.0
2.2/7.6
20.7/34.6
+D*S
5.3/20.7
0.1/0.8
20.3/50.8
79.0/95.0
35.9/59.2 22.5/44.3
3.3/10.7
23.8/40.2
Table presents the alignment results of the different LLMs across seven reasoning benchmarks, using GRPO and GSPO as the base algorithms. Our observations are fourfold. First, the intro- duction of D'S consistently improves performance across all backbone models, demonstrating its strong adaptability across various types of backbones. Notably, D achieves the highest average scores of 35.8% for pass@1 I and 56.8% for @ on the Qwen2.5-Math- demonstrates a significant performance advantage over both the original method and PODS. For example, when Qwen2.5-Math provement of 4.5 points in pass @1 and 3.7 points in pass compared to the original GRPO. Simi- larly, with Llama3.1
model. Second, D
is used as the backbone. GRPO+D*S achieves average im-
as the backbone, GRPO with D'S surpasses the original GRPO and

---

Published as a conference paper at ICLR 2026
GRPO+PODS by 3.3 and 3.1 points on pass and by 7.8 and 5.6 points on pass@8, respectively. Third, even for the strong baseline GSPO, incorporating D'S still yields improvements of 2.6 and 1.5 points on pass @ and pass@8. respectively. This demonstrates that the D down-sampling strategy can be seamlessly generalized to and further enhance other algorithms leveraging group- relative advantages. Fourth, for the smaller-scale Qwen2.5-Math- 1.5B model, D still achieves the best average performance among all compared methods, showing its scalability and effective- ness across LLMs of varying sizes. We provide more experimental results, including pass@ 16 metrics, in Appendix D.3 which further validate the effectiveness of our approach.
4.3 ABLATION STUDY OF S
D integrates sample-level, token-level, and dynamic down-sampling schedule to effectively train the policy model. To systematically evaluate the impact of each component in the D strategy, we conduct a series of ablation studies by progressively incorporating more sophisticated selection strategies into the GRPO baseline, as listed in Table Here, we utilize Qwen2.5-Man --tth-- as the base model. The notation for each method is explained in detail in Section 1.1
Table 2: Ablation studies of different components in D strategy. Performance is evaluated using Pass @8 8 across various benchmarks. The experiments utilize Qwen2.5-Math-7 the base model and GRPO as the foundational algorithm.
Model
AIME24
AIME25 AMC23 GSM8K
MATH
Minerva
Olympiad
Average
Base
8.9/33.2
2.3/13.4
22.8/70.4
30.1/83.2 27.9/64.6 8.4/33.7
4.1/14.6
14.9/44.7
GRPO
13:2/37.6
5.5/21.6
47.0/83.5
64.9/94.3
48.5/70.2 19.8/45.0
9.7/19.8
29.8/53.1
+D's
13.2/42.9
5,9/20.2
50.6/84.4
68.5/94.9
50. 1/7.5 20.8/46.4
10.3/20.4
31.3/54.2
+DC
17.3/40.0
7.7/25.6
51.9/83.3
73.495.4
52.8/70.9
25.0/47.1
10.6/20.6
34.1/54.7
+D's
16.9/42.2
6.0/21.2
49.6/82.8
66.3/94.9
49.5/70.7 20.9/46.7
10.1/20.3
31.3/54.1
+D'S
20.3/48.2
7.9/25.8
54.4/87.1 71.3/95.7 52.2/71.5 23.4/48.2 10.7/20.8
34.3/56.8
The ablation results reveal that the contribution of individual components is not strictly monotonic. For instance, DåS occasionally underperforms D'S on AIME24 but achieves greater improvements on AIME25. Nevertheless, the complete DOS configuration consistently delivers the best perfor- mance. This provides strong evidence that the dual-level design. combined with dynamic down- sampling schedule, effectively balances exploitation and exploration, leading to robust improve- ments across tasks. Beyond the Qwen2.5-Math-77 model, additional ablation studies and analyses on Llama3.1 models, detailed in Section D.1 further demonstrate the generalizability of the D
4.4 ABLATION STUDY ON DYNAMIC SCHEDULING HYPER-
HYYPER-PARAEETE PARAMETER
Table 3: Ablation of parameter (numbe of selected samples) and ((ratio of reserved tokens) used in dynamic scheduling (Equation 9) on GRPO algorithm finetuning Swee2.5-Mathh Results presented in form of Pass @ 1/Pass @8 scores. The best results are highlighted in bold while the second-best are underlined.
Configuration
AIME24
AIME25
AMC23
GSM8k
MATH
Minerva
Olympiud
Avernge
D's
20.3/48.2
7.9/25.8
544/87.1
71.3/9577
$2.2/7155
25.0/48.2
10.7/20.8
34.3/56.8
N_ 16
20.1/46.2
7.9/33.2
56.5/85.0
76.1/95.9
55.5/71.7
23.0/469
11.1/21.0 0
35.8/55.7 7
N 16 - 32
18.4/467
7.6/25.2
55.085.5
73.6995.8
53.1/71.4
22.5/47.6
10.7/2099
34,4/56. 2
300%
19:7/43.4
8.9/25.6
$7.471.3
77.6/95.
23.6/47.6
11,3/112
36.455.2
K = 5%
+ 40%
18.1/41.1
7.6/24.6
$5.2/82.5
72.5/95.4
$4.1/71.5
21.7/6.9
10.2/20.8
$4.2/54.7
K- 5%
50%
17.5/44.5
7.6/25.8
52.0/82.8
74.4/95.6
54.271..7
1/47/
11.3/14
34.4/55.7
K 10%
+ 20%
16.8/42.3
7.5/23.8
56.0883.6
74.3/95.6
54.6/71.6
21.7/46.4
10.5/20.5
34.5/54.8
K- 10%
+ 30%
17 3/46.1
8.1/24.8
$3. 7/.7/86.4
72.7/95.2
53/1/71.5
22 1/47.0
10:3/208
34.0556.0
K 10%
40%
18 6/45.0
7.7/26.1
$2.5/83.
70.2/95.6
52.2/71.4
2/.5/48.2
10.4/20.6
33.4/55.7
K 10%
-> 50%
17 5749544.5
8.0/27.8
52. 3/83.6
72.1/95.6
53.9/71.5
22.2/47.2
10.7/20.7
34.0555.8
To assess the sensitivity of D dynamic down-sampling schedule to its key hyper-parameters (Nmit: NN Kinit, K finat), we conduct a detailed ablation study. We use (owen2.5-Math-77 as the backbone, and the results are presented in Table 3 Our baseline D'S configuration (used for main results in Section 4.2) sets G = 32 (large enough in favor of GRPO advantage estimation, fixed in this ablation), N = 8 + 32 and 5% + 20%. achieving an average Pass@ I/Pass @g of 84.84/56.766 76. This configuration heuristically draws inspiration from empirical conclusion from
7

---

Published as a conference paper at ICLR 2026
Wang et al. (2025). The ablation results show that DÒ is not overly sensitive to these choices. When testing difterent sample-level configurations, such as N - 16 (35.74/55.70) and N = 16 - 32 (34.41/56.10) the average performance remains remarkably stable and comparable to our baseline. We also explored a wide range of token ratios. All configurations maintain strong performance, while K 5% 30% achieves
higher Pass @1 1 on multiple benchmarks, which indicates training might benefit from more "low-signal" data. This improvement further shows the advantage of
schedule strategy of D
This analysis demonstrates that the effectiveness of D is robust across a reasonable range of hyper- parameters. D'S does not require extensive, task task-specific tuning to achieve significant performance and efficiency gains.
4.5 TRAINING EFFICIENCY
Table 4: Comparison of training efficiency. We assess the performance gains brought by integrating D into the GRPO and GSPO, along with the time acceleration needed to achieve these gains.
D vs GRPO
D vs GSPO
Methods
@v 32
Time
Avg@32
Time
DDwe22.-Maath-2
+6%
2.04x 1.57
+17%
5.51x
Qwen2.5-Math-Il
+4%
+2%
1.10x
higghhghs the remarkable training efficiency of D For instance, on the S.we2.55MMtth75. model, D'S achieves an average accuracy (Avg@3) 32) that 6% higher than GRPO, while reducing the training time by half to reach the same performance level (2.04x speedup). The benefits are even more pronounced compared to GSPO, with a 5,51 x faster training speed and a 17% accu- racy improvement. On the smaller Qwen2.5-Math- alongside 1 1.57x speedup. These findings underscore D'S's ability to not only accelerate model convergence but also enhance final performance, partieularly for larger models.
model, D²S delivers 4% accuracy boost
4.6 DYNAMIC DOWN-SAMPLING SCHEDULE MITIGATES OVERFITTING
To better understand how D impacts policy optimiza- tion, we track key metries during the training process. The first metric, sample usefulness rate (SUR), measures the proportion of groups in each batch with non-zero ad- vantages, reflecting the percentage of samples that ac- tively contribute to policy updates. The second metric, KL divergence (DKL.), quantifies the difference between the policy and reference model distributions. A higher DKT. indicates greater divergence from the reference model, which may signal an increased risk of overfitting.
Sample Useful Ration))
100
90-
80
70
60
Epoch
0.6
Figure 2 illustrates the training dynamics under various D settings, as detailed in Section 4.. Four key ob- servations emerge: First, the original GRPO and D'S (without cross-group oper- ation) maintain SUR of approximately 70%, with minor fluctuations. Second, introducing cross-group operation S w/Cross and D²S) significantly boosts the SUR to nearly 100%, indicating that cross gross mechanism effectively filter out ambiguous data within the training batch. Third, as shown in Figure 2h] the DKL curves for S'S, S w/Cross, and D2S rise more steeply in the early stages compared to GRPPO but eventually converge to similar values. This suggests that while these methods initially deviate more from the reference model, they ultimately reach comparable limits, potentially increasing the risk of overfitting. Fourth, the SUR gradually declines from 100% to 70% as training progresses, aligning with the design goal of dynamic down-sampling wwwmpling-iinrrmmmmtttee incrementally increasing samples and tokens to mitigate overfitting in later stages. The DKL. curve also demonstrates that D achieves smaller deviations from the reference model compared to other methods.
0.4
shown in Figure 2a both
0.2
0.0
Epoch
Figure 2: Training dynamics of dap sam- ple usefulness and (b) KI. divergence.
8

---

Published as a conference paper at ICLR 2026
GRP
GRPO
GAPO
ERPO
0.25
0'5 Cross
0.25
17%
0.20
0.20
0.30
0.20
0.15
0.35
0.15
0.15
0.10
0.30
0.10
0.10
0.05
0.05
Epoch
Epech
Spoch
Epoch
Figure 3: The Avg@32 test performance of AIME24 Owen2.5-Math-78 under various settings. Methods without dynamic down-sampling schedule (ablec accelerate convergence in the early stages but suffer from overfitting later. In contrast, the dynamic down-sampling schedule ( not only accelerates convergence initially but also outperforms other methods in the later stages.
Figure B provides further comparison of Avg@32 32 performance under different down- wamn-sampling configurations. It can be observed that down-sampling strategies without a dynamic schedule con- sistently accelerate convergence but exhibit varying degrees of overfitting, with their Avg @ 32 curacies eventually being surpassed by GRPO the later stages of training (Figure 3a b c In contrast, D (Figure which incorporates dynamic schedule, not only accelerates convergence in the early stages but also achieves significantly better results in the later stages. This highlights the critical role of dynamic down-sampling schedule mitigating overfitting.
4.7 ENTROPY ANALYSIS OF D
We investigate entropy dynamics across various base models and RL algorithms to understand learn- ing behaviors. As illustrated in Figures 4a and (bb] D³S consistently achieves lower and more stable policy entropy compared to baseline algorithms. This improvement stems from the token- level selection mechanism described in Section 3.2 By prioritizing updates on tokens with a high advantage-entropy product (|Ai,r) x H,,)), D focuses learning on resolving high-impact uncertain- ties. This targeted strategy enables the model to converge more efficiently toward a decisive policy, as evidenced by its reduced average entropy.
Conversely, the Llama3. I--Ba...-BB--.. 1-8B--8BB--st Instruntt model exhibits a distinct behavior, as illustrated in Figure 4c We hypothesize that this phenomenon arises from the base capabilities of Llama3.1 and its lack of adaptation to mathematical reasoning tasks, which leads D²S to promote more effective and productive exploration. The baseline GRPO algorithm only begins to slowly increase entropy when training is nearly halfway complete, indicating inefficient and insufficient exploration. In contrast, D²S, through token-level selection mechanism (prioritizing tokens with high A,, H,,)) able to encourage productive exploration both earlier and more effectively, thereby enabling the model to escape local optima more efficiently. To validate this hypothesis, we introduce OpenMath2- Llama3.1 1-BB Toshniwal et al. 2024), a model fine-tuned specifically for mathematical tasks, point comparison. As illustrated in Figure 4d D exhibits entropy dynamics consistent with those observed in Figures 4a and 4b reinforeing our hypothesis. In Figure 4d while the baseline GRPO still shows a sharp and unstable entropy spike early in training, D ensures smoother and more stable learning trajectory with consistently low entropy. Comparison between results in Figure 4d and 4d strongly indicates that D is able to adaptively encourage exploration (carlier,
Qwen-GRPP
Owen-GSPO
Llama-GRPO GRPO
Llama
OpenMath2-GAPP
Gard
GaPO
iPa
DPs
Epoch
Epoch
Epoch
Epoch
Figure 4: Entropy dynamics of different base models and RL algorithms. D' effectively balances exploration and exploitation, fostering confident policies in well-aligned backbones b and db while driving essential exploration in less-aligned ones (c).

---

Published as a conference paper at ICLR 2026
sharper entropy increment) on less-aligned models and accelerate exploitation (lower entropy curve) on well-aligned models. Additional analyses are provided in Appendix D.I
In conclusion, the entropy dynamics reveal that D effectively balances exploration and exploita- tion, fostering confident policies in aligned models while encouraging necessary exploration in less- aligned ones. This further underscores the robustness of the proposed framework.
5 RELATED WORKS
Data Selection for Enhancing Training Efficiency As models and datasets scale, selecting data with high informational value becomes crucial for improving training efficiency. |aziin et al. (2024) 2025) suggests that increasing reward signal variance sharpens the optimization landscape, thereby accelerating convergence. [u et al. (024)) introduces SEAM, an automated metric for quantifying PM-RM differences, which enhances training reliability by filtering out samples where RM mis- judges PM outputs. Several works focus on curriculum-bast based selection. Curry-DPO (Pattnaik et al.] 2024) and LPPO (Chen et al. 2025) dynamically adjust sample weighting based on difficulty or learning progress. Similarly, Sun et al. (2025) proposes DOTS, which filters questions based on a heuristic difficulty metric (pass rate) estimated by auxiliary predictor. Coresets (Mirzasoleiman (ct al. 2020) aim to select a weighted subset to provably approximate the full dataset's 's gradient. Unlike these approaches, D&S does not rely heuristic difficulty gradient approximation. In- stead, our sample-level strategy maximizes the Advantage Variance of rollout groups, which directly maximizes the upper bound of the policy gradient norm for efficient optimization.
Token-Level Entropy Utilization Entropy reflects the unequal importance of tokens within a se- quence. Waan et al.] (2025) observes that policy gradients are sparse and driven by minority of high-entropy tokens. To address the risk of entropy collapse caused by high-signal tokens, Cui et al. (025) identifies tokens with high covariance between probability and advantage the cause of instability, proposing Clip-Cov to suppress their updates. Similarly, Hao et al. (2025) intro- duces STEER to stabilize training by woinhwiigg tokens that contribute significantly to tropy change. Wen et (2024) introduces ETPO to improve rngrapy-regulariiedd credit assignment. Similarly, [Shen] (2025) proposes AEnt, which calculates clamped entropy over subset of high- probability tokens and employs an adaptive coefficient to balance the entropy reward. While these methods focus on stability by suppressing high-impact tokens, D'S adopts distinct philosophy of efficiency via prioritization. Prioritizing high-signal data has been utilized by Prioritized Experience Replay (PER) (Schaul et al.] (0166). which samples and replays uncertain transitions basing on their TD-error magnitude. Cheng et al. (2025) shapes GRPO advantages with entropy to improve explo- ration, instead of using entropy regularization. In our study, D integrates entropy with advantage magnitude (IA| x H) to precisely locate tokens that are both critical and uncertain. Rather than sup- pressing these high-signal tokens, D explicitly prioritize them to accelerate convergence, together with employing dynamic schedule to mitigate the overfitting risks.
6 CONCLUSION
In this study, we reveal that the theoretical upper bound of the policy gradient norm in group-relative advantage-based algorithms (e.g., GRPO) is positively correlated with advantage variance. Lever- aging this insight, we introduce the Dynamic Dual-level Down-Sampling (D*S) framework to en- hance training efficiency. At the sample-level, D selects rollout responses to maximize advantage variance, while at the token-level, it prioritizes tokens with a high product of entropy and advan- tage magnitude, directing updates to regions where the model is both uncertain and impactful. To mitigate overfitting, D& adopts dynamic down-sampling schedule inspired by curriculum learn- ing, gradually relaxing sampling criteria over time. Extensive experiments on the Qwen2.5 and Llama3. models show that D consistently surpasses baseline methods across diverse reasoning benchmarks. Ablation studies and training dynamic analyses reveal that the synergy between D components is crucial and S is robust against introduced syper-paramette These results high- light the significance of fine- -raingr sample utilization in RLHF and emphasize the pivotal role of entropy in managing the exploration-exploitaion trade-off.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our study introduces the Dynamic Dual-Level Down-Sanpling (D'S) framework to improve the ef- ficiency and performance of RL algorithms. We affirm that this research raises no significant ethical concerns. All methodologies strictly adhere to ethical standards and responsible research practices. The datasets used are publicly available, widely recognized within the research community, and uti- lized in full compliance with their terms and conditions. Additionally, we declare no conflicts of interest, sponsorships, or external influences that could compromise integrity of this work. In line with our commitment to transparency and reproducibility, we will release our code publicly to facilitate further research and innovation in this domain.
REPRODUCIBILITY STATEMENT
Experimental details are presented in Section 4.1 The code and data used in this study are included in the supplementary materials and will be publicly released. Proofs for the main theoretical results (Propositions) and Lemma I can be found in Appendices B.I and B.3 respectively.
ACKNOWLEDGMENT
This work was supported by the WeChat Search team, Tencent, which provided necessary re- search environment for this work. This work was motivated by challenges in AI search systems at WeChat. This work was also supported by Shenzhen Science and Technology Program (No.
077020024000310005500
REFERENCES
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41-48, Montreal Quebec Canada, June 2009. ACM. ISBN 978-1-60058-51141 doi: 10.1145/1553374. 1553380.
Xinjie Chen, Minpeng Liao, Guoxin Chen, Chengxi Li, Biao Fu, Kai Fan, and Xinggao Liu. From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization. arXiv,, arXiv:2507.0657. July 2025. doi: 10445500axXx:2557..667..
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with Exploration: An Entropy Perspective, November 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reúchiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers Solve Math Word Problems. arXiv, (xXi::210011411 November 2021. doi: 10.48550/axxvv.1100..2
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng. Lei Bai, Wanli Ouyang. Yu Cheng, Bowen Zhou, and Ning Ding. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv, arXiv:2505.226 May 2025. doi: 10448550axXi..25552267..
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and Aiesha Letman. The Llama 3 Herd of Models. arXiv, 2024. doi: 10.48550/arXiv. 2407.21783.
/rr/abss 2407.21733
Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking Entropy Interventions RLVR: An Entropy Change Perspective. https://rriiv.rr/aa////// 10150v1, October 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual
11

---

Published as a conference paper at ICLR 2026
Multimodal Scientific Problems. arXiv, arXiv:2402.1400: June 2024. doi: 10.48550/arX1..24022 14008.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv, arXiv:2103.03874 November 2021. doi: 10.48500/rxxi.22000.088/
Hynek Kydlíček. Math- Vert-ferify: Math Verification Library, September 2025.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving Quantitative Reasoning Problems with Language Models. arXiv, arXiv:2206. 14858, July 2022. doi: 10.48550/arXiv.2266. 14858.
Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, and Huaxiu Yao. It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF. arXiv, arXiv:2406.07971, June 2024. doi: 10.48550/arXi..2266.77999
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl. Technical report, agentica, 2025.
Mathematical Association of America MAA. Math-ai/ume23 Datasets Hugging Face. http://uuggmnaaaeeoooaaasstttmbu
2023.
Mathematical Association of America MAA. Math-ailaime24 Datasets Hugging Face. 2024.
Mathematical Association of America MAA. Math-ai/aime25 Datasets Hugging Face. 2025.
https:/huuggingffce.
co/dattset//math-alaaiiii http:://gggnngiiie.cooootttseeee
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for Data-efficient Training of
Machine Learning Models, November 2020.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv, 2.Xiv22033.02155. March 2022. doi: 10. 10.485500aaXxi.2203.2211
Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Mad- husudhan. Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences. arXiv, 2XXv:2403.072230 November 2024. doi: 10.485500arXx..2403.772222
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, and Etai Littwin. Vanishing Gradients in Reinforcement Finetuning of Language Mod- els. arXiv, arXiv:2310.2770 March 2024. doi: 10-455500axxX.2231.2077.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What Makes Reward Model a Good Teacher? An Optimization Perspective. arXiv, a5X March 2025. doi: 10448550anXx..25555557777
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay, February 2016.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeckMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv, arXiv:2402.03300, April 2024. doi: 10.485500axXx.22022.33000
Han Shen. On Entropy Control in LLM-RL Algorithms. arXiv, arXiv:2509.03490 September 2025. doi: 10.48550anXiv.2550.54444
12

---

Published as a conference paper at ICLR 2026
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty- targeted Online Data Selection and Rollout Replay. http://ariivarraas55566..666... June 2025.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 Rule: High-Entrope Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv, 2025. doi: 10.48550/arXi..5506 01939. URL httthhtt:///rtt
org/ abs / 2206 939
Muning Wen, Cheng Deng, Jun Wang, Weinan Zhang, and Ying Wen. Entropy-Reggllrizee Token- Level Policy Optimization for Large Language Models. arXiv, arXiv:2402.06700, February 2024. doi: 00448500/rXxi224022007700
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not All Rollouts are Useful: Down- Sampling Rollouts in LLM Reinforcement Learning. arXiv, arXiv:2504.138 13818, June 2025. doi: 10.485500/rX1v.2244.3338
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mate technical report: Toward mathematical ex- pert model via self- coff-mpproveeent. arXiv preprint arXiv:2409. 12122, 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hur Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group Sequence Policy Opti- mization. arXiv, 1807,,,, July 2025. doi: 104485500axxiw.2577.1888
13

---

Published as a conference paper at ICLR 2026
A USAGE OF LARGE LANGUAGE MODELS
In this study, we leverage LLMs to summarize and refine academic papers, ensuring clarity, preci- sion, grammatical accuracy, and correct spelling. These models also offer suggestions improve coherence and readability. Our aim is to elevate the efficiency and quality of academic writing.
B PROOF
B.1 PROOF OF PROPOSITION I
We analyze the gradient at the reference policy without considering the clip- ping operation, as it is inactive when the importance ratios equal 1.
Proof of Proposition L
The gradient becomes:
vd A Vo log (o
V VoJaRO Be &
(11)
G
V/
i=1
E
A Ve log softmax(20 vi,ce))
(12)
G
BgnDi p
(T v0(-(,,)i.<z)) Vo
(13)
G
In following analysis, simplify the average of token-wise advantages Σ 13s1
Etsl Ar, into sequence-
wise advantage A, On the other hand, is also description of the more commonly used sequence- wise outcome-rewarded GRPO implementations in math tasks, Formally:
1
V JaJroo ( B (1)(
Σ
A.Vo log mo
(14)
G
where V lo 1oul
Ve log To (yr, (x, yi,ce).
For any c O, we decompose the sum based on the magnitude of the standardized advantage:
VoNGRPO(O)
1
Σ
Aog
G
+ E
Σ
A log
(II)
G
</A:/>
Bounding term
For samples with [Ag] S c:
[[(]]] <
Ad V log (x+
(15)
<
E
(Ione-hot =0(112))² Voze
(Vll))))\Vozlll (v11=))
(16)
G
< E
Σ
c. (Igeeeuu
na(- [|V=
(17)
Considering non-negative property of one-hot vector and likelihood vector, get:
Tone-hor T0(+1>)]1
Tome-bor T-(-12)]1=
(18)
14

---

Published as a conference paper at ICLR 2026
where || ||1 denotes the L1 norm.
Apply Equation 18 to Equation
17
we can get:
[ S c- c.2 0)
(19)
where (x;; O) denotes a static parameter related to input query x and model parameters O. Similarly, we apply Equation 18 and (2:; 0) in bounding term
[II] = E
A. log (o(((2))
(20)
G
A
< E
| To(il()))
(21)
2 7. E
Σ A
(22)
:||:
2. 2 B[[AA| A|/cel
(23)
For samples with |A: > c, we use the refined Chebyshev inequality. Since A, is group-normalized
with E[A,] and Var(A.) we have E[A2]
1. By the refined Chebyshev bound:
E E
E|AA,1 1A1d
(24)
Therefore:
1
||II|| s 2- ((x; 0)
(25)
Combining the bounds:
V VoJaRPo (0)1l s ||O]|| + +|(DD|
(26)
S. 2 2-; 0) + 2.77 7z; 0)
(27)
= 2> 0)
c
(28)
The right-hand side objective is minimized when c 1, we obtain: [[VeJGRPO < - >x;; 0) 2 47
(29)
B.2 PROOF OF PROPOSITION 2
Proof of Proposition Considering Equation 24 the premise for it to be valid is E[
=
0, VAr(A) = thus E
A Var( + E[A]² =
which is property of standardized advantages.
Thus, applying E[A2 and [A,| > c, we can proof Equation 24 as:
B| 1 1AAd
AAlf(A)dAA
(30)
AA
S(A)dA
(31)
|A,
(AP)AAA)AA
(32)
C
IAPs((A)AA
(33)
E[
(34)
15

---

Published as a conference paper at ICLR 2026
However, considering |A|-driven down- fown-sampling, E[A*]] = 1 no longer holds for subset ./'.
We first estimate the upper bound [As]. In GRPO, advantages are standardized as Equation When reward signals are fixed in set 0, 1, the maximum of |A,] within group of size G can be obtained when only 1 sample i rewarded with 1/0 with other G samples rewarded with 0/1 correspondingly. Formally:
R² E[
A max
(35)
std
R
1 1
c V
(36)
VG-1
Apply Equation
36 to Equation 23 can get:
E||A4| 1aypd VG vG- 1 c))
(37)
|||||| S 2. 22;7(z;0) VG-I. Var[A"
(38)
c²
|| (o-roo < (z: 0) c+ V- Var(A')
(39)
c²
where we try to choose optimal c:
d de
Var(A')
= 2VG Var(A')
c+ VG-I.
(40)
= 2VG Var(A1)
(41)
c' = 12 -I Var(A') 1/3
(42)
Thus
(oJGRro(0)) 3. (xx;0) (VG- (Var(A'))
(43)
B.3 PROOF OF LEMMA J
Proof of Lemmal ] 2.
We proceed by backward induction on N, starting from N = M down to N
Base case (N = .M: Take A' = Then Var(A')
1.
Inductive step: Assume for some n with S n + S M that there exists subset A/211 S A with | = n + and Var(A', 2 1. We will show that there exists a subset A', C A with |A, and Var(A() > by removing an element a E A/412 to form A', A/ \ {a).
Considering definition variance:
n Var(A() + (a =2+11)²= (n + 1) Var(A
(44)
We need Var(Ar) 2 thus:
Var(A' (a Mmx1)) 21 Thus we need to find an a E A,+1 that
+ 1 n
(45)
n
(a - (n+1)² )² < (n + 1) Var Var(A'nt1)
(46)
When Var Var(Ar (A/+1) = 1, it is easy to find an a makes (a 2n+1)² < (n+1) n Var(A
hold since Var(A( is the average of (a (0-mx+1)2 the distance of elements to average, over A/+11
When Var A (4 > 1, we assume there is no makes Equation 46 hold. Formally:
Va Ah (a 2241)² (n 1) Var A A
(47)
16

---

Published as a conference paper at ICLR 2026
Thus
Var(A > (n + +) Var(A - 1> Var(An+1)
(48)
(49)
which is conflict with Var( (4/+1) 2 1.
So assumption 47 does not hold, which means when Var( (1/+1)1)) 1:
3a A/+11 (a pa+1)² S (n + Var (ar(A))1)
(50)
So Equation 46 always holds when Var(A'2+1) 2 and Var(An) 2 1. By induction, Lemma holds for all N with 2 < N < M.
C TRAINING HYPER- PARAMETERS
Table 5 and 6 list hyper-parameters used in experiments.
Table 5: Generation configuration different base models according to their official release.
Model
Epoch
Temperature
Top-p
Learning rate
Qwen2.5-Math-7B
1.0
0,9
5e
Swwan.55-Math-
1.0
0.9
5e
Llama3. 1-B...--BBBiii -nstruutt
2
0.6
0.9
1e-8
OpenMath2-LLanma 1-BB
2
0.7
0.95
-7
Table 6: Hyper parameters used in finetuning.
Parameter
Description
Value
G
Group size of GRPO and GSPO
32
Clip thereshold of importance ratio
0.2
Ninit
Init size of D'S selected responses
8
Ninal
Final size of D selected responses
32
Kinit
Init ratio of D selected tokens
5%
Kinal
Final ratio of D selected tokens
20%
D DETAILED EXPERIMENT RESULTS
D. ANALYSTS ABOUT EXPERIMENT RESULTS OF LLAMA MODEL
In our ablation studies, a noteworthy phenomenon emerged when applying the D framework the purposs 1- model, which has been specifically aligned for mathematical reasoning. As detailed in Table 7 strategies based on intra- gutra-group sampling (D's and D²S-I) demonstrated markedly superior performance compared their pross-group counterparts S-C and S) Specifically, DIS-I, which omits the tross-group sampling component, achieved an average Pass@ I/Pass@ @8 score of 24. 24.0/40.2, surpassing the full D configuration.
provide a clear explanation for this discrepancy. The cross-group D'S strategy (dark blue line) induced extremely high- high-variance and unstable policy gra- dients, as shown in Figure 5a] accompanied by sharp increase in DRL. (Figure 5c). This suggests that for an unaligned model with highly heterogeneous output quality across different prompts, the global selection mechanism over-concentrates the learning signal on few outlier samples, leading to an unstable optimization process. contrast, D (purple line) maintained a policy gradient that was both high in magnitude and remarkably stable, with much milder policy distribution mi- gration. This indicates that for unaligned models, providing a stable, localized learning signal within each prompt's ss context is more effective than pursuing the globally maximal signal.
The training dynamics, depicted in Figure 5
This behavior changes when the framework is applied to OpenMath2- 1aam2---laa3.111 1-B.....BBBBBBB domain- aligned version of the same base model. The results in Table 7 show that the performance gap
17

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study by incrementally applying different part of strategies of D to Llama3.1-88 IB-Instruct and OpenMathh-Llamaal 1-BB. Performance across benchmarks measured in pass @ 1/pass@8.
AIME24
AIME25
AMC23
GSMSK
MATH
Minerva
Olympiad
Average
Llamas. Inst-usttuct
base
L7/1099
0.4/2.8
13.047..
57.7/928
29.3/55.9
14.7/3..5
2.278.2
17.3/36.6
GRPO
2.0/5.0
0.0/0.0
13.7/33.4
78.6/93.5
31.5/52.0
15.9/35.6
2.1/7.2
20.5/32.4
D'S
4,1/14,6
0.7/5.1
21.8/56.8
76.9/94,4
37,5/59.6
19.9/41.9
38/11.6
23.5/40.6
D'S-C
2.7/9.3
0.140.8
14.4/35.7
78.4/93.4
31.3/1.7
16.2/35.5
1.9/7.0
20.7/33.3
D
1.96.5
0.1/1.0
13.1/32.3
77,6/93.5
32.3/53.0
15.9/36.8
2.3/7.7
20.5/33.0
D'S
5.3/20.7
0,1.0.8
20.3/50.8
79.0/95.0
35.9/59.2
22.5/44.3
3.3/10.7
23.8/40.2
D
4.4/18.8
0,5/4.2
23.0/57.8
77.8/95.0
36,2/59.6
22.8/44.6
3.3/10.6
24.0/40.2
OpenMath2-L
1-8B
base
3.3/12.2
2.0/10.2
35.5/60.7
89.1/96.2
49 69.8/65.6
11.8/26.4
7.8/15.0
28.5/40.9
GRPO
5.8/17.8
2.0/10.0
35.8/59.9
85.7/95.6
50.7/65.3
10.0/22.9
7.4/14.0
28.2/40,
D'S
6,9/20.7
3.1/16.2
40.7/67.5
90.5/96.1
52.6/66.2
14.027.4
8.9/16.0
31.0/4.3
D'S-C
6.8/20
2.5/11.6
35.9/63.6
89/95.9
52.2/66.5
9,7/210
7.1/13.0
29.0/41.7
D 5.5/18.4
3/12.9
35,9/60
88.7/96
51.1/65.7
9.9/2188
7.4/14.1
28.8/41.3
D 5.6/18.6
2,0/9.0
89.4996.2
19.6/64.9
11.3/26.
8.1/1.2
28,7/41.3
D'S-T
6,7/19.7
3.V/1.55
39.1765.6
90.2/96.2
52.7/66.4
13.5/27.5
9.0/16.4
30.6/43.6
between intra-group and cross-group sampling strategies narrows considerably. While the intra- group variants D (31.0/44.3) and D' (30.6/43.6) remain among the top performers, highlight- ing their robustness, the cross-group methods also yield competitive results. This suggests that as the model becomes better aligned and its response quality more consistent, the risk of instability from cross-group sampling diminishes, allowing its benefits be more effectively realized.
This comparative analysis strongly indicates that the model's degree of alignment is a critical vari- able in determining the optimal sampling strategy. In this context, the D variant stands particularly robust framework. By combining the stability of intra-group sampling with the precision of token-level selection and dynamic scheduling, delivers excellent performance and stable train- ing dynamics across models with varying levels of initial capability, making it a more universally applicable solution.
b
C
GRPO
D
GRPO
D DĪ5
0.3
GRPO
D
Gradient Norm
D2S-I
D
D?5-I
D/5-1
Entropy
0.2
5
0.1
0-
0.0
0
Epoch
Epach
0
Epoch
Figure 5: Detailed training dynamics D strategies with Llama3. 1eanaa..1-B-BIA 1-8B-Instruct as base model, where S-I denotes D without cross-group down-sampling strategy. (a) gradient norms, (b) pol- icy entropy, and (c) DKT. restrain. Compared to the original GRPO, the integration of D signif- icantly increases norm of policy gradient. Since Llama3.1-8B, --S--nrttrutt model lacks pre-alignment, D'S-I provides better balance in exploitation and exploration through smoother policy gradient and milder policy distribution migration measured in DKL.
D.2 COMPARISON WITH PODS
To address the specific inquiry regarding the performance difference between D&S and PODS (Xu al. 2025), provide detailed comparison of their training dynamics.
Sample Usefulness Rate (SUR). As shown in Figure 6a the SUR of PODS remains around 70% throughout the training process, exhibiting pattern almost identical to the baseline GRPO. This indicates that PODS, which relies on intra-group selection, fails to utilize high-signal samples when they appear in "invalid groups" (groups where relative advantages near zero). In contrast, D achieves a near 100% SUR the early stages via cross-group group sampling, ensuring maximum data efficiency when the model is weak.
18

---

Published as a conference paper at ICLR 2026
Training Stability (DKL). Figure 6b illustrates the KL divergence dynamics. PODS exhibits significantly sharper rise in DKL compared to D`S, suggesting more aggressive and potentially unstable policy updates. D maintains more controlled D trajectory, demonstrating that our dynamic scheduling effectively balances acceleration with training stability.
Training Performance. Figure compares the Avg @.2 aceuracy the validation set. While PODS provides a marginal acceleration over GRPO, D demonstrates a significantly steeper learn- ing curve in the early stages and achieves higher final convergent performance. This validates our theoretical analysis that maximizing Advantage Variance (V (ar()) yields a superior optimization landscape compared maximizing Reward Variance (V (r(r)) used in PODS.
b
Sample Useful Ratio(%)
100
0.6
0.4
PODS
0.2
70
PODS
60
0.0
Epoch
Epoch
Figure 6: Training dynamics of a sample usefulness and GRPO and PODS.
KL divergence compared among D
GRPO
0.25
D
PODS
Avg@32 32
0.20
0.15
0.10
0.05
Epoch
1
Figure 7: The Avg@ test performance of AIME24 on Qwen2.5-Math-77 under various settings compared among DåS, GRPO and PODS.
19

---

Published as a conference paper at ICLR 2026
D.3 PASS @ 16 METRICS
Table 8: Performance across benchmarks measured in pass @ 16 calculated from 32 parallel runs.
Model
AIME24
AIME25
AMC23
GSMSK MATH
Minerva
Olympiad
Average
Qwwn22.5-Maid
base
42.8
19.4
82.4
92.2
71.1
42.7
18.2
52.7
GRPO
42.7
28.0
89.0
96.3
73.1
50.2
22.3
57.4
PODS
48.1
30.8
85.1
96.4
73.5
51.9
23.2
58.4
D
52.3
27.4
89.9
96.5
73.2
52.7
22.8
59.3
D'S-C
45.3
32.1
73.4
51.6
23.1
58.6
D
48.8
27.0
73.3
52.0
D'S
54.6
31,6
91.2
96,9
73,9
53.4
23.3
60.7
Llamaš.1 8-88 Instruct
base
17.9
4.6
59.8
95.3
62.0
45.1
11.0
42.2
GRPO
6.7
0.0
44.7
94.9
$7.2
41.2
9.7
36.3
PODS
14.7
5.0
94.9
44.5
39.7
D'S
18.5
8.8
47.4
D'S-C
13.8
1.7
47.4
56.8
40.5
37.8
D
9.4
2.0
42.2
95.1
58.1
43.0
10.2
37.1
D'S
28.4
1.7
59.9
96,3
64.7
49.9
13.6
44,9
OpenMarh22 Jama..
1-KB
basc
15.0
14.3
68.6
97.0
68.7
30.9
17.0
40.9
GRPO
22.5
13.7
27.2
44.2
PODS
22.6
17.7
67.9
96.7
68.3
25.8
45.0
D'S
26.9
24.7
74.1
96.8
69.0
31.3
18.2
48.7
D'S-C
24.2
15.7
70.2
96.8
69.6
24.5
14,6
45.1
D'S
25.9
18.1
67.6
96.7
68.8
25.8
15.9
45.5
D
24.0
13.2
65.3
96.9
68.1
31.3
17.0
45.1
20


Reviews:
[{"review_id": "V4Fh9SgARG", "reviewer": "Reviewer_5NNR", "summary": "This paper proposes a down-sampling approach for fine tuning LLMs with RL to speed up learning and convergence called D3S. The authors argue that using all the available interaction data can slow down learning because critical learning signals are overshadowed by many uninformative transitions. The proposal is to down-sample training data at two levels: (1) select a small set of trajectories to have large advantage variance, and (2) within each trajectory select transitions whose advantage magnitude multiplied by policy entropy is large. Finally, to avoid overfitting to the sub-selected data, the sub-sampling is decreased over training to promote large updates in early learning, while maintaining diversity and generalization toward the end of training.\n\nThe contributions are as follows,\n1. Proposing a down-sampling framework to improve training efficiency of critic-free policy optimization algorithms for fine tuning LLMs.\n2. Theoretical analysis to prove that maximizing the advantage variance increases the upper bound of the policy gradient norm, which can speed up learning.\n3. Experiments show improved sample efficiency, and better final performance on benchmarks compared with previous methods\n4. Ablation studies show different components of the algorithm contribute to its improved performance", "questions": "- The paper claims state of the art generalization for D3S. I don't think this claim is supported by the evidence. Maybe this claim needs further elaboration on what is meant by generalization in this case.\n- D3S also introduces some hyperparameters such as N_init, K_init, N_final, K_final. Are these values hard to choose or require additional resources to find good values for? Are they sensitive to the task?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I believe this paper is a good contribution to the RL and LLM literature and should be considered for acceptance. Down-sampling training data to improve training efficiency is not common in prior work and may be interesting to the community. Furthermore, maximizing advantage by sub-selecting trajectories is a novel idea and it is nicely investigated in this work.\n\n- The paper's overall quality is good. The writing and content organization is good. The exposition and motivation is clear. The paper mostly does a good job covering prior work and placing their contribution in the context of the broader literature. The proposed algorithm is supported by intuitive and theoretical justification and the experiments are accompanied by sufficient details and analysis.", "weaknesses": "- The paper may benefit from further discussion of prior work such as coresets for machine learning. For example: Mirzasoleiman, B., Bilmes, J.A., & Leskovec, J. (2019). Coresets for Data-efficient Training of Machine Learning Models. International Conference on Machine Learning.\n- The paper may also benefit from some discussion of prioritizing transitions in RL. For example: Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized Experience Replay. arXiv: Learning.\n\n- some typos in the abstract and introduction\n  - line 017 “efficient”\n  - line 019 “proven”\n  - line 078 “data pool is gradually expanded”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swsxY9mhto", "reviewer": "Reviewer_wBN3", "summary": "The paper focuses on enhancing the learning efficiency of current LLM RL algorithms by training on subsets of rollouts with high advantage variance and tokens that exhibit both high advantage value and high entropy. Meanwhile, the algorithm is equipped with a dynamic schedule that includes more rollouts and tokens to prevent overfitting. An empirical comparison with GRPO, GSPO, and PODS is provided on multiple math reasoning benchmarks.", "questions": "1. How is the sample usage rate compared to PODS?\n2. Is there any analysis on which tokens are masked off? Do they contain less information?\n3. Is there any evidence that, without a downsampling schedule, the model is overfitting? As shown in Figure 2(a), the sample usage is low again. Why can such zero-advantage samples help if they do not modify the gradient?\n4. The KL result where $D^3S$ is closer to the reference model is interesting. Why does it happen? Does it contribute to the performance of $D^3S$?\n5. Usually, entropy decreases during RL training, but why does Figure 4 show even increasing entropies?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The algorithm is intuitive and well-explained. \n\nThe paper presents a thorough ablation study, demonstrating the contribution of each component.", "weaknesses": "PODS already proposed to downsample rollouts. The advantage of $D^3S$ over the previous model is not immediately apparent.\n\nThe analysis of token masking and dynamic scheduling is not thorough enough to understand their effects. (Check the Question section)\n\nWith the additional cost of hyperparameters, the performance improvement from PODS is not obvious.\n\nThe paper can be better prepared, for example, by adding a comparison to PODS for analysis and including y-axis labels for Figures 3 and", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJny9DNSVa", "reviewer": "Reviewer_Bn5B", "summary": "This paper proposes a framework called Dynamic Dual-Level Down-Sampling ($D^{3}S$), aimed at making critic-free reinforcement learning algorithms such as GRPO more efficient. The core problem is that these methods are often slow to converge because critical learning signals are diluted by an abundance of uninformative samples and tokens.\nIts core contributions are:\n1) Theoretical Justification: It first theoretically proves that the upper bound of the policy gradient norm is positively correlated with the advantage variance ($Var(A)$) of the sampled subset. This justifies its strategy of selecting samples to maximize this advantage variance, rather than reward variance ($Var(R)$) as other methods do.\n\n2) Dual-Level Down-Sampling: The $D^{3}S$ framework operates at the sample-level and token-level to select only the most valuable data for updates.\n\n3) Dynamic Sampling Schedule: To prevent overfitting to this high-signal data, $D^{3}S$ employs a dynamic schedule inspired by curriculum learning. It starts with aggressive down-sampling (using fewer, high-signal samples) to accelerate early learning and then gradually relaxes this (using more data) to promote robust generalization.\n\nExperiments on models like Qwen2.5 and Llama3.1 show that $D^{3}S$ achieves state-of-the-art performance on reasoning benchmarks while using significantly fewer tokens and achieving major training speedups.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) This paper clearly defines the problem and presents a clear methodology, proposing a three-part strategy ($D^{3}S$) within critic-free frameworks like GRPO/GSPO to focus on the most informative samples and tokens from the source.\n2) This paper provides sufficient theoretical proof, presenting the upper bound of the GRPO gradient norm (Proposition 1). It further proves that when maximizing advantage variance on a subset, this upper bound increases with the advantage variance (Proposition 2), which forms the theoretical basis for the sample-level selection. Subsequently, the dual selection levels are integrated into a PPO-style objective.\n3) The experimental validation is thorough, conducted on models including Qwen2.5-Math-7B/1.5B and Llama3.1-8B, across a total of 7 benchmarks.", "weaknesses": "1) This paper introduces the Dynamic Down-Sampling Schedule, which in turn brings at least four key hyperparameters: $N_{init}$ (initial sample size), $N_{final}$ (final sample size), $K_{init}$ (initial token ratio), and $K_{final}$ (final token ratio). The paper demonstrates in Figure 3 that this dynamic schedule is crucial for preventing overfitting. This implies that the model's final performance may be highly dependent on the careful selection of these four new hyperparameters. However, the paper does not provide a sensitivity analysis for these hyperparameters. This could make the cost of reproduction and application to new tasks very high.\n\n2) In Figure 4, $D^{3}S$ lowers policy entropy in well-aligned models (Qwen, OpenMath2) but sharply increases policy entropy in the unaligned Llama model. The paper attributes both of these contradictory behaviors to the ability of $D^{3}S$ to effectively balance exploration and exploitation. Why does the exact same algorithmic mechanism lead to exploration in one model and exploitation in another? The underlying mechanism for this is not fully substantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SurbOZXYgV", "reviewer": "Reviewer_CB3f", "summary": "This study uncovers that in group-relative advantage-based RL algorithms (e.g., GRPO), the theoretical upper bound of the policy gradient norm scales with advantage variance. Building on this insight, the authors propose Dynamic Dual-level Down-Sampling (D3S), a novel training framework that improves efficiency and performance through two key mechanisms:\n\nSample-level: selects rollouts that maximize advantage variance to enrich signal diversity;\nToken-level: prioritizes tokens with high entropy × |advantage|, focusing updates on regions that are both uncertain and influential.\nTo prevent overfitting, D3S employs a curriculum-inspired dynamic down-sampling schedule, gradually easing selection criteria as training progresses. Experiments on Qwen2.5 and Llama3.1 demonstrate consistent gains over baselines across multiple reasoning benchmarks.", "questions": "Please refute the above weaknesses through discussion or experiment. I will adjust the score based on the author's feedback and the opinions of other reviewers.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The thesis is well-written\n(2) The theoretical proof is reasonable and supports the contribution of this paper\n(3) The proposed algorithm is relatively concise and has certain generalization potential\n(4) The experiment was conducted from multiple perspectives and achieved a good empirical analysis", "weaknesses": "Recently, there have been many works on trajectory screening based on entropy and advantage, and the methods are relatively homogeneous. Please distinguish the differences between this work and previous ones through detailed comparative experiments and discussions. \n\nIn my view, the amplitude of the advantage value represents the difficulty of the problem (like [3]), and the positive impact of high-entropy trajectories on the optimization of strategies has also been discussed in a large number of studies [1,2]. The combined influence of advantage and entropy was also discussed in the paper [4].\n\n[1]: Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\n[2]: The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models\n[3]: Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay\n[4]: RETHINKING ENTROPY INTERVENTIONS IN RLVR: AN ENTROPY CHANGE PERSPECTIVE", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)