Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
Authors: Xinyu Tang
Zhenduo Zhang
Yurou Liu
Xin Zhao
zujie wen
Zhiqiang Zhang
JUN ZHOU
Venue: ICLR 2026 Poster (2026)
Keywords: Data Efficiency
Reinforcement Learning with Verifiable Reward
Primary area: reinforcement learning
TL;DR: We propose a Data-Efficient Policy Optimization pipeline for Reinforcement Learning with Verifiable Reward that integrates optimized strategies for both offline and online data selection, namely DEPO.

Abstract: Recent advances in large language models (LLMs) have utilized reinforcement learning with verifiable rewards (RLVR) to improve reasoning capabilities.
However, scaling these methods typically requires massive data and extensive rollout computations, leading to high training costs and low data efficiency.
To mitigate this issue, we propose DEPO, a Data-Efficient Policy Optimization approach that combines optimized strategies for both offline and online data selection.
In the offline phase, we curate a high-quality subset of training data based on multiple objectives, including diversity, influence, and difficulty.
During online RLVR training, we propose a sample-level explorability metric to dynamically filter out samples with low exploration potential, thereby reducing substantial rollout computational costs.
Additionally, we employ a replay mechanism for under-explored samples to ensure sufficient training, which enhances the final convergence performance.
Experiments on five reasoning benchmarks show that DEPO consistently outperforms existing methods in both offline and online data selection scenarios.
Notably, using only 20% of the training data, our approach achieves a 1.85 $\times$ speed-up on AIME24 and a 1.66 $\times$ speed-up on AIME25 compared to GRPO trained on the full dataset.

Published as a conference paper at ICLR 2026
TOWARDS HIGH DATA EFFICIENCY IN REINFORCEMENT LEARNING WITH VERIFIABLE REWARD
Xinyu Tang Zhenduo Zhang" Yurou Liu Wayne Xin Zhao 1.3
Zujie Wen', Zhiqiang , Jun Zhou²
'Gaoling School of Artificial Intelligence, Renmin University of China PAt Group 'Beijing Key Laboratory of Research on Large Models and Intelligent Governance
ABSTRACT
Recent advances in large language models (LLMs) have utilized reinforcement learning with verifiable rewards (RLVR) improve reasoning capabilities. How- ever, scaling these methods typically requires massive data and extensive rollout computations, leading to high training costs and low data efficiency. To mitigate this issue, we propose DEPO, a Data-Efficient Policy Optimization approach that combines optimized strategies for both offline and online data selection. the offline phase, we curate high-quality subset of training data based multiple ob- jectives, including diversity, influence, and difficulty. During online RLVR training, we propose sample-level explorability metric to dynamically filter out samples with low exploration potential, thereby reducing substantial rollout computational costs. Additionally, employ replay mechanism for under-explored samples to ensure sufficient training, which enhances the final convergence performance. Experiments on five reasoning benchmarks show that DEPO consistently outper- forms existing methods both offiine and online data selection scenarios. Notably, using only 20% % of the training data, approach achieves 1.85 x speed-up A AIME24 and a 1.66 x speed-up on AIME25 compared to GRPO trained on the full dataset. The code is available at / com//UCAIBOxXODEOOOOO
x1.66 Time
Accuracy
Accuracy
GRPO (100%)
GRPD (100%)
DEPO (20%)
DEPO (20%)
1250 1ire x000
2ir
585
1093
2000
1002
1920
Training Time (min)
Training Time (min)
(a) AIME24
(b) AIME25
Figure 1: Training Accuracy GRPO and DEPO. DEPO uses only 20% of the training data and reduces training time by at least 1.6 x while achieving comparable performance to GRPO.
INTRODUCTION
Recently, reinforcement learning with verifiable rewards (RLVR) (Ouyang al. 2022 Shao et al. 2024: Tang et al., 20256) has emerged prominent technique to unlock the reasoning capabilities of large language models (LLMs) (Jaech et al., 2024 Zhao al., 2023a). In RLVR training, LLMs are required to explore multiple reasoning trajectories (i.e., rollouts), and receive binary rewards based on the correctness of the final answer. This process enables I.LMs to iteratively refine their reasoning strategies. A common way to enhance RLVR is to scale the amount of training data and the number rollouts, which allows LLMs to discover more diverse reasoning paths and improve
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
performance. Despite its effectiveness, this scaling strategy introduces significant drawbacks, that is, substantially increases training cost and results in low data efficiency.
To mitigate this problem, prior work has explored ways improve data efficiency through both offline and online data selection strategies. In offline settings, existing methods often rely on a single metric like reward trends (Li et al., 2025b), reward variance (Wang et al. 2025b), and gradient alignment (Li et al. (025a) to select data, which fails to fully capture the complex characteristics of the training data. Furthermore, these approaches often require prior training to compute the metries, which incurs high computational costs. On the other hand, online data selection methods aim to improve the rollout efficiency, which is the major bottleneck in RLVR. Zheng al. (2025b) employs probabilistic filter to exclude samples with historical zero reward variance. Although this method reduces rollout costs, treats all historical non-zero variance samples equally and lacks finer-grained metric to evaluate their exploration potential. Additionally, all existing methods enhance data efficiency from either the offline or the online perspective, resulting in suboptimal data efficiency.
In this work, we are the first to introduce Data-Efficient Policy Optimization approach that integrates optimized strategies for both offline and online data selection for RLVR, namely DEPO. During the offline phase, we apply a multi-objective high-quality data selection strategy. Specifically, to determinantal point
overcome the redundancy of training data, we propose a PageRank-weighhe process method to prune the dataset and preserve diverse and influential samples. Besides, to better align the difficulty of the dataset with the model's current capabilities, we perform offline rollouts on this pruned subset and select samples whose difficulty scores approximate normal distribution. In the online RLVR training process, we tackle the computational inefficiency of exploring low-potential samples by proposing a sample-level explorability metric. This metric dynamically quantifies a sample's exploration potential based on its historical training dynamics, which allows us strategically skip rollouts for wwweellorability samples and allocate computational resources to samples with higher exploration potential for rollouts and policy updates. Moreover, to ensure all samples are adequately trained, we employ dynamic replay for under-explored samples to further improve the final convergence performance.
To validate the effectiveness and efficiency of our approach, we conduct experiments on five reasoning benchmarks. Experimental results show that DEPO outperforms several competitive baselines in both offline and online data selection settings. In particular, when using only 20 % of the training data, DEPO achieves 1.85 times speed-up AIME24 and a 1.66 times speed-up on AIME25 with DeepSeek-FR -iieeckk-R--iiiie R1-Distill-O Sariill--ee---7- 7B compared to GRPO trained on the full dataset. Our main contributions are summarized as follows:
To the best of our knowledge, we are the first to integrate both offline and online data selection strategies enhance data efficiency in RLVR training.
In the offline phase, we employ a multi dineii-dmmesioonll data curation strategy based on diversity, influence, and difficulty. Then, during online training, we dynamically filter samples by their explorability and replay under-explored samples to further improve training efficiency.
Extensive experiments across five reasoning datasets and three LLMs demonstrate the effectiveness and efficiency of our proposed method under both offline and online data selection scenarios.
2 RELATED WORK
Our work is related Reinforcement Learning with Verifiable Reward (RLVR) and data efficiency. More detailed related work is presented in Appendix F
Reinforcement Learning with Verifiable Reward. Reinforcement learning with verifiable reward (RLVR) effectively improves reasoning in LLMs, particularly for mathematics and code genera- tion (Tang et al., 2025a Zhan et al., 2025 Chen et al., 2026). It uses simple verification to provide binary rewards, eliminating the need for learned reward models. DeepSeek-R1 (DeepSeck-Al et al.)
2025) first proposed the GRPO algorithm within this framework. Building on GRPO, subsequent
work have further advanced RLVR by refining various aspects, including loss functions (Liu et al.
2025a Yu et al.) 2025 Zheng et al., [2025;: Chen et al. (015a), token-level entropy (Wang et al.
2025a Hao et al. 2025). advantage estimation (Cheng al. 2025 Chen et al. 2025b), and hyperpa-
2

---

Published as a conference paper at ICLR 2026
(a) Training Pipeline of DEPO
Offline Selection
Online Data
Selection
Update Policy
A
Raw Data
Selected Data
Batch
Selected Ratch
Policy Model
(b) High-quality Offline Duta Curation
Explorability-guicc Online Rollout Pruning
Kaw Data
Final Selected Dato
History Accumecy
11M
Difficult--nuuee Distribution Sumpleg
Difficulty
RM
0.0
1
MO 0 0.2
0.5
6.8
6
Nember
Somple-Jesel Explorabilay
Mestarement
Emmeedded Sumples
...
Accuracy
Explorability
0.0
0.9
0.0
5.7
3,5
Repretentatiinnnnneeeeee
Peperank-neishee Determinnantel
Sample Graph Constraction
Point Process Pruning
Sampling
Probability
Difflesity
Rollout
Dynomic Replez of Under.
explored Sumpler
Sample
Asausey
Samples
Samples Rethined Pruning
Selected Samples
Policy Gradiien Updates
Figure 2: Overview of our approach DEPO. (a) Our approach improves the data efficiency in RLVR training via an end-to-end offline and online data selection strategy. (b) In the offline phase, we first construct sample graph based the representations, then apply BaggeRank-weiighhe Point Process to select a diverse and influential subset, and finally sample from this subset with difficulty following normal distribution. (c) In the online phase, we evaluate the explorability of each sample based on its historical training dynamics and retain high- eigghhrxllrrbbiity ones for rollout, and actively replay under-explored samples ensure sufficient training of all samples.
Determinantal
rameter (Liu al. 2025b An et al. 2025 Xi et al. 2025). In this work, we focus on improving the data efficiency of RLVR to reduce computational costs while maintaining performance.
Data Efficiency for RLVR. Improving data efficiency RLVR requires strategic selection of high- quality samples, incorporating offline and online data selection methods. Offline data selection methods focus on identifying a high-quality subset of data prior to training. Some studies select samples based model reward trends (Li et al., 2025b), reward variance (Wang al.) 2025b), and gradient alignment (Li et al. 2025a). While effective, these methods require training the dataset for several epochs for selection. On the other hand, online data selection methods dynamically filter samples during the training process. GRESO (Zheng et al., (025b) excludes samples with historical zero reward variance, but lacks finer-graie finer-grained distinction among non-zero variance samples. In this paper, we integrate optimized offline and online selection to improve data efficiency for RLVR.
METHODOLOGY
In this section, we present DEPO, a method to enhance data efficiency in reinforcement learning with verifiable reward (RLVR). We first define the problem formulation, then describe a two-stage data selection process: (a) offline curation based on diversity, influence, and difficulty, and (b) online rollout pruning guided by explorability. Finally, we discuss the effectiveness and efficiency of our approach. The overall framework of DEPO is illustrated in Figure 2
3.1 PROBLEM FORMULATION
In this work, we focus on large language model (LLM) parameterized by 0 E NR which has been pretrained large-scale corpora and will be further trained RLVR dataset D [x:

---

Published as a conference paper at ICLR 2026
to enhance its reasoning abilities. A key challenge in this process is the high computational cost associated with RLVR training, which typically requires extensive rollouts and policy updates over large datasets. To accelerate reasoning improvement, achieve comparable performance with less training data and fewer rollouts) by leveraging 'D more efficiently without modification augmentation. To achieve this, we first reduce data redundancy by selecting high-quality subset offline. Then, to further speed up RLVR training, we select samples with high exploration potential during training. The two stages are detailed below.
goal is to optimize data efficiency (i.e.,
3.2 MULTI-DIMENSIONAL OFFLINE DATA CURATION
In this part, we present an offline data curation method select high-quality subset of RLVR data based on three criteria: diversity, influence, and difficulty. First, we construct a sample graph based on the representations. Next, prune redundant samples by applying a PageRank-weighter Determinantal Point Process to retain a diverse and influential subset. Finally, we further refine this subset by selecting samples whose difficulty levels follow normal distribution.
3.2.1 DIVERSITY AND INFLUENCE-AWARE DATA SELECTION
Representatioo-augmeenee Sample Graph Construction. The first step of our approach to model the relationships among samples using graph. Previous studies (Hendel et al. 2023 Stolfo et al. 2025) demonstrate that internal model representations can effectively capture sample characteristics. Inspired by this, follow [iu al. (2044) and use the last token embedding from the final layer as the sample representation, as it aggregates the entire model information and input semantics. Based on these representations, we construct sample graph denotes a sample, an edge e(i,j) E E connects node i; and viy and edge weight matrix P encodes the pairwise similarities between their embeddings. This representation-uuse auggrieted graph is subsequently used for our offline data selection process.
(V, E, P), where each vertex U; V
Pagerank-weighted Determinantal Point Process Data Pruning. We prune redundant data from the dataset by considering two properties: diversity and influence. Diversity ensures broad information coverage, while influence reflects the importance of samples in the graph. To promote diversity, Determinantal Point Process (DPP) (Kulesza & Taskar 2012al to identify subset that maximizes the determinant of the corresponding similarity submatrix: maxy CP det(Y), where Y is the submatrix of the full similarity matrix P. and det(-) represents its determinant value. This refers to selecting samples that form a larger volume in the feature space, with a larger volume indicating greater diversity. To measure sample influence, we use PageRank (Brin & Page 1998) to assign a weight wo E w to each sample, reflecting its representativennss.. Our goal is to maximize the influence of the selected samples: m II wy. To unify both objectives, we combine diversity
iEY
and influence into single kernel matrix Ly and maximize its determinant:
de) II w/b 'EY
max det YCP
diag(w 123 Y (itagww//'
Ly
where Ly denotes the kernel matrix that integrates both diversity and influence of the selected subset. We provide detailed theoretical proof this optimization objective in Appendix
Since the optimization problem is Na-hard, we follow Kulesza & Taskar (2012b) and employ greedy algorithm to approximate the result. The algorithm iteratively selects samples based on a dynamically updated probability distribution and updates the probabilities of the remaining samples via Graa--Shhiie orthogonalization. The pscudo-code of this algorithm is presented Algorithm This process effectively reduces redundancy while retaining influential and diverse samples.
3.2.2 DIPHIULTY--WARE NORMAL DISTRIBUTION DATA SAMPLING
After pruning data via weighte DPP, we obtain representative and diverse subset Y. However, in RLVR training, samples that are too easy or too hard provide limited learning signals and contribute little policy optimization. To better align the difficulty of training data with the
4

---

Published as a conference paper at ICLR 2026
current model's capability, we propose a difficulty aware sampling strategy that prioritizes samples of moderate difficulty. Specifically, for each sample i in pruned subset Y, we generate G offline trajectories using the current policy model TT and compute its accuracy score Ace,:
Acc; = 1 [V(oj, a:)l
Here, { ~ no are G responses generated by Ty for question i, and V(oj. a:) denotes a verifier that evaluates whether model output 0j matches the ground-truth answer dz-
Using accuracy score as a measure of difficulty, we sample from according to a normal distribution Ni,, o²), where μ and are the mean and standard deviation of the accuracies in the final selected subset Dsub, Thus, the sampling probability for each sample is proportional to the standard normal density function, which assigns higher probabilities to samples near the mean difficulty:
A
Pi
where
((x)
1
2//2
ΣREY Φ
Acc
22T
The final RL training subset D is constructed by sampling from Y according to the probability distribution P = key, which is subsequently used for RLVR training.
3.3 EXPLORABILLTY-GUIDED ONLINE ROLLOUT PRUNING
In RLVR training, rollout generation is computationally expensive and becomes the main bottleneck for training speed. To further improve data efficiency, we propose an explorability metric to quantify the exploration potential of samples. Using this metric, we select high explorability samples for rollout generation and policy gradient updates and selectively skip unnecessary rollouts. Additionally, to avoid missing under-explores samples that may become valuable, implement dynamic replay mechanism to reuse them in the training process. The pseudo-code is presented in Algorithm 2
3.3.1 SAMPLE-LEVEL EXPLORABILITY MEASUREMENT
In RLVR training. high-entropy samples encourage exploration, and low-entropy samples may lead to overfitting. Leveraging this effect, we define the explorability of sample as the average entropy of its rollouts over a sliding window of recent epochs evaluate its exploration potential. However, although high-entropy samples generally promote exploration, certain negative samples with excessively high entropy (i.e. pathological trajectories) can introduce noise and destabilize training. To mitigate this, we apply threshold À to exclude them. Specifically, define the filtering function as:
(gg, a, of) = I ( a) = 1 (v(of.a) = 0A e(of) S )e(o
where I denotes the indicator function, (g, a) is the question-answer pair, and of is the i-th rollout in epoch The verification function V returns if of matches the ground truth a, and 0 otherwise. e(of) is the average entropy across tokens, e(o,+t) is the mean entropy of positive rollouts, and A serves as the threshold to filter high-entropy negative rollouts. We then compute the explorability of single rollout of by weighting its entropy by the absolute advantage |A, and the filtering indicator [gg, of):
Fl a, ) = A e(ot) T(g, a,)))
where A, mean( (r.) is the advantage. This explorability metric prioritizes samples of moder-
std (r (r.)=
ate difficulty, which tend maximize the sum of absolute advantages De= |Á,)- In contrast, samples with all rollouts either entirely correct or entirely incorrect yield zero advantage. Therefore, our explorability metric effectively captures both the exploration potential (entropy) and the difficulty (absolute advantage) of each sample. Finally, to evaluate a sample's 's exploration potential over time, aggregate the explorability scores across the group over sliding window of recent epochs to obtain the sample-level explorability E.
E(g, a, (0'
1
Σ
Σ
Elg,a, a, of),
(6)
G
1=c-841
where (0*) {of) are the rollouts in epoch I, and is the sliding window size of recent epochs.
5

---

Published as a conference paper at ICLR 2026
Table 1: Computational time on 4xRTX 3090 GPUs.
Component
Graph Construction
wagerran--eiighted DPP
Difficulty-aware Sampling
Time (h)
0.75 (x 8.33)
0.09 (x 1.00)
44.33 (x 492.56)
3.3.2 DYNAMIC REPLAY OF UNDER-EXPLORED EXPLORED SAMPLES
During training, samples with consistently low explorability may be overlooked, even if they become valuable in later stages. To mitigate this, we introduce a dynamic replay mechanism that proactively reintroduces explo-e samples. Specifically, each pruned batch Prruned consists of two types of samples: (I) the top % of samples from B ranked by their explorability E, and (2) the bottom p% of samples from ranked by their historical exploration frequency (i.e., those selected the least number of times up to the current epoch e) Thus, the optimization objective for DEPO is:
(0) Bg (..) mo (1g) 1/0.0, 10-))=---
1 G
Oi
min (rr,(0)A,,, clip 1
3DxL (Tol (Tol(Tee))
{
{
E(g, a, (0')i--0'=- is top-ae% [O' is bottom-p% else,
where a 10' 1i=--s11 =
(0) To(ou q, ct,
and 0. 0. d e.
(8)
(0i, 9, 01,=1))
To adaptively shift the training focus from broad exploration in the early stages to specialized refinement in later phases, we dynamically reduce the proportion of high-explorability samples per epoch using linear decay rate d. Here, B is the samples in training batch from the offline selected subset D ag denotes the initial sampling rate, d is the decay rate, and e is the current epoch. The online rollout pruning strategy optimizes computational resource allocation by focusing samples with high exploration potential and strategically replaying under-explored samples to ensure all samples sufficiently trained, significantly improving training efficiency.
3.4 DISCUSSION
Effectiveness of DEPO. DEPO enhances the data efficiency of RLVR via both offline and online data selection methods. During the offline phase, DEPO selects high-quality subsets by jointly optimizing three criteria: diversity, influence, and difficulty, which overcome the problem of prior methods that rely on single metrics such as reward trends (Li et al., 20256), reward variance (Wang et al., 2025b), and gradient aligmment (Li et al., 2025a). Besides, these metrics rely only early-stage training dynamics, which limits their effectiveness in later training phases. In the online phase, DEPO dynamically prioritizes samples based on their explorability, which is fine-grained measure of exploration potential. In contrast, GRESO (Zheng et al. (025b) probabilistically removes samples with historical zero-variance rewards, which treats all historical non-zero variance samples equally and underperforms when zero variance samples are scarce.
Efficiency DEPO. DEPO improves data efficiency by jointly optimizing offline and online data selection, thereby reducing both the amount of training data and the rollout numbers. In contrast, existing methods typically address data efficiency from only one perspective (i.e., either offline or online). In the offline phase, DEPO employs a two-step strategy to minimize computational overhead, which first prunes the dataset using a AngeRank-weighhe DPP, and then performs offline rollouts only this reduced subset.
This method is more efficient than performing rollouts over the full dataset (An et al., 2025). ) as the based filtering significantly reduces the dataset size, thereby lowering the cost of offline

---

Published as a conference paper at ICLR 2026
Mean Raspanse Length
DEPO
Rollout Numbers
Entropy
Rowardd
0.44
Training Staps
Training Steps
Training Steps
Training Staps
(a) Entropy
(b) Reward
(c) Response Length
(d) Rollout Numbers
Figure The RLVR training dynamics of GRPO and DEPO.
rollouts. As shown in Table | difficulty sampling accounts for a significantly larger portion of the total time compared to other components. Moreover, unlike offline methods that require multiple epochs of training on the original or warm-up up dataset to guide selection (Li et al. 2025b; Wang al. 2025bb Li al. 2025a), our approach avoids such additional training costs. In the online phase, DEPO dynamically estimates sample explorability based on historical training dynamics, which helps minimize rollout costs and improve training efficiency. contrast, some methods Yu et al. (2025) rely extensive rollouts before each update, incurring significant computational costs.
EXPERIMENTS
In this section, we first describe the experimental setup, then present the main results and provide detailed analysis.
4.1 EXPERIMENTAL SETUP
DeepSSe--RI-Disill-Llamm AI et al. 2025), Qwen2.5-Maan --tth--- 7B (Yang et al. 2024). We use DAPO-Math (Yu et al. 2025 as the training dataset and apply the GRPO algorithm trai the models. For evaluation benchmarks, use three mathematical reasoning benchmarks (i.e., AIME24, A AIME25, Math500 (Hendrycks et al., 2021 and two other reasoning benchmarks (iee, GPQA (Rein al. 2023) and LiveCodeBench (Jain et al. 2025) ). We follow Zhengg et al. (2025b) to evaluate models every 50 steps and report the performance that achieves the best average performance across five benchmarks. We repeat the test set 32 times for all benchmarks and report the average accuracy. More experimental details are provided in Appendix C To enable a systematic comparison, we include several representative offline and online data selection methods as baselines. For offline selection methods, we compare our method with random selection, supervised fine-tuning (SFT)-based methods (i.e., PPL.-Top (Laurençon et al. 2022) and PPI-Middle (Ankner et al. (025)), ) and RIVR selection methods (i.e., LIMR (Li et al. 2025b) and Learnalign (Li et al. (2025a)) For online selection methods, we integrate them into our offline selected subset and compare DEPO with random online selection and GRESO (Zheng et al.)
We run experiments on DeepSee--R--Disill--ueee
(DeepSeek
20256). Detailed descriptions of these baselines are provided in Appendix
4.2
MAIN RESULTS
In this section, we present the main experimental results. Figure iillsttrates the training dynamics of DEPO and GRPO, and Table presents the main results.
DEPO selects high-quality subsets offline for RLVR training, As shown in Table reinforcement learning on mathematical data not only enhances mathematical reasoning but also improves the performance on other reasoning tasks. Among offline data selection methods, although SFT-based data selection methods (i.e., PPL-Top and PPL-Middle) incorporate additional information from training data, their performance performs poorly. This may stem from the mismatch between SFT and RL objectives. SFT maximizes the likelihood target outputs, making perplexity a natural indicator of sample difficulty. In contrast, RL aims to maximize rewards, requiring samples match the model's current capability. Moreover, RLVR methods (i.e., LIMR and Learnalign), which perform training before selection, lead to further improvements. However, these approaches tend to select
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison of various data selection methods. "Offline" and "Online" refer to the offline and online data selection methods, respectively. "Ratio, "Time", and "RN" denote the ratio of selected data, total training time, and total rollout numbers, respectively. We highlight the best performance across different data selection methods. Numbers marked with * indicate that the improvement statistically significant compared with baselines (1-test with p-value < <.005).
Accuracy
Emiieecc
Model
Method
AIME 24
AIME
MATH500
GPQA
LiveCodeBench
Average
Ratio
Time
Base
51,5
38.5
91.2-11
45.9
37.0.a1
52.8110
Full
63.4
52,7:03
96,3-1
51.6-14
44.7-14
61,7-12
100%
100%
100%
Random
56.8
43.0am7
94,6.12
47.3.m
39.5.14
$6.3.1
20%
98%
100%
PPI-TTp
57.5-06
45.3-07
95.0-03
48-2:07
40.4-1
S7.3455
20%
101%
100%
Deepseck-R1-
Offline
Middi
57.5
45.4
95.2.u1
48.6.nu
40.8.a
57.5.36
20%
97%
100%
Distil-Quenn7I
LIMR
59.9tns
46,4
95.1=13
48.5-07
40.9-ns
58.2-39
20%
99%
100%
Leanslign
60.1s05
46.8x00
95.5=04
41.9:01
58.7
20%
DRPO-OMine
63.17:13
51.7"=L1
96.1'=03
f1.7
44.5
61.4"
20%
99%
100%
Random
58.7
45.3
93.1=05
47 47.2-113
39.3-a7
56.7=38
20%
58%
40%
Online
GRESO
60.2aa1
47.4
94.3
48.1
40.6:a19
58.1
20%
55%
40%
+ DEPO
62.8"
50.9"=1e
95.9"=14
5154
=4.3"=15
61.1"=44
20%
57%
40%
Base
41.1
304105
18.5
37.3
4.
48.3
Full
56.9
45.1
94,8-07
44.4:04
49.6-
58.2
100%
100%
100%
Random
47.6us4
38.7
90.5.11
39.6.nu
44.5.ns
52.2.s7
20%
100%
100%
PPL-TTpp
48.2-01
39.3.01
90
39 lin
45.0-1
52.4-39
20%
102%
100%
Deepseek-RI-
Offiine
Midee
49.9
39.2st.
91.1=09
39.4
45.7=a1
53.1=38
20%
102%
100%
DistillLLlama-
LIMR
52.3a11
40.9
91.621a
41.0:1
45,3-13
54.2s1h
20%
97%
100%
Leanaliinn
54.7a11
11.8
91. 11
40.8:11
46.2-01
55.0s11
20%
98%
100%
DEPO-Oflie
$7.6"
14.
94.2 .as
43.6*
49.3*...
$7.9"
20%
100%
100%
Randon
50.2
38.4
90.1=12
39.7:03
44.8:99
52.6e38
20%
55%
40%
Online
GRESO
$2.6ass
40.2
92.0cu
46.6.15
54.4aa7
20%
DEPO
$6.8 ao
44.4% 1.
93.7 as
48.8 .as
17.3"
20%
56%
40%
Base
13,4sa
6.4207
$4.5=11
28.7 28.7
5.6ans
21.7
Full
30.2a04
20.3:11
86.8.ns
35.7
13.6.01
37.3
100%:
100%
100%
Random
22.5
13.3et
72.5-n5
30.3:00
8.2e05
29.4226
20%
98%
100%
PPp
24.1a10
13.8
76.2:1.
31 1100mm
9.5uu
30.9:ax
20%
102%:
100%
Qwen2.5. 7B-Math
Offline
PPI Middle
24.8
14.3:11
76.0-ox
30.6-n7
9:9.n
$1,1-98
20%
98%
100%
LIMR
15.8
78.0.ns
32.2.11
10.6.a7
32.6.37
20%
101%
I Leamalign
27.1-1.0
17:2-11
80.5-05
33.6-111
10.9-a7
33.9-39
20%
98%
100%
DEPO-OMine
30.0"=a77
19.4
85.8"
35.2
13.2"=93
36.7"=444
20%
99%
100%
Random
24.3001
14.5:11
74.3.11
31.1.n1
9.3inx
30.7.38
20%
57%
40%
Online
GRESO
27.7-03
16.8.05
80.7=09
33.4
10.9-a4
33.9:33
20%
57%
40%
+ DEPO
29.8"=a.
19.2"=as
86.3
34.5*
12.8"
36.6"286
20%
55%
40%
samples that match the initial model capabilities, resulting in improvements during early stages but limiting performance in later phases. Additionally, they often overlook interdependencies among problems. In contrast, DEPO achieves the best performance across all methods, nearly matching the performance of training on the full dataset. One possible reason is that DEPO selects samples with diversity, influence, and appropriate difficulty, ensuring rapid improvement during the early stage. Besides, the diversity data difficulty supports sustained improvement in later stages. As illustrated in Figure 3 our method selects samples with higher initial entropy. lower initial rewards, and longer response lengths. This ensures that our selected data matches the model's current capability and offers diverse exploration paths for effective RLVR training.
DEPO saves training computational costs and maintains comparable performance. As shown in Table 2 randomly reducing rollouts severely degrades performance, which highlights the importance of careful online sample selection. Moreover, GRESO improves model performance by filtering out historical zero-variance samples that contribute little training. However, it does not account for the differences among historical non-zero- nan-zero-varianee samples, which limits its overall performance. In comparison, DEPO achieves performance comparable full rollouts while using less than 60% of the training time and 40% of the rollout budgets. This is because DEPO dynamically estimates sample explorability based on historical training dynamics, prioritizing highly explorable samples for rollouts and policy updates. Furthermore, ensure sufficient training across all data, which leads to better final convergence performance. As illustrated in Figure 3 our rollout pruning strategy consistently selects samples with higher entropy. faster reward growth, and longer responses throughout the training process, demonstrating that DEPO efficiently improves reasoning performance.
incorporate a replay strategy for under-explored samples to
8

---

Published as a conference paper at ICLR 2026
manmmlign DEPO
CRESO BEPO
Accuracy (%)
Accuracy (%9)
Accuracy (50)
Pacuune 1%0
Tandom JesmAlgr
RE10 Pd
Rancom 64550
BEPO
30
HARP
DAPO
Open-R1
HARP
DAPO
Open R1
Sampling Ratio (%)
Rollout Number Ratio (%)
(a) Offline Selection
(b) Online Sclection
(a) Offline Selection
(b) Online Selection
Figure Different RLVR datasets.
Figure Different sampling and rollout ratios.
62
Accuracy (%)
Accuracy (%)
Accuracy
Random
Stratified
0.25
Curticulum
p=03 0.5
c1 0.2
Ours
0.75
32
2000
52g
1203
Training
Training
Training
(a) Different Sampling Strategy
(b) Different Mean
(c) Different Standard Deviation
Figure 6: Performance comparison across data of different difficulty levels.
4.3 DETAILED ANALYSIS
In this part, we conduct a detailed analysis of our proposed method. Unless stated otherwise, we report the average accuracy across five benchmarks using Dear-Sekk-RR--iiil-------ee We provide additional detailed analysis in Appendix D
DEPO performs well using different RLVR training datasets. conduct experiments on three datasets of varying sizes (i.e., HARP Yue et al. 2024) (5k samples), DAPO (Yu et al.) 2025) (17k samples), and Open-R1 |ace 2025) (30k samples)). As shown in Figure 4 we observe that training on the DAPO dataset yields the best performance, indicating that higher data quality is more beneficial than larger data quantity in the RLVR training process. Moreover, DEPO outperforms competitive baselines in both offline and online settings across all datasets. These results confirm that DEPO is capable of improving data efficiency with different volumes of data.
DEPO performs well under different offline data sampling ratios. Figure 5a compares offline data selection methods under varying sampling ratios. As we can see, DEPO consistently outperforms Random and LearnAlign across all ratios. Notably, using only 20% of the data, DEPO matches the performance of full-dataset training. This indicates that our approach identifies high-value samples, allowing the model to efficiently improve its reasoning capabilities with limited data. Moreover, we observe that the performance of our method initially improves with increased sampling ratios, and it plateaus at around 20%. This also suggests that the dataset contains a substantial portion of redundant and low-value samples, which contribute little to the model's 's reasoning performance.
DEPO achieves the best performance across different rollout numbers. Figure 5b presents the performance online data selection methods under different rollout ratios. As can see, our proposed method consistently outperforms other online data selection baselines under different rollout budgets. This demonstrates that selectively performing rollouts on samples with high explorability significantly improves training efficiency without sacrificing performance. These results show the effectiveness of using explorability metric select samples in the RLVR process.
Normal distribution sampling aligns dataset difficulty with model capability. To evaluate the impact of different sampling strategies, we compare our approach with random, stratified sampling, and an easy-to-hard curriculum learning strategy. Random sampling preserves the original U-

---

Published as a conference paper at ICLR 2026
shaped difficulty distribution, with many samples being entirely correct incorrect. Stratified sampling selects equal proportions from each difficulty level, and the easy-to-harg tor-drddddddd--eo curriculum strategy progressively increases the difficulty of training samples over time. As shown in Figure 6a random sampling performs worst, followed by the easy-to-hard curriculum and stratified sampling, and our method achieves the best results. The easy-to-hard curriculum strategy leads slow initial progress because overly simple samples provide limited learning signals. Its performance improves noticeably in the middle phase when moderately difficult samples are introduced, but the model fails achieve higher final accuracy, as overly difficult samples also contribute little to learning. These results confirm that extremely easy or hard samples offer limited training signals, but DEPO prioritizes moderately difficult samples that are more beneficial.
Medium codiiumdifficullty samples accelerate learning, while the inclusion of challenging samples improves final convergence performance. To further analyze the effect of difficulty normal distribution sampling. we vary the mean and standard deviation of the sampling distribution in Figure [bb] and Figure [6c] Results indicate that relatively easier samples (i.e. μ 0.75) lead to lower convergence performance, while harder ones (i.e., μl = 0.25) improve final results but learn slower. Additionally, smaller standard deviation (i.e., 0.05) speeds up early learning but limits the final performance, whereas larger standard deviation (i.e., = 0.5) slows initial progress but achieves higher convergence. These findings confirm that medium-difficulty samples facilitate rapid improvement, and incorporating some challenging samples is essential for the final peak performance.
4.4 ABLATION STUDIES
To evaluate the effectiveness of each compo- nent in our method, we conduct ablation studies on three math benchmarks using DeepSeek-RR R1- Destill-Qeen785 As shown in Table remov- ing any component leads performance degra- dation, demonstrating all components are essen- tial. In offline selection, removing Difficulty- aware Sampling leads to the most significant drop, which indicates that sample difficulty is crucial factor for selection. For online selection, replacing explorability-bann based filtering with ran- dom filtering substantially reduces performance. indicator a sample's value in RLVR training. Furthermore, removing Under-cxplored Sample Replay notably impairs performance on more challenging tasks such as AIME 25 (performance drops from 50.9 to 48.4), suggesting that replaying challenging and under-trained samples is critical for enhancing the model's 's ability solve hard problems.
Table 3: Ablation study on three math benchmarks.
Dataset
AIME
DEPO
62.8
50.9
95,9
Offline Data Selection
w/o Peggrank--eiggtee DPP
62.1
50.0
95.6
w/o Difficulty aware Sampling
60.3
47.8
95.1
Online Selection
w/o Explorability Measurement
58.7
45.3
93.1
Absolute advantage wio Entropy wio Under- Sample Replay
61.9
60.6
48.4
94.6
62.3
48.4
95.2
This confirms that explorability is an important
CONCLUSION
In this paper, we proposed DEPO, a data-efficient policy optimization pipeline that integrates offline and online data selection strategies. By first constructing high-quality subset of training data that emphasizes diversity, influence. and appropriate difficulty, then dynamically filtering rollouts based on sample-level explorability, our approach significantly reduced both data volume and computational costs while maintaining strong performance. Extensive experiments across multiple reasoning benchmarks and LLMs demonstrate that DEPO consistently outperformed competitive baselines in both offline and online settings. We hope our work inspires future research toward developing more data-efficient methods to accelerate RL for LLMs.
ACKNOWLEDGMENTS
This work was partially supported by the National Natural Science Foundation of China No. 92470205 and Beijing Major Science and Technology Project under Contract No. Z251 10000442500. This work was also supported by Ant Group Research Fund. Xin Zhao is the corresponding author.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning advanced reasoning models, 2025. https:////kku github... io/
blog/2025/Polaris
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-basee -bseed data pruning with small reference models. In ICLR. OpenReview.net, 2025.
Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. Comput. Networks, 30(1-7):107-117 1998.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang. Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang. Chu Qiao, Chunhao Zhang. Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song. Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng, Ke Yang. Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang, Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi. Shitong Weng, Shuchang Guan, Shuqi Yu, Sichen Li. Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang, Yiqi Shi, Yiran Zhong, and Yonghong Duan. Minimax-ml: Scaling test-time compute efficiently with lightning attention. CoRR, abs/2506. 1 13585, 2025a.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass @k training for adaptively balancing exploration and exploitation of large reasoning models. CORR, abs/2508. 10751, 2025b. doi: 10.48500ARXXV22588005555 URL |https
//doi.org/19 0ttp//00.4555///
48550/ ariiv..arii.... 2508. 10751
Zhipeng Chen, Xiaobo Qin, Wayne Xin Zhao, Youbin Wu, and Ji-Rong Wen. Adaptive ability decomposing for unlocking large reasoning model effective reinforcement learning. arXiv preprint arXiv:2602.00759, 2026.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2250.114558 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilm via reinforcement learning. CoRR, 0..//2501.12455 2025.
Hugging Face Open rl: A fully open reproduction of deepseck-rl, January 2025. URL https //gi wtg.thhub.cmmmmmo// cot///ggingngaaacceeeee ccpe//e--r
11

---

Published as a conference paper at ICLR 2026
Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen. Rethinking entropy interventions in rlvr: An entropy change perspective, 2025. URL https::/aaxii...ootttt wttp:/iwww...rr///////ooooo wtw.g//ga../.......//g 10150
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In EMNLP (Findings), pp. 9318 9933. Association for Computational Linguistics, 2023.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS Datasets and Benchmarks, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Pas- s08, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Ku- mar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Ben- jamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong. Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song. Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao. Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Iige Akkaya. Openai ol system card. CoRR, abs/2412.167200 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In ICLR. OpenReview.net. 2025.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. CoRR, abs/1207.608$ 007776683,222 2012a.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Found. Trends Mach. Learn., 122-3):123-286, 2012b.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, pp. 61 1-166.6. ACM, 2023.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Sasko, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben Allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo. Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq. Suzana Ilic, Margaret Mitchell, Alexandra Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6tb composite multilingual dataset. In NewrlPS, 2022.
Shikun Li, Shipeng Li, Zhiqin Yang, Xinghua Zhang, Gaode Chen, Xiaobo Xia, Hengyu Liu, and Zhe Peng. Learnalign: Reasoning data selection for reinforcement learning in large language models based improved gradient alignment. CoRR, abs/2506.1 1480, 2025a.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. abs/2502. 11886, 2025b.
LIMR: less is more for RL scaling. CoRR,
12

---

Published as a conference paper at ICLR 2026
Yifan Li, Zhenghao Chen, Ziheng Wu, Kun Zhou, Ruipu Luo, Can Zhang, Zhentao He, Yufei Zhan, Wayne Xin Zhao, and Minghui Qiu. Unleashing perception-ting time scaling to multimodal reasoning models. CoRR, abs/2510.08964 2025c. doi: 10.48500AARXW2550000066 URL
https://doi http:///ooo.rg////// 405500/555a/ri 25100686166666
Yifan Li, Yukai Gu, Yinggian Min, Zikang Liu, Yifan Du, Kun Zhou, Min Yang. Wayne Xin Zhao, and Minghui Qiu. VIPER: process-aware evaluation for generative video reasoning. CoRR,
abs/2512.24952, 2025d. doi: 1044855/ARXX..2512222995 URL https:////do
arXiv 2512. 24952
Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. In ICML. OpenReview.net, 2024.
Yurou Liu, Mingyang Li, Xinyuan Zhu, Rui Jiao, Yiming Dong, Xinyu Tang, Yang Liu, Jieping Ye, Bing Su, and Zheng Wang. Drugtrail: Explainable drug discovery via structured reasoning and druggability-tailen taillore preference optimization. In The Fourteenth International Conference Learning Representations, 2026. URL https: httpss//ee/eeeeeeeieeeeee http:eet//orrum//////////y/yyn.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. CoRR, abs/2503.2078. 2025a.
Zihe Liu. Jiashun Liu, Yancheng He, Weixun Wang. Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong. Ju Huang, Jian Hu, Shengyi Huang, Siran Yang. Jiamang Wang, Wenbo Su, and Bo Zheng, Part Tricks or traps? deep dive into rl for Ilm reasoning, 2025b. URL httphttss///axxiv org// abs/2508. 08221
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311. 120222 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingehuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.033000, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In EuroSys, pp. 1279-1297. ACM, 2025.
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving cetructiio-followiinn in language models through activation steering. In ICLR. Open- Review.net, 2025.
Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. Unlocking general long chain-of-thouge reasoning capabilities of large language mod- els via representation engineering. In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Viennu, Austria, July 27 6849. Association for Computational Linguistics, 2025a. URL httpss//accanttont wrw.l/ogooorg//2200 /twgg//22..0222
August 1, 2025, pp. 6832-6849,
339/ Xinyu Tang, Yuliang Zhan, Zhixum Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou Rethinking sample polarity in reinforcement learning with verifiable rewards. CoRR, abs/2512.21622 2025b. doi: 1044550/RXXX.551222655, URL https!
//doi.org/1i 40l..88550/aaa ariiv 2 2512.21625
13

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, Abs/2500.019399 2025a.
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng. Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning large language models with one training example. CoRR, abs/2504.20571, 2025b.
Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Hua Wu, and Haifeng Wang. BEE-RAG: balanced entropy engineering for /ttps::ss:://o abs/2508.05 05100, 2025e. doi: 1044850/ARXIV2250..555 050000000000 2508.
generation. CoRR.
Yuhao Wang, Ruiyang Ren, Yucheng Wang, Wayne Xin Zhao, Jing Liu, Hua Wu, and Haifeng Wang. Reinforced informativeness optimization for long-form retrieval-augmennte generation. CoRR, abs/2505 20825. 2025d. doi: 1048550ARXIV25500.0000 20825. URL httphhtts:://o.. https://doi.org///// arXiv 2505.
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, Miao Zheng. Guoteng Wang. Shuo Zhang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, and Xuanjing Huang. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping, 2025. URL httph: org/ httg//b//25555 2510. 18927
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Inftythink: Breaking the length limits of long-context reasoning in large language models. In The Fourteenth International Conference on Learning Representations, 2026. URL https:// htwprrviee..net/ooummiiie.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409.12122 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system scale. CoRR, abs/25003.14476 2025.
Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K. Singh. HARP: A challenging human-annotated math reasoning benchmark. CoRR, abs/2412.088 2024.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofci Zhu, Wenyuan Xu, Jiaze Chen, Theng-Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu. Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. CoRR, abs/2504.051188 2025.
Yuliang Zhan, Xinyu Tang, Han Wan, Jian Li, Ji-Rong Wen, and Hao Sun. L2v-cot: Cross-modal transfer of thfin-of-thought reasoning via latent intervention. CoRR, abs/2511.11911 2025. doi: 10.48550/ARXIV.555 17910. URL https: //ooi org/r0 10. 48550/arXiv.. 2511.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji- Ji-Rong Wen. A survey of large language models. CoRR, Abs/2303.118223 2023a. doi: 10.48550/ ARXIV.2303.1 12.3.1882 18223. URI //doi. org/10. www../ 2303. 18223
14

---

Published as a conference paper at ICLR 2026
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright,, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow., (6(12)3388-3388 3860, 2023b.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu. Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. CoRR, abs/2507.18071, 2025a.
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. CoRR, abs/2506.02177/ 2025b.
15

---

Published as a conference paper at ICLR 2026
Algorithm 1 Pageraankweighhee Sequential Data Pruning(S, w, n.k))
Inputs:
Similarity matrix E R= Pagerank weighted vector w E I8" Total node n, Sample size S n
Initialize:
0. C.C (1, ................................ n). QAQ". A diag(A1. ..n........................... An). V Q diagg(X 1/2 A...., 1/2)
Keree Matrix L = diag( (w 1/ S..--Sgdaa''' (iag(w-//))..
for 1 to k do
p < [ for = to n do
Initialize zero vector: p E [R P. 0
if i E C then
p|1] < [[Vil
end if
end for
< p/ Σ, plj]
+ Sample(C, p)
Y < (úu
C < C) {ic)
ift < l then
u Vi
Select pivot vector
+- Ve
Vo
Gram-Schmide orthogonalization
end if end for return Y
Algorithm Explorabillty-gudded Rollout Pruning(B, 8, X, G, (o')
Inputs:
Raw batch (q:, a:) )= Window size s, Threshold filtering poor negative rollouts X, Replay ratio p.
Decay rate d, Current epoch number Rollout numbers per sample G, Rollout history {O*]e=
Initialize:
Pruned batch grruned
for each sample (q,, a, in B do
E, + E.(gg, a,, {0, (i=e-et1)
Calculate sample- level explorability using Equation ??
end for
Oe 00 -d
Calculate high explorahility ratios for this epoch
Sort B descending order by &
Eigg +- top [a. x [B]] samples from sorted B BRuplay sample [p [B] samples from B with the smallest [(0*) (f=l
Brrundd + Exigh U Replay
return Pruned
A PSEUDO CODE
In this section, present the detailed pseudo code for Pagerank-weighted sequential data pruning in Algorithm and explorability-guiee rollout pruning in Algorithm 2
B THEORETIC PROOF
In this part, we provide theoretical derivation the optimization objective aimed at maximizing the determinant of kernel matrix that incorporates both diversity and influence in Equation] [
B. OPTIMIZATION OBJECTIVE REFORMULATION
begin with the original weighted determinant maximization problem:
max
det(Sy)
II
(9)
YCP
KEY
16

---

Published as a conference paper at ICLR 2026
where Sy R denotes the similarity matrix over the subset Y, and w, represents the influential weight (i.e. the PageRank score) of each sample i. This objective aims to select a subset Y that is both diverse (as captured by det(Sy )) and influential (as promoted by the product of weights Teey ue)) To combine these two factors more naturally, we express the product of weights in matrix form. Note that:
II w; det(diag(wy)) 'EY
(10)
where diag(wy) (wy) is the diagonal matrix formed by the weights w, To incorporate the weights directly into the kernel matrix, consider the square root of the weights.
diag(w>? 2/2) = diag( (oog((W)))er
(11)
We then observe that:
max YCP
det(Sr) wi
II SEY
max det (diag(w3/??) Sy diag(w?/2)
(12)
YCP
This equality follows from the multiplicative property of the determinant and the fact that diag (w 1/2) is a diagonal matrix. Now, defining the weighted kernel matrix for the subset Y as:
Ly diag(w) w 1/3 SY diag(w,
(13)
we can rewrite the optimization problem as:
max det (Lx) YCP
(14)
This form corresponds to standard determinantal point process (DPP) with kernel L, where the joint effects of diversity and influence are captured by Ly.
B.2 THE DETERMINANT AS A MEASURE OF SAMPLE DIVERSITY
be a m samples, where each x; E Rd represents feature vector. Yhe vectors are normalized such that ||x;|| = for all i. The similarity matrix Y is defined as:
, Let Xg....: Xm
Yu x Xj
(X:; xz).
(15)
Under the normalization assumption, Y, for all i, and Yú, cos 0.j, where 0,y is the angle between x; and Xj-
The key insight connecting diversity to the determinant arises from the geometric interpretation of the similarity matrix. The mn-dimensional volume of the parallelotope spanned by the vectors XI, Xm is given by:
Volm .,,, xm) = de)(Y).
(16)
Equivalently.
det((Y) = =(Vol (Xi,..., xm))
(17)
This can be derived by letting X be the x m matrix whose columns are X,,..., Xmp- The volume is inherently /det(X
We now state and prove the main theorem connecting the determinant diversity. Let S span ,x,,..,, x;-1) denote the subspace spanned by the first & vectors, and let Φv be the angle between X₁ and its orthogonal projection onto St-1. The volume of the parallelotope can be
expressed recursively as:
m
m
m II
Volm (X)
Xm)
xx
sin de
sin de.
(18)
where the second equality follows from the normalization ||x quantifies the orthogonal component of x relative to the previous vectors:
1. Each factor sin ds E [0.1]
17

---

Published as a conference paper at ICLR 2026
sin Øy 1 if and only if x;; is orthogonal Sk-1 (maximal orthogonal component)
sin Øj: = 0 if and only X3: E Sk-1 (linear dependence, zero orthogonal component)
Since dY) Vol - sin 2 Φk. we observe:
det(Y)) approaches its maximum value when all sin Φx + (i.e., when each vector is orthogonal to the span of previous vectors, yielding pairwise orthogonality).
det(Y)) approaches zero when any sin dg + 0, indicating linear dependence and minimal diversity.
Thus, det(Y) strictly increases as the vectors become more orthogonal.
Since Y is real symmetric matrix and represents inner products of normalized vectors, it is positive semidefinite. The diagonal entries are all 1. By the Hadamard inequality for positive semidefinite matrices, the determinant is bounded by the product of the diagonal entries:
det (Y) < Y =1
(19)
This establishes the upper bound. Now, we prove the equality condition. Hadamard's 's inequality holds with equality if and only if Y is diagonal matrix. A diagonal Y with Y,, 1 means:
Y = Im. where Im is the identity matrix.
(20)
This implies Y, for all i + j. Since Yú, = (x, x2), this means x; 1 x; for all i + The vectors mutually orthogonal. From geometric perspective, when the vectors orthogonal, the parallelotope they span forms an m-dimensional hyperrectangle and achieves its maximum volume.
C DETAILED EXPERIMENTAL SETUP
Models. We run our experiments on DeepSeek -iikk-RR--- RI-Distill- Distill-Qwenn7B (DeepSeek-AL et al. (025). DeepSeek-Re R 1-Distill-LLlama-88 (DeepSeek-Al et al. (025), and Qwen2.5- (Yang et al. 2024). For DeepSeek-FP -I---Riiillll-eeeeeee length to 16384. For Qwen2.5-Math-7B models, we set the context length to 4096, as it is the maximum context length for this model.
and Deepssek-RR--isstll--ne
we set the context
Training. Our method is implemented based the Verl (Sheng al. 2025 pipeline and uses vLLM (Kwon et al., 2023) for rollout. We train DeepSeek-R ---iiiii -weesill--ee--8 7B and DeepSeek- RI-Distill-Llam--8 8B on 64xH200 GPUs, and Owen2.5-Math-77 on 32xH200 GPUs. For training datasets, we use the DAPO-Math Yu et al. 2025) as the training dataset. During rollouts, we set the temperature to and sample responses per prompt. The training batch size is set to 256. We apply the on-policy GRPO algorithm to train the model. Similar to Yuue et al. (2025), we remove both the KL divergence loss and the entropy loss. We train models for 1000 steps, and we optimize the actor model using the AdamW (Loshchilov & Hutter 2019) optimizer with constant learning rate of 2e-6 for DeepSeek-RB DeepSeek-RI-Distil--00 -i-iiitll--Qeenn- and Deepseek- RI---R--D-s L----itill-LLlama--- Owen-Math-7BB The actor module is optimized using Fully Sharded Data Parallel (FSDP) (Zhao ct al. (023b) for efficient distributed training. The chat template we use is "User: \n [question] \n Please reason step by step, and put your final answer within \boxed(]. ). \n An Assistant:" For offline data selection, we first apply PageRank-weighter Determinantal Point Process (DPP) to reduce the sample set to 50% of its original size. We then perform difficulty-aware sampling based on a normal distribution to select the final subset, which constitutes 20% of the full dataset. The mean and standard deviation o of the difficulty distribution for the final selected subset are set 0.5 and 0.2, respectively. In the online data selection phase, we set the window size for recent epochs to 5, the replay sample ratio p to 0.05, and the threshold for filtering out poor negative rollouts to 1.5. For the linear decay of rollout pruning, we initialize the sampling rate ogg to and apply a decay rate d of 0.05, gradually reducing the proportion of samples used for rollout and policy update until it reaches 20% of the batch size.
and le-6 for
Evaluation. For evaluation benchmarks, we use three widely used complex mathematical reasoning benchmarks (i.e., AIME24, AIME25, Math500 (Hendryeks al. 202T)) and two other reasoning
18

---

Published as a conference paper at ICLR 2026
Rollout Number (x100)
Rollout Number (x10s)
Accuracy (%)
Accuracy (%)
Accuracy
Accuracy
53
60
50
53
0.1
6.69 6.55
10
Decay Rate
Replay Ratio
Threshold
Window Size
(a) Decay Rate
(b) Replay Ratio
(c) Threshold
(d) Window Size
Figure Hyperparameter analysis
benchmarks (i.e., GPQA (Rein al. 2023 and LiveCodeBench (Jain al. 2025)) to evaluate the model performance. We follow /Zheng et al. (025b); Yan et al. (2026) to evaluate models on those benchmarks every 50 steps and report the performance the checkpoint that obtains the best average performance on five benchmarks. All evaluations are conducted in zero-shot setting. Following DeepSeek-Al al. (2025); Wang et (2025c); Li et al. (022dd) evaluate all models setting lemperature to 0.6 and top-k to 0.95. We repeat the test set 32 times for evaluation stability for all benchmarks and report the average accuracy.
D ADDITIONAL EXPERIMENTS
D.1 THE EFFECT DIFFERENT SCALE MODELS
To evaluate the scaling ability of our proposed method, we conduct experiments on Qwen3-8B, Qwen3-14B, and Qwen3-32B using the DAPO algorithm. The training was conducted with batch size of 512 and mini-batch size of 32, resulting 16 gradient steps per training batch. We report average performance across three mathematical reasoning benchmarks (i.e., AIME24, AIME25, and Math500). As illus- trated in Figure 7 we observe that our approach demonstrates strong scalability across different model scales. This improvement can be at- tributed the offline data selection strategy of DEPO, which selects high-quality subset based diversity, influence, and difficulty, which is beneficial for RLVR training. Furthermore, we select high explorability samples for rollouts and policy updates, and incorporate under-explored samples for replay, which significantly improves training efficiency without sacrificing performance.
Sandam
DEFO
Accura (60
Accuracy (%6
Offline Sclection
(b) Online Selection
Figure 7: Different scales LLMs.
D.2 HYPERPARAMETER ANALYSIS
In this work, adopt a linear decay strategy gradually decrease the proportion of rollouts in each batch over training epochs. As shown in Figure Sal setting the decay rate d to 0,05 leads to slight performance drop while substantially reducing rollout numbers. the decay rate is too high, many samples may not be sufficiently trained, leading to suboptimal final performance. Conversely, an excessively low decay rate increases the number of rollouts, thereby reducing training efficiency. With respect to the replay ratio, setting it to 0.05 allows the model to achieve an optimal balance between final performance and training efficiency, as illustrated Figure 8b) An excessively high replay ratio introduces unnecessary computational overhead due to redundant rollouts, while ratio that is too low may prevent challenging samples from being adequately trained, thereby limiting the model's 's reasoning capability.
We further analyze two hyperparameters: the threshold for selecting high-quality negative rollouts and the window size As shown in Figure performance initially improves and then declines
19

---

Published as a conference paper at ICLR 2026
the threshold increases. This indicates that when the threshold is set too low, potentially useful samples that could help exploration may be excluded. Conversely, an excessively high threshold may lead to the inclusion of noisy rollouts (e.g., nonsensical text rollouts), which model performance. Regarding the window size s, Figure 8d indicates that the model performs best when 8 = 5. This suggests that the window size should be chosen within an appropriate range. One possible explanation is that a very small window may not capture broader historical training dynamics, while an overly large window may not focus on recent training trends.
adversely affect
D.3 THE EFFECT OF DYNAMIC OFFLINE DATA CURATION STRATEGY
To further investigate the effectiveness of dynamic offline curation strategies, we conduct additional experiments be- yond our static difficulty-based selection with approaches that periodically update the training subset. Specifically, we implement two dynamic strategies: (1) Dividing training into two (i.e., 10% data per phase, resample every 1250 minutes) and four phases (i.e., 5% data per phase, resample every 625 minutes). (2) First training on a static 20% subset until convergence, then continuing with newly resampled 10% subset.
(%)
Resample Times
Resample Times
No Resemple (Durs)
Resample 10%
Training Time
Figure D compares the performance of these dynamic strate- gies against our static baseline. As we can see, updating the Figure 9: Dynamic Offline Curation. subset with only 5% data per phase results in performance degradation, as the model tends to overfit the current phase's 's data distribution and forget previously learned data. Besides, the two-phase ap- proach strategy with 10% data per phase performs comparably to our static 20% baseline. This reveals a critical trade-off in dynamic curation between maintaining alignment with difficulty-macceed data distribution and preventing catastrophic forgetring.
Furthermore, when extending training with new 10% subset after initial convergence on 20% data, we observe only slight improvements, indicating that data freshness through resampling is effective. However, the limited improvement suggests that the DAPO dataset is nearly saturated for this model's performance potential.
E DETAILED DESCRIPTION OF BASELINES
In this part, we provide detailed de- scriptions of all the baselines used in our experiments. For offline data selection methods, we com- pare our method with random se- lection, conventional supervised fine- tuning (SFT) data selection meth- ods (i.e., PPL-Top (Laurençon al. 2022) and PPL-Middle (Ankner et al. 2025 ) RLVR selection method (i.e., LIMR (Li al. 2025b) and Learnalign (Li al. 2025a
Table 4: Comparison of RLVR data selection methods.
Method
Offiiee
Offiine Method
Online
Online Method
Reward
One RLVR
Reward variance
Leurnalign
Gradient aligument
Polaris
Difficulty
GRESO
X
History reward variance
DEPO (Ours)
Multi dimension
Explorability guideed
Table presents a comparison of our method with others.
Random: Randomly samples data from the training set.
PPL.TTpp (Laurençon et al.
y022): Selects the data with the highest perplexity.
PPL-Middle (Ankner al.
2025 Selects the data with the middle perplexity.
LIMR (Li et al.
2025b): Selects the data whose learning patterns complement the model's overall
reward trajectory.
Learnalign (Li et al. 2025a): Selects the data based on representativeness (measured via gradients during warmup training) and difficulty (determined by rollout accuracy).
For online data selection methods, we incorporate them into our offline selected subset and compare against random online selection and GRESO (Zheng et al. 2025b).
20

---

Published as a conference paper at ICLR 2026
Random: Randomly filters 40% of the data at each batch prior to rollout during training.
GRESO (Zheng et al. 2025by: Probabilistically filter historical samples with zero variance at cach batch before rollout during training.
F
RELATED WORK
F.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD
Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a promising paradigm for enhancing the complex reasoning capabilities of large language models (LLMs), particularly in domains such as mathematics and code generation. The key advantage of this approach is its reward design, which relies solely on simple verification functions provide binary rewards without requiring learned reward models. DeepSeek-RI (DeepSeek-Al et al. 2025) introduces the GRPO algorithm under the RLVR framework and demonstrates its effectiveness in significantly scaling the reasoning abilities of LLMs. Building on GRPO, subsequent work have further ad advanced RLVR by refining various aspects, including loss functions (Liu al.. [225a Yu al. 2025 Zheng et al.) 2025a Chen et al. 2025a), token-level entropy (Wang et al. 2025a Hao et al., 2022). advantage
estimation (Cheng et al. 2025 Wang al. [025d] Li al. 2025c), and hyperparameter (Liu et al.)
2025b; An et al. 2025 Xi et al. 2025 Liu et al. 2026). In this work, focus on improving the data
efficiency of RLVR to reduce computational costs while maintaining model performance.
F.2 DATA EFFICIENCY FOR RLVR
Data efficiency aims to enhance model performance by strategically selecting high-quality training samples. Existing RLVR data selection approaches can be broadly categorized into offline and online strategies. Offline data selection methods focus identifying high-quality subset of data prior to training. Some studies select samples based on model reward trends (Li et al.) (022b)), reward variance (Wang al. 2025b), and gradient alignment (Li et al.] (2025)). While effective, these methods require training the original or warmup dataset for several epochs for selection. Another line of work (An et al.] (025)) uses offline rollouts estimate sample difficulty relative to the current model's 's capabilities. However, this approach requires computationally expensive rollouts across the entire dataset and fails to account for the diversity and influence of the training subsets. On the other hand, online data selection methods aim to reduce training overhead by dynamically filtering samples during the training process. These approaches target samples that contribute little to improving reasoning capabilities yet require costly rollouts. GRESO (Zheng et al. 2025b) employs probabilistic filtering strategy to exclude samples with historical zero variance. Although it improves efficiency, this approach only focuses among other samples. In this work, we enhance the data efficiency of RLVR by integrating optimized strategies for both offline and online data selection.
historical zero- zero-variance samples and does not distinguish
G
CASE STUDY
In this section, we provide detailed case studies five reasoning benchmarks.
21

---

Published as a conference paper at ICLR 2026
case study from AIME24
[Question:]
Every morning Aya goes for 595-kilameter long walk and stops at coffee shop afterwards Wher she walks constant speed of 5s5 kilometers per hour, the walk takes her hours including 515 walks spent the coffee shop. miautes spent hours and misutes. including [H[2JS kilometers per hour. Find the namber of minutes the walk takes her, including the StS minutes spent in the coffee shop. Aalease resson by step, and your final answer within 1 b[xed[[])-
kilometers per coffee shop. Suppose Aya takes 4aalerrae
[Answer:]
Okay,
figure this out step by step Anissi let parse the information Vatal. Every morming. . Aya goes for 9-Kilomeier-Ine walk and then stops
problem about Aya's morning
coffee stops.
coffee shop An2 When she walks at constunt speed of 's takes her 4 hours. Wait, hold mentions including
kilometers per hour, the entire walk, including her coffee problem says minutes spent in the coffee shop. 1mm, maybe need to clarify AntnWait, maybe the total time includes both walking and coffee time. So, when she walks at speed s km/h, her tatal time is 4 hours. when walks includes minutes
hours.
hours 240 minutes. Bat
which includes time takes her when she walls at 1/2 km/h, again including the minutes coffee stop.
the coffee shop Anknnimilarly
2 km/h. the total AnknOur geal
minates.
e Final Answereata boxed 224 zinlaCiven that Aya goes for a 9-kilometer -long walk every coffee shop. we need to determine the total time she lakes when she walks at AnkaFirst.
murning and stops Wiracc (1H21 W km/h. including the
is the coffes given information stannl. When walking at speed VC W) km/bsla Total time = hours = 240 minutes An Let W W) be the time spent in the coffee shop Equation: Wr Wirac [540]fs]
speed of Wi
up equations
240 W An'a2. When walking at speed We
Equation: W (54c W. km/h:in
Total time
144 W. Wtrac [540][s] fruc (5401ls + 2) 96 WhnM 540 Wleeft Wifrac [1H(s] - V.frac (1lls + 21 Wright] 96 WhaW 540
hours 24 minutes 144 minutes.le
atnSsbtracting
second equation
elsminate
W)
Wleett V (2 + 231 W Wright) = 96 W[JaMlf W.rre (efraa((0s0]]]]
WHlD [1050][s(s + 2)1 = WJKM 1 frac
(1080]196)
WANMM 11.25
11.25
WilhatnSavving the Wirrc Upm 7H(2)
guadratic equation Discriminant:
45
Solutions
W) 'n- Posilive solution: VC x 2.5 W) km/h am/h.Annnnssinn W. s 2.5 W) to find WC :t=- From WE i (r (540112.5) frac (11(2) 3 W) km/h:kn- Time walk 9 km: WE W.rra = W) hours
240 W Aa WE 216 240 Wn WW t. 24 11) minutes. inlnween
walking W
minutes. An- Toral including coffee: 180
11) minutes Aainnhus. the total time
taken is WCW boxed (204]W) IW) minutes.
A case study from AIME25
[Question:]
Find the sum integer bases Sbo9s which by step . and put your final answer within booeed
$9n_b.StnPlease reason step
[Answer:]
Atright. "e see. have this problem here: need to find the of all isteger bases W b okay. So, bases greater remember that when numbers
for which 17_b () is divisor of Wi 97_b means we'r re looking at bases starting from upwards. right? intnFirst. are written in a certain hase VK b W) their value in decimal (bose 10) ean be eatculated. For example, number like Wy 17.b W) is equal to WC Wrimes b 7 Wtimes I = Similarly. Wi 97_b problem is asking for that when we convert both numbers to base 10, WE 97_6 W. should be divisible by WE 17_b W) in base 10 terms.
which
() in base 10. 10. tainSo,
would W Wtimes
Wtimes
9b W) ia
W) where WE 17_b W. divides We 97.b W. That meaos
Finzl Answers conversetalln hoxed 70 sin'aTo solve the problem, we need to find all intager bases W)) which Vi 17_b
divisor 10:10- WC 1 Wrimes 7
97_b VninFirst.
numbers Wtimes
from WC W)
te W T 9b 7 Wi Antawe need W b + 7. W1 to divide Wi 9b + W). This translates to the congruence let WE
Antil 9b 7 Weguiy Wpmoddb 7) WiKaannewwitiing this
W. W b 7) Weguiv Wpmodin] WJNMW
Wequiv (immadin) WHall 9a 56 Weguiv Wpmod]a] Wpal -56 Wequiv Wpmod]a] WJ aThis implics WC W. is a divisor of 56. Since WM b. 11) B1 = b- > 16 15.. The divisors of 56 greater
Suberituting back,
gersAnii
than 16 28 and IntsThss solving for We h 90 stn- 1f WC 28 15). then W. b 28 7 56
W b satisfy digit constraints their respective numbers. Therefore, the som of these bases AW 21 49 70 WiB a' boxed[70]any)
Nataoothh 21
valid bases they
22

---

Published as a conference paper at ICLR 2026
case study from MATH500
[Question:]
1MM MMamm_ 1jaWin (1fraf11]]ag Vquadd Weext [and] Viuuad Ntsum_K 11 infty Witrae (1 (1) Find way to Wo = 1jnWintty Wsum_[k 1j-Wintty fra [T] =3433jjja
SinPlease step step, pur your final
within Woxxed
[Answer:]
Okay. so have this problem here where need to express the double sam W0 15 1/WWinryy " sum_th 1J-Winfty W [IHH k)433)0 terms of W(pp)) and W(gh), where Wip Wsam (k Wsum Ik Wjaiinfty remember that WPl) is related to the Basel problem and equals W(l frac (1p1)216611) and W(gH)) is Apery 's constant, the double approximately 1,202... But maybe I don In need their exact values just to WGJ()
6339)
terms of W(Pl) and Will) AntaSo, the double sum is two indices. WG/G k)43)).
WKkW)) clever way to rewrite this ur mayhe relate infinity to the known sertes W(pW)) and W( Let think about how to handle double sams. Often, double sums can sometimes be converted into single sums by using integral representations AntaLet write down the doable explicitly clever manipulation (j=lj*iinfty Hsom [aum_ik=] [kel]"Winfly Wreae (I]( 12A3/t/lllaaaa
starting
such sums, sometimes we can change variables to simplity the expression. Maybe ser We
kW)..
For each W(ll)). W(]W) can range from Win 1W so we rewrite the double sum as single sum num:kn/n:1//e Msum MMsmmij=ll [jinitt [ssm_[k=]] [k=l]"Winfly Wfrac [I](( ky*3 Noum [n=] (n=2]"iiffy [smm_jj=)][aa
Wrae (1][n*3ja Waaaaaoo each Wall). there Win 190
WMG. k)1)) Anln Woum (m=2) (ttgghteaalBBaaallllll Biaahis can be written as as:inialiiaa u M (1)nn() N
soctatn =nsum__n=22 Wisiityy
(fracel) Nia split
mNMD frac(n [][a3] [3 = = (affrellllng))) W.rrac (a
infly Weett Wfrrac [1][=2] (ffra(11]
left( Waum [m=2]*Minfty Wfra (1) Wright) Wleff( (n=2__n==]"
series Wip [asania-1])Waatty W-lltWinfty Wfaac (3H(0*3)1), we
Wig Waom Wfrae (1]fa22) =p Miuuad textland] Wquad [ssm_[n2]] [n=ni 1 Wfraat [1]a* = M Aninti[AnS (p -)
giatt/aala final
Woxeed 9)
23

---

Published as a conference paper at ICLR 2026
case study from GPQA
[Question:]
Answer the following muttiple choice question. The last line of your respoase should be of the following format: 'Answer: SLETER' (Without quates) where LETTER is one of AKD. Think step by step before answering An'uTwa quamtum
with energies El want to clearly distingnish these two energy levels. Which one of the
have lifetime
and 105-9
10--8 see. respectively following options could be their energy difference so that they can be clearly resolved NakaataA) 10--9 eVinR) 10s-11 eVinC) 10s-8 eViataD) 10--4 eV
[Answer:]
Okay, so have this quantum mechanics problem is frost of me. Hmm. let's 's try te think through. The question is about distingaishing two quantum states with different lifetimes. Right, remember that in quantom mechanics, the uncertainty principle plays role is determining the observable wait, perhaps it
differences beiween energy levels. Or maybe it 's related to the line width due to the energy ancertainty affecling the transitiun between states Aninso the problem states that two quantum states have energies El and B2 with
Heeiseaber uncertainty principle?
lifetimes of le-9 seconds and le-3 seconds. respectively. The goal options
difference find which D'inall 'm
would
clearly distinguish key here is the concept the spectral line width. Oh wait
energy levels.
resty on this, but think remember something about the width of spectral lines being related to the unceriainty in energy... or perhaps the Heisenberg uncertainty principle. Because if the lifelimes differences might lead differeat spectral principle says that the product of the uncertainty in energy ('delta B) and the uncertainty in time (edeltal) is approximately "22.AnnnAleeeaative line (due 'deliaf is approsimately deltaE would be on the order of ta Aninão if we have two states with lifetimes Haul and Mauu2 their respective spectral linewidths would be IdeltaEl = h Viaul and AdeltaE2 = h Itau2. So the difference their energy levels. Adeltaf = E2 Bl, needs to larger than the of their linewidths. think, to distinguish the if the separation is less than the sum of their widths. they might merge isto one broud peak, making them indistinguishabll AnlaWait. but I might have this reversed. Mayhe the difference in energies < delta between the uncertainties (line widths) to resalve them two separate peaks. Because larger than the sum of the two unsertainties they can be distinguished. Otherwise, the two peaks would overlap into one broad peak. An'aLet's 's clarity: For two spectral lines, each with width about 1 deltaEl and IdeltaE2. the energy separation needs be less thas the
different. their that can reselved aniisseeeenerr uncertainty
remember that the natural lingwidth of spectral the inverse the lifetime. So the linewidth energy
uncertainty principle) is related the lifetime. Because
Ita. where
uncertainty
lines they
't overlap and can resolved. Because
levels should be greater
the of their individual energy
the separation
than ideliaki IdeltaB2 for them their linewidths, their peaks
distinguishable. Because if their separation might overlap. making it impossible to tell them apart.ininšo. the energy difference ideltak should be greater than (vdeltaEl (deltaE2) AaknBat wait. wait. Alternatively perhaps the requirement is ddelta ('deltaEl (deltaE2), but let think about than the sum their widths. they can be resolved. Because each line width at half maximum, IWM. and for them to be distinct. , the distance between them must be greater than the sam of their PwEaS Otherwise, the secoad line would start before the first has ended, and they marge
If
separated by width (say, the
Alternatively. mayhe question is considering the inverse situation: energy differeace should more than each e-7, and no option meets that except D. So likely. the correct answer is D.ins/think stalalo determine which energy difference allows the two quaatum states to be clearly resolved. we analyze the relationship
individual widths. but doesn 't sense because the larger one 6.5
their lifetimes spectral line related to the lifetime W WGamma W) is the Linewidrh and Mt Wias W) is the lifetime. For the two states the lifetimes are WC Wis_l 1091-9 B. Wrext(s) W) and WC Wisu 2 109[-X] frac [Whbaa [[Wtau Wquudd WGamma.22
energy widths using the uncertainty principle. The state by the uncertainty principle: Vnin
linewidth
WGama Wfrrre [Wbbar ) tWat J JA
[ext([s] W). Their respective linwwdrhhs We aHGGmma [1thbar (Maaau2 JaJ.] we exteulate Aala Gamma] Wrrae (6,582 timees 101[-16]11100--911 -911 = 6,382 Weimes 10*[-7) W. Wieetlee I 1.I 6.582 Wrimes 104[-8] n. Weex J An WGammss 2 - Wrra (6.582 Wtimes 10 [10[]]][[009[8]] WGGamma22 6,582 Wtimes 109[7] 6.582 Witees ohee som of
linswddths (a Mat the two states be clearly resalved. 1091-8] 1.316 Wilmes 109f-71 W.
the energy difference W W Delta E W) between them must he greater than the sum of their linewidths W Deita E> W.Gamaa + 1GGmmaa22 Wapprox 1,316 Witimes 198[77 W. (eVxr(eV W 10419) W. WeexrjeV] W) much smaller than Mi 1,316 Wrimes 109[-7) W. [ee W) s0 does not satisfy the condition Anin- Dptionn B: WE 104-111 R. texriev) Wi ln This is even smaller than We 1091-9) M. Wrext (eV) W. so it does not satisty the condition Anin- ecOption C: W 104[-8 W, Wrexi(aV) W) Wrext(eV) condition
nNow. evaluate the given options stain- +pOption
This is
which is smaller than Optiio D: WV 109[-4] W. text]eV] Wise ia This is W 1.0 times 109 U. [eV] W). which is much larger than W 13316 Wrimes 10M[-7] W. (eV W. it satisties the condition AayaThus, the only energy 10*[-4) W. (eexx(eVV W).m An)
This 109[-7) W. Wsext]ev] W), so does not sausfy
Wrimes 10"[-8]
1,316 Wrimes
difference allows the states clearly resolved nAnswer:
24

---

Published as a conference paper at ICLR 2026
case study from LiveCodeBench
[Question:]
axd Question Nayon are given as integer sequence AWiotss .A.N) of length N. Here. A.I. A,2, W Idots.. A.N are all Aestintt.Aabiie element in A the second largest largest Phatainputiaaaann input following format:aaN \r30A_1 A.2 Widors t. the integer X such that the X-th element
Standard Input
the second aaggoottaattssstriinns ta'aaa- Weleg Wleg 1000e- I Wleg A Wleg 10 A.I. 2.2, Widets A.N are all distinct Ja- All input values are integers AnAnSample Taput Walas45rka8 2 Dok'nSampl Ourput Snlnnirtataaii second largest A,3,
print 3.AntaSample laput 2in'o8irial 3 nineau Format: Read inputs from stdin solve the problem and write the answer stdout (do directly test on the sample inputs). ). Enclase your code within delimiters as follows Aa" python'ns YOUR CODE HEREin" "Waaate Answer: (use the provided format with backticks)
element
IitatuSample Outpat 21a3n61
[Answer:]
Okay, I need to salve this problem where have to find the secund largest element in a given integer sequence and then determine which position it occupies in the list specifically the index in the I- based sotation. Iam, let 's think through how approach this AnlaWait, wait. The problem says that element distinct
worry about duplicates. then output the index (but wait, the problem says the X-th element, which the position in the array. not zere-based), ). Oh right. the first sample iapet: 8 5 L. The largest is s. 5
find
largest namber
then 5, then then 1. Wait 9. Wait, no.
wail the second sample sample input
11, The largest is the
array 11.2.3.4.5.10.2 largest is at position 8, the second largesi is 10 at position 6. the output is 6. So the task is to find the second largest element in the list which is the element that is just smaller than the maximem but it could integers. An2. Find the moximum value the list Ann, Then find all elements that are less than the maximam. Among these. the second largest will be the maximum of that subset. But wait, wait, maybe the second largest 2,3.1. max those 5, which correct. sample 2, the
larger that
there might be elements that AninSu.
after the
smaller
plan isstatal. Read
necessarily the of the
For example. in the first sample. the 11, the
8, the rest are 1.2.3,4.5. 10... The mas of those is 10, which is correct lnlaSo. the approach is:
The approach involves finding the maximum element. then determining the second largest element from the remaining elemeats, and finally lacating its position is the original list Aalnees Approachinl. read the input values. which include the length elements An2. aaFind Maximum Value se; Compute
Read Inputes: First.
list and list An3. Filter Candidates
maximun value ee: Create a sublist of all elements excluding the maximum value An4, sePind Second Largest Prom the filtered sublist. find the maximum value whith is the second largest element in the original index the output.Navaen Solution iaa = list (map(int, isput() (ppt((.ssppii split() (1) Ammas,__ val mxx(a) Anaandidates [x for x in a max_val B asscond__val val maxtccanddates ) \nindde a. indexx muxx(secome vall + a Convert to I-based indext " "atnete Explanation leagth the list and built =in max()) function we determine the largest value is the list lis Filtering Candidates
Aa5. ><Determine Position
largest element ppyhon'kna intrinput
original
and convert
aptint (index
Readiin laputes: We read the integer "n" which the integers "a" AAn- aaFinding Matimmass: Using Python
We create new list "candidates" that excludes the maximum value to focus the remaining elements An- e* Finding Second Largestas: seladex Conversion eo; index using iidexx which returns a O-based index. so we add to convert it to 1-based indexing before printing the result, AninThis approach ensures that we efficiently find the second largest element and correctly determine position in original list, adhering the problem constraints and requirements.
"aandidates" list
largest element second largest element found
oziginal
25


Reviews:
[{"review_id": "eT0IfUdNwl", "reviewer": "Reviewer_L77o", "summary": "This paper addresses improving sample efficiency in Reinforcement Learning with Verifiable Reward (RLVR). It introduces DEPO, a method that integrates offline and online strategies to reduce the data required during finetuning. DEPO is evaluated on three pre-trained models across three mathematics benchmarks, compared with baselines --- both offline and online variants of RLVR, and demonstrates strong performance.", "questions": "1. Related works do not sufficiently cover the literature. For instance, for RLVR, the reviewer believes that after DeepSeek-R1, there are many representative works that have used it in different places. But there are only about 5 papers are cited.\n2. What is $\\mathcal{B}$ in equation (5)?\n3. Could you give a more detailed explanation for the exploration metric in (4)? The equation is too complicated without an intuitive explanation to validate each term.\n4. In equation (6), what does top-α_e% means, which term should be the top of which set? The same for \"bottom-ρ%\"\n5. In the final objective function (5), the reviewer didn't see the role of the final offline dataset D^{sub} if the reviewer didn't miss something. So what are the roles of the offline and online phase?\n6. Could the author explain more about this claim/intuition around line 177: \"To promote diversity, we use Determinantal Point Process (DPP) (Kulesza & Taskar, 2012a) to identify a subset that maximizes the determinant of the corresponding similarity submatrix: maxY ⊆P det(Y ), where Y is the submatrix of the full similarity matrix P, and det(·) represents its determinant value. This refers to selecting samples that form a larger volume in the feature space, with a larger volume indicating greater diversity.\" Why a larger determinant of the similarity matrix P may lead to diversity? Since intuitively, if some entries in P_ij is small, that means the corresponding sample i and j are different, giving more diversity. So should we just find those regions with small elements in P? Why the determinant is related to the diversity of samples?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experimental results are comprehensive. Many ablation studies of different components, different hyperparameters are done to clearly show the contributions and robustness of the proposed components.", "weaknesses": "1. The writing could be improved, particularly in the methodology section. Additional clarification questions are provided below.\n2. The work introduces many components—three in the offline phase and one in the online phase—but the related work for each component is not sufficiently discussed. For example, while the introduction touches on high-level prior work for both phases, detailed, technical comparisons are missing. This makes it somewhat difficult for reviewers to assess the novelty of the proposed algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svY67lrh5j", "reviewer": "Reviewer_Gqob", "summary": "This paper addresses the high data and computational costs associated with Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing the reasoning capabilities of Large Language Models (LLMs). The authors propose DEPO (Data-Efficient Policy Optimization), a pipeline that combines offline data curation and online rollout pruning. In the offline stage, a PageRank-weighted Determinantal Point Process is used to select a diverse and influential data subset, which is then refined to favor samples of moderate difficulty aligned with the model's current capabilities. In the online stage, an \"explorability\" metric is used to dynamically skip rollouts for low-potential samples, thereby improving computational efficiency.", "questions": "1, To assess the reliability of the core results, could you please repeat the key experiments in Table 1 (e.g., for the DeepSeek-R1-Distill-Qwen-7B model) using at least three different random seeds and report the mean and standard deviation? This is crucial to verify whether the performance difference between your method (DEPO) and full-dataset training is statistically insignificant.\n\n2, The offline \"Difficulty-aware Normal Distribution Sampling\" strategy (Section 3.2.2) statically prioritizes moderately difficult samples. This overlooks a comparison with a simpler, dynamic curriculum learning strategy (e.g., moving from easy to hard samples as training progresses). Could you provide a comparative experiment against such a classical curriculum strategy to demonstrate the superiority of your proposed sampling method?\n\n3, The \"explorability\" metric (Equation 4), central to your online pruning, combines normalized reward (Ât) and entropy (E(o_i^t)). To isolate the contribution of these two factors, could you add to your ablation study (Table 2) a variant that performs pruning based solely on entropy, removing the reward term (Ât)? This would help determine whether the combination with the reward signal is essential.\n\n4, The offline data curation is a two-step process: DPP pruning to 50%, followed by difficulty sampling to 20%. What is the advantage of this two-step approach over a single-step method that could, for instance, incorporate a difficulty score directly into the DPP kernel? Could you provide an experimental comparison or a stronger justification for this design choice?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The proposed method, DEPO, achieves performance nearly equivalent to GRPO training on the full dataset while using only 20% of the data and significantly reducing training time. Specifically, on the DeepSeek-R1-Distill-Qwen-7B model, the full DEPO pipeline achieved a comparable average accuracy (61.1% vs. 61.7%) using only 57% of the training time and 40% of the rollouts compared to training on the full dataset (Table 1). This is a substantial practical contribution that could lower the barrier to entry for RLVR training.\n\n2, The paper presents one of the first approaches to systematically integrate both offline data selection and online rollout pruning. This dual optimization strategy, which considers diversity, influence, and difficulty offline (Section 3.2) while performing dynamic sample filtering online (Section 3.3), is a distinguishing feature compared to prior work that has typically focused on only one of these aspects.\n\n3, The effectiveness of the proposed method is validated across three different LLMs and five reasoning benchmarks (Table 1). The paper provides extensive empirical evidence to support its claims, including sensitivity analyses on varying dataset sizes (Figure 4), sampling ratios (Figure 5a), and rollout ratios (Figure 5b), as well as a clear ablation study demonstrating the contribution of each component (Table 2).", "weaknesses": "1, The core components of the proposed method are largely a combination of well-established ideas. The use of Determinantal Point Processes (DPP) and PageRank for offline selection (Section 3.2.1), and difficulty-based sampling aligned with model capability (i.e., curriculum learning) (Section 3.2.2) are not new concepts. The entropy-based \"explorability\" metric for online pruning (Equation 4) appears to be an intuitive extension of existing RL techniques that leverage entropy to encourage exploration. While the authors claim the \"integration\" of offline and online strategies as a key contribution, the lack of originality in the individual components limits the overall novelty of the work.\n\n2, The most significant weakness is the lack of statistical validation for the experimental results. Most results, including the main ones in Table 1, present figures from a single run without reporting means and standard deviations (or confidence intervals) over multiple seeds. For instance, the performance difference between DEPO-Offline and the Full dataset on the Deepseek-R1-Distill-Qwen-7B model is a mere 0.3 percentage points (61.4 vs. 61.7). It is impossible to determine if this difference is statistically significant or simply due to random variance. This severely undermines the credibility of the paper's central claim of \"maintaining comparable performance while improving efficiency.\" \n\n3,The \"explorability\" metric (Equation 4) is intuitive but designed in a somewhat heuristic manner. The paper lacks a theoretical justification or deeper analysis for the specific combination of normalized reward (Ât), entropy, and a hard threshold (λ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ci8WucoCMW", "reviewer": "Reviewer_1LYZ", "summary": "This paper proposes DEPO, a two-stage data selection pipeline, to address the high training costs and low data efficiency often encountered in Reinforcement Learning with Value Response (RLVR). The process begins with offline data curation, where the data is meticulously chosen based on three distinct criteria: diversity, influence, and difficulty. Following this, the second stage, online rollout pruning, selects appropriate rollout samples by utilizing a metric focused on explorability. The empirical results demonstrate that DEPO achieves sample efficiency and faster learning performance compared to conventional methods.", "questions": "* It would be very helpful if the authors could provide a quantitative measure of the specific pre-computation cost required for the offline curation stage.\n* Given that the offline subset $D^{sub}$ relies on a static difficulty measurement, it would be interesting to know if the authors considered or experimented with a dynamic offline curation strategy? For instance, periodically re-sampling $D^{sub}$ to reflect the evolving capabilities of the policy might be an interesting alternative to explore.\n* Could the authors provide a bit more detail on how the specific filtering threshold $\\lambda=1.5$ was determined? Since this value multiplies a dynamic term that changes during training, a justification for why fixing the value of $\\lambda$ is an appropriate and robust choice would be appreciated.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper proposes a comprehensive learning paradigm in RLVR that systematically integrates both an offline data curation phase and an online data collection phase.\n* The metrics used for data selection during both the offline and online processes are highly refined and logically compelling.\n* The efficiency and efficacy of the proposed method are validated through extensive experimental results conducted across a diverse range of datasets.", "weaknesses": "* Regarding the significant reduction in overall training time, it would be beneficial for the authors to clarify whether the initial cost of the offline data curation is included in this calculation. Since the graph construction and DPP application likely involve a notable overhead, quantifying this pre-computation cost would provide a more complete picture of the method's overall efficiency.\n* The offline curation relies on difficulty-aware sampling from a fixed distribution determined before training. While this is shown to be effective, this dataset could become less challenging for the policy as it improves. The paper's claims on sample efficiency would be even more compelling with a brief discussion or analysis of why this static approach remains effective throughout the entire training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)