Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR
Authors: Hao Yi
Yulan Hu
Xin Li
Sheng Ouyang
Lizhong Ding
Yong Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning; Large Language Model; Active Learning; Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR

Abstract: Large Language Models (LLMs) have recently improved mathematical reasoning through Reinforcement Learning with Verifiable Reward (RLVR). However, existing RLVR algorithms require large query budgets, making annotation costly. We investigate whether fewer but more informative queries can yield similar or superior performance, introducing active learning (AL) into RLVR. We identify that classic AL sampling strategies fail to outperform random selection in this setting,  due to ignoring \textbf{objective uncertainty} when only selecting by subjective uncertainty. This work proposes an \textbf{uncertainty consistency} metric to evaluate how well subjective uncertainty aligns with objective uncertainty. In the offline setting, this alignment is measured using the Point-Biserial Correlation Coefficient (PBC). For online training, because of limited sampling and dynamically shifting output distributions, PBC estimation is difficult. Therefore, we introduce a new online variant, computed from normalized advantage and subjective uncertainty.  Theoretically, we prove that the online variant is strictly negatively correlated with offline PBC and supports better sample selection. Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30\% of the data, effectively reducing the cost of RLVR for reasoning tasks.\footnote{The code is available at \hyperref[https://github.com/yihao-123/uncertainty-consistency]{https://github.com/yihao-123/uncertainty-consistency}.
}

Published as a conference paper at ICLR 2026
LEARN MORE WITH LESS: UNCERTAINTY CONSIS- TENCY GUIDED QUERY SELECTION FOR RLVR
Hao Yi Yulan ; Li2, Sheng Ouyang!.2 Lizhong Ding", Yong Liu
1 Renmin University of China, Gaoling School of Artificial Intelligence, Beijing
Amap, Alibaba Group
3 Independent Researcher
yi yihao @ruc.edu.cn
ABSTRACT
Large Language Models (LLMs) have recently improved mathematical reasoning through Reinforcement Learning with Verifiable Reward (RLVR). However, exist- ing RLVR algorithms require large query budgets, making annotation costly. We investigate whether fewer but more informative queries can yield similar or supe- rior performance, introducing active learning (AL) into RLVR. We identify that classic AL sampling strategies fail to outperform random selection in this setting, due to ignoring objective uncertainty when only selecting by subjective uncer- tainty. This work proposes an uncertainty consistency metric to evaluate how well subjective uncertainty aligns with objective uncertainty. In the offline set- ting, this alignment is measured using the Point-Biserial Correlation Coefficient (PBC). For online training, because of limited sampling and dynamically shifting output distributions, PBC estimation is difficult. Therefore, we introduce a new online variant, computed from normalized advantage and subjective uncertainty. Theoretically, we prove that the online variant is strictly negatively correlated with offline PBC and supports better sample selection. Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30% of the data, effectively reducing the cost of RLVR for reasoning tasks] [T]
INTRODUCTION
Large Language Models (LL.M)) (Guo et al. 2025 Team et al. 2023 Achiam et al. 2023 Yang al. 2024) have recently advanced complex mathematical reasoning. key driver Reinforcement Learning with Verifiable Reward (RLVR) (Shao et al. 2024). which leverages explicit, verifiable re- wards in math tasks (correct vs. incorrect). This property allows us to obviate the need for learned reward models and critics commonly used in actor-critic methods. Instead, group- bbsed normal- ized rewards are used to estimate sequence-level advantages, as implemented methods such as GRPO (Shao et al. 2024), DAPO Yu et al. 2025). REINFORCE+ (Hu 2025). and RLOO (Ah-
madian et al. 2024).
However, these algorithms typically require tens of thousands of queries to reach optimal perfor- mance, while annotating answers for mathematical reasoning tasks incurs substantial cost. More- over, previous RL research has overlooked the influence of query selection on reasoning ability. Poor query selection can bias models, induce entropy collapse, gradient instability, and hinder con- vergence (Yu et al.] 2025: Cui al.) 2025) [heng et al., 2025 Wang et al.] (025). So this leads to a pivotal question: Can we achieve comparable or superior performance with fewer but more informative queries in the RL reasoning lask? Active learning (AL) (Wang & Shang 2014) offers
a promising approach.
In classic AL, model selects budgeted subset of unlabeled queries for annotation based un- certainty (Citovsky et al. 2021 Wang & Shang 2014 Geifman & EI- Yaniv 2017) or feature-space
"Corresponding author.
'The code is available at http://cthhccooommmioeeeeiiittte

---

Published as a conference paper at ICLR 2026
2017). The goal is to label the most valuable examples, thus improving genera with minimal annotations. Similar ideas ap- pear in LLMs for in-context learning (Diao et al. 2023 Du et al. 2024), preference alignment (Chen) et al. 2024, Muldrew et al. 2024), supervised fine-tuning [Bayer] 2025), and knowledge distilla- tion (Zhang et al.j 2023). Therefore, we assess classic AL strategies in the offline reasoning RI. (Qwen et al. (2025) on the MATH dataset (Hendrycks et al., 202I). as shown in Table [ The results clearly show that classic AL strategies fail to improve performance in our setting. This raises an important question: why do tra- ditional AL methods fail here? To investigate this, we perform an additional analysis focusing on the relationship between subjective uncertainty (e.g., perplexity estimated by the model) and objective uncertainty (i.e., rule-based reward), shown in Figure Ta Consistency samples (orange curve in Fig- ure Tay are those which subjective and objective uncertainty are simultaneously high or low. How- ever, Figure [a] reveals that samples with high subjective uncertainty but correct answers produce extreme policy gradients. These outlier gradients induce high variance, destabilizing RL training and hindering reasoning performance. In contrast, consistency samples yield more stable training dynamics. For concrete illustration why inconsistent samples tend to induce larger variance in gradient norms, consider that in a consistent positive sample, the average generation probability of tokens might be around 0.9, whereas in inconsistent positive sample it may be around 0.3. Since policy gradient methods tend to increase the probabilities of positive sample tokens toward 1, incon- sistent samples have a larger margin for probability improvement, thereby making targe-norm and unstable gradients more likely. The same reasoning applies to negative samples. Motivated by these findings, we propose that focusing on consistency samples rather than simply those with the highest subjective uncertainty would be more beneficial for stable and effective RL training.
coverage(Ash al., 2021 Agarwal et al. 2020: Sener & Savarese
setting. We first conduct a pilot experiment with Qwen2.5-0.5
In the offline setting, we measure the uncertainty consistency using the Point-Biserial Correlation Coefficient (rps) (MacCallum) 2002), and select the top 5% samples with minimal rys for RL. How- ever, estimating rph becomes challenging in online settings due to limited sampling and dynamical change in output distributions (Guo et al. 2025). To address this, propose online uncertainty consistency metric, ponfine, calculated from the normalized advantage and the current model's un- certainty. Theoretically, we prove that the online metric is strictly negatively correlated with its offline counterpart, and under mild conditions, its maximization matches optimizing sample-wise uncertainty. These results provide a solid theoretical foundation for the proposed method.
Experimental results show that in the offline setting, selecting samples with lower offline consistency metries noteblely outperforms random selection and classic active learning strategies. In the online setting, our method achieves performance comparable to better than RL with the full dataset by training on only 30% of the data. This validates our metrics' efficacy in query selection for RL. reasoning.
Table 1: Results of different AL strategies on MATH task using GRPO on Qwen2.5-0.50 Full de- notes the use of the full dataset, while other strategies employ offline sampling with sampling ratio of 10%. Every experiment is conduct 5 times using different random seed. The result shows that classic AL strategies, whether based on uncertainty (PPL, Entropy (Wang & Shang (014)). features
(K-means" K-center (Sener & Savarese (017)), or L.LM-based prompting (AskLI.M (Sachdeval
et al. 2024)), demonstrate no marked difference compared to the Random strategy. AskL.LM
prompt is shown in Appendix C and more experimental setup refers to Appendix
Q2.5-0.5B
Full
Random
PPL
Entropy
K-center
K-means
AskLLM
33.2
31.0
30.8
29.9
31.1
30.4
30.9
MATH
(#:0.1
(+0.2)
(+0.6)
(+0.3
(+..3
(=0.5)
(+0.7)
RELATED WORK
Reinforcement learning base on verifiable reward (RLVR). Building on the noteble perfor- mance of reinforcement learning methods in aligning LLMs with human feedback, for example
Featured by of k-means.
(Zhang et al. 2025 Choosing the points to be labeled as the centers
2

---

Published as a conference paper at ICLR 2026
grrsle
02
(b) Correlation between ponliie and rpbh (7 = 1): Each point corresponds to one training instance. K 64 responses are sampled for that instance and used to estimate rps and r onisme
pb
(a) Gradient Norm Dynamics (Inconsistent consistent samples).
pb
Figure/ I: (a) Gradient norm dynamics for inconsistent vs. consistent samples.
Correlation
between online and offline uncertainty consistency metries.
PPO (Schulman et al. 2017), Shao al. (2024) extends Reinforcement Learning base on Veri- fiable Reward (RLVR) to improve LLM mathematical reasoning capabilities. GRPO (Shao et al.] 2024) dispenses with reward model and replaces it with rule-based rewards. It also removes the trainable critic model used to estimate sequence returns and advantages. Instead, it estimates pol- icy gradients using group-wise normalized advantages, which greatly reduces GPU memory usage during RL training. RLOO (Ahmadian et al. (024) estimates the advantage with a leave-one-out method, improving sample efficiency. REINFORCE+ (Hul (025) follows PPO by incorporating KL regularization into the advantage estimate, mitigating reward backing issues that arise with the GRPO group-wise estimation. DAPO (Yu et al. 2025) improves sample efficiency and reduces reward noise through empirical techniques such clip higher, dynamic sampling, and overlong reward shaping. However, these algorithms typically require tens of thousands of queries to reach optimal performance, while annotating answers for mathematical reasoning tasks incurs substantial cost. Moreover, previous RL research has overlooked the influence of query selection on reason- ing ability. Poor query selection can bias models, induce entropy collapse, gradient instability, and hinder training Yu et al. 2025 Cui et al. 2025 Cheng et al. 2025 Wang et al. 2025).
Active Learning (AL). The core idea of Active Learning (AL) is: we can maximize post-training performance with a small and limited labeling cost by selecting the most informative queries. Clas- sic AL strategies fall into two categories: uncertainty-based methods and feature-space methods. Uncertainty-based methods assume that the samples with the highest uncertainty are the most in- formative. Representative approaches include Least Confidence (LC), Margin Sampling (MS), and Maximum Entropy (Wang & Shang, 2014: Geifman & El- Yaniv 2017). However, these methods ignore the distribution of samples in feature space, so the selected subset may lack diversity. To address this, Core-Set (Sener & Savarese 2017) and CDAL (Agarwal et al. 2020) expand the cov- erage of the selected subset in feature space. This increases diversity while preserving the overall information content of datasets. Although Kaplan et al. (2020) shows that LLMs are data-hungry during pretraining and benefit from more data, additional data can hinder optimization and limit achievable performance in many downstream tasks. In in-context learning (ICL), Activate Prompt- ing (Diao et al.] 2023) highlights inefficiencies in example selection: the most effective examples are not directly discoverable. CAL (Du et al.) (2024) argues that datasets exhibit biased diversity and over-optimization. In supervised fine-tuning (SFT), ActiveLLM (Bayer) 2025) finds that the cold- start phase requires large amounts of data, which limits its utility. In preference alignment tasks, Chen et al. 2024) argues that most current online algorithms still rely on human preference labels provided as feedback to update the policy model. This reliance leads to substantial expert query costs. In knowledge distillation (KD), Zhang et al. (2023) shows that LLMs easily influenced by erroneous signals, and the high cost of annotation restricts their applicability in domain-specifis tasks. However, the effectiveness of AL strategies in the RL reasoning task is underexplored.
3

---

Published as a conference paper at ICLR 2026
3
PRELIMINARY
In this section, we first present the unified mathematical formulation of the RLVR loss Section Section 3.2 we introduce the method for estimating subjective uncertainty.
3.1
3.1
REINFORCEMENT LEARNING BASED VERIFIBLE REWARDS (RLVR)
RL methods epitomized by PPO (Schulman et al. 2017 already serve in InstructGPT (Quyang et al., 2022 to align LLMs with human feedback. To eliminate the memory overhead of training a critic and reward model, Shao et al. (2024) introduces RLVR to improve the reasoning capabilities of LLMs. the on-policy setting, RLVR can unify into the following form:
(
1 K
A log To ( (0))
(1)
Uk
where K denotes the sample number for each query, ma represents the policy model, [yb is the length of the tth response and Ag., is the token-level advantage function. Usually, the response level advantage function is adopted. For GRPO an example, Å Ag,
meenn RR {=]]]]]]]]]]]]]] R is the reward function. mathematical reasoning tasks, rule- std([ ((aue ke] (]]))
based reward function is typically adopted:
[1, a=a a
R(c, (lc,a/z)
0, otherwise
Here, c, a, and a" represent the reasoning content, the predicted answer, and the reference answer, respectively.
3.2 UNCERTAINTY ESTIMATION
To estimate the subjective uncertainty of L.LMs, we first use the reference model Tref to sample ( each sample z for K times, obtaining K responses, denoted as {0; fj=1- For each response, compute the PPL as follows:
log Toggrerly (y (2 (8) Mu.ce)
PPI. = 10
larger value of PPL indicates that the probability of Tref sampling Vr is lower, which corre- sponds higher degree of subjective uncertainty for the sample r. More uncertainty estimation method (Margin Score, Entropy) can refer to Appendix B For convenience in subsequent discus- sions, use ( to uniformly denote of the above metries.
METHOD
In this section, introduce the query selection method based uncertainty consistency under two scenarios, offline and online. Section 4.1 describes how to characterize uncertainty consistency in the offline setting: in Section 4.2 we extend the concept of uncertainty consistency to the online setting, and demonstrate the connection between the offline and online metrics as well as how the online metric affects the single-step optimization process theoretically.
4.1 UNCERTAINTY CONSISTENCY IN OFFLINE SCENARIOS
In Section I we have shown that, in mathematical reasoning tasks within RL scenarios, simply applying classic AL strategies, such as selecting samples where the LLM exhibits the highest sub- jective uncertainty for training, does not lead to better enhancement of reasoning ability compared to randomly choosing training samples. We argue that the key reason for this ineffectiveness lies in the neglect of the relationship between objective and subjective uncertainty.
4

---

Published as a conference paper at ICLR 2026
Offline Query Selection
Online Query Selection
Input: Query set D (= reference model Ter, oftline generations K, sample ratio p. training steps T, mini-batch size B, online generations
Input: Query set D = (x ). reference
model weet sample ratio p. training steps T, mini-batch size B, online generations
Output: Policy K0r
Output: Policy F0r Initialize Oo from Tha
Initialize 0p:
for
1 to N do
for L 1 T do
Generate K responses
Sample mini-batch B (
(200)
{yk 1K merr- (x))0))
foreach x²¹) E B do Generate G responses
Compute (t) via Formula
( Tg=1 TT,_ T0,_
Select the p% queries with smallest (6) form
Compute 77o (plline(x via Formula
D:
for = 0 1too do Sample {ar 1 DD: for j to do Generate G responses J==1 ( na (: (1)0))) Compute VoL via Formula Update 0z+1 + 0r Vol
Select top-p% queries with largest rg form B;
Compute VoL via Formula 1 on B: Update O, 0.-1 VoL:
10 return Regi
10
11
return :o.
Figure 2: Offline (left) and online (right) query selection procedures.
In Equation 2 this Bernoulli variable reflects the degree of objective uncertainty: a lower accuracy rate over K samples indicates higher objective uncertainty. Through our experiments, we found that samples with higher aligmment between subjective and objective uncertainty tend to be more valuable for the RL reasoning task, Since the reward is binary variable, for a training sample x, we use the PBC to characterize the relationship between subjective uncertainty U (Equation 3) and objective uncertainty (Equation 2h:
U1 Uo V
KoKı K2
( Lu )K
(4)
Here, Uj denotes the mean subjective uncertainty for correct responses for ,0(0) Up for incorrect responses, şк is the standard deviation of { } and K₁ and Ko are the numbers of correct and incorrect responses, respectively. A more negative value of Tpb indicates a stronger negative correlation between the two variables, and vice versa. Therefore, when subjective and objective uncertainty are aligned, U and R should exhibit negative correlation, i.e., Tpb should be as small as possible. In the offline setting, we pre-select the top p% of samples with the smallest 7po values from the training set as the dataset for RL. The detailed procedure is shown in Figure
4.2 UNCERTAINTY CONSISTENCY IN ONLINE SCENARIOS
In the previous section, propose offline uncertainty consistency metric (Equation[4) However, this metric presents the following challenges in online RL settings:
Estimation accuracy. Because the calculation of rpb relies on large number of samples K, it is not feasible to estimate this value accurately in practice;
Model update. The calculation of subjective uncertainty U depends probability outputs the model, and online settings, the sampling distribution the behavioral policy is changed as the model is updated.
5

---

online samples are Published as a conference paper at ICLR 2026
Therefore, we propose an equivalent online uncertainty consisteney metric that can mitigate the above issues:
onllnn pb
(8) 1 (o Lv, 1k=1) K
(5)
A,20
A
Here, A, is the advantage estimate assigned by the RL algorithm (e.g.. GRPO) for the sample (x(0), 1/ (1)), U. is the current model To's estimation of the uncertainty metric (see Equation 3 and Y > is hyperparameter balance the ratio between positive response and negative responses.
Based on our experiments (Figure [b) and theoretical analysis Theorem D we find the negative correlation between Tpb and pb rondine Figure lb shows the relationship between the offline metric rp and the online metric pp (>=1) There is a noteble negative correlation between rpb rooline and rpb in six combinations of models and datasets. Theorem shows that the covariance between Tpb and pontine is strictly negative. Therefore, to select the most valuable training sample, we choose the top pb
2% of samples with the largest ro rontine from the minibatch at every step during the online training, which means the highest subjective and objective uncertainty consistency. The detailed procedure refers to Figure (right).
Theorem (Negative Correlation between Tyb and yonline), For the same model wa, the covariance 72
between Ti and go is less than zero, i.e., Cov(C (T'p,, pibline) < O.
The proof is shown in Appendix A.1.1 Furthermore, to explain why the largest r pb effective in the RL reasoning task, have proven Theorem 2
Theorem 2 (Equivalent between Maximizing Decrease in Sample Uncertainty and Maximizing ponkin). Suppose (1x;0) E U. denotes the subjective uncertainty for sample Under pb
sample gradient orthogonality assumption (Assumption) TT and bounded gradient norm assumption (Assumption 2. in one optimization step of an on-policy RL algorithm (e.g.. GRPO), selecting sam- ples in the minibatch with largest rr can maximize the decrease sample uncertainty.
Assumption (Sample Gradient Orthogonality). For any i.j E [K] and + j. the derivatives of the sample uncertainties with respect to the parameter 0 are orthogonal, namely NoU! VoU? >= O.
Assumption (Bounded Gradient Norm). For any i [K], we have 0 < m 1 M.
The explanation of the assumption and the proof is shown in Appendix A.1.2
EXPERIMENT
5.1 SETUP
Model. We select the following models to evaluate the effectiveness of uncertainty consistency metric in both offline and online settings. These models cover different sizes, different architectures,
and include both Pretrained and Instruct versions: 1) Qwen2.5-7B (Q-7B) (Qwen et al. 2025) 2)
Qwen2.5-3B (Q-3B) (Qwen al. 2025)> 3) Llama-3. --8---ntttttttt (L-8B-I) (Dubey al. 2024p.
Dataset. We choose two mathematical reasoning tasks, MATH and GSM8K. MATH consists of 7,500 training examples and 5,000 test examples, while GSM8K consists of 7,474 training exam- ples and 1,319 test examples. All evaluations focus generalization within each task, without considering cross-task generalization. In order to utilize the CoT ability, we use CoT prompt shown in Appendix C
Baseline & Metric. In the offline setting, we include the following baselines for comparison: 1) Full: Using the entire training dataset. 2) Random: Randomly selecting p% of the data. 3) PPL & For each query, generating K = 64 inferences, then selecting the top of examples based on the highest PPL (Equation 3) ENT (Equation 7p scores. 4) K-center (Sener & Savarese 2017): Utilizing Qwen3-88 Beet338---mmm BB-Embedding (Zhang et al., 2025) to extract language features from training queries, and then applying the K-center algorithm select p% of the data. 5) AskLLM (Sachdeval
ENT:

---

Published as a conference paper at ICLR 2026
al.
2024
The ask prrm used for LLM is shown in Appendix C 6) Active Prompt Diao
et al.
2023
7) CAL (uu et al. 2024).
In the online setting, Random, PPL, and ENT are performed within each minibatch. All other baselines remain the same as the offline setting. All experiments report the greedy decoding @ass I score the test set.
RL Algorithm Hyperparameter. We mainly use GRPO to validate the effectiveness of our method. Training is conducted with sampling temperature set to 1.0, maximum response length of 2048, and selected from the set {0.1, 0.5, 1.0, 1.5, 2). Adam W (oshchilov & Hutter 2017) is used as the optimizer with constant learning rate of 1e-6, and k3 KI. divergence regularization with a coefficient of 0.001. The batch size is set to 256, and total training steps fixed at 50. In Section 5.3 further discuss the effects of different RLVR algorithms. The hyperparameters for RLOO and REINFORCE+ are identical to those of GRPO. As for DAPO, we set the high clip ratio and low clip ratio 0.28 and 0.20, respectively, with a long buffer length of 1024 and long penalty coefficient of 1.0: Other hyperparameters are the same GRPO. All experiments are run on x 96GB NVIDIA H20 GPUs.
K 8. 7 in Equation
5.2
MAIN RESULT
Table 2: Main result in offline and online setting.
Model
Method
GSMSK
MATH
OFF(p=30)
ON(p=30)
OFF(p=30)
ON(p=30)
Full
91.5
91.5
73.2
73.2
Random
88.6
88.1
70.8
68.2
PPL
88.9
90.4
71.0
72.1
Q-7B
ENT
88.4
90.3
70.3
71.8
K-center
88.1
70.5
AskLLM
87.8
69.8
Active Prompt
85.2
65.1
ZS-CAL
84.3
64.0
(urr)
90.1(+1.5%)
72.1(+1.3%)
pontine (Ours)
91.7(+2.4%)
72.9(+4.7%)
Full
85.2
85.2
63.8
63.8
Random
82.4
81.2
62.2
58.8
PPL
81.6
83.4
61.6
62.5
Q-3B
ENT
82.7
83.2
62.9
62.8
K-center
83.0
62.5
AskLLM
82.8
61.9
Active Prompt
80.5
54.1
ZS-CAL
79.2
54.5
Tp (Ours)
83.6(+1.2%)
63.3(+1.1%)
yontine pb
(Ours)
84.9(+2.5%)
64.0(+5.2%))
Full
90.2
90.2
52.0
52.0
Random
87.0
88.0
50.9
50.6
PPL
86.5
90.5
49.8
51.6
L-8B-I
ENT
87.0
88.7
50.7
51.3
K-center
87.3
51.0
AskLLM
86.2
50.5
Active Prompt
84.1
49.5
ZS-CAL
84.5
48.6
Tp (Ours)
88.7(+1.7%)
51.5 (+0.6%)
online (Ours)
89.9 (+2.9%)
52.5 (+1.9%)

---

Published as a conference paper at ICLR 2026
The main experimental results are reported in Table 2 We focus three types of models and conduct comparative studies on two mathematical datasets under both online and offline settings. In all experiments, the sampling ratio is consistently set to 30%.
Offline. In the offline setting, classic uncertainty metrics (PPL, ENT), feature-based methods (K- center), and the prompt-based uncertainty method (AskLLM) perform similarly random selection and are noteblely worse than using the full dataset. These findings are consistent with our pre- liminary experiments in the warm-up stage. Applying AL strategies in ICL provides only limited improvements to model reasoning ability. In contrast, directly selecting the 30% of samples with the lowest rpp yields much better performance than random selection, although still falls short of the results obtained with the full dataset. This gap can be attributed to estimation bias caused by the changes in model output distribution during RI. parameter updates.
Online. In the online setting, selecting samples with high PPL and ENT values during RL leads to better reasoning performance than random selection, but does not reach the performance achieved with Full. However, dynamically selecting the top 30% samples with the highest roplin in each minibatch not only noteblely outperforms random selection. but also closely matches the perfor- mance of Full. In some cases, such as Qwen2.5-7B on GSM8K and Qwen2.5-3B on MATH, even surpasses Full by 0.2%. These results indicate that in RL--based reasoning training, using only 30% of the data is sufficient to achieve nearly the same performance Full. Samples with consistent subjective and objective uncertainty are more valuable, supporting both our empirical observations and theoretical hypotheses.
5.3 ABLATION STUDY
To further evaluate the validity and robustness of our method, we conducted additional experiments using Qwen2.5-7B.
Choosing Top 5% or Bottom p% Tpo samples. investigate whether samples with consistent subjective and objective uncertainty are more valuable for RL reasoning training, we selected the bottom 30% (consistency samples, lowest rpt) and top 30% (inconsistency samples, highest Typ) in the offline setting. We then evaluate model performance
the test set throughout training, as shown in Figures 3 (1-a) & (1-b). The results indicate that, for both MATH and GSMSK, using consistency samples consistently outperforms using inconsistency samples. Moreover, training with inconsistency samples even performs worse than randomly selecting the same number of samples. This suggests that inconsistency samples may be detrimental model reasoning in RL.
Cid NUTH
Goal MAA Lanhi
Gob MAlli--Aaap
2ed MLTT
114
4
kmm0
Figure 3: Ablation study and discussion on Qwen2.5-7B: (I-b)&((-b) Choosing Top 30% or Bottom
30% rp samples. (1-))&(1-d) Sensitivity of y in Equation 5 (2-a)&(2-b) Responses length and entropy during online training. (2-c)&(2-d) Different Sample Ratios.
Sensitivity of y in Equation 5
We explored the impact of different y values, ranging from (0.1, 1.0, 1.5, 2.0], as illustrated in Figures 3 (1-c) (1-c) & (1-d). The results show that our approach favors smaller 7 values, such as 0.1. With larger 7, performance lower and does not match that of Full, although it still surpasses the online random selection baseline.
Different Sample Ratios. While our main experiments used a sampling ratio 30%, here we examine how different sampling ratios affect the consistency criterion under both online and offline
8

---

Published as a conference paper at ICLR 2026
Table Online RL results on hard dataset DAPO-MATH-I7K, Sample ratio is 30%.
Model
Dataset
Method
GRPO
R1.O0
DАPO
REINFORCE++
Random
68.2
72.0
70.6
71.7
MATH
Full 73.2
74.8
73.5
73.9
pnlline pb
72.9
74.3
73.2
73.9
Qwen2.5-7B
Random
88.1
90.1
88.5
89.4
GSM8K
Full 91.5
92.0
91.4
91.7
ponline pb 91.7
92.1
91.0
91.4
Table 4: Impact on different RLVR algorithms.
Model
Dataset
Random
Full
PPL
ENT
oonline pb
Qwen2.5-7B
DAPO-MATH-11K
27.1
34.0
33.0
32.1
33.8 (+6.7%)
(2-c) & (2-d). Our results show that with smaller ratios, such as 10%, the reasoning performance of the model decreases noteblely. For example, compared to Full, performance drops by about 1.8% (MATH) and 1.5% (GSM8K) the online setting, and by about 3.1% and 2.4% in the offline setting. This indicates that too little training data seriously impair RL training. On the other hand, increasing the ratio to 50% does not provide clear improvement and approaches the full data result. These findings highlight the importance of selecting proper query sampling strategy for RL-based reasoning training.
settings, shown in Figures
Different RLVR Algorithms. Our main experiments adopt GRPO the default RLVR algorithm. Here, we further assess whether consistency sampling benefits other RLVR algorithms under the online setting. Results in Table demonstrate that, with 30% sample ratio, consistency sampling reliably improves the reasoning performance of all tested algorithms, achieving results comparable Full.
Hard Training Data. We also increased the difficulty of the mathematical tasks by validating the importance of consistency sampling on the DAPO-MATH-17K datase shown in Table Even with 30% sampling ratio, we observed similar findings in the main experiments. This confirms that uncertainty consistency query selection is equally effective for more challenging mathematical tasks.
Selected by Highest Objective Uncertainty. the online setting, if we select the top p% samples with the highest objective uncertainty within batch, these samples are very likely to have rewards of zero under RLVR algorithms. According to the RLVR loss in Equation [ such samples will contribute no gradient signal, severely hindering model optimization. So we only consider it in offline setting. In the offline setting. we compare two selection strategies on MATH dataset: a) Selecting the top 30% samples with the highest objective uncertainty (Top 30% Hard); b) Selecting 30% samples according to the offline metric Tpb- The result is shown in Table
Table 5: Training on highest objective uncertainty dataset (Top Hard) vs uncertainty consistency dataset (rpb) in the offline setting.
Model
Tph
Top Hard
Qwen2.5-7B
72.1
68.3
Qwen2.5-3B
63.3
57.8
Llama3. Inst-lnstt
51.5
50.4
The result demonstrates that filtering samples solely based objective uncertainty will induce dif- ficulty imbalance in the training set, which can affect the model's generalization ability. Therefore,

---

Published as a conference paper at ICLR 2026
framework explicitly considers uncertainty consistency, rather than relying single notion of uncertainty.
5.4 DISCUSSION
We further analyzed how consistency sampling affects response length and policy entropy during RL training on the Qwen2.5-7B MATH task, as illustrated in Figures (2-a) & (2-b):
Response Length. We find that consistency samples help maintain the model's 's response length at a level comparable to RL with the full dataset. This suggests the difficulty of the selected samples matches that of the full dataset, while random sampling may pick samples that are too easy and less beneficial for improving reasoning ability.
Entropy. With full-data RL, the model's entropy drops rapidly and noticeably lower than other methods at the early stage of training. This phenomenon limits the model's exploration ability and restricts gains in reasoning ability, even with more data. In contrast, consistency sampling maintains higher entropy in the early phase, promoting better exploration and improving the sample efficiency of RL training.
CONCLUSION
This work mainly explores what kind of data is more valuable for RLVR training and how to achieve the reasoning effect of full-data RL with less data. Key findings are: 1) Classic active learning strate- gies underperform full-data RLVR because they ignore objective uncertainty, i.e., the probability that the model answers correctly. 2) We introduce offline uncertainty consistency metric, which is the point-biserial correlation between correctness and model perplexity; 3) Because of limited sampling and dynamically shifting output distributions, from normalized advantages and current subjective uncertainty 4) We prove that online metric is strictly negatively correlated with its offline counterpart and maximizing online uncertainty consis- tency is equivalent to maximizing decrease of sample uncertainty, providing principled selection criterion. Experiments show that selecting by low uncertainty consistency metric already surpasses random and classic AL baselines, while online selection with only 30% of the data matches or ex- ceeds the accuracy training on the full dataset. Our consistency-driven query selection thus offers a scalable path to data-efficient RL for complex reasoning tasks.
estimate the online uncertainty consistency metric
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. European Conference on Computer Vision, pp. 137-153. Springer, 2020.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in Ilms. arXiv preprint arXiv:2402. 2024.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34:8927- 8939, 2021.
Markus Bayer. Activellm: Large language model-based active learning for textual few-shot sce- narios. In Deep Learning in Textual Low-Data Regimes for Cybersecurity, pp. 89-112. Springer, 2025.
Yifang Chen, Shuohang Wang, Ziyi Yang, Hiteshi Sharma, Nikos Karampatziakis, Donghan Yu, Kevin Jamieson, Simon Shaolei Du, and Yelong Shen. Cost-effective proxy reward model con- struction with on-policy and active learning. arXiv preprint arXiv:2407.02 19, 2024.
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14755 2025.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Ros- tamizadeh, and Sanjiv Kumar. Batch active learning at scale. Advances in Neural Information Processing Systems, 34:1 1933-11944, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505. 226/7, 2025.
Diao, P Wang, Lin, and T Zhang. Active prompting with th-hout for large language models. arxiv. arXiv preprint arXiv:2302. 12246, 2023.
Li Du, Zhouhao Sun, Xiao Ding, Yixuan Ma. Yang Zhao, Kaitao Qiu, Ting Liu, and Bing Qin. Causal-guided active learning for debiasing large language models. arXiv preprint arXiv:2408. 12942, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yonatan Geifman and Ran Yl-va..v. Deep active learning over the long tail. arXiv preprint 17110:1711.00941, 2017.
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Fric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 1XXIv:2103303874 2021.
Jian Hu. Reinforce+:: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03260 2025.
11

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.08366 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
MacCallum. On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1):19-40, 2002.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. arXiv preprint arXiv:2402.081 08100, 14, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint urXiv:2402.09668 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1708. 00489, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2442..33300, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312./1800 2023.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International Joint Conference on Neural Networks (IJCNV), pp. 112-119, 2014. doi: 10. 20.1.004 6889457.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint urXiv:2506.01933 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. /2 122, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Jiaxin Zhang, Zhuohang Li, Kamalika Das, and Sricharan Kumar. Interactive multi-fidelity learning for cost-efffective adaptation of language model with sparse human supervision. Advances in Neural Information Processing Systems, 36:78261-78274, 2023.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/70 2025.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PROOF
A.1.1 PROOF FOR THEORAM 1
Theorem Negative Correlation between Tpb and pb For the same model TT, the covariance between Pbb and mo is less than zero, i.e., Cov(r pbs pobine) < O.
Proof. We first define r2o = -Ū, -b) KaKı K2
Since the variance 8K > 0, Cov ro
and Cov(r phs ph have the same sign. We now show that Cov pbs ripliiee 0. For clarity of exposition, we abstract the above theorem as follows:
Let U be random variable such that P(U > 1) = 1, and consider i.d K samples {u;)e-) Let P = pb - Cjtli we have Cov Q)Q) O. pb
p-nllie
K Σi Zi=l d: 2, where c,d, for any E [K]. Then,
First, expand the expectation of the product:
]
[[QQ]
Σ dj
1=1
K K
Σ
Edd
1/ug]-
i= )=
For: j. Elu, 1/u4]
For * J. if us and u, are independent samples from U, then
Elu; 1/ug] = B[uu] [[1/u,]-
So.
E[PQ] = Σ
qd + Σ
qd,
iml
in
K
cdde
Σ
c,E(uu]
Ed d,ElL/us))
c.d,E [./1/// [[1/u]]
The expectations of P and Q are
K E| Σ
K E[Q]
03 B [ull
Σ d
f=1
So,
K Σ j=1
E[
Σ
L /uy]
Subtracting, the covariance becomes:
Cov(P, Q) = E[PQ] - [[
K
Σ cd. Σ c.d.Blu, BdlBB/uBB///uu
i=l
So.
k Cov(P,Q) = Σ
c,dy (1 E|ug [((uu,))
13

---

Published as a conference paper at ICLR 2026
Now, because u, 1 almost surely, know E/u,] > 1 and E[1/u] < 1. More importantly, by
Jensen's ' inequality or the Souwwy-Schwarz inequality,
E Rlu F [u]] 2
with equality only if u, is constant. Therefore,
1 - E/u,/E[//uu] <
and, since D c, > 0, it follows that
Cov(P. Q) < 0.
So. Cry popline)
pb
< O.
A.1.2 PROOF FOR THEOREM
Theorem
2 (Equivalent between Maximizing Decrease in Sample Uncertainty and Maximizing ronkie). Suppose Ox; = Σn U denotes the subjective uncertainty for sample x. Under pb
sample gradient orthogonality assumption (Assumption D and bounded gradient norm assumption (Assumption 2 in one optimization step of an on-policy RL algorithm (e.g., GRPO), selecting sam- ples in the minibatch with largest pb
ooliie can maximize the decrease sample uncertainty.
First, assumption 2 common and mild. Now explain why assumption practice. We use Qwen2.5-0.5B to perform eight inference runs on the same question. For each response, compute the sample derivative of the uncertainty with respect the parameters and the normalized gradient inner products for each pair of derivative. As Figure Shown,, apart from the diagonal, the gradient inner products between different responses are close zero. This indicates that, although the samples are generated for the same question, the uncertainty gradients in the high- dimensional parameter space remain approximately orthogonal. Next, we begin to prove Theorem
reasonable in
2
8 gradient inner product
1.00
0.08
0.06
0.07
0.09
0.12
0.08
0.06
0.08
1.00
0.00
0.10
-0.01
0.08
0.08
0.06
0.06
0.00
1.00
0.06
0.06
0.11
0.06
0.08
0.07
0.10
0.06
1.00
0.02
0.07
0.06
0.04
0.09
-0.01
0.06
0.02
1.00
0.05
0.05
0.11
0.12
0.08
0.11
0.07
0.05
1.00
0.02
0.03
0.08
0.08
0.06
0.06
0.05
0.02
1.00
0.07
0.2
0.06
0.06
0.08
0.04
0.11
0.03
0.07
1.00
Figure 4: Gradient inner product heatmap.
14

---

Published as a conference paper at ICLR 2026
Proof. For any query x, we have K responses {1/ Ji=1 and set RL algorithm (e.g., GRPO), the loss is defined as follows:
Ar2 m And according to on on-policy
K
Mi
C
log Ne (x,
K
2/1
K
T821 Ia log (1x
A, In
K
1
A
In
Equation
K
Taking the derivative of the above equation with respect to 0, we have:
A
VoL
VoU.
According to Gradient Decent, the parameters in the next step is:
0
where 77 is learning rate. the decrease in sample uncertainty after this update is:
(U U(x; 0¹) (x;
-n (x))
VoL
VoL
First-order Taylor Estimation
A
VOU!
VOU!
-
K
k
VOUi
VoU?>
K
7
V
Vo 1m
Assumption
K
A
A
1
Σ
U9
K
U
U.e
1,30
4,50
A
17
M2
Σ
Assumption
K
U
A
Å,co
Å
A
Σ
+
Σ
K
U
U
A
A.cu
vpm
2 online
pb
So, maximize the decrease in sample uncertainty is equivalent to maximize onliiee
pb
B MARGIN SCORE AND ENTROPY ESTIMATION
Similarly, we can use the Margin Score (MS) or Entropy (ENT) (Wang Shang 2014) to represent the subjective uncertainty of LLMs:
MS@ (
1
Tre (ym, ((2 Tre (rer(yacl(o))) (8)
(6)
(5)
10k
t=0
15

---

Published as a conference paper at ICLR 2026
10
ENT.
(9) R
H( H(11x000 (
Here, Vm.t and Vs,t denote the tokens with the highest and second second-highest probabilities at position in the current sequence, respectively. H/- y) represents the entropy at each position of the current sequence, A larger MS or a smaller ENT reflects greater subjective uncertainty of the LLMs.
C
TASK PROMPT
MATH Task Prompt
[User]
< <Question>>>
Let's think step by step and output the final answer within \)boxed{).
GSM8K Task Prompt
[User]
<Question>>
Letá think step by step and output the final answer after ####.
AskLLM (Sachdeva et al. 2024) Prompt
###
Question ###
Does the previous reasoning question demarcated within ### and ###lt contain infor- mative signal for reasoning reinforcement learning training ? An informative datapoint should be well-formatted, contain some usable knowledge of the world, and strictly NOT have any harmful, racist, sexist, etc. content. This reasoning question should have clear answer, and you should consider it solvable for you, while also ensuring that it is not an overly simple question.
OPTIONS:
yes
no
D WARM UP EXPERIMENT SETUP
In Section we assess classic AL strategies in the offline reasoning RL setting. We check Full, Random uncertainty-basee methods (PPL, Entropy (Wang Shang, 2014)), feature-sapce cov- erage methods (K-means, K-center (Sener & Savarese, 2017)) and LLM prompting base meth- ods (AskLLM (Sachdeva et al. 2024)) using GRPO on Qwen2.5-0.5B (Qwen et al., (025) in MATH (Hendrycks et al. 2021) dataset. The detailed experimental setup is as follows:
Full: Training conducted on the entire 7,500 samples from the MATH dataset.
Random: 10% of the training data are randomly sampled for training.
PPL: Training samples are selected as the top 10% of the training data with the highest average perplexity (Equation B)
Entropy: Training samples are selected as the top 10% of the training data with the highest average entropy (Equation) 7
16

---

Published as a conference paper at ICLR 2026
K- K-means: After obtaining vector representations from the Qwen3- S-we33-B--Emen 1----mmedinng (Zhang et al. 2025) model for each query, k-means++ (Arthur & Vassilvitskii 2006) initialization and k-means clustering applied, and 10% of the samples are unitormly drawn from each cluster for training.
K-center: After obtaining vector representations from the Qwen2-88-Embedding
model (Zhang et al. (025). the K-center (Sener & Savarese 2017) algorithm is applied to select the top 10% of samples for training.
Using the AskLLM prompt (See Appendix C the probability of the token "yes" appearing in the model's s response is recorded, and the top 10% of samples with the highest "yes" probability are selected for training.
AskLLM:
All the hyperparameters in the warm up experiment is the same as those main experiment (See Section Besides, every experiment is conduct times using different random seed and we report its mean and standard deviation of greedy accuracy on 5000 MATH test dataset.
E THE USE OF LARGE LANGUAGE MODELS (LLMs)
The Abstract, Introduction, and Method sections of the paper are polished with the assistance of large language model.
17


Reviews:
[{"review_id": "PFz6TQAEnc", "reviewer": "Reviewer_eynX", "summary": "The paper introduces a new query selection strategy for Reinforcement Learning with Verifiable Rewards (RLVR) that allows training mathematical reasoning models using far fewer queries, without sacrificing performance. The key insight from the paper is that not all queries are equally informative. Standard active learning methods often pick samples with high subjective uncertainty (e.g., high perplexity), but this fails in RL reasoning because these samples frequently produce unstable or high-variance gradients, which hurts training stability. To address this problem, the paper proposes selecting samples where Subjective uncertainty (model confidence) and Objective uncertainty (whether the answer is correct) are consistent. They define it as “uncertainty-consistent” samples. The Experimental results show that using only 30% of the training data, the model reaches the same or better performance on the reasoning tasks compared to the full datasets RLVR training.", "questions": "Do the uncertainty consistency proposed in this paper still hold in open-ended, multi-step reasoning tasks where correctness is subjective or less binary (e.g., instruction following, safety, dialogue)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper identifies a practical limitation in current RL-based reasoning training pipelines: query selection methods that rely solely on subjective uncertainty (e.g., perplexity) often select examples that are uncertain but uninformative, leading to unstable gradients and inefficient learning. This motivation is clearly articulated and supported by empirical evidence. This insight is both intuitive and impactful—valuable training samples are those where uncertainty meaningfully reflects correctness, rather than those that are merely hard. The theoretical analysis establishing their negative correlation and training benefit is clearly developed and strengthens the contribution.\nThe method achieves full-dataset performance using only ~30% of the training data, while maintaining or improving generalization on standard math reasoning benchmarks. This is a practically meaningful result, especially given the rising cost of RL-based reasoning training.", "weaknesses": "The consistency metric assumes that the examples used for selection reflect the distributions during RL optimization. If the underlying data distribution shifts over training (which is common in RLVR), the effectiveness of selection may degrade unless the scoring is frequently recomputed.\n\nThe evaluation is rather limited to only Math reasoning. For query selection methods, it would be great to draw broader insights on whether the methods can be generalized beyond Math reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RrlALOmAQp", "reviewer": "Reviewer_ntf7", "summary": "This paper investigates query selection strategies for reinforcement learning from vision and reward (RLVR). The authors observe that standard active learning (AL) sampling methods often fail to outperform random selection in this context. To address this, they introduce an uncertainty consistency metric to guide sampling. In the offline setting, they use PBC (policy–behavior consistency) to measure alignment, while for online training—where estimating PBC directly is difficult—they propose a variant based on normalized advantage and subjective uncertainty. The paper also provides a theoretical analysis suggesting a negative correlation between offline and online PBC. Empirically, the proposed approach outperforms both random and classic AL baselines, reaching near–full-dataset performance using only 30% of the data.", "questions": "Please comment/justify the weakness part above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is well-motivated and relevant to current challenges in RLVR.\n2. The paper offers an interesting empirical observation that inconsistent samples can lead to extreme gradients, which explains why standard AL can underperform random sampling.\n3. The introduction of two alignment metrics—one for offline and one for online settings—is insightful, and the accompanying theoretical analysis provides some grounding.\n4. Experiments are extensive and demonstrate strong results, achieving competitive performance with significantly fewer samples.", "weaknesses": "While the paper is promising, several points could benefit from deeper clarification or justification:\n\n1. The link between sample inconsistency and extreme gradient behavior is intuitively explained but lacks theoretical support or formal analysis.\n2. It is unclear why the offline setting cannot also leverage the online metric $r_{pb}^{online}$, which appears to yield stronger performance in experiments.\n3. In some cases, training on the full dataset leads to worse results than using only 30% of the data; the paper should provide more discussion or intuition for why this happens.\n\nTypo: In line 451, “Table ??” is not rendering correctly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dla5DNNj85", "reviewer": "Reviewer_4djn", "summary": "This work introduces the active learning (AL) process with a new acquisition metric into reinforcement learning with verifiable reward (RLVR) for large language models (LLMs). They first confirm the query selection's impact on the stability of the gradients during fine-tuning via RLVL and reveal that the classic AL cannot select the most informative queries as well as random sampling. After assessing the inconsistent between an LLM's output with the lowest probability and a reward model's evaluation of the samples' accuracy effects on the gradient norm, they propose the smallest inconsistent metric $r_{pb}$ by the Point-Biserial Correlation Coefficient (PBC). Moreover, they extend the idea to online training. The experimental results show that the AL with an inconsistent metric could enhance both offline and online training as well as different RL algorithms.", "questions": "1. While you mention that *Because the calculation of $r_{pb}$ relies on a large number of samples $K$, ...* in Sec 4.2, your experimental settings of $K = 8$ in Sec 5.1 seems not large. Could you give stronger motivations for using Online Query Selection? For example, the **Model update** is the key point to address.\n2. Following 1., what are the key components in Online Query Selection for addressing sampling distribution shift in **Model update**?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The story (writing) is good to easily follow the authors' idea and refresh the utilization of the AL for the emerging field.\n2. This paper highlights that the importance of the query selection metric should not only rely on the LLM itself but also require consistency with the reward model's evaluation.", "weaknesses": "1. **My concern about using uncertainty.** After reviewing the Eqs. (2) and (3), IIUC, the definition of the subjective uncertainty is the low average probability of a policy model's responses $\\log \\pi_\\mathrm{ref}(y_{k, t}^{(i)} \\vert x^{(i)}, y_{k, <t}^{(i)})$ and the objective uncertainty is the low accuracy of a reward model's evaluation of a model's response, respectively. However, why can we call these two terms uncertainty? For example, if an LLM's response gives a response with lower probability but a 'consistent response' for the same (or similar) prompt $x^{(i)}$, could we still say the sample is uncertain?\n2. Follow 1., I feel that the metric of these terms is more like 'difficulty of reasoning the sample $x^{(i)}$', i.e., the degree of the probability that LLM can give the response (reasoning) and get the high reward (correct answer) for a sample. If so, what you check is the consistency between an LLM's output and the reward model's output.\n3. **My concern about the comparison with other AL methods.** Follow 2, if the core idea is evaluating a degree of the informative sample requires both LLMs and reward models, the proposed comparison with Entropy, K-center, K-means, and AskLLM might be insufficient, which only considers the LLMs' response and ignores the reward models' evaluation. To strengthen it, I suggest that the authors consider adding an alternative ablation study on the 'uncertainty' of the reward models.\n4. **Make a consistent symbol for equations.** For example, Eq. (1) uses $y_i$ but Eq. (3) uses $y_k^{(i)}$, the index of the sample and the index of generations should be differentiated.\n5. In Figure 1, the authors present that the inconsistent sample would give high gradient norm dynamics, i.e., these samples would cause gradient instability. However, the (degree of) impact of these gradient instabilities on the final performance is unclear. To highlight the gradient instability would be a significant issue for RLVR, I suggest that the authors provide some illustrations or examples of this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wzht3e2h5d", "reviewer": "Reviewer_KvRw", "summary": "The authors are proposing to use Active Learning (AL) to mitigate the large sample requirements of Reinforcement Learning with Verifiable Reward (RLVR) used in the LLM alignment process. \nThey find, that samples where the predicted model uncertainty diverges from the actual accuracy on the sample (measured by sampling K outputs and computing the reward) are detremental to model convergence in RLVR.\nThe authors propose an AL score to find samples with high alignment of uncertainty and accuracy and use this score to sample a subset of the full training data (offline setting).\nSince this proposed score is difficult to obtain in the online setting, the authors also propose an approximation of their score and demonstrate its theoretical properties.\nBoth proposed methods show strong empirical results on 3 models and 2 math datasets.", "questions": "- Impact of higher gammas (ablation study): Higher gamma values mean a stronger deterrence of predicted negative advantage in Eq. 5. Does this mean, we focus on samples the model knows to improve upon (positive advantage for some output y)? If so, does this not direcetly counteract exploration behaviours in the RL training? Would this be comparable to directly influencing exploration/exploitation ratios in the training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Very streamlined description of research gap, related work and proposed method\n- Excellent line of reasoning:\n\t1. Preliminary experiment to show shortcomings of existing solutions to the problem\n\t2. Starting with a simplyfied setting (offline RL) and showing a principled advantage of the proposed method\n\t3. Generalizing to realistic settings (online RL) by finding an approximation of their method that has theoretical guarantees\n\t4. Demonstrating strong performance of the approximation on 3 models and 2 datasets\n\t5. Providing ablation studies on the additional properties of their method\n- Providing believable evidence for the non-standard assumption of \"Sample Gradient Orthogonality\"\n- No additional computational overhead, as sampling K inferences for each x is already part of RLVR", "weaknesses": "- the authors claim \"(non-)significant lifts\" multiple times (line 78,88,373,381), but do not provide hard evidence for this claim in the form of standard deviations of results, critical difference diagrams or p-tests. We urge the authors to either provide one of these metrics in the appendix, or use a less mathematically loaded term instead of \"significant\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)