Sample Lottery: Unsupervised Discovery of Critical Instances for LLM Reasoning
Authors: Zhiping Xiao
Yusheng Zhao
Qixin Zhang
Jiaye Xie
Wanjia Zhao
Weizhi Zhang
Xiao Luo
Philip S. Yu
Ming Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reinforcement Learning with Verifiable Reward
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with Verifiable Reward (RLVR) has equipped large language models (LLMs) with the capability of reasoning over complicated logical problems through policy optimization. However, conventional methods require complete annotation of the entire dataset and allocate computation uniformly over all samples. We articulate the lottery sample hypothesis in policy optimization of LLMs: a large training set contains a small subset that, when trained alone, yields performance comparable to that of the full dataset. This paper therefore explores the following question: How can we identify these lottery-winning samples from the original dataset without access to answers? Unlike prior efforts that analyze the effect of different samples in the training set with complete annotation, this paper focuses on the unsupervised discovery of critical instances for LLM reasoning and proposes a novel framework termed Complementary Conformal Selection (CONST). Specifically, CONST evaluates the importance of samples by considering two complementary components: procedural volatility and outcome volatility. Procedural volatility measures the potential variations during the LLM’s reasoning process, while outcome volatility captures inconsistencies in the final answer. Subsequently, conformal prediction is used to obtain a prediction set whose cardinality serves as the criterion for selecting the lottery-winning samples for annotation. We also provide a theoretical analysis, showing that CONST can effectively approximate the optimal policy. Extensive experiments on various LLMs across different datasets demonstrate the effectiveness of CONST.

Published as a conference paper at ICLR 2026
SAMPLE LOTTERY: UNSUPERVISED DISCOVERY OF CRITICAL INSTANCES IN RLVR OF LLMs
Zhiping Xiao"; Yusheng Zhao?: Qixin Zhang", Jiaye Xie? Wanjia Zhao",
Weizhi Zhang'. Xiao Luo Philip S. Yu Ming Zhanga
Paul G. Allen School of Computer Science and Engineering, University of Washington
2 State Key Laboratory for Multimedia Information Processing.
School of Computer Science, PKU-Anker LLM Lab, Peking University
3 College of Computing and Data Science Nanyang Technological University
4 Department of Computer Science, Stanford University,
5 Department of Computer Science, University of Illinois Chicago,
6 Department of Statistics, University of Wisconsin-Madison
patxiao@uw. edu, wwt.eeegw..gaagggggooooc..aa.... ott.://eeaaaaaccccoooccccciaaaaa
com
Syyeahofficial@mmann com, wan wwn.eaa.aaaaaahttttttaooordd [wzhan42, psyu) edu,
edu, xiao www.aa....w.scc....ooo mzhang edu.cn
ABSTRACT
Reinforcement Learning with Verifiable Reward (RLVR) has equipped large lan- guage models (LLMs) with the capability of reasoning over complicated logical problems through policy optimization. However, conventional methods require complete annotation of the entire dataset and allocate computation resources uni- formly over all samples. We articulate the lottery sample hypothesis in policy optimization of LLMs: a large training set contains a small subset that, when trained alone, yields performance comparable that of the full dataset. This pa- per therefore explores the following question: How can we identify these lottery- winning samples from the original dataset without access answers? Unlike those prior efforts that analyze the effect of different samples in the training set with complete annotation, this paper focuses the unsupervised discovery of critical instances for LLM reasoning and proposes a novel framework termed Complementary Conformal Selection (CONST). Specifically, CONST evaluates the importance of samples by considering two complementary components: procedu- ral volatility and outcome volatility. Procedural volatility measures the potential variations during the LLM's reasoning process, while outcome volatility captures inconsistencies the final answer. Subsequently, conformal prediction used to obtain prediction set whose cardinality serves the criterion for selecting the lottery-winning samples for annotation. We also provide a thcoretical analysis, showing that CONST can effectively approximate the optimal policy. Extensive experiments on various LLMs across different datasets demonstrate that CONST is at annotation-efficcent, tigh-peeforming and model-agnostee The code is available com/ Yusheng SampleLoterry.
INTRODUCTION
Reinforcement learning (RL) has recently become an essential tool for post-training large lan- guage models (LLMs) (Anil et al., 2023; OpenAI, 2024; 2025; Guo et al., 2025; Du et al., 2025). Pol- icy optimization algorithms (Schulman et al., 2017: Shao et al., 2024) have significantly enhanced the logical reasoning capabilities of LLMS (DeepMind, 2024; Wang et al., 2025a; Ren al., 2025). For logical problems, directly verifiable answers provide straightforward rewards for reinforcement learning, enabling effective outcome supervision of L.LMs. This approach, known as reinforcement learning with verifiable reward (RLVR) (Gao et al., 2024; Lambert et al., 2024), is commonly im-
"Equal contribution.
Corresponning authors.

---

Published as a conference paper at ICLR 2026
Suppose that (og) is sequence numbers sariefying Mera Zide " and that lly lig = 128: What is ux?
Questions
Ophimization
Tedious Annotation Over
Uniform Allocation of Resources
Entire Datoset
(a) Conventional Apereaches
Comporeéble Performance
Unlobeled Dataset
Complementory
Questions
Answers
Optimization
Conformal Selechion
Minimal Annotation Effort
Concentrated Computation
(b) The Proposed Paradign
Figure I: Conventional approaches require tedious full annotation over the entire dataset and allocate computation resources uniformly across the training set. By comparison, this work selects lottery- winning samples from the training set in an unsupervised manner, and then optimizes the model using these critical instances only, achicving comparable performance.
plemented using algorithms such as Group Relative Policy Optimization (GRPO) (Shao al., 2024) and its variants (Liu et al., 2025d; Chen et al., 2025; Pang & Jin, 2025; Zhang et al., 2025b).
Despite the significant improvement, conventional approaches (Hao et al., 2025; Wu et al., 2025; Di et al., 2025) demand full annotation over the entire dataset for verification, and often allocate computation resources uniformly across the full dataset. Nevertheless, some recent studies in RLVR LLMs (Chen et al., 2025; Wang et al., 2025c; Vanlioglu, 2025) and many prior works on the more general field of data subset selection and valuation (Paul et al., 2021; Das et al., 202 lb; Guo et al., 2022: Das et al., 2024) suggest that the instances in the training set are not equally important, and that training on a small subset may also lead to satisfactory results (Wang et al., 2025c). Based on these findings, we articulate the lottery sample hypothesis in RLVR of LLMss:
A large training set for RLVR on LLMs contains small subset that, when trained alone, can achieve performance comparable to that of the full dataset.
With this hypothesis, it is possible to break conventional approaches from two aspects (as illustrated in Figure 1): (i) full annotation of the dataset no longer required, and ground truth answers of sev- eral lottery-winning samples sufficient; (ii) computation can be concentrated on several critical instances. Therefore, this paper explores a central question of this new paradigm:
How can we find the critical instances (the lottery-winiing samples) for RLVR on LLMs from the original training set without annotation?
To answer this question, this paper proposes a novel framework named Complementary Conformal Selection (CONST) for the unsupervised discovery of critical instances in the training set. CONST evaluates the value of each instance from two complementary perspectives: procedural volatility and outcome volatility Procedural volatility assesses potential variations in reasoning chains by ex- amining how different segments of reasoning affect the final answer. Outcome volatility measures inconsistencies in the final answers produced by different reasoning paths. Both yield multisets (i.e. sets that allow duplicating elements) of results, which are then merged and fed into a conformal pre- diction module. The conformal prediction produces a prediction set, whose cardinality is used as the criterion for selecting boitery-winning samples. A theoretical analysis is also provided demonstrat- ing that CONST can effectively approximate the optimal policy. We conduct extensive experiments across datasets with various LLMs, showing that CONST outperforms various baselines and enables comparable performance with < 0.5% of the samples. Our contribution summarized as follows:
New Perspective: We present probabilistic perspective for the unsupervised identification of critical instances in the full dataset for further annotation and RLVR optimization on LLMs, en- abling an unnoaaioon-iinime
data-efficient and performance-competinee optimization procedure compared with training on the entire fully annotated dataset.
Novel Methodology with Theoretical Analysis: We propose CONST, a probabilistic approach based on conformal prediction that considers both procedural volatility and outcome volatility in
2

---

Published as a conference paper at ICLR 2026
LLM reasoning. select Nottery-winninn samples for annotation and optimization. Notably, provide rigorous theoretical analysis of CONST demonstrating that it can effectively approximate the optimal policy parameter setup under the lottery sample hypothesis.
Empirical Validation: We conduct extensive experiments across four mathematical datasets on various LLMs against competing baselines, showing that CONST is (1) annotation-efficient, achieving similar performance to the full dataset with < 0.5% of the annotation, (2) high- performing, outperforming competitive baselines by 10.97% on average, and (3) model- agnostic, showing consistent improvement across three different architectures.
2 PRELIMINARIES
Problem Definition. Given training set of questions Q X2......... XN} from the input space X. conventional RLVR approaches first annotate the dataset with ground truth answers = (Yı, .2...., Yx} with RL algorithms (e.g., GRPO), i.e., x= *(TT, A,, where π F is the policy optimized with full data annotation and ( is the optimization process of RLVR. Our goal to find subset of Q with budget b, i.e., Q' C and Q' = b. and then annotate the selected data with answers A' so that the optimized policy P + (o Q A') achieves comparable performance to .f.
in the output space V and then optimize the original LLM (i.e., the policy) To
Reinforcement Learning with Verifiable Reward. During the training process of reinforcement learning with verifiable reward, the LLM generates list of outputs {0,, Oz...., On, for question X, and the outputs are verified against the ground truth answer Y to obtain the rewards 12,72.... : where correct answers receive 1 and incorrect ones In the widely- andely-adopted group relative policy optimization (Shao et al., 2024; Liu et al., 2025c; Guo et al., 2025), the advantage function of each output O computed as:
a mean((ry)
std((r)))))
With this, the GRPO optimization objective can be formulated as follows:
22
min (T( To. (O, di, clip
(O,l)) To (O,
0: )
LGRPO -BoinT
E, 1 +
(2)
+DLL (Tol(T)))
where Hy is the current policy, Ter is the old policy, and To is the reference policy.
Conformal Prediction. Conformal prediction (CP) (Vovk al., 2005; Papadopoulos et al., 2007; Angelopoulos et al., 2023; Barber et al., 2023; Su et al., 2024) is a model-agnostic solution to obtain prediction sets (or intervals in regression tasks) that are mathematically guaranteed cover the ground truth answer with high probabilities. Concretely, given an input from X, prediction from ) and the prediction model x XX) a scoring function f X x y + R defined quantify the disagreement between the input and the predicted answer. The scoring function is applied to calibration set Deal = ((Xçal ) which independently and identically distributed as the dataset under consideration, to obtain the calibration scores f (Xça), y,al), 1, 1. m. For [(++)(1-a] quantile
given confidence level 1 a, threshold can be determined by taking the of these scores. For given input E X, and the predefined error rate a, the conformal prediction set is guaranteed to cover the correct answer with probability of 1 - defined as follows:
( = 1 (X, p}.
(3)
3 THE PROPOSED CONST
3.1 FRAMEWORK OVERVIEW
The overall idea of this work is to first select the critical instances Q' from the training set of ques- tions @ using the proposed Complementary Conformal Prediction (CONST), then annotate these crit- ical instances with answers A', and finally optimize the LLM using off-the-shelf RLVR algorithms
3

---

Published as a conference paper at ICLR 2026
Selection Scores
Passible
Suppose that d'sequence real numbers satisfying "a-2 20 uh and that uy and 16 128 What is us?
Question Tokens
Reasoning Tokens
Answer Tokens
Andwers
Ranking
(a) Procedural Volatillity
Labeling
Conf ormal Prediction
A''
Critical Samples
Unlabeled Dataset
Question
Dirferent Reasoning
Answer
Tokens
Paths
(b) Outcome Volatility
Prediction Set
Optimization
Figure 2: The overall framework of the proposed CONST. CONST selects critical samples via confor- mal prediction based on both procedural volatility and outcome volatility. The selected samples are then annotated with ground truth answers for standard RLVR optimization LLMs.
such GRPO. The proposed CONST evaluates the importance of each instance perspectives: procedural volatility and outcome volatility. Procedural volatility measures how the final answer to the input question changes when the reasoning paths are truncated at different stages, while outcome volatility captures the inconsistencies in the final answers produced by different rea- soning paths. Conformal prediction is then used to obtain a prediction set each instance, and cardinality of these sets serves the criterion for selecting the critical instances. We illustrate the overall framework in Figure
Q from two
3.2 PROCEDURAL VOLATILITY
Questions important for policy optimization are expected to induce reasoning trajectories of sig- nificant complexity, straightforward thinking chains are less likely substantially enhance the model's logical reasoning abilities. Therefore, we propose measuring the volatility in the reasoning process by truncating the thinking chains different stages and prompting the LLM provide the final answers directly. Simple and straightforward reasoning paths are more likely to yield consistent answers, whereas complex, intricate ones are more likely to exhibit volatility.
Specifically, given question X E Q. we first deterministically sample an output, denoted as O = [fi, f2. tL; Y] XX) where t1, t1,t2....,47 tt are tokens of the reasoning path and Y is the predicted answer typically enclosed in special formats like {box(]. We truncate the reasoning path at different stages obtain set of truncated reasoning trajectories, defined as follows:
T(X)= {14 12, 2.
1,2, np).
where n is the number of stages. Subsequently, we query the LLM with these truncated trajectories and ask the model to directly output single final answer for each truncated trajectory without rea- soning. Formally, we obtain a multiset (bag) for each sample E through response truncation and LLM re- re-querying:
= x
(5)
3.3 OUTCOME VOLATILITY
In addition to procedural volatility, we also consider variations in the final answers. During pol- icy optimization, diverse answers to question induce gradients from multiple directions, helping the model avoid pitfalls from different sources. To quantify this diversity, we introduce outcome
Different from sets, multisets (bags) allow repetition of elements, with each element associated with count of its appearance. For convenience, use | to denote the size of the multiset, defined as the sum of the counts of all elements, and denote the union of multiscts, where the counts of each element added.

---

Published as a conference paper at ICLR 2026
volatility, which evaluates the inconsistencies in the final answers (outcomes) produced by different possible reasoning trajectories sampled from the policy.
In particular, given the original policy (the LLM before RLVR training) wo and question X Q under considcration, we directly sample no outputs from the policy to obtain a multiset, i.e.,
B (X)
(x) = (1 i = 1.2........ no. P, (((1)
3.4 CONFORMAL PREDICTION
Procedural volatility and outcome volatility generate multisets for each instance, encompassing the possible answers of LLMs. We are now concerned with how many of these answers are likely to be the correct answer. To address this, we employ conformal prediction (Vovk et al., 2005; Su et al., 2024) theoretically grounded solution. The overall procedures of computing conformal prediction sets have been discussed in Section 2, and the following, we will present the design of the scoring function f and how the prediction sets (X), X E Q are obtained.
As discussed in Section 2, the scoring function (x, E R is designed to quantify the dis- agreement between the input question X and the final answer Y. In other words, when the model To XY is certain that Y is the correct answer given the input X, f (X, Y) will be low, and vice versa. Conformal prediction does not require the scoring function to have theoretical guarantees of the measurement of certainty, although a good measurement is preferred. For each input question X E Q and predicted answer Ý E ( (X) U (X), we compute the scoring function by comparing Y with other elements in B(X). Specifically, consistent predictions are natural sign of certainty (Wang et al., 2022), we use the negative frequency of Y in B(X), i.e.,
count (())
[NF(X, 9) (Y:
[B()]
where counts(x) (-) returns the number of elements in the multiset. Nevertheless, using negative frequency alone as the scoring function may cause the scores to be concentrated on certain values, and therefore, entropy is used for fine tieeggained measurement, defined as follows:
(B))))
- Exc(((())) (Yeq(Y'; (YX))( log freq((";
fem 8)
(8)
log [xx
log [B(X]
where sct(.) returns the non-repeating elements in the multiset. Finally, the negative frequency and entropic scores are combined to obtain the final scoring function:
fx. fNF(X, F) + X (x,,
(9)
where À is a hyperparameter balancing the two terms. With the scoring function, we then calibrate with a calibration set described in Section to obtain the threshold p. Note that when computing the calibration scores, if the correct answer Y does not appear in the final multiset ((xxaa)), we set the score (X y.al)) ) to +oc Thus, we obtain the conformal prediction set for each X E Q as:
(xa(( 18 E set(B(X)) f (X, s Ø).
(10)
3.5 MODEL OPTIMIZATION
The size of the conformal prediction set naturally measures how many of the answers that the model considers likely to be correct. larger size indicates richer and more effective optimization sig- nals associated with the correct answer during model training. Therefore, we use the size of the conformal prediction set as the criterion for selecting critical samples. Additionally, to encourage sample diversity, we cluster the set of questions Q into b groups Q1, Q2..... Qs before selecting the samples from each group. Formally, the selection process can be written as follows:
Q arg max (X)I 1.2,...,b XEQ.
(11)
With the selected set of questions Q' Q, we annotate this small subset to obtain its ground truth answers A'. Finally, we use the standard RLVR algorithm (i.e., GRPO), as described in Section 2, to optimize the model To with Q' and A' to obtain " as our final model. A summary of the execution pipeline of the proposed CONST is presented in Algorithm 1.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1:
The execution pipeline of CONST
Input: The set of questions Q, the original policy To, the calibration set Deal {(X;a), y Initialize Ca rell and zelli st empty lists do Calculate the score fn (X;",, Y(" with Eq. and append it to Ca Scoreliist
for iteration
// Calibrate the scoring function
end
Find the [(=++))()=)] quantile of CalscoreList a for iteration + to do
// Obtain the conformal prediction sets Calculate the multiset of possible answers B(XX,) according to Eq. 5 and Eq. 6
Calculate the score f (X,, Y.) for each P. E B(X.) with Eq. 9
Obtain the prediction set (X) with uusng Eq. 10 and append the size of SizeList
10 end
11 Cluster Q into b groups 21, . Q
12 Select the question with the largest size from each group to form o' Select critical samples
13 Annotate O' with ground truth answers A'
// Annotate several samples
14 Optimize with Q' and A'' to obtain the optimized model n² // Optimize the policy using RL
15 return x" as the optimized model
3.6 THRORETICAL ANALYSIS
Here, we aim to provide theoretical understanding of our proposed CONST under the lottery sam- ple hypothesis. Before going into the details, we first review the basic notions of ergodic Markov decision processes and mixing time (Puterman, 1990).
Definition 3.1. A Markov decision process M & (S,AP n) is ergodic if the induced Markov chain under any stationary policy admits unique stationary distribution Pac. Moreover, the under- lying Markov chain of an ergodic MDP is said to mix in time (mia(())
tmixe (c) min 2 0 max (max((Pr(Y E AA% = Poo (A)))) - #ES
<e),
ACS
where (Yo, YI..., Y.... denotes an induced Markov chain under any stationary policy and e >> 0.
Remark 3.1. When < 1/2, choosing a different only changes the mixing time up to constant factor (Levin & Peres, 2017) and so one often fixes = 1/4 and simply writes fmii Emix (1 /4).
With the concept of ergodic MDP, we next establish generalization bound for our proposed CONST under the lottery sample hypothesis. Before doing so, we first introduce some frequently used no- tations. Specifically, since CONST only utilizes a subset of training instances Q' C Q to optimize policy me where | follows:
So, for any subset C Q, we define the empirical GRPO loss on as
min (O,
(X) at clip (O,lx) To (O,
1- =, 14) + +D To).
n1|5
XES
where Yi Y--mean (rybes) is the advantage calculated based on relative rewards. With this std(r)))es)
symbol, we then present the approximation assumption about the chosen subset Q', that say,
Assumption 3.1 (Lottery Sample Hypothesis). Subset Q' is said to be an e-approximation of the full training set Q & {X1, X2...., ,,Nv} V (C V Lo (0) & e holls for any parameter vector 8 where the symbol l|z denotes norm and
Next, make some standard assumptions in optimization theory (Li et al., 2018; Yue al., 2023). 20 Assumption 3.2 (Smoothness). The objective Eo Ponro(() is L-smooth, that is, Lo (0) is differen- tiable and there exists constant L > 0 such that |/ L2 (x) V V < L|| ullz- Assumption 3.3 PPlyykkLoojssieed Condition). There exists a constant P such that 2 (L2 (0) Lonro < 2 where O'RPO A arg wrgminn L2 22 (

---

Published as a conference paper at ICLR 2026
Furthermore, we suppose, at line 14 of Algorithm 1, the policy parameter vector 0 is updated by stan- dard gradient descent, i.e., f _ 0x I V GRPO (0), where L denotes the smoothness parameter. With all these preparations, we can have the following generalization theorem:
Theorem 3.1 (Proof is deferred to Appendix A). Under Assumption 3. 1-3.3, if the underlying MDP M P,T, is ergodic with mixed time Emi and the gradient V CRPO is bounded, i.e., || E CRPO (0)ll S G. then the following inequality holds with probability greater than 8, that is,
+0( Immi je
+) In(
In(g) Nn(1-
(O)) CORPO(0") 5 4R(FoR) 4R(FFR))+())
Nn
(1 1 k++ (ORDO + 2
2G
2²
+ - 241'
where 0* arg ming (O), R(FGR) is the Rademacher complexity of the group-relative loss function space FGR. N denotes the size of full training set Q. n. is size of outputs for each question and R is upper bound of variance of the return (r, ( i.e.. Var (T:) S of v0.
Remark 3.2. Note that Rademacher complexity serves as a measure of the policy network's 's capacity to fit the training data, reflecting the richness of the function class it can represent. As result, under the Lottery Sample Hypothesis, Theorem 3.1 implies that with sufficiently large question dataset and verified rewards, our proposed CONST can effectively approximate the optimal policy parameter 0*,
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and Evaluation. During model training, we use the BigMath-sub dataset, randomly selected subset containing 2048 instances of the BigMath dataset (Albalak et al., 2025). During the test phase, we use four mathematical reasoning datasets widely adopted in RLVR evaluation, i.e., AMC 23 (problems & solutions, 2023), MinervaMath (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), and MATH500 (Hendrycks al., 2021c; Lightman et al., 2023), with details deferred to Appendix B.I. In the experiments, we report the avg0256 6 accuracy metric for the smaller AMC23 dataset, and avgê32 for other datasets following prior works (Zuo et al., 2025; Wang et al., 2025c). To reduce randomness in instance selection, we repeat the algorithms three times and report the average result. More details can be found in Appendix B.2.
Baselines. We compare the proposed CONST against various basclines, i.e., (1) NoFinetuning, which uses the original model for inference, (2) RandSelect, which annotates randomly selected instances for training, (3) active learning algorithms, including EntSampling (Settles, 1995), BADGE (Ash al., 2020), and CEC (Safaei Patel, 2025), and (4) peasoning-specifee selection strategies, includ- ing SCF (Wang et al., 2022) and EWS (Beygelzimer et al., 2009). More details about the baseline methods can be found in Appendix B.3 and Appendix C.3.
(Grattafiori et al., 2024), 5B (Guo et al., 2025), Qwen2.5-Mate 5-Math-1. and Owen2. 5-::2.55MMthe 5-atth-7B (Yang et al., 2024) for RLVR training. For the calibration set, we use 1024 instances randomly selected from BigMath (ensuring no overlap with Big.Math-sub), and to justify the robustness of CONST to the choice of the calibration set, MMLU (Hendrycks et al., 2021abb) is used as an alternative. For procedural volatility, we set the number of stages np to 20, and for outcome volatility, we no 20. In conformal prediction, we the error rate a 0.1 and À to 0.02. For the budget of annotation, we report the results of 4 and 8 instances. For the clustering step, we use Sentence-BERT (Reimers Gurevych, 2019) to obtain the embeddings of the input queries, and use the K-means algorithm to obtain the clusters. The number of clusters is set to b, which is the budget of annotation. For the RLVR optimization hyperparameter setup, we generally follow the training configuration of Wang et al. (2025c). By default, we set the maximum number of tokens to 8192 in training and 3072 in inference, the learning rate to x 10 10-6, the weight decay to 0.01, the hyperparameter of 3 in GRPO (Eq. 2) to 1 x 10-3, the batch size to 64, and 8 gradient updates for each rollout. We train the model for at most 500 iterations and evaluate the model every 20 iterations. During training, framework (Sheng et al., 2024) for RLVR training and inference. For the computation hardware, we use 4 NVIDIA H800 for both training and inference.
Implementation Details. We adopt 3.
duplicate the samples to occupy single batch. We use the VERL
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison with various baselines and training on the full dataset under the
avge 32 6 for AMC23) metric. We mark the best in bold and runner-ups with underline.
Datasets
AMC23
MinervaMath
OlympiadBench
MATH100
AVG
Budget
8
LLaMA 3.1 8D Inst ruct
NoFinctuning
18.03
14.37
12.28
35.79
20.12
RandSelect
19.48
18.34
16.77
13.54
13.87
39.02
39.03
22.20
22.32
EntSampling
20.42
19.70
16.66
18.12
13.43
13.14
37.29
36.33
21.95
21.82
BADGE
19.34
20.02
20.25
19.28
13:27
13.81
41.72
42.68
23.95
23.65
CEC
20.25
21.24
16.66
18,50
13.05
15.02
38.03
42.97
21.79
24.43
CONST
20.62
24.27
21.68
24.19
16.83
17.61
43,46
47.17
25.65
28.31
FullDataset
24.30
20.99
18.23
48.58
28.03
DeepSeek Disstill- QWen- 1. 5B
Nol'inetuning
30.94
14.17
17.68
50.03
28.21
RandSelect
40.96
54.55
19.28
23.31
25.02
33.68
64.29
72.96
37.39
46.
EntSampling
54.91
54.88
22.23
22.55
32.68
32.98
71.79
71.97
45.40
45.60
BADGE
42.84
50.13
21.39
23.20
27.62
30.76
67.73
71.04
39.90
43.78
CBC
49.75
51,46
21.88
22.14
30.50
31,50
69.96
71.44
43.02
44.14
CONST (ours)
55.84
59.16
23.17
23.66
33.66
34.90
73.61
74.84
46.57
48.14
FalaDataset
60.27
24.55
36,30
75.49
49.15
Owen2 5-Math-2
1.8
NoFinetuning
31.74
9.47
21.72
36.23
24.79
RandSelect
44.66
46.88
15.19
21.35
30.94
30.46
64.32
65.74
38.78
41.11
EntSampling
40.64
42.43
17.69
20.62
27.24
27.32
59.79
62.26
36.34
38.16
BADGE
44.61
46.73
19.91
20.97
28.87
29.54
63.69
64.88
39.27
40.53
CEC
42.42
45.32
13.67
21.24
28.00
30.88
59.11
68.87
35.80
41.58
CONST (ours)
47.19
48.42
24.54
32.05
32.75
67.68
69.58
41.73
43.82
FatlDataset
49.13
24.30
33.33
70.88
44.41
Owen2, -5
NoFinctuning
56.04
33.90
37.28
81.03
52.06
RandSelect
57.05
57.48
35.26
35,91
38.29
38.77
81.58
81.90
53.05
53.52
EntSampling
56.95
57.87
35,36
35.37
38.46
38.63
81.88
81.89
53.16
53.44
BADGE
$4.36
56.76
34.12
35.08
37.63
38.92
80.81
82.25
51.73
53.25
CEC
56.03
58.28
34.73
35.21
38.41
39.18
81.70
82.16
52.72
53.71
CONST (ours)
58.21
59.05
35.83
36.97
39.56
40.19
82.94
83.55
54,14
54,94
FullDataset
58.70
36.66
41.04
83.61
55.00
4.2
PERFORMANCE COMPARISON
We compare the proposed CONST with various baselines across different LLM architectures on four mathematical reasoning datasets with varying budget sizes, and report the avg032 32 (5vg@256) 2.6) metric in Table 1. According to the results, we make several observations (Obs.) listed as fol- lows: Obs. CONST significantly improves vanilla models across various scenarios, outperform- ing baselines. For example, using only 8 critical instances, CONST significantly improves vanilla LLaMA-3, 1- Instrnnt by 40.71% 71%, SeepSeek-R1-Dis-e RI--Distill--een---11 5B by 70.65%, and Qwen2. 5-Math-1. 1.. 5B by 76. 76%, surpassing competitive active learning baselines not de- signed for reinforcement learning on LLMs (e.g. BADGE and CEC). Obs. Training on critical instances discovered by CONST achieves comparable results to training on the full dataset. The results in Table demonstrate that with less than 0.5% of the samples selected by CONST an unsu- achieve very similar performance
pervised manner, in terms of avgêk accuracy for budget 8. This confirms the value of these lottery-winning samples and our method that discovers them without access ground truth answers.
average: less than 1.09% difference
4.3 ABLATION STUDIES
We then investigate how the different components or mechanisms in CONST affect the final perfor- mance. In particular, we design six variants of CONST, denoted as VI to V6. VI removes conformal prediction and randomly selects instances from each cluster. V2 skips clustering and chooses sam- ples with the largest conformal prediction sets. V3 uses entropy in Eq. 8 instead of conformal prediction sets select instances in each cluster. V4 explores alternative configurations in Eq. 11 by clustering instances into b/2 groups and selecting the top 2 items with the highest conformal
8

---

Published as a conference paper at ICLR 2026
Olympiad MANIlgo0
Avarage
AME23
Olympiad
Avergge
Calib-
BM Callbrg
Minerva
Minerva
MATHooo
MIMU
48
48
Accuracy (%6
50
Accuracy (36)
Accuracy (96
40
40
30-
32
32
24
24
10-
16
16
AMC23
Minerva
Olympiad
MATH$00
Average
10
20
30
10
20
25
30
R0
Figure 3: Left: performance under different numbers of stages (np). Middle: performance under different numbers samples (no). Right: robustness to different choices of the calibration set.
prediction sets. V5 removes procedural volatility, and V6 removes outcome volatility. We perform experiments on LLaMA-3. 1-1..88---- 1S9B-Innsuutt with the budget of 8, and the results in terms of avg@32 (avg@256) are shown in Table 2, from which we have the following observations: Obs. All components or mechanisms in CONST contribute to the final performance. As shown in the table, removing any of the proposed techniques consistently decreases accuracy across datasets, demonstrating the effectiveness of conformal prediction, proceduralloutcome volatility, and clus- tering. Obs.9 Conformal prediction plays an important role. The results show that replacing conformal prediction with alternatives such random selection entropy selection leads to severe performance degradation (i.e., 5% drop in terms absolute accuracy).
4.4 FURTHER ANALYSIS
Performance under Different Hy- perparameters. We then show the model's performance (in terms of avg@32/avgle (26) under different hy- perparameters: the number of stages (i.e. np) in procedural volatility (Eq. 4) and the number of samples (i.e., no) in out-
Table 2: Ablation study of CONST with the budget of 8.
Minerva Math
Olympiad Bench
MATH
Variants
AMC23
AVG
500
VI
20.87
16.89
13.76
40.53
23.01
V2
22.03
21.37
15.18
43.60
25.54
V3
19.98
17.64
14.79
40.79
23.30
V4
22.97
25.02
17.11
45.31
27.60
come volatility (Eq. 6). The experimental results on LLaMA-3. 1--B-IInstrit shown in Figure (Left and Middle).
V5
23.96
20.21
16.56
44.80
26.38
V6
21.16
16.34
14,50
40.16
23.04
CONST
24.27
24.19
17.61
47.17
28.31
From the results, we observe that: Obs.6 Both np and no achieve best performance at 20. The number of stages np controls the granularity of procedural volatility: when the granularity is too coarse (small s)s)) it may be difficult to capture the twists and turns in the reasoning trajectories; when the granularity small (large nps). it may frequently interrupt the logic fragments. On the other hand, no controls the balance of procedural volatility and outcome volatility in the multiset (XX,, which affects the scoring function and thus conformal prediction sets.
Robustness to Different Choice of Calibration Sets. CONST requires calibration set, and there- fore, we also investigate the method's robustness under different choices of the calibration sets. Specifically, we adopt three sets: (1) BM Calib-1, which is the original one: (2) BM Calib-2, which also sampled from the BigMath dataset with no overlap with the training set; (3) MMLU, which contains mathematical questions from the MMLU dataset. The results LLaMA are presented in Figure 3 (Right), and we observe that: Obs. The propused method is robust to the choice of calibration sets. Using calibration set with identical distributions the training set (i.e., BM Calib-/ and BM Calib-2) yields similar high accuracy on average. When using a calibration set with different distribution (i.e., MMLU), there is slight decrease, but the difference is marginal on average, showing that CONST is robust to the different choices of the calibration set. The results also indicate that is possible use existing datasets that are already annotated (e.g., MMLU) as the calibration set to avoid the need to annotate the calibration set.
5 RELATED WORKS
Reinforcement Learning for LLM Reasoning. LLMs and their transformer architecture have become promising solution in many fields, including multimodal understanding (He et al., 2021;

---

Published as a conference paper at ICLR 2026
Zhao al., 2022; 2025b; Huang et al., 2025), formal languages (Zhao et al., 2025e), time-series understanding (Zhao et al., 2025d), and graph data (Zhao et al., 2025h). Reinforcement learning (Sutton et al., 1999; Havrilla et al., 2024a; Wen et al., 2024; Liu et al., 2025a) has significantly enhanced the reasoning capabilities of L.MMs via rewards from verifiable answers (Shao et al., 2024; Mroueh, 2025; Wen al., 2025) or reward models (Dong et al., 2024; Setlur al., 2024; Qu et al., 2025). Early efforts (Sprueill et al., 2023; Deng et al., 2024; Wang et al., 2024) mainly focus on supervising the LLMs' reasoning process, often involving the value functions (Havrilla et al., 2024b; Zhai et al., 2025; Yuan et al.. 2025; Zhang et al., 2025a). More recently, outcome supervision, with the reward obtained from verifiable ground truth answers, has received increasing attention (Shao et al,, 2024; Liu et al., 2025b; Su et al., 2025; Liu et al., 2025c), due to its simplicity and the immunity from reward hacking (Gao et al., 2024; Fu et al., 2025; Miao et al., 2025).
Conformal Prediction. Conformal prediction (Vovk et al., 2005; Tibshirani et al., 2019; An- gelopoulos et al., 2023; Straitouri et al., 2023; Kiyani et al., 2024a; Gibbs et al., 2025) is a model- agnostic and distribution-free solution of uncertainty quantification (Stracuzzi et al., 2017: Wang et al., 2019; Psaros et al., 2023), with solid mathematical foundations (Fontana et al., 2023; An- gelopoulos et al., 2024). It generates prediction sets that contain the ground truth answer under predefined error rate. While most prior efforts focus on conformal prediction with smaller classi- fication regression models (Correia et al., 2024; Jeary et al., 2024; Cresswell al., 2024; Zhou et al., 2025), its adoption in natural language processing (Campos et al., 2024), and particularly LLMs (Cherian al., 2024; Kiyani et al., 2024b; Su et al., 2024; Mohri & Hashimoto, 2024; Wang et al., 2025b; Chankaev & Ilyushin, 2025), has received increasing attention. Compared to these prior works, this paper uses conformal prediction to guide the selection of critical samples.
Active Learning. Active learning (Cohn et al., 1994; 1996; Baram et al., 2004; Castro et al., 2008: Ren et al., 2021) aims to optimize deep learning models with limited annotation efforts. It is particularly useful when the ground truth answers can only be obtained with relatively high costs (Konyushkova et al., 2017; Yuan et al., 2023; Xiao et al., 2023; Chen et al., 2024), and facilitates other tasks like test-time learning (Zhao et al., 2025a:g; Zuo et al., 2025). With the success of LLMs, efforts have been made in both LLM for active learning. which uses LLMs for active an- notation (Margatina et al., 2023; Melo et al., 2024; Li et al., 2024; Kholodna et al., 2024; Ceravolo et al., 2024; Astorga et al., 2024; Xia et al., 2025), and active learning for LLMs, which adopts active learning for optimizing LLMs (Muldrew al., 2024; Sun et al., 2024; Zhang et al., 2024a; Hübotter et al., 2024; Zhang et al., 2024b). In this paper, we explore active learning to optimize the reasoning capability of LLMs with data-efficient and performance-competiivv reasoning framework.
Data Selection and Valuation. Data selection and valuation (Das et al., 2020; Paul et al., 2021; Wang & Jia, 2023; Das et al., 2024; Ebiele et al., 2025) aim find the most valuable data in the training set to save computational resources, which is also an important part of data-centric methods (Zhao et al., 2025f;c). For example, Paul et al. (Paul al., 2021) use the loss function and its gradients to select important examples very early in training. Guo et al. (Guo et al., 2022) provide comprehensive code library in addition to extensive evaluation for data subset selection and valuation. Das et al. (Das et al., 2024) propose CheckSelect, flexible, accurate, robust, and efficient method for extracting the high-value subsets. Nevertheless, many works on data subset selection and valuation assume complete annotation of the training set by computing the loss function or its gradients (Paul et al., 2021; Wang & Jia, 2023; Das et al., 2024), and conducted primarily in the domain of vision (Das et al., 2020; 2021a; Paul et al., 2021). By comparison, this work proposes CONST that aims to find important training data (the critical instances) without the annotation, and only annotate the important instances achieve annotation-efficient RLVR optimization of LLMss
6 CONCLUSION
This paper investigates an important question: how can we identify the lottery-winning samples from the original dataset without access to answers, thereby enabling an annotation minimal, data- efficient and performance-competiity alternative for optimizing the reasoning capability of LLMs. Through the design of our novel CONST framework, a probabilistic method grounded in the mathe- matical foundation of conformal prediction and incorporating complementary considerations of pro- cedural and outcome volatility, we demonstrate that the unsupervised discovery of critical instances in full datasets can achieve comparable performance with significantly less annotation efforts.
10

---

Published as a conference paper at ICLR 2026
ACKNOLEDGEMENT
Ming Zhang and Yusheng Zhao are supported by grants from the National Natural Science Founda- tion of China (NSFC Grant Number 62276002). The authors are grateful the anonymous review- ers for critically reading this article and for giving important suggestions to improve this article.
REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: large-scale, high-quality math dataset for reinforcement learning in language models, 2025.
Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and trends in machine learning, 16(4):494-591, 2023.
Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:241..1182/ 2024.
Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1188 2023.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Reprexentations, 2020.
Nicolás Astorga, Tennison Liu, Nabeel Seedat, and Mihaela van der Schaar. Active learning with Iims for partially observed and cost-aware scenarios. Advances in Neural Information Processing Systems, 37:20819 19-89992877 2024.
Yoram Baram, Ran El Yaniv, and Kobi Luz. Online choice of active learning algorithms. Journal of Machine Learning Research, ((Mar):255 2004.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816-844 2023.
Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning, pp. 49-56, 2009.
Margarida Campos, António Farinhas, Chrysoula Zerva, Mário AT Figueiredo, and André FT Mar- tins. Conformal prediction for natural language processing: A survey. Transactions of the Asso- ciation for Computational Linguistics, 12: 12:149711166. 2024.
Rui Castro, Charles Kalish, Robert Nowak, Ruichen Qian, Tim Rogers, and Jerry Zhu. Human active learning. Advances in neural information processing systems, 21, 2008.
Paolo Ceravolo, Fatemeh Mohammadi, and Marta Annamaria Tamborini. Active learning methodol- ogy in Ilms fine-tuning, In 2024 IEEE International Conference Cyber Security and Resilience (CSR), pp. 743-749. IEEE, 2024.
Shamil Chankaev and Eugene llyushin. Calibration of large language models based on the conformal prediction. International Journal of Open Information Technologies, 13(6):10-15, 15, 2025.
Jiayi Chen, Benteng Ma. Hengfei Cui, and Yong Xia. Think twice before selection: Federated evidential active learning for medical image analysis with domain shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11439-11449, 2024.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty- policy optimization. arXiv preprint arXiv:2505.12346, 2025.
John Cherian, Isaac Gibbs, and Emmanuel Candes. Large language model validity via enhanced conformal prediction methods. Advances in Neural Information Processing Systems, 37:1 14812- 114842, 2024.
11

---

Published as a conference paper at ICLR 2026
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):001-221, 1994.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129-144 145, 1996.
Alvaro Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. An information the- oretic perspective conformal prediction. Advances in Neural Information Processing Systems, 37: 17:100000-101041, 2024.
Jesse C Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. In International Conference on Machine Learning, pp. 9439-9457. PMLR, 2024.
Soumi Das, Sayan Mandal, Ashwin Bhoyar, Madhumita Bharde, Niloy Ganguly, Suparna Bhat- tacharya, and Sourangshu Bhattacharya. Multi-criteria online frame-subset selection for au- tonomous vehicle videos. Pattern Recognition Letters, 133:349-3555 355, 2020.
Soumi Das, Harikrishna Patibandla, Suparna Bhattacharya, Kshounis Bera, Niloy Ganguly, and Sourangshu Bhattacharya. Tmcoss: Thresholded multi-criteria online subset selection for data- efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Com-
puler Vision, pp. 6341-6350, 2021a.
Soumi Das, Arshdeep Singh, Saptarshi Chatterjee, Suparna Bhattacharya, and Sourangshu Bhat- tacharya. Finding high-value training data subset through differentiable convex programming. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 666-681. Springer, 2021b.
Soumi Das, Manasvi Sagarkar, Suparna Bhattacharya, and Sourangshu Bhattacharya. Checkselect: Online checkpoint selection for flexible, accurate, robust, and efficient data valuation. 1EEE Transactions on Artificial Intelligence, 2024.
DeepMind.
Ai achieves selver-medal standard solving international mathemati- cal olympiad problems.
httts:://eepmiinet google hwt.ggle/diiieeeeeee stleer///obbbo/////////oo/ sil-solve--o vemleei------- propploso-seeeee tetessat-ssiie sos-silvers -edlvee-mddallleeeelll 2024.
Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, and Weipeng Chen. From novice to expert: Llm agent policy optimization via step-wise reinforcement learning. arXiv preprint arXiv:2411.038/7, 2024.
Xinhan Di al. Enhancing math reasoning in small-sized Ilms via preview difficulty-aware inter- 2025.
vention. arXiv preprint
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2455.07865 2024.
Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chen- zhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.
2025.
Malick Ebicle, Malika Bendechache, and Rob Brennan. Quantitative data valuation methods: A systematic review and taxonomy. ACM Journal of Data and Information Quality, 17(2):1-39, 2025.
Xiequan Fan and Qi-Man Shao. Self-normalized cramér type moderate deviations for martingales and applications. Bernoulli, 31(1):130-161, 2025.
Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23, 2023.
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang. Qi Han, and Yanghua Xiao. Reward shap- ing mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.1877 2025.
12

---

Published as a conference paper at ICLR 2026
Jiaxuan Gao, Shusheng Xu, Wenjie Ye. Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115. 2024.
Isaac Gibbs, John J Cherian, and Emmanuel Candès. Conformal prediction with conditional guar- antees. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. qkaf008, 2025.
Alessandro Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXiv:240.221783, 2024. includes LLaMA-3.1-8FF -8--rurtuuct among other LLaMA 3.1 variants.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec- tion in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Yifan Hao, Fangning Chao, Yaqian Hao, Zhaojun Cui, Huan Bai, Haiyu Zhang, Yankai Liu, Chao Deng, and Junlan Feng. Jt-math: A multi-stage framework for advanced mathematical reasoning in large language models. arXiv preprint arXiv:2507. 19748, 2025.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yoe Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0400 2024a.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yie Yu, Maksym Zhuravin- skyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve Ilm reasoning via global and local refinements. arXiv preprint arXiv:2402. 10963, 2024b.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lews bilingual mul- timodal scientific problems, Xrxiv preprint arXiv:2402. 14008, 2024.
Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transre- fer3d: aadity-and-relatte aware transformer for fine-grained 3d visual grounding. Proceedings of the 29th ACM International Conference on Multimedia, pp.
2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv 2021c
preprint
Jinsheng Huang. Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, et al. Mmevalpro: Calibrating multimodal benchmarks towards trustworthy and efficient evaluation. In Proceedings the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, volume 1, pp. 4805-4822, 2025.
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. arXiv preprint arXiv:2410.08020 2024.
Linus Jeary, Tom Kuipers, Mehran Hosseini, and Nicola Paoletti. Verifiably robust conformal pre- diction. Advances in Neural Information Processing Systems, 37:4295-4314 2024.
13

---

Published as a conference paper at ICLR 2026
Nataliaa Kholodna, Sahib Julka, Mohammad Khodadadi, Muhammed Nurullah Gumus, and Michael Granitzer. Lims in the loop: Leveraging large language model annotations for active learning in low-resource languages. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 397-412. Springer, 2024.
Shayan Kiyani, George J Pappas, and Hamed Hassani. Conformal prediction with learned features. In International Conference on Machine Learning, pp. 14749-24766 PMLR, 2024a.
Shayan Kiyani, George Pappas, and Hamed Hassani. Length optimization in conformal prediction. Advances in Neural Information Processing Systems, 37:99519 19-99563, 2024b.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. Ad- vances in neural information processing systems, 30, 2017.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training, arXiv preprint arXiv:2411.15122 2024.
Guanghui Lan. First-order and stochastic optimization methods for machine learning, volume 1. Springer, 2020.
David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathe- matical Soc., 2017.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura. Ac- tive learning for abstractive text summarization via lim-determined curriculum and certainty gain maximization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 8959-8971, 2024.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in werrparameterizzed matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pp. 2-47. PMLR, 2018.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Xiaoqian Liu, Ke Wang, Yongbin Li, Yuchuan Wu, Wentao Ma, Aobo Kong, Fei Huang, Jianbin Jiao, and Junge Zhang. Epo: Explicit policy optimization for strategic reasoning in llms via reinforcement learning. arXiv preprint arXiv:2502. 12486, 2025a.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505. / 2025b.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503. 2025c.
Ziru Liu, Cheng Gong. Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning. arXiv preprint arXiv:2507. 10628, 2025d.
Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi- Yu. Active learning prin- ciples for in-context learning with large language models. Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 501 1-5034, 2023.
14

---

Published as a conference paper at ICLR 2026
Luckeciano C Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling in large language models. Advances in Neural Information Processing Systems, 11FF118002-1100000 2024.
Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The en- ergy loss phenomenon in rlhf: A new perspective on mitigating reward hacking. arXiv preprint arXiv:2501.19358, 2025.
Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge university press, 2017.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guar- antees. In Proceedings of the 41st International Conference on Machine Learning, pp. 36029- 36047, 2024.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.06633 2025.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. arXiv preprint 17Xiv:2402.08114. 2024.
OpenAI.
Learning to reason with llms. 2024.
https://openal..oom/mee
OpenAl.
Introducing openai o3 and 04-mini. 2025.
htrp:://ppeaal.cmm/nddee
Lei Pang and Ruinan Jin. On the theory and practice of grpo: A trajectory-correetted approach with fast convergence. arXiv preprint 2iti::2500.02833.3 2025.
Harris Papadopoulos, Volodya Vovk, and Alex Gammerman. Conformal prediction with neural networks. In 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2007), volume 2. pp. 388-395. IEEE, 2007.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 14:20596-20607 2021.
Ame problems and solutions. Art of problem solving. com/wt wi ki /e/iidddxxx////// hhtppptittleeooPP.rooeeeeeeedddd
https://attofppoobeemmllll
2023.
Apostolos Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncer- tainty quantification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physies, 477:1 1902, 2023.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang. Chang Liu, and Xiangyang Ji. Latent reward: Lim-empowered credit assignment in episodic reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 20095-20103, 2025.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.1 10084, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. peepseek-prover-eee Advancing formal mathematical rea- soning via reinforcement learning for subgoal decomposition. arXiv preprint arXiv:2504.2180. 2025.
15

---

Published as a conference paper at ICLR 2026
Bardia Safaei and Vishal M Patel. Active learning for vision-language models. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 19902-4122 IEEE, 2025.
John Schulman, Filip Wolski, Pratulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 11XX.v:77770044777 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alckh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08146, 2024.
Burr Settles. Active learning literature survey. Science, 10(3):377-33 1995.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory algo- rithms. Cambridge university press, 2014.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhí framework. arXiv preprint arXiv: 2409. 19256, 2024.
Henry W Sprueill, Carl Edwards, Mariefel V Olarte, Udishnu Sanyal, Heng Ji, and Sutanay Choud- hury. Monte carlo thought search: Large language model querying for complex scientific reason- ing in catalyst design. arXiv preprint arXiv:2310. 14420, 2023.
David John Stracuzzi, Maximillian Gene Chen, Michael Christopher Darling, Matthew Gregor Pe- terson, and Charlie Vollmer. Uncertainty quantification for machine learning. Technical report, Sandia National (SNL Albuquerque, NM (United States), 2017.
Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In International Conference Machine Learning, pp- #2633-32653 PMLR, 2023.
Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. Api is enough: Conformal prediction for large language models without logit-access. arXiv preprint 2rXiv:2403.01216, 2024.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding with verifiable rewards across diverse domains. arXiv preprint ar arXiv:2503. 23829, 2025.
Zhouhao Sun, LI DU, Xiao Ding, Yixuan Ma, Yang Zhao, Kaitao Qiu, Ting Liu, and Bing Qin. Causal-guider active learning for debiasing large language models. In Anmual Meeting of the Association for Computational Linguistics, 2024.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuro- science, 1(1):126-134) 1999.
Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal pre- diction under covariate shift. Advances in neural information processing systems, 32, 2019.
Ilya O Tolstikhin and Yevgeny Seldin. Pac- berobbyye--emmiiall--eeeseenin inequality. Advances in Neural Information Processing Systems, 26, 2013.
Abdullah Vanlioglu. Entropy-guided sequence weighting for efficient exploration in rl-based I1m fine-tuning. arXiv preprint MXXv:22003.22500 2025.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world.
Springer, 2005.
Bin Wang, Jie Lu, Zheng Yan, Huaishao Luo. Tianrui Li, Yu Zheng, and Guangquan Zhang. Deep uncertainty quantification: A machine learning approach for weather forecasting. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 2087-2095, 2019.
16

---

Published as a conference paper at ICLR 2026
Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying, Zekai Zhu, et al. Kimina-prov preview: Towards large formal reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 11354, 2025a.
Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang. and Yi Wu. Offline reinforcement learning for Ilm multi-step reasoning. ar preprint 0rXiv:2412.16145 2024.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference Artificial Intelligence and Statistics, pp. 6388 6421. PMLR, 2023.
Sean Wang, Yicheng Jiang, Yuxin Tang, Lu Cheng, and Hanjic Chen. Copu: Conformal prediction for uncertainty quantification in natural language generation. arXiv preprint arXiv:2502.12601 2025b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //171, 2022.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprini urXiv:2504.2057/ 2025c.
Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing Ilm agents via policy optimization with action decomposition. Advances in Neural Information Processing Systems, 37: 103774-103805, 2024.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXiv:2506. 14245, 2025.
Lixin Wu, Na Cai. Qiao Cheng, Jiachen Wang, and Yitao Duan. Confucius3-mathh A lightweight high-performance reasoning 1lm for chinese k-12 mathematics learning. arXiv preprint arXiv:2506. 10.55066.133... 18330, 2025.
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Demoncourt, et al. From selection to generation: survey of llm-based active learning. arXiv preprint arXiv:2.00.. 1767, 2025.
Ruixuan Xiao, Yiwen Dong, Junbo Zhao, Runze Wu, Minmin Lin, Gang Chen, and Haobo Wang. Freeal: Towards human-free active learning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14520-14535, 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.55math technical report: Toward mathematical ex- pert model via self- eefooimprovemett. arXiv preprint arXiv:2409.12122 2024.
Di Yuan, Xiaojun Chang, Qiao Liu, Yi Yang, Dehua Wang, Minglei Shu, Zhenyu He, and Guang- ming Shi. Active learning for deep visual tracking. IEEE Transactions on Neural Networks and Learning Systems, 33110):13244-133
2023.
Yurun Yuan, Fan Chen, Zeyu Jia, Alexander Rakhlin, and Tengyang Xie. Trajectory bellman residual minimization: A simple value-based method for I1m reasoning. arXiv preprint arXiv:2505.15311 2025.
Pengyun Yue, Cong Fang, and Zhouchen Lin. On the lower bound of minimizing polyak-lojasiewiez functions. In The Thirty Sixth Annual Conference on Learning Theory, pp. 2948-2968. PMLR, 2023.
Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang. Bo Ding, and Huaimin Wang. Enhancing decision-making for Ilm agents via step-level q-value models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 27161-27161 2025.
17

---

Published as a conference paper at ICLR 2026
Hongbo Zhang, Han Cui, Guangsheng Bao, Linyi Yang, Jun Wang, and Yue Zhang. Direct value optimization: Improving chain-of- thought reasoning in llms with refined values. arXiv preprint MrXii::5022.33233, 2025a.
Linhai Zhang, Jialong Wu, Deyu Zhou, and Guoqiang Xu. Star: Constraint lora with dynamic active learning for data-efficient fine-tuning of large language models. In Annual Meeting of the Association for Computational Linguistics, 2024a.
Xiaoying Zhang, Hao Sun, Yipeng Zhang. Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng- Critique-grpo: Advancing IIm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.00 0310,, 2025b.
Yifei Zhang, Bo Pan, Chen Ling, Yuntong Hu, and Liang Zhao. Elad: Explanation-gun guided large language models active distillation. arXiv preprint 4xX1v2242..10998 2024b.
Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang. Haibing Ren, Huaxia Xia, and Si Liu. Target-driven structured transformer planner for vision-language navigation. In Pro- ceedings of the 30th ACM International Conference on Multimedia, pp. 4194 4203, 2022.
Yusheng Zhao, Junyu Luo, Xiao Luo, Jinsheng Huang, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. Attention bootstrapping for multi-modal test-time adaptation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 39, pp. 2282222282222 2025a.
Yusheng Zhao, Junyu Luo, Xiao Luo, Weizhi Zhang, Zhiping Xiao, Wei Ju, Philip S Yu, and Ming Zhang. Multifaceted evaluation of audio-visual capability for milms: Effectiveness, efficiency, generalizability and robustness. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 1026-1041, 2025b.
Yusheng Zhao, Xiao Luo, Junyu Luo, Wei Ju, Zhonghui Gu, Zhiping Xiao, Xian-Sheng Hua, and Ming Zhang. Spot: Spectral optimal transport for graph domain generalization. ACM Transac- tions on Knowledge Discovery from Data, 20(1):1-22, 2025c.
Yusheng Zhao, Xiao Luo, Haomin Wen, Zhiping Xiao, Wei Ju, and Ming Zhang. Embracing large language models in traffic flow forecasting. In Findings of the Association for Computational Linguistics, ACI. 2025, Vienna, Austria, 2025d.
Yusheng Zhao, Xiao Luo, Weizhi Zhang, Wei Ju, Zhiping Xiao, Philip S Yu, and Ming Zhang. Marco: Meta-reflection with refseennnn for code reasoning. arXiv preprint arXiv:2505. 17481, 2025e.
Yusheng Zhao, Changhu Wang, Xiao Luo, Junyu Luo, Wei Ju, Zhiping Xiao, and Ming Zhang. Traci: deta-centric approach for multi-domain generalization on graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 13401-13409, 2025f.
Yusheng Zhao, Qixin Zhang, Xiao Luo, Junyu Luo, Wei Ju, Zhiping Xiao, and Ming Zhang. Test- time adaptation on graphs via adaptive subgraph-based selection and regularized prototypes. In Forty-second International Conference on Machine Learning. 2025g.
Yusheng Zhao, Qixin Zhang, Xiao Luo, Weizhi Zhang, Zhiping Xiao, Wei Ju. Philip S Yu, and Ming Zhang- Dynamic bundling with large language models for zero-shot inference on text- attributed graphs. The Thihh Annual Conference on Neural Information Processing Sys- tems, 2025h.
Xiaofan Zhou, Baiting Chen, Yu Gui, and Lu Cheng. Conformal prediction: A data perspective. ACM Computing Surveys, 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuckai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
18

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM 3.1
In this section, we prove our Theorem 3.1 in two steps. At first, in subsection A.1, we analyze the convergence standard gradient descent, ie., 0x+1 _ 8x - 7 G where v) denotes the step size and t (0) represents the empirical GRPO loss function utilized by CONST method. Subsequently, in Subsection A.2, we establish the generalization upper bound between the expected return (0) A E [R(T)) and the empirical return, i.e., -20 LGRPO
A.1 CONVERGENCE OF GRADIENT DESCENT
Lemma A.1. Under Assumption 3.1-3.3, if the gradient V C bounded, i.e. [ & G, then the Gradient Descent algorithm with a constant step-size L. that is,
Ok+1
O2
V
eo'
Caro (0):
L
has linear convergence rate. We have
2G e² (
C9 (PGROO) S (1
4 22
2p
where URRO arg ming (O).
Proof. According to the L-smoothness (Lan, 2020), we have that
Lo
(0x+1 v
22 LORPO + (V CERPO (0k), Ok+1 - 0x) + 2 10++1 - 0x
CooO (0s)
1 I
V 22' (
(0))
+ 2L
Co' G (0x)1
where the final equality follows from 0x+1
02 1 v
GRPO (0).
Next, we show that
(V CeRrO (Ox). v &o (0) (V (0+), v V (0)) (V CRPO (0x), V V V V (LCrro( (0))
2
|/V CORPO (0x) 112 - || CERPO I22 C (0) -
2 |V CERo (0x) - eG,
where the final inequality from the Assumption 3.1 and boundedness. Morcover,
M
IV GRPO (0x)
L GRO (0+) 1/ + 2(V Co
@ORPO (0x), Vo GRPO (0x) V VL & (0)) + ||V GRPO (0x) V L CORO 10
& I/V E (1)) 112 + 2G. + +
where the final inequality from the Assumption 3.1 and boundedness.
As result, we have that
2G
à
CERP 28 (0s)
GRPO ( (1+))12+
c
2L
L
2L
2G
2
< L (0x)
(CoRPo (0x) -200-2o CORPO
L
21
where the final incquality follows from Assumption 3.3. Finally, we have
2G 6²
11 1. (CCrro) - CeReoOOORRP
(1
+ +
2L
....
1)tt1 (Co (
2G
?
Σ By
(1
I
E
2L
j=0
M Mytt1 (2O (00)
2G
< (1
+
+
I
2p
19

---

Published as a conference paper at ICLR 2026
where the final inequality follows from Σ)=o(1 F)" p
A.2 GENERALIZATON OF GRPO
Lemma A.2. 1he underlying MDP M - (S. A. show that, for any 8 (0,1) 1), with probability 6, the following inequality holds, that is,
Prr,7) is ergodic with mixed time Emizs then we
sup CRPO ()) - LaRPO(0) S 2R(FoR) 2R(FOR) o In()) 0E6
In(
Nn
Nn(1-
where è denotes the parameter space, R(FGR) is the Rademacher complexity of the group-relative loss function space FOR, N denotes the size full training Q, is size of oulputs for each question and R is an upper bound of variance of the return {ri i.e., Varn, (ri) S on v0 E e.
Proof. At first, we introduce a theorem from Tolstikhin & Seldin (2013), that is.
Theorem A.1 (Lemma 1 in Tolstikhin & Seldin (2013)). For any function fn. H x (X' x 81 R and for any distribution P1 over H. such that Pi is independent of dataset S P (1i,,)),... (In, yny], with probability greater than 1 - over a random draw of S. for all distri- butions P2 over H. simultaneously:
(n S KL(P2)P) +
+Im( in())
in (Enwr [s' ND [efulh,s (1))
(12)
where S TDd represent a n-size independent dataset s' drawn from data space D.
Then, we set (x
(o)) and investigate the expectation Eo (( for any fixed 0 E θ and > 0. From the classic Bernstein-type self-bounding inequality(e.g. Theorem 2.1. in Fan & Shao (2025)), can have that
X²V
Ee (
< exp(
2(1 nNA
where V = =E E [ERRO (0) - Love (0) ) GRPO (0) 1oix; (0)
< A deterministically
for any j E [N] and 201x; (0) represent the leave-one-quesiion-ou loss.
In standard MDP, we usually use the discount return such that we could infer that A O (ااا where y is the discount parameter (Puterman, 1990). Moreover, from the variance conversion for mixing ergodic MDP (Levin & Peres, 2017), we also can show that V < Amin 20 !.)
Nn
From Eq. 12, we can have that
Ea-r, [A[Sx (0, Q) (0,
KL + In EBrP (Eo(eNGW(RR)--RRlllee
X2 R2 -
< (I) + 2Nn(1 - - (11-))-133)
Let P1 Ps, thus can show that
Ea-r, (ero (aRPO(0)]
In(
21 - 1)
Eg-Py [ (0) (oR +
2Nn(1 - A(1 17)
< &Eolsup 2 GRPO (0) - LGRPO (0) in(1)
Xtmiz 2f 1)
BE0
A
2Nn(1 X(1 - -7)-1/33
where the final inequality follows from 10 G (0)
LGRPO(0) (0) S supace 1 (0) LGRPO(0))
20

---

Published as a conference paper at ICLR 2026
Like the structure of the proof of Theorem 3 of Tolstikhin & Seldin (2013),
investi- 2Nn(1 X(1 (1-7))1133 where À E (0, 1-x): The minimum
In(
Afmiz (1-
gate the function ((A)
2Nn log( (1+88))
and 3 (1-7) 1x" such that g(A*) =
of g(A) often occurs at X
Aman A
3
OOV Amir 21 1n())
In(
As result, we have that
Nn
Nm(1-7)
Ea [iRPO (0) - (O)P
[Splsup G (0) [aRPO [oRPo(0)]]+ bmizo o +) In()) OEG
In(
+
Nn
Nn -
Furthermore, from the classical symmetrization lemma in statistical learning theory (Shalev- Shwartz. & Ben-David, 2014; Mitzenmacher & Upfal, 2017), we have that Eg(suPsee LaRrO (0)]] s 2R(FGR) where R(Fan) is the Rademacher complexity of the group-relative loss function space FoR. Therefore, we have that
ICo (
Eg |I (0) - (GRP S 2R(FoR) + O(V Emir o R2( - (n
In(#)
Nn
+
Nn(1 Nn(1-7)
Finally, due to the randomness of P2 over θ, we get the result of lemma A.2.
With the results of Lemma A.1 and Lemma A.2, we also prove Theorem 3.1, that is,
LGRPO (0x) (0")
= LoRrO G + LO (ORP - LO (f') + (()) - (f))
$ sup CORPO (0) LaRFO (0) + CORPO CORPO AE
1m 1. 1n())
In(
2G e²
< 4R(FFR) (1 + y co (0n) + +[V
+
Nn
Nn - 7) /
24
where the first inequality comes from ORRPO
arg ming
LQ (O) and 0* arg ming LCRPO(
21

---

Published as a conference paper at ICLR 2026
B DETAILS OF THE EXPERIMENTAL SETUP
B.1 MORE DETAILS ABOUT THE DATASET
Our experimental evaluation is conducted several widely recognized mathematical reasoning benchmarks. For model training, we utilize a subset the BigMath dataset, while for the calibration of our scoring function, evaluation spans four distinct datasets to ensure a comprehensive assessment of performance. Below, provide detailed description of each dataset.
instances from both BigMath and MMLU. For the test phase,
Big.Math-sub. For the training phase of our experiments, we use BigMath-sub, which is ran- domly selected subset containing 2048 instances from the large-scale BigMath dataset (Albalak et al., 2025). BigMath is high-quality dataset specifically curated for reinforcement learning in large language models on mathematical tasks.
MMLU. The Massive Multitask Language Understanding (MMLU) dataset (Hendrycks et al., 2021a;b) a comprehensive benchmark to evaluate the reasoning ability of LLMs. In our experi- ments, it serves as altermative calibration set to justify the robustness of our method against the choice of calibration data. To maintain relevance to our mathematical reasoning task, we utilize specific subset of MMLU, comprising five math-related subjects.
AMC23. The American Mathematics Competitions (AMC) dataset (problems & solutions, 2023) is a collection of challenging problems from the official mathematics competitions for students in the US. These problems require both the knowledge of mathematical concepts and excellent problem-solving ability.
This dataset (Lewkowyez et al., 2022) is a benchmark focused on quantitative reasoning, containing problems that require step-by-step logical inference. The problems are sourced from various STEM disciplines and are designed to test the model's ability to perform complex, multi-step calculations and reasoning.
MinervaMath.
OlympiadBench. This is a highly challenging benchmark consisting of problems from interna- tional science and mathematics Olympiads (He et al., 2024). The dataset is designed to push the limits of LLM reasoning capabilities, as the problems often require creative and non-standard approaches to solve.
MATH500. The MATH dataset (Hendrycks al., 2021c; Lightman al., 2023) is a widely adopted benchmark for mathematical problem-solving, composed of 12,500 problems from high school math competitions. The problems are categorized by difficulty and subject, covering topics such algebra, geometry, number theory, and more. We use 500-instance subset for our evaluation.
B.2 MORE DETAILS ABOUT THE EVALUATION METRICS
In the experiments, we the avg@k accuracy metric for evaluating the performance. This metric formally defined as follows. Given an input X and the ground truth answer Y, the model predicts a total of answers, i.e., Y,, F2,.., Pe. and the metric can be computed as:
avgêk Σ 1l Y]
(13)
k
where 1l is the indicator function. In the evaluation, we consider the size of the datasets when deciding k: for the smaller dataset of AMC23, we set & to 256, whereas for larger datasets of MinervaMath, OlympiadBench, and MATH500, set le to 32.
B.3 MORE DETAILS ABOUT THE BASELINE METHODS
We compare the proposed CONST against various baselines. These methods are grouped into three categories: a non-finetuning baseline, random selection baseline, and several active learning algo- rithms. detailed description of each baseline is provided below.
NoFinetuning. This baseline directly the original large language model for inference with- out any fine-tuning. serves as fundamental benchmark to evaluate the performance improve- ment brought by different instance selection and training strategies.
22

---

Published as a conference paper at ICLR 2026
RandSelect. This is simple yet crucial baseline where subset of instances is randomly selected from the entire training set for annotation and subsequent model optimization. This method helps to gauge the effectiveness of more sophisticated active learning strategies against naive selection approach.
EntSampling. a classic uncertainty-base active learning method, Entropy Sampling (Settles, 1995) selects instances for which the model has the highest predictive entropy. Our implemen- tation adapts this principle for generative LLMs. For each instance, we first generate a multiset of potential answers by sampling the model's output n. times. The entropy is then calculated based on the frequency distribution of the unique answers within this multiset. This procedure directly mirrors the Outcome Volatility component of our proposed CONST method, ensuring fair comparison.
BADGE. Deep Batch Active learning by Diverse, Uncertain Gradient Lower Bounds (BADGE) (Ash et al., 2020) is state-of-the-art active learning strategy that selects batch of samples that are both uncertain and diverse. It computes hypothetical gradient embedding for each unlabeled sample with respect the parameters of the final layer. The magnitude of this gradient represents the model's uncertainty. To ensure diversity, it then uses k-MEANS++ seeding on these gradient embeddings to select batch of points that are far from each other in the gradient space.
CEC. Calibrated Entropy-weightt Clustering (CEC) (Safaei & Patel, 2025) is active learning framework designed for vision-language models that selects informative samples by leveraging both self-uncertainty and neighbor-aware uncertainty. The method first calibrates the predictive entropy to counteract the known miscalibration issue in large pre-trained models. It then com- bines this calibrated self-uncertainty with the uncertainty of neighboring samples to obtain more reliable score. Finally, CEC employs an uncertainty -eerriinty-wegghedd clustering approach to ensure di- verse sample selection from various regions of the feature space. For our LLM-based tasks, we apply its core mechanisms directly to the language model's final-laye embeddings.
23

---

Published as a conference paper at ICLR 2026
C CASE STUDY
In this section, we present three case studies qualitatively demonstrate the effectiveness of our proposed CONST method. For each case, we show the question, the response from the original LLM (Before Training), and the response from the LLM fine-tuned critical instances selected by CONST (After Training). An analysis is provided to highlight the specific improvements in the model'' 's reasoning capabilities.
C.I EXAMPLE
Example I:
Question
Given that the sum of the distances from any point inside regular tetrakedron with edge length a its faces is constant value, this value.
Example 1: Before Training (Original Model)
find the constant value of sum of distances from any point inside regular tetrahedron to its four faces, we can use the following steps:
1. Consider regular tetrahedron with edge length a.
2. Let P be any point inside the tetrahedron. We want to find the sum of the distances from to the four faces.
3. The sum of distances can expressed as di + da da.
4. We can use the fact that the sum of the distances from point inside tetrahedron to its four faces equal the height of the tetrahedron multiplied by 3.
5. The height a regular tetrahedron with edge length can be found using the formula 2/8
6. Therefore, the of the distances is 3/ = 3. evE av6.
The final answer is: avb
Example 1: After Training (Fine- tuned Model)
Step 1: Understanding the Problem. We given regular tetrahedron with edge length The problem asks
constant value that the sum distances from any point inside the tetrahedron to its four faces.
Step Identifying Key Geometrie Properties. A regular tetrahedron has four equilateral triangular faces. key result (a generalization of Viviani's theorem) states that for any interior point, the sum of the distances to the four faces a constant, equal to the tetrahedron's 's height.
Step 3: Considering the Centroid. The distance d from the centroid cach face be calculated using the formula d av8
Step Calculating the Height and Summing the Distances. Since there are four faces, the sum of the distances from the centroid (or any point inside the tetrahedron) to the four faces 4d = eve6 eVi
12
Step Conclusion. Therefore, the constant value that is the of the distances from any point inside the regular tetrahedron to its four faces is avē 3
V6
The final answer is:
The original model makes a fundamental error in geometric reasoning, incorrectly stating that the sum of distances is three times the height (Step 4). This conceptual mistake leads directly to wrong final answer. After fine-tuning. the model demonstrates complete conceptual and procedural correction. It correctly identifies the key geometric principle: the sum of the distances is equal to the height of the tetrahedron (Step 2). Furthermone, executes sound, step-by-step calculation
Analysis:
24

---

Published as a conference paper at ICLR 2026
correctly derive the height's value as aye and provides the correct final answer. This successful transformation from a flawed understanding to fully correct solution showcases the effectiveness of the training sample selected by CONST.
Example 2: Question
A school selects 50 students from all 500 senior students for study situation questionnaire survey. The 500 students numbered from to 500. The interval number student selected of every 10 students. A mumber randomly selected from I and the number is selected, the numbers that should taken from 125 to 140 are
10, which means
Example 2: Before Training (Original Model)
To find the numbers that should be taken from 125 to 140, we need to determine which numbers this range correspond to the interval of 10.
Since the interval number 10, can divide the range 125 140 into intervals of 10. number is selected, the numbers that should taken from are This means that we need to select every 10th number starting from 125.
problem states that
So, the numbers that should be taken from 125 to 140 are: 125, 135.
The final answer 125, 135
Example 2:
After Training (Fine-tuned Model)
Step 1:
Understand the selection process. The school is using systematic sampling. One student is chosen from every 10 students.
Step
Determine the selection pattern. A random start from to was chosen, and it was This means the selected students will be those numbered 6, 16, 26, 36, and so on. The general form for selected student's ss number is 10n 6 for some integer n 2 0.
Step Identify the relevant range students. are interested in the numbers that should be taken from 125 to 140.
Step Apply the selection pattern the relevant range. We need find numbers of the form 10m + that fall between and 140.
If 11, the number is 10(11) + 6 116 (too low).
If n 12, the number is 10(12) + 6 126 (in the range).
If n 13, the number 10(13) + 6 136 (in the range).
If 14. the number 10(14) + 6 146 (too high).
Step 5: List the selected numbers in the range 125 140. The selected numbers are 126 and 136.
The final answer is: 126 and 136
The original model completely misunderstands the concept systematic sampling: ignores the provided random start (6) and incorrectly assumes the selection begins at the start of the local range (125), leading to wrong answer. In sharp contrast, the model after fine-tuning demonstrates perfect understanding of the procedure. correctly identifies the general form of the selected numbers (10n + 6) and flawlessly applies this rule to the specified range to find the two correct numbers, 126 and 136. This case illustrates clear and complete correction from flawed logic to precise procedural execution, with both the reasoning and the final answer being correct.
Analysis:
Example 3: Question
Given that the sequence fbnk is an arithmetic sequence with the first term as -34 and a common difference of I, and the sequence {an satisfies ant1 an 2n (n N²",, and bs7, then the maximum value of the sequence
is
25

---

Published as a conference paper at ICLR 2026
Example 3: Before Training (Original Model)
Given that (bn} J arithmetic sequence with the first -34 and common difference of we can write formula b, as: ba, -31 + (n 1). -33 n. Given that 0x++1 0n 2",
write the formula for an On d1 + >>= 2 + + -2. -3
Since d1 byz. we can substitute n into the formula b, find a1 01 037 So, an = +
37
2 2
Now, we can find the value of Am 2+2" -3++
To find the maximum value of the sequence, we can try different values of n: n = 5, -28
-27
34
We can see that the value of n that makes the denominator close to the numerator is not integer. We
try the value of 3: =30 -3. ... [The model continues plug in values without clear strategy and fails to find maximum.]
Example 3: After Training (Fine-tuned tuned Model)
Step 1: Determine the general term of the arithmetic sequence (b.). Given bi 34 and
mon difference 1. general term b,
1)d = 34 (n- 1)( = n
Step 2: Determine the general term of the sequence [an}. J. We are given the recurrence dn+1
2". The general term an can be found by telescoping sum: On 01 + Σズ= 1 (Ax+1 04.
d1 + > A=1 2 This a geometric series sum: Σ 1 2 2(2" On = 01 + 2
1)
So,
Step 3: Find the value of a1. We are given d1 bu7. Using formula for b.: a1
37 35 2. Step 4: Finalize the formula for Ont Substituting a1 into the expression from Step 2: a. (2) + 2" 2". 2".
Step 5: Find the ratio b=, The sequence we want to maximize
I1-35 2
Step 6: Determine the maximum value. Let f'n) 2n The denominator grows much faster than the numerator. n > 35, value is positive and eventually decrease towards
35, the value is negative. For n 35, value For
maximum value
will occur for a small n 2 36. Let's check n = 36: f(36)
38-35
35
Let's check
230
n 37: (37)
37 35-35- 237 230 Let's check 38: f(38)
38-35
288 Since
237
238
236 value decreases after n 37. maximum value 230.
The final answer is: 2
Analysis: This multi-step problem requires correctly deriving formulas for two sequences before optimizing their ratio. The original model makes errors in deriving both b, (off-by-one) and ans making the rest of the problem unsolvable. After training, the model flawlessly executes the setup phase: it correctly derives 6, significant improvement. then correctly analyzes the behavior of the resulting function n-35 2" and, by testing points near the transition from negative to positive, finds the correct maximum value. The ability to correctly formulate the problem is the critical learned skill here.
35 and a, 2n This mastery of the prerequisite steps is
C.2 ANALYSIS OF PROCEDURAL VOLATILITY FOR A CRITICAL INSTANCE
To further illustrate how our proposed CONST method identifies critical instances, we present de- tailed analysis of the procedural volatility for Stratified Sampling problem. This process involves truncating the model's 's reasoning process at various stages and extracting an answer from each partial thought process. The analysis demonstrates how the model's 's path to the correct answer is unstable, with its intermediate conclusions fluctuating significantly. This volatility highlights a fragile grasp of the multi-step procedure, which makes this sample high-value critical instance for fine-tuning.
26

---

Published as a conference paper at ICLR 2026
Procedural Volatility Analysis
Full Reasoning Process:
To find the number of sophomores, we first need to find the total number of students in the school. We know that there are 450 freshmen and 250 seniors, but we don't know the total mumber of students (n). However, the stratified random sampling find the number sophomores. In stratified random sampling, the sample divided into subgroups based on the population characteristics (in this case, grade level). The number of students each subgroup is proportional to the number of students in that subgroup in the population. Let's denote the number of sophomores
use the information about
The total number of students in the sample 60, and the number of freshmen the sample 27, can set up a proportion to represent the ratio of freshmen the sample to the total number of freshmen in the population: 27/60 = 450/n We can cross-multiply and solve for n: 27n = 60 450 = 27n 27000 = 1000. Now that know the total number of students (n 1000), can find the number of sophomores. The total number students the sum of freshmen, sophomores,
seniors: 1000 450 + x 250. Subtract 700 from both sides: 300 x. So, the number of sophomores is 300.
Analysis Truncated Processes:
Stage (25% Process)
Truncated Reasoning: "To find the number of sophomores, we first need to find the total num- ber of students in the school. We know that there 450 freshmen and 250 seniors... we use the information about stratified random sampling... In"
Extracted Answer: (b7xe
Stage 2 (50% % of Process)
Truncated Reasoning: "...th sample is divided into subgroups... The number of students in each subgroup proportional to the number students that subgroup the population. Let's denote the number of sophomores as x. The total number of students in the sample is 60,"
Extracted Answer: \boxed(73}
Stage 3 (75% of Process)
Truncated Reasoning: "... We can set up proportion... 27/60 450/n. We can cross cruss-multiply and solve for n: 27n = 60 450 + 27n = 27000 = n= 27000/277
Extracted Answer: (boxed
Stage 4 (100% of Process)
Truncated Reasoning: "...n 1000. Now that we know the total number of students... we can find the number of sophomores... 1000 450 250. Subtract 700 from both sides: 300
So, the number of sophomores 300."
Extracted Answer: (boxed(300}
Analysis. The model's reasoning process exhibits high procedural volatility. The extracted answer fluctuates from a hallucination (173), another incorrect guess (73), to correct intermediate result mistaken for the final answer (1000), before finally reaching the correct solution (300). This unstable path, despite culminating in correct answer, reveals a fragile and non-robust understanding of the multi-step problem-solving procedure. This makes the instance a quintessential "critical" or haluevalue sample for corrective fine-tuning, as it exposes a weakness that simpler, more direct problems would miss.
C.3 PROMPT TEMPLATE
Our framework utilizes several prompt templates tailored for different tasks, including guiding the model's 's reasoning process and standardizing its final output for evaluation. The core templates used in our experiments are detailed below.
27

---

Published as a conference paper at ICLR 2026
Main Instruction Prompt for Mathematical Reasoning
This template is appended every mathematical prohlem to instruct the model to generate detailed, step-by-step solution. The final prompt the model concatenation of system prompt, the question, instruction.
You helpful assistant. You are asked solve the following question. <question>
Let's s think step by step and output the final answer within [boxed().
2. Answer Extraction Prompts
After the model generates a free-form reasoning trace, use dedicated extraction prompt parse the trace and isolate only the final answer. This ensures standardized format for automated evaluation.
For General Problems:
You are an expert mathematician and a precise answer extractor. Your task is to analyze the provided mathematical reasoning and extract only the final numerical answer. Do not provide any explanation or preamble. Your final output should ONLY the answer enclosed in a \boxed().
MMLU Multiple-Choice Questions:
You precise answer extractor. Your task is to analyze the provided reascning for a single-choice question and determine the correct option.
Your final output must be ONLY the letter of the correct option (e.g., A, B, C, D) enclosed in \bexed().
3. MMLU-Specific Instruction Prompt
For the MMLU calibration which multiple-chen choice format, specialized instruction prompt is used guide the model's response.
You will be presented with a single-choice question. Please analyze the question and the provided options to determine the single correct answer.
Your final response should be ONLY the letter of the correct option (e.g., A, и, C, or D) enclosed in \boxed For example, if the correct option is B, your response must be (b)xed(B).
D ADDITIONAL EXPERIMENTS
D. DIFFERENT SIZES OF CALIBRATION SETS
In this part, we present the model's performance under different sizes the calibration set (i.e., m.. We conduct experiments using of 8. We vary the size m from 256 to 1024, and the results are shown in Table 3. As can be seen from the table, the performance generally improves the size increases. Specifically, increasing m from 256 to 1024 leads to consistent improvement across all datasets. This observation aligns with the intuition that larger calibration set Deal provides more accurate estimation of the scoring
-3.
with fixed annotation budget
28

---

Published as a conference paper at ICLR 2026
Table Model's performance under different sizes of the calibration set (i.e., m). The best results are marked in bold.
Datasets
AMC23
MinervaMath
OlympiadBench
MATH500
AVG
m 256
22.19
20.54
13.88
42.85
24.87
512
23.87
22.87
16.73
45.59
27.27
m 1024 (default)
24.27
24.19
17.61
47.17
28.31
Table 4: Performance comparison on undergraduate-level STEM problems (OCWCourses) using
LLaMA-3. 1 ----BB-nneee Instruct with a budget of 8. We report the avg032 accuracy. The best results are marked in bold and runner-ups with underline.
Methods
Astronomy
Solid Chem.
Dynamics
AVG
NoFinetuning
7.5
15.5
26.9
16.63
EntSampling
9.4
11.3
11.5
10.73
CEC
3.8
10.3
26.9
13.67
CONST (ours)
11.3
17.5
38.5
22.43
function's distribution, thereby enhancing the quality of the generated prediction sets Ci-o(X) Therefore, we set m = 1024 in our main experiments to ensure robust performance.
D.2 RESULTS ON SCIENTIFIC PROBLEMS
To evaluate the effectiveness of CONST in scientific domains, we analyze performance on the OCW- Courses (Lewkowyez al., 2022) subsets. We report the results of LLaMA-3. 1-s with budget of 6 = across three representative science courses: Introduction to Astronomy, Solid State Chemistry, and Dynamics and Control. As shown in Table 4, CONST demonstrates substan- tial improvements over the base model (NoFinetuning) and significantly outperforms other active learning baselines. Notably, classic uncertainty-base these hard domains (average accuracy drops to 10.73%) likely because high entropy in these tasks correlates with total model confusion rather than learnable uncertainty. In contrast, CONST achieves the highest performance across all three subjects, with an average accuracy of 22.43%, providing relative improvement of 34.9% % over the original model (16.63%) showing that CONST better identi- fies critical instances. While there may be linguistic noise generated by the model in these domains, CONST is less affected and still outperforms baselines.
methods like EntSampling perform poorly in
D.3 ADDITIONAL BASELINES
To further validate the effectiveness of our proposed framework, we compare CONST with two addi- tional bascline methods on LLaMA-3. 8B Instruct with a budget of 8:
SCF (Self-Consistency Filtering) (Wang et al., 2022): This method evaluates the uncertainty of each instance based on the disagreement among the model's outputs. Specifically, SCF calculates the frequency of the majority answer and selects instances with low self-consistency scores (i.e., high reasoning variance), assuming these instances lie on the decision boundary.
Sampling) (Beygelzimer et al., 2009): Instead of deterministically se- lecting the instances with the highest variance, EWS samples instances probabilistically, where the probability of selection is proportional to the predictive entropy.
EWS
The results are presented in Table The results show that while SCF and EWS achieve competitive performance (25.87% and 22. 14% average, respectively), surpassing the NoFinetuning baseline, the proposed CONST performs better compared to these baselines across all datasets, achieving an average accuracy of 2..31%. These results further demonstrate the superiority of selection criterion based on conformal prediction considering both procedural and outcome volatility.
29

---

on the MATH500 dataset, (ii) filter them using the scor- Published as a conference paper at ICLR 2026
Table Performance comparison with additional baselines on LLaMA-
ruct with
a budget of 8. We report the avge 32 (avg@25 256 for AMC23) accuracy. The best results are marked bold.
Methods
AMC23
MinervaMath
OlympiadBench
MATH500
AVG
SCF
22.83
21.63
16.36
42.64
25.87
EWS
21.38
16. 14
13.71
37.31
22.14
CONST (ours)
24.27
24.19
17.61
47.17
28.31
Table Performance comparison between the likelihood- bascline (LogProb) and CONST us- ing LLaMA-2 3, -B- Instruct with budget of 8. We report the avg032 (avg8256 for AMC23) accuracy. The best results are marked in bold.
Methods
AMC23
MinervaMath
OlympiadBench
MATH500
AVG
LogProb
22.53
23.38
15.23
46.19
26.83
CONST (ours)
24.27
24.19
17.61
47.17
28.31
D.4 ADDITIONAL EXPERIMENTS ON THE SCORING FUNCTION
We also perform ablation studies on the scoring function in conformal prediction. Specifically, we the model's intrinsic output likelihood as the scoring function:
f (X, P) - log P(Y,O|X) O|X),
(14)
where y the final answer and O is the reasoning trajectory. Note that we also incorporate O since the final answer is often obvious given the reasoning trajectory. The performance comparison is pre- sented Table 6. We observe that while using log-probability achieves decent performance (26.83% average accuracy), it consistently underperforms CONST across all datasets. Specifically, CONST surpasses this alternative design by 1.48% on average, with notable margins on OlympiadBench (+2.38%) and AMC23 (+1.74%) A possible explanation for this result is that the log- pogobbbbbiyy prone to linguistic noise (e.g., linguistic unpredictability. rare vocabulary. or stylistic variations), while the proposed method better reflect logical uncertainty. For example, model might be "sur- prised" by a token simply because it is an uncommon phrasing, even if the reasoning logic is sound. In contrast, our proposed CONST relies on both procedural volatility and outcome volatility, which abstract away from token-level noise to capture higher-level inconsistencies in the reasoning process and the final answer.
D.S DIFFERENT CLUSTERING CONFIGURATIONS
We also provide ablation studies of alternative configurations of clustering, where we compare "b clusters, instance in each cluster" (default configuration) and "b/2 = clusters, instances in each" (alternative configuration). As can be seen from the results in Table 7, the default configuration slightly better than the alternative.
D.6 PASS RATES OF ANSWER SETS
We also conduct detailed analysis the pass rates of the generated prediction sets. Specifically, for each test question, we: (i) generate N 40 candidate answers using SeroSSee-----istll----e-
ing function and the threshold derived from the calibration set (with size m = 100), and (iii) mea- set. We compare CONST against two baselines:
Seff-Consistency and Entropy-based based Selection. We set the target error rate = 0.2, implying a target coverage of 80%. The results are presented Table 8. Note that the average set size of conformal prediction is k 3.53. Since both Self-Consistency and Entropy-Based Selection cannot naturally decide the sizes of the candidate sets for each test question, we fix the size of the candidate sets of these two methods to both & = 3 and prediction increases the pass rate of the candidate sets. Conformal prediction can dynamically de- cide the size of the candidate sets for each question: easy questions will have smaller candidate sets
sure the pass rate of the final answer
4, for fair comparison. As can be seen from the results, conformal
30

---

Published as a conference paper at ICLR 2026
Table Performance comparison under different clustering configurations of CONST using
LLaMA-3. ------------------ 11--B-Innne Instruct with a budget of 8. We report the avg0 32 (avgê256 256 for AMC23) accuracy. The best results are marked in bold.
Methods
AMC23
MinervaMath
OlympiadBench
MATH500
AVG
alternative
22.97
25.02
17.11
45.31
27.60
default
24.27
24.19
17.61
47.17
28.31
to ensure high probabilities of covering the ground truth answer, and hard questions will have larger ones. this paper, we use the size of the conformal prediction sets to guide sample selection.
Table 8: Pass rate comparison of the candidate sets generated by different selection strategies on MATH500. We use Deesseee--L--.isil---veeee
with N 40 and
0.2. The
best results are marked in bold.
Methods
Pass Rate
Self-Consin Consistency (k 3)
76.75%
Patropy-Busee Selection (A
76.50%
Self ColiiCCosistency (k 4)
78.50%
Entropy-Base Selection (k 4)
77.75%
Conformal Prediction (ours, & 3.53)
80.75 %
31


Reviews:
[{"review_id": "pzcTGAZ4k2", "reviewer": "Reviewer_rXws", "summary": "This paper describes a new data subset selection method called CONST. CONST identifies the critical instances from training data by measuring their procedural volatility (variations in the reasoning chain) and outcome volatility (inconsistencies in the final answer), combining these metrics using conformal prediction to determine sample importance. Experimental results demonstrate that training on the small subset selected by CONST achieves comparable performance to full-dataset training, showcasing its effectiveness for data-efficient policy optimization.", "questions": "Why are the authors not reporting results on qwen 2.5 7 b or qwen 3 8b ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- The problem is important and interesting. It is good that the proposed method requires minimal annotation from experts and also does not use any LLM annotations.\n\n- The authors analyse their algorithm theoretically.\n\n- The authors show that using a small number of examples gives almost as good a performance as the whole dataset.", "weaknesses": "- The main drawback is that the authors do not discuss the whole of data subset selection and valuation literature. Example papers include:\n\nPaul, Mansheej, Surya Ganguli, and Gintare Karolina Dziugaite. \"Deep learning on a data diet: Finding important examples early in training.\" Advances in neural information processing systems 34 (2021): 20596-20607.\n\nGuo, Chengcheng, Bo Zhao, and Yanbing Bai. \"Deepcore: A comprehensive library for coreset selection in deep learning.\" In International Conference on Database and Expert Systems Applications, pp. 181-195. Cham: Springer International Publishing, 2022.\n\nDas, Soumi, Manasvi Sagarkar, Suparna Bhattacharya, and Sourangshu Bhattacharya. \"CheckSelect: Online Checkpoint Selection for Flexible, Accurate, Robust, and Efficient Data Valuation.\" IEEE Transactions on Artificial Intelligence (2024).\n\n- I am not sure about the validity of assumption 3. The authors cite another paper, calling it a standard assumption. However, in my opinion, this assumption is very strong and is often invalid. Also, I am not sure if the proof is novel under this assumption.\n\n- The authors did not show results on bigger reasoning models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePji5E9wuJ", "reviewer": "Reviewer_H6dA", "summary": "This paper proposes CONST, a framework for identifying lottery-winning samples that are most critical for RLVR in LLMs. Instead of requiring full annotations for all training data, CONST selects a very small subset of informative instances without access to ground-truth answers. It combines two complementary measures: procedural volatility (instability of reasoning paths) and outcome volatility (variability in final answers), and leverages conformal prediction to quantify uncertainty through the size of prediction sets. Samples with higher uncertainty are selected for annotation and used for RLVR optimization. Theoretical analysis under the lottery sample hypothesis shows CONST can approximate the optimal policy, and experiments on several mathematical reasoning benchmarks demonstrate that CONST achieves near full-dataset performance with less than 0.5% of annotated samples, outperforming existing active learning baselines.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The method is valuable because it helps us understand which samples truly improve model performance and which do not, providing insight into data efficiency for RL-based reasoning.\n\n2. The experiments are comprehensive and convincing, covering multiple models, datasets, and ablation settings to clearly show the method’s effectiveness.\n\n3. The paper is easy to follow and well organized, with a clear narrative from motivation to theory and experiments, making complex ideas accessible.", "weaknesses": "1. I want to know whether CONST is suitable for logic reasoning datasets with discrete or small answer spaces such as multiple-choice tasks with only four options, where outcome volatility may be artificially low and conformal prediction less informative.\n\n2. Theoretical results justify that an optimal subset can approximate full-data training, but the analysis stops short of proving that CONST reliably finds such a subset, the connection between the proposed selection criterion and the theoretical gradient proximity assumption remains heuristic.\n\n3. Is the method sensitive to the number of instances in the calibration dataset?\n\n4. The ablation (V2) skips the clustering step in Algorithm 1, but the paper never explains how clustering is done—what features or metrics are used and how the number of clusters is chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GLEQC0gPVe", "reviewer": "Reviewer_d26V", "summary": "The paper investigates whether a very small subset of training problems can drive reinforcement-learning-with-verifiable-reward (RLVR) fine-tuning of large language models to the same accuracy obtained from the fully-annotated corpus. It fuses procedural volatility (how unstable an answer is to reasoning truncation) and outcome volatility (how inconsistent full answers are across rollouts) into a single sample-utility score using conformal prediction. This mechanism is intuitive, computationally light compared to full uncertainty modeling, and agnostic to the base model or reward function—making it broadly applicable.", "questions": "1. Why not incorporate log P(Y | X) from π₀ into fπ₀(X,Y)? Did likelihood-based scoring perform worse?\n\n2. Which embedding and K were used? How sensitive is CONST to these settings? Provide ablations with identical budgets but varying clustering to isolate diversity effects.\n\n3. Could CONST operate in an active loop where the fine-tuned policy re-selects new samples? Any preliminary results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. If validated at larger scale, CONST could substantially reduce reward annotation costs in reasoning RL pipelines, which is an increasingly important direction for sustainable LLM alignment.\n\n2. Novel formulation: Using conformal prediction to combine procedural and outcome uncertainty for sample scoring is an elegant, model-agnostic idea.\n\n3. The figures and algorithms are cleanly presented; the method’s intuition is easy to follow even for non-specialists.", "weaknesses": "1. Similar ideas have appeared under names like self-consistency filtering, or uncertainty-guided selection. CONST’s originality mainly comes from framing these within conformal prediction and the RLVR objective.\n\n2. The “ε-approximate lottery-sample” assumption is interesting but unverifiable. The authors don’t measure ε or show gradient proximity, so the bound doesn’t really illuminate why CONST works.\n\n3. Procedural and outcome volatility both depend on stochastic decoding variance. Without separating linguistic noise from reasoning uncertainty, the method could mis-rank samples for harder domains.\n\n4. Compared mainly to classification-style active learning. RL- or reasoning-specific selection strategies (e.g., self-consistency filtering, entropy-weighted sampling, or value-driven selection) are absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKhSZeWgXf", "reviewer": "Reviewer_fBZw", "summary": "This paper focuses on identifying data subsets that achieve performance comparable to using the full dataset. The proposed method, CONST, leverages procedural and outcome volatility to construct conformal prediction sets, enabling unsupervised discovery of critical instances. Experimental results demonstrate that CONST is both efficient and effective.", "questions": "How should  n_p be selected in practice? For each truncated CoT, how many final answers are sampled? Only one or multiple ones?\n\nDoes outcome volatility functionally overlap with procedural volatility? In other words, is outcome volatility a special case of procedural volatility? \n\nWhat are instances? Is that data samples? Like 4 instances means 4 questions in the math dataset.\n\n\nCould you also report the experimental results where you (i) generate multiple answers per test question, (ii) filter them by the scoring function and the threshold (from Conformal Prediction) to form a candidate subset, and (iii) measure the pass rate of the final answer set? A comparison against self-consistency and entropy-only baselines would be helpful for understanding the motivation for using the calibration set.\n\nIn addition, using a calibration set incurs a cost as well: you use m = 1024 instances (vs. 4 or 8 for training). Are these 1024 instances annotated? Finally, the choice of  m likely matters—why fix m=1024? A sensitivity analysis over m would be informative.\n\nAssumptions are central to the validity of your claims; please state them explicitly and tie each result to the assumption it requires: permutation tests and exchangeability. Also, please discuss the practical applicability. Why can we use the MMLU subset as a calibration set for the BM set?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem addressed is important and compelling. Identifying significant samples within datasets can reduce computation, conserve resources, and clarify how data-collection choices influence training. \n\n- Notably, the method requires minimal human annotation, which enhances its generalizability and practical applicability.", "weaknesses": "The paper would benefit from clarifying its scope earlier. My understanding is that the method is developed and evaluated specifically for the RLVR setting. To help readers set the right expectations, I suggest explicitly stating this focus in the title and/or abstract. Doing so would also avoid any impression that the approach covers broader RL or non-RL problems.\n\nI also have several additional questions/suggestions, which I listed in the Question part. I did not examine the theoretical part in depth and just provisionally assume its correctness; I’m happy to discuss if other reviewers raise concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)