UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
Authors: Yang Zhao
Kai Xiong
Xiao Ding
Li Du
YangouOuyang
Zhouhao Sun
Jiannan Guan
Wenbin Zhang
Bin Liu
Dong Hu
Bing Qin
Ting Liu
Venue: NeurIPS 2025 poster ()
Keywords: Data Selection
LLMs
Reinforcement Learning
Primary area: deep_learning
TL;DR: UFO-RL efficiently selects LLM RL data via fast uncertainty measure, matches full data performance using 10% data, and greatly speeds up training (up to 16x).

Abstract: A primary impediment to scaling reinforcement learning (RL) for large language model (LLM) training is the substantial computational cost, predominantly arising from the necessity of multi-sampling for policy optimization and evaluation. This underscores the critical yet challenging nature of efficient training data selection. Drawing inspiration from the Zone of Proximal Development (ZPD) theory, which posits that learners acquire knowledge more effectively from tasks of intermediate difficulty, we hypothesize that LLMs exhibit optimal learning from data they have not yet mastered but demonstrate the potential to comprehend. Conventional methodologies for assessing data difficulty or informativeness typically rely on computationally intensive multi-sampling or iterative procedures. To address this limitation, we introduce UFO-RL (**U**ncertainty-**F**ocused **O**ptimization for **R**einforcement **L**earning), a novel framework that employs a computationally efficient single-pass uncertainty estimation technique to identify informative training instances. This method, requiring only a single forward pass and obviating the need for iterative next-token computation, achieves a significant acceleration (up to 185$\times$) in data evaluation compared to multi-sampling approaches. UFO-RL leverages this efficient metric to select data within the model's estimated ZPD for training. Extensive experimentation across diverse LLMs and mathematical benchmarks demonstrates that training with a mere 10\% of the data, carefully selected by UFO-RL, yields performance comparable to or even surpassing that of full-data training. Furthermore, this targeted data selection results in up to a 16$\times$ reduction in overall training time, concurrently enhancing training stability and improving generalization capabilities. Thus, UFO-RL presents a practical and highly efficient strategy for scaling RL fine-tuning of LLMs by focusing learning efforts on the most informative and valuable data, thereby mitigating the computational bottlenecks associated with traditional RL training.

UFO-RL: Uncertainty-Pouused Optimization for Efficient Reinforcement Learning Data Selection
Yang Zhao Kai Xiong ,Xiao Ding LLi ? YangouOuyang ,Zhouhao Sun
Jiannan Guan R Wenbin Zhang* Bin Liu Dong Hu Ting Liu c and Bing Qin
Research Center for Social Computing and Interactive Robotics
Harbin Institute of Technology, China
O Beijing Academy of Artificial Intelligence, Beijing, China
Du Xiaoman Technology (Beijing) Co., Ltd.
(yangzhao, kxiong, xding, oyyo, hzsun, jnguan, tliu, qinb) hit. edu. cn duli@baai ac. changwenbin, hudongguuxiaammm..nn com
Abstract
A primary impediment to scaling reinforcement learning (RL) for large language model (LLM) training is the substantial computational cost, predominantly arising from the necessity of multi multi-sampling for policy optimization and evaluation. This underscores the critical yet challenging nature of efficient training data selection. Drawing inspiration from the Zone Proximal Development (ZPD) theory, which posits that learners acquire knowledge more effectively from tasks of interme- diate difficulty, we hypothesize that LLMs exhibit optimal learning from data they have not yet mastered but demonstrate the potential comprehend. Conven- tional methodologies for assessing data difficulty or informativeness typically rely on computationally intensive multi- sampling or iterative procedures. To address this limitation, we introduce UFO-RL (Uncertainty -Fccuse Optiizzttion for Reinforcement Learning), a novel framework that employs a computationally effi cient single-pass uncertainty estimation technique to identify informative training instances. This method, requiring only a single forward pass per instance, thereby obviating the need for repeated sampling runs, achieves significant acceleration (up to 185 x) data evaluation compared to halll-saamping approaches. UFO-RL leverages this efficient metric to select data within the model's estimated ZPD for training. Extensive experimentation across diverse L.MSs and mathematical benchmarks demonstrates that training with mere 10% of the data, carefully selected by UFO-RL, yields performance comparable to or even surpassing that full-data training. Furthermore, this targeted data selection results in up to 16x reduction in overall training time, concurrently enhancing training stability and im- proving generalization capabilities. Thus, UFO-RL presents practical and highly efficient strategy for scaling RL fine-tuning of LLMS by focusing learning efforts the most informative and valuable data, thereby mitigating the computational bottlenecks associated with traditional RL training. The source code for this work is publicly available at: https//githu..com/zy (ttt 25413/UFO_RLL
Introduction
RL has emerged as a powerful paradigm for fine-tuning LLMs, enabling them to acquire advanced capabilities and align their outputs with desired behaviors [8
4
14 28 By optimizing decision-
"Corresponiing Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

making strategies based on reward signals, RL equips LLMs with complex reasoning abilities, especially for challenging tasks like mathematical problem-solving [6
Despite its immense promise, applying RL to LLMs, especially for complex reasoning tasks, faces significant challenges. Primary among these is the high computational cost from multi-sampling per instance for robust policy gradient estimation and evaluation [23] 22 [3] creating a major training bottleneck. Additionally, many current state-of-the-art RL methods for reasoning, such as those using policy gradient with outcome- saacome-bbsed rewards, process data uniformly, and lack mechanisms to prioritize instances yielding the most impactful learning signal [T6. Effective training data selection is thus crucial for optimizing RL efficiency, minimizing redundant computation by focusing on high-value samples.
Existing work addresses the data efficiency issue. For instance, DAPO 23 explores removing data points with consistently correct or incorrect outcomes across multiple samplings, filtering out seemingly "easy" or "impossible" tasks. While this approach reduces computational waste associated with samples providing little learning gradient, it suffers from two main limitations. First, the identification of consistently successful or unsuccessful samples still requires the costly multi- sampling process it seeks to mitigate. Second, data exhibiting such extreme consistency constitutes only a small proportion, this filtering technique only removes limited amount of data, leaving the majority, where learning is most needed, in the training set, thus offering limited efficiency gains. These limitations highlight the need for more efficient and comprehensive data selection strategy that can effectively prioritize learning opportunities across a broader range of data.
To address the limitations of current RL. data selection, we draw inspiration from ZPD theory [55. ZPD posits that optimal learning occurs when learner engages with tasks challenging but achievable with guidance, rather than tasks that are either trivial or insurmountable. Applied to LLMs and RL fine-tuning, we hypothesize that the most informative training samples for RL are those within the model's current ZPD - tasks it has not yet fully mastered but where it shows potential for improvement. These correspond to what we term "fuzzy data", where the model's understanding or execution is incomplete or uncertain. To empirically investigate the relevance of the ZPD concept in LLM RL. we initially measure data difficulty using the accuracy observed across multiple model samplings. Our findings reveal a non- mo-monooooicc relationship between estimated data difficulty and learning effectiveness: training on overly simple data yields diminishing returns, while attempting tasks that are too complex can lead to training instability and suboptimal performance. Optimal learning occurs on data of intermediate difficulty.
While our preliminary study used multi-sampling accuracy (derived from average reward) as a difficulty proxy, this approach faces significant drawbacks. The high computational cost and the discreteness of rule-based reward signals result in a coarse granularity for data difficulty assessment, which hinders its effective practical application. To overcome these, we propose novel, efficient uncertainty evaluation method that directly estimates the model's confidence in predicting the correct tokens in single forward pass. This approach offers a significant computational advantage. achieving up to 185x speedup compared to multi-sampling-tae based methods. Critically, this direct confidence score provides a more precise and fine- tineggrineed signal of the model's 's comprehension and potential for learning on given sample, enabling a more accurate identification of the "fuzzy data" region. Leveraging this efficient uncertainty evaluation, we introduce the UFO-RL (Uncertainty- Focused Optimization for Efficient Reinforcement Learning Data Selection) framework, designed to proactively select training samples from the model's dynamic ZPD, thereby significantly optimizing the efficiency and effectiveness of the RL fine-tuning process.
We conducted extensive empirical validation across multiple mathematical reasoning benchmarks and diverse LLM architectures of varying scales. Our results demonstrate the remarkable efficiency of UFO-RL: by judiciously selecting merely 10% of the available data, UFO-RL requires less than 1/16 of the computational resources needed for full-data training, yet achieves performance comparable to, and often exceeding, the full-data baseline. Furthermore, UFO-RL exhibits enhanced training stability and improved generalization capabilities to unseen problems.
The main contributions of this work are summarized as follows:
We introduce a novel principle for efficient RL fine-tuning data selection for I.LMs, grounded in cognitive ZPD theory, advocating focus on data exhibiting intermediate model uncertainty.

---

We propose UFO-RL, a lightweight and scalable framework that implements this principle, leveraging a novel, efficient single-pass uncertainty evaluation method to drastically reduce computational overhead compared to multi-sampling alternatives.
We provide extensive empirical evidence, demonstrating state-of-the-art training efficiency across multiple benchmarks and I.I.M architectures: matching or surpassing full-data performance using only 10% data and less than 1/16 computational resources, coupled with improved training stability and generalization.
2
Related Work
RL-based Post-training for LLMs RL has become key technique for fine-tuning LLMs, particu- larly complex tasks such as mathematical reasoning [18] 4 17]. Standard RL algorithms, including policy gradient methods like PPO [14, are computationally intensive primarily because they require multiple samples per instance estimate gradients accurately and reduce variance. Methods like GRPO [6] improve training efficiency by leveraging group-relative policy optimization. Building on this, DAPO 23 further enhances efficiency by filtering out samples that are consistently answered correctly or incorrectly across multiple samplings. However, despite these improvements, DAPO still depends on computationally expensive multi-sampling to identify such samples, which limits scalability. In contrast, our approach tackles this fundamental bottleneck by introducing novel single-pass uncertainty estimation method that efficiently evaluates data without requiring repeated sampling, thereby significantly reducing computational overhead during data selection.
Data-Driven "Less is More" Beyond algorithmic optimization, data has emerged as a pivotal factor in acquiring and enhancing LLM capabilities [25] 24). The "less is more" principle highlights that carefully selecting high-quality data can substantially improve training efficiency 19 22 26]. For instance, in the Supervised Fine-Tuning (SFT) stage, empirical evidence shows that using small, high- quality datasets [2) can effectively enhance model performance, even surpassing fine-tuning on massive amounts of ordinary data, thereby achieving significant lеap in LLM capabilities. Notably, such high-quality data often correspond to complex samples, and prioritizing these challenging
examples boost reasoning abilities 20
However, identifying and leveraging "high-quality data" within the context of RL poses unique and significant challenges. While methods like LIMR [T0] attempt data selection by focusing samples with intermediate rewards or performance metries cerriic--onncppualll aligned with identifying challenging- ceatinng-uuttcchie aut-achie samples (similar to the Zone of Proximal Development, ZPD) these approaches share a critical limitation: the computational cost of evaluating data remains prohibitively high. This is because estimating the value informativeness of instance in RL typically requires expensive multi-step interactions or rollouts to observe outcomes and compute rewards or performance signals. The resulting overhead makes it difficult efficiently apply data selection strategies in large-scale RI. processes and can offset or even undermine the efficiency gains brought about by data selection.
To overcome this challenge, our UFO-RL framework addresses this challenge with an efficient single-pass uncertainty estimation for data evaluation. This approach enables us to efficiently select informative samples (i.e.. fuzzy data) in a scalable manner, completely avoiding the prohibitive computational cost of traditional methods.
Fuzzy data is beneficial for the learning process in RL.
This section presents preliminary study that uses sampling accuracy as a quantifiable proxy for model uncertainty to investigate its relationship with RL learning outcomes, thereby motivating our uncertainty-aware approach.
3.1 Definition Sampling Accuracy
Comparing with the ground-truth answer, common and objective reward signal in RL for mathemat- ical problems, allows us to gauge model consistency and infer the difficulty of individual data points by assessing answer accuracy across multiple generations.
3

---

Consider a dataset of M examples, denoted as (.1...., EMM To define the sampling accuracy pa for each example 24, we generate N 16 independent answers using fixed LLM. Let af") be the j-th generated answer for and y; be the ground-trute answer. The sampling accuracy is then calculated as:
N Pi R
(1) 0 :..)
where R(-) ) is the binary reward function (I for conrect, 0 otherwise). Intuitively, a lower p; indicates lower confidence in the model's responses for example 2,, suggesting that example 2; is more difficult for the model. Conversely, higher p; suggests higher confidence in the model's answers, suggesting that example z; simpler for the model.
3.2 Experimental Setup
To empirically investigate the relationship between data difficulty and model learning efficiency, we conducted experiments using the GSM8K dataset []] with several LLMs: four Qwen2.5 variants (0.5B, 1.5B, 3B, 7B) [21], Llama3.1-8BB --8-Inntruut
3 and Mistral-7B-Instren
The experimental
procedure is follows:
First, for each model, we computed the sampling accuracy p; for all examples in the GSM8K training set by performing N 16 independent inference passes. During this process, the temperature was set 1. P; thus serves as our preliminary proxy for model consistency and inferred example difficulty.
Next, to analyze how this proxy relates to learning, we sorted all training examples based on their computed p; values and partitioned them into K 10 equally-sized bins So..... Gg. Each bin contains one-tenth of the total number of examples in the dataset. These bins are ordered by sampling accuracy from lowest to highest: Bin Go contains the 10% of examples with the lowest accuracy (corresponding to the highest inferred difficulty), Bin G9 contains the 10% of examples with the highest accuracy (corresponding to the lowest inferred difficulty), while Bins G1 through G3 contain examples corresponding to the intermediate deciles.
Finally, using the Open-RI R1 RL fine-tuning framework 2]. conducted separate training using only data sampled from each respective bin (Gk) to observe the learning efficiency achieved by training data of different difficulty levels.
3.3 Results and Analysis
Figure 1 presents the learning effectiveness when training data subsets corresponding to each sampling accuracy bin. The experiment demonstrates a clear non-monotonic relationship between inferred example difficulty (as proxied by sampling accuracy) and learning outcomes. Performance is generally lowest when training on the easiest data (highest accuracy bins), improves significantly when training data of intermediate difficulty, and then declines again when training the hardest data (lowest accuracy bins). This strongly supports the hypothesis that data points within the model's ZPD provide the most effective learning signal during RL fine-tuning. Intermediate sampling accuracy serves as a preliminary proxy for this optimal difficulty Ievel.Analysis across models varying scales further suggests that the optimal difficulty level for learning be model- model-dependent, with larger models showing more robustness when trained on harder examples compared to smaller models.
Table Distribution of Multiple Sampling Accuracy on the GSM8K Training Set for the Qwen2.5 family (0.5B, 1.5B, 3B, 7B), - Instructt and -B-sral-B-Insttrutt..
Acc(%)
Qwen2.5-0.5B
Qwen2.5-1.5B
Qwen2.5-3B
Qwen2.5-7B
Llama3.1
Mistral
0
15.91
4.82
2.46
1.36
0,90
20.02
[0, 15)
37.31
15,83
8.26
3.21
2,40
34,20
(85, 100]
4.83
5.34
12.40
75.71
53.74
14.36
100
0.73
0.47
1.99
49.07
21.53
3.12

---

Qwen2.5-0.55
Qwen2.5-1.58
Qwen2.5-38
74
40
80-
Accuracy
72
30
78-
71
76
20
69
G
qwen2.5-78
Llama-3 3.1-8B-Instruct
Mistral 78-Instruct-v0.3
89
85.0
54
Accuracy
88
Accuracy
52
84.0
50
87-
48
Ram Accuracy Accuracy Thend
83.0
46
8
Figure 1: Impact of Sampling Accuracy-Based Data Subset Training on Model Learning Efficiency. Evaluation across Qwen2.5 (0.5B-7B), Llama3. I--BB, 1-8B, divided into 10 bins by accuracy. Lines represent RL performance from exclusive training on each bin (lowest to highest accuracy), showing the relationship between inferred difficulty and learning efficiency.
and Mistral 7B. GSM8K training data was
more detailed analysis of the distribution of multiple sampling accuracies for various models on the GSM8K dataset, as presented in Table [ reveals important insights into the models' current capabilities on these problems. A considerable fraction of examples for many models is concentrated in the extreme accuracy bins (0% 100%). For instance, a 0% accuracy rate (e.g., 20.02% for Mistral 7B) indicates instances that are consistently challenging or currently insurmountable for the model, reflecting current failure mode. Conversely, 100% accuracy rate (e.g.. 49.07% for Qwen2.5-7B) suggests instances the model has mastered, indicating robust initial competence. This distribution of sampling accuracies, clearly correlating with the model's demonstrated ability solve problems (from consistently failing to consistently succeeding), reinforces sampling accuracy as suitable and intuitive proxy for data difficulty relative the model's 's' current state.
Understanding this difficulty profile is crucial. As suggested by the non-monotonic relationship observed in Figure [ optimal learning is likely occur data of intermediate difficulty those instances falling between these extremes. Consequently, focusing training efforts on data within this 'fuzzy" middle ground, where the model shows partial understanding but not mastery, is essential for efficient RL fine-tuning, underscoring the necessity of judicious data selection strategies.
4 Uncertainty-Focuser Optimization for Efficient RL Data Selection
While sampling-based methods are effective in assessing data uncertainty, their prohibitive cost and coarse granularity limit their applicability improve the efficiency and stability of RL for LLMs, we propose UFO-RL, novel framework for strategic training data selection based on model uncertainty. This section first introduces our confidence estimation method, then validates its consistency with sampling-basee based metries, analyzes its computational efficiency, and finally describes its application in uncertainty-driin driven data filtering.
large-scale RL training. To address this, and to
4.1 Confidence Estimation via Average Log-Softmax
Although multi-sample accuracy provides a direct approach to estimating sample difficulty, computationally expensive and discrete, making it impractical for scaling RL training. To address this limitation, we propose an efficient confidence estimation technique based on token-level logit statistics obtained from a single forward pass.
5

---

Given an input 2i, we generate a complete answer sequence {11. : :.r using a decoder-onl, language model and record the conditional probability P(y. Fy Me.) at each decoding step. We define the confidence score for I; as the average log-probability across all output tokens:
1 (onf( = T Σ log P(u The Vet).
This confidence metric offers several practical advantages. First, it requires only single forward pass, avoiding repeated inference. Second, it is fully parallelizable across batches and examples, making highly scalable. Third, demonstrated in section 4.2 it demonstrates strong correlation with sampling accuracy. Furthermore, unlike reward functions, this approach provides a continuous uncertainty signal. This signal lightweight and computationally inexpensive. Crucially, is distinct from typical reward values; it is domain-agnoste yet model-specific, enabling its application across various reasoning tasks.
4.2 Comparison with Confidence and Sampling Accuracy
Table Similarity Analysis of Multi-Sample Accuracy and Confidence Scores across the Qwen2.5 Family (0.5B, 1.5B, 3B, 7B), 1lama-3.1-8B Innttutn,tt and ---nul77-tnnttrut--0.....
Model
Qwen2.5-0.5B
Qwen2.5-1.5B
Qwen2.5-3B
Qwen2.5-7B
Llama3.1
Mistral
Similarity 0.68
0.78
0.86
0.82
0.80
To validate that confidence measure serve useful and efficient proxy for estimating example difficulty, we conducted experiments to assess its consistency with sampling accuracy. As shown in Table the confidence scores and multi-sample accuracy calculated for different models on the GSM8K training set exhibit high consistency, demonstrating strong alignment in ranking examples by inferred difficulty.
Table 3: Comparison of Qwen2.5-7B training performance using the easiest data deciles determined by multi-sample accuracy versus confidence.
Method 10% Easiest
10%-20% % Easiest
20%-30% Easiest
30%-40% % Easiest
Accuracy 90.37
89.92
90.51
89.87 91.50
Confidence 71.09
88.01
91.28
Furthermore, the discrete nature of accuracy and rule-based rewards limits their granularity for nuanced data selection. As illustrated in Table [ substantial portion of data for models like Qwen2.5-7B (49%) yields 100% accuracy even with multiple samples, making finer distinction based on accuracy impossible.
(alleeecomparet the learning performance when training on successive deciles of the 'easiest" data (lowest inferred difficulty). identified separately by sampling accuracy and confidence. Accuracy-based selection shows minimal performance differentiation across these bins, whereas the confidence-basee method reveals clear performance variations, indicating its superior ability to segment data difficulty within seemingly 'easy' ranges.
Computational efficiency is paramount for scaling RL to large models and datasets. Table presents the computational cost comparison for evaluating difficulty across models using single-pat pass con- fidence vs. multi-sample accuracy (16 samples per instance, accelerated with VL..M {8] on the GSM8K dataset. As shown, confidence estimation achieves a speedup up to 185x x compared to multi-sample accuracy under equivalent resource conditions. This dramatic difference highlights its advantage as a practical evaluation metric, making confidence-bart evaluation highly suitable for large-scale data filtering and dynamic data selection within RL training loops.
4.3 Confidence-Based Data Filtering
Drawing upon 7PD, we posit that the most informative samples reside within an intermediate difficulty spectrum that is contingent upon the model's current capabilities. To delineate this ZPD,

---

Table 4: Computational Efficiency Comparison between Multi-Sample Accuracy and Confidence Estimation on a Single A100 GPU for Qwen2.5 family, Llama-3. 1--m--3.1-8-- 1-18B-Insttuct. and Mistral-7B- Instruct-v0.3. Our method achieves up to 18 x speedup over accuracy-bant baaue methods.
Method
Model
Time
SpeedUp
Model
Time
SpeedUp
Accuracy
3337s 18s
3712s
Qwen2.5-0.5B
x 185
Qwen2.5-1.5B
x82
Confidence
45s
Accuracy
41 1186s
6827s
Qwen2.5-3B
x47
Qwen2.5-7B
x39
Confidence
89s
175s
Accuracy
11426s
8335s
L1ama3.1-8B
Mistral
x57
Confidence
186s
146s
metric capable of fine-grained differentiation is necessary. To this end, we leverage the continuous nature of the confidence score Conf(x;) (z)) and define derived score:
exp exp(Conf(zz))
where 8; E (0, 1). This score can be interpreted as the geometric mean of the token probabilities, serving continuous measure of the model's certainty regarding the sample. Higher values indicate greater confidence, while lower values signify increased difficulty.
For each sample in the dataset, we compute "fuzziness score" to evaluate its suitability for training. This score is formally defined as: Score(s;) = 1- (s; μ)², where represents the mean confidence score of the candidate dataset. This function is designed to assign higher scores samples whose confidence 8; is close to the mean confidence μ., reflecting the moderate uncertainty characteristic of samples within the ZPD.
Ultimately, the top 10% samples, as ranked by this "fuzziness score", are selected from the candidate dataset to constitute the training data for RI..
5 Experiments and Analyses
To evaluate the effectiveness and efficiency of uncertainty-drivet sample selection in RL, we con- ducted a comprehensive empirical study across multiple language models and benchmark datasets. This section provides a detailed description of the experimental setup, training details, core perfor- mance findings, and computational cost analysis.
5.1 Training Setup
Our experiments utilized several models ranging from 0.5B to 8B parameters, Data selection was conducted on the commonly used RL datasets GSM8K (1 and DAPO-MATH-I7K [23]. Training was performed using the open-source GRPO framework from open-rl. Details regarding models, datasets, training environment, and hyperparameters are provided in Appendices A.3
A.4 and A.S
For comparison, we conducted experiments evaluating several distinct data selection strategies for RL fine-tuning. These strategies aimed to assess the impact of different data subsets model performance and training efficiency. We included the Full Data strategy, training using the entire available training dataset the standard performance and efficiency baseline. We also evaluated training on a 10% subset of examples exhibiting the highest confidence scores, termed High Conf, representing focus on data the model already finds 'casy'. Conversely, we assessed training on 10% subset with the lowest confidence scores, termed Low Conf, representing focus on data the model finds 'hard' or highly uncertain about. As baseline for data reduction without targeted selection, we used Random sampling. training on 10% uniformly random subset. Furthermore, we included Accruters training on data remaining after filtering out examples with extreme multi-sample accuracy (precisely 0% or 100%) to explore excluding consistently failed or mastered data. Finally, we evaluated UFOours: our proposed method, which trains on a 10% subset specifically selected based on intermediate uncertainty using our efficient confidence approach, consistent with the Zone of Proximal Development (ZPD) principle. ensure robustness, the results for the Random sampling strategy were averaged over 5 independent runs.
7

---

Table 5: Performance of Different Data Selection Strategies. Performance is measured by accuracy (%). Results are reported separately for models trained on the GSM8K and DAPO-MATH-IT7 datasets. We evaluated several strategies: Full Data (baseline), High Conf, Low Conf, Random,
Accriters and Ourr.
Model
Dataset
Full Data
High Conf
Low Conf
Random
AcCriller
UFO
Trained on GSM8K
GSM8K
52.66
45.67
41.57
41.96
50.61
46.27
Qwen2.5-0.5B
Math500
30.80
30.40
28.20
29.64
30.40
30.60
MMLU
42.06
41.90
41.93
41.97
41.94
42.14
GSM8K
76.78
73.22
74.12
74.83
76.48
76.63
Qwen2.5-1.5B
Math500
53.00
54.00
54.00
54.40
55.20
55.40
MMLU
57.86
58.06
58.20
58.14
58.01
58.32
GSM8K
83.33
62.67
82.17
82.25
83.61
84.37
Qwen2.5-3B
Math500
66.20
65.00
64.40
65.84
65,40
67.40
MMLU
64.56
64.51
64.61
64.55
64.89
64.65
GSM8K
91.88
71.09
91.20
90.93
91.35
92.03
Qwen2.5-7B
Math500
75.00
74.40
75.60
75.64
76.20
76.40
MMLU
69.64
69.25
69.44
69.14
69.26
69.43
GSM8K
88.62
86.05
87.55
87.29
88.01
88.30
LLaMA3.1-BMMthh50
51.00
50.80
52.00
53.14
52.00
55.00
MMLU
62.63
59.34
61.65
60.05
59.12
61.67
GSM8K
53.87
51.74
47.95
52.81
55.46
56.67
Mistral-7B
Math500
11.60
13.40
12.00
13.14
13.80
14.20
MMLU
58.46
59.38
59.44
59.61
58.96
59.66
Model
Dataset
Full Data
High Conf
Low Conf
Random
AcCFilter
UFOours
Trained on DAPO-MATH-17X
GSM8K
12.44
40.82
17.06
35.71
14.26
41.43
Qwen2.5-0.58
Math500
29.40
30.02
28.60
29.80
29.20
33.40
MMLU
42.28
42.17
42.17
42.19
42.27
42.18
GSM8K
55.01
55.84
47.95
49.27
$5.39
55.87
Qwen2.5-1.5B
Math500
54.20
53.60
52.00
52.94
$4.60
54.80
MMLU
58.28
58.30
58.11
58.22
58.46
58.17
GSM8K
60.69
60.55
50.70
61.81
60.77
65.70
Qwen2.5-3B
Math500
63.60
66.00
65.80
66.24
67.60
68.20
MMLU
64.83
64.63
64.57
64.62
64.57
64.55
GSMBK
92.03
84.75
81.34
87.43
88.09
91.16
Qwen2.5-7B
Math500
75,80
76.40
77.20
75.46
75.60
77.40
MMLU
70.33
68.88
68.88
69.17
68.91
69.69
GSM8K
86.56
59.41
86.12
82.67
86.12
86.20
LLaMA3.I-8BMalle
45.80
47.40
49.80
49.40
$1.80
52.40
MMLU
61.47
60.30
60.64
63.06
60.78
59.59
GSM8K
44.40
49.54
48.72
47,60
49.62
50.38
Mistral-7B
Math500
10.60
12.40
11.80
12.38
11.60
14.80
MMLU
58.71
59.78
58.95
59.28
59.30
59.11
5.2 Main Results:
Performance Evaluation
In Table 5 we conducted two main sets of experiments based on the training dataset. When trained on GSM8K, our UFO strategy demonstrates robust performance across various task domains:
On the in-domain GSM8K test set, UFO consistently achieves results comparable slightly exceeding full-data fine-tuning across all evaluated model scales.
On the near-domain Math500 benchmark, UFO generally outperforms full-data RI., indi- cating enhanced generalization to more challenging mathematical reasoning.

---

On the out-of-domain MMLU benchmark, performance variations across different data selection strategies are relatively marginal, suggesting limited impact of domain- -specificc RI. on general language understanding, irrepecctive of the selection mechanism.
Analysis of the baseline strategies reveals patterns consistent with our central hypothesis. On the one hand, training exclusively on tigh-confidence samples, compared to training subset, frequently leads to performance stagnation or even degradation, particularly pronounced for larger models, likely due overfitting signals. On the other hand, training on low-confidence samples tends to destabilize the optimization process, especially evident in models with lower capacity. In contrast these detrimental extremes, strategies like random sampling and Accymtee show improved performance. Specifically, the Accritee method (which trains on data after removing examples with 0% or 100% multi-sample accuracy) demonstrates that excluding these extreme examples is an effective approach to improve RL efficiency. However, while random sampling and AcCritter strategies generally outperform training extreme confidence subsets, they mostly underperform our UFO approach. which specifically targets the intermediate uncertainty range. Therefore, this underscores the efficacy of targeted data selection over indiscriminate sampling.
random
overly simplistic instances that offer diminished learning
To further substantiate the generalizability of our proposed methodology, replicated our experi- mental protocol on the more challenging and diverse DAPO-MATH-I17K dataset, which presents a higher degree of complexity compared to the elementary-level mathematics of GSM8K. The results obtained on DAPO-MATH-I77K mirrored the trends observed on GSM8K, providing additional empirical validation for our central thesis. Notably, the challenging DAPO-MATH-I7K dataset, which can potentially cause catastrophic effects for smaller models (as mentioned (9), our method demonstrates resilience and avoids such drastic performance drops seen with less targeted selection or on smaller data subsets in some cases, further highlighting the benefits of focusing training on informative data.
5.3 Computational Cost Analysis
Beyond performance benefits, a key advantage of UFO-RL lies in its efficiency. Prior work 13 4 has demonstrated that the computational cost per instance tends to increase significantly as RL progresses, primarily because the inference length, especially with multi-sampling, grows considerably during training. Our method leverages effective data selection achieve significant performance with only 10% of the data, which leads to a drastic reduction in the overall computational cost of RL fine-tuning. As detailed Table 6 our method achieves speedup of up to 16x compared to RL on the full dataset. This substantial speedup is direct result of processing significantly fewer training instances in the RL loop, thereby mitigating the amplified cost associated with the longer inference sequences in later training stages.
Table 6: RL Fine-tuning Time Comparison: UFO (10% Mid Confidence Data) vs. seconds, on an 8 A100 GPU server).
Full Data (in
Method
Model
Time (s)
Speedup
Method
Time (s)
Speedup
UFO
140
407
Qwen2.5-0.5B
x13
Qwen2.5-1.5B
x x14
Full Data
1815
5694
UFO
739
1154
Qwen2.5-3B
x14
Qwen2.5-7B
x11
Full Data
10224
12959
UFO
1219
1454
Llama3. 1-BBB
x12
Mistral 7B
x16
Full Data
14040
22955
6 Conclusion
In conclusion, this work introduces UFO-RL, a novel and efficient framework for RL fine-tuning of LLMs, inspired by the ZPD theory. By employing a computationally lightweight single-pass uncertainty estimation method, UFO-RL efficiently identifies and prioritizes training samples within the model's "fuzzy data" regime, where it exhibits the most significant potential for learning. Our comprehensive empirical evaluation across diverse mathematical reasoning benchmarks and various LLM architectures demonstrates the remarkable efficiency and effectivenees of UFO-RL. Specifically,

---

we demonstrate that UFO-RL achieves performance comparable to, and often exceeding, full- data training by training on only 10% of the data. This targeted selection results drastic reduction in computational cost, requiring less than 1/16 of the resources needed for full-data training. Furthermore, our findings indicate that UFO-RI. enhances training stability and improves generalization to unseen instances. These results strongly suggest that focusing RL fine-tuning on samples within the model's estimated ZPD offers promising avenue for significantly improving the efficiency and efficacy of training LLMs for complex reasoning tasks.
7
Acknowledgements
The research in this article is supported by the New Generation Artificial Intelligence of China (2044YE00203700), National Natural Science Foundation of China under Grants U22B2059 and 62176079.
References
[1] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[2] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https://gi tttgg.thub.... com www./oooo/gaee..oooooo huggihggacceeeeeeo oppnrrr
[3] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[5] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[6] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, al. Openai ol system card. arXiv preprint 2rXi::2412.167200 2024.
[7] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Banford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.. org/ htt.b///311008222
[8] Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu. Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 -626, 2023.
[9] Hung Le, Dai Do, Dung Nguyen, and Svetha Venkatesh. Reasoning under 1 billion: Memory- augmented reinforcement learning for large language models. arXiv preprint arXiv:2504.022755 022773.. 2025.
[10] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.1 18866 2025.
[11] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2055:22305.20050. 2023.
10

---

[12] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08566 2023.
[13] Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. 2025. Accessed: 2025-01-24.
[14] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.0634: 2017.
[15] Karim Shabani, Mohamad Khatib, and Saman Ebadi. Vygotsky's zone of proximal development: Instructional implications and teachers' professional development. English language teaching, 3(4):237-245 2010.
[16] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
[17] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12591 2025.
[18] Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models, arXiv preprint N5XYv:2503306777 2025.
[19] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.044: 043333 2024.
[20] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
[21] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024.
[22] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03388, 2025.
[23] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arviv:2503. 14476, 2025.
[24] Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. Supervised fine-tuning achieve rapid task adaption via alternating attention head activation patterns. arXiv preprint vxXiv:2409.15820 2024.
[25] Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Shi Jun, Ting Liu, and Bing Qi. Deci- phering the impact of pretraining data on large language models through machine unlearning. In ACL (Findings), 2024.
[26] Yang Zhao, Li Du. Xiao Ding, Yangou Ouyang, Hepeng Wang. Kai Xiong, Jinglong Gao, Zhouhao Sun, Dongliang Xu, Yang Qing, al. Beyond similarity: A gradient-based graph method for instruction tuning data selection. arXiv preprint arXiv:2502. 10002, 2025.
[27] Chunting Zhou, Pengfei Liu, Puxin Xu. Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems,
2023.
[28] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 2019.
arXiv preprint
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper introduces uncertainty-focused optimization approach within the abstract and introduction. Furthermore, we propose Efficient RL Data Selection. The introduction concludes by outlining the key contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The Limitations section is presented in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer: [Yes]
This study proposes a hypothesis based on cognitive theory (e.g., ZPD) that models learn most effectively on data with intermediate uncertainty. To implement this principle, we propose an efficient metric based on single-pass confidence to measure data difficulty and model uncertainty, and through experiments (Table 2 Table we validate the consistency of this metric with multi-sample accuracy (as a difficulty proxy) and its significant computational efficiency, demonstrating its effectiveness as difficulty proxy. Based on this metric, we propose the UFO-RL method for data selection, and through extensive experiments [ Table demonstrate the effectiveness of this method in improving RL fine-tuning efficiency and performance.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification:AApendi A.3 Appendix A.4 and Appendix A.5 provide detailed descriptions of key information for reproducing our results, ensuring the reproducibility of the paper's findings. We have also provided the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce
13

---

reproducible, they the model (e.g., with an open source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have submitted code and data sufficient for reproduction. We utilized publicly available datasets, which are detailed in Appendix A.4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
cwt.iiissssooiyiiiiooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttpp/.cc/pbllc///uiiiiii httpollccggiies///e/ wtw.ii/deeeeeeeeeeeeeeeeeeeeeeee
ssioonolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Appendix A.4, and Appendix A.5 provide detailed descriptions of key information for reproducing results, ensuring the reproducibility of the paper's findings
Justification: Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
14

---

Justification: As shown in Figure we have plotted the corresponding confidence intervals. For the experiments presented in Table []] the random baseline was independently run times, and we report the average value.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our hardware setup is described in Appendix A.5 the time overhead of our method in Table and Table
A.5 Additionally, we discuss
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All our research utilizes publicly available datasets, adhering to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
15

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Justification: Answer: [Yes]
A.2 includ- ing positive societal impacts (such computational efficiency gains, environmental benefits, increased research accessibility) and potential negative impacts (such as lowering the bar- rier to malicious use), considering the specific context of our work domain (mathematical reasoning).
Justification: We discuss the broader impacts of this research work in Appendix [..2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not involve releasing new data or models, and therefore does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
16

---

Answer: [Yes]
Justification: The data, models, and code used in this paper are all fully open-source, and appropriate citations have been provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsppppeaaaeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The provided code for this work includes complete documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
17

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our work investigates the rapid learning capabilities of LLMs in RL. Appendix A.5 providess detailed information on the key hyperparameter settings for the LLMs used during our training and evaluation phases.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtandard components.
Please refer to our LLM policy (https //nerrip///ups..... cc/Conferences/ /LLM) for what should or should not be described.
18

---

A Technical Appendices and Supplementary Material
A.1 Limitations
Although the UFO-RL method proposed in this paper has been effectively validated across multiple models and datasets, due to the substantial computational overhead of RL, our experiments were limited smaller-scale language models (up 8B parameters) and relatively small training datasets (up to 17K examples). The computational and training costs RL significantly increase when scaling up to larger models (e.g., those with more than 8B parameters) and larger datasets (e.g.. those containing millions or tens of millions of samples). As a result, experiments on these larger-scale models and datasets were not conducted in this work. Therefore, the experimental results presented in this paper do not cover the scenario of large-scale models and datasets, which remains an open challenge.
A.2 Broader Impacts
The UFO-RL method significantly reduces the computational resources required for RL-based fine- tuning of LLMs by improving data selection efficiency and reducing costly multi-sampling. This leads to faster training times and a substantial reduction in energy consumption, aligning with the principles of Green Al and promoting more sustainable Al development. It lowers financial and environmental costs, enhancing the accessibility of cutting-edge AI research for institutions with limited resources.
Potential Negative Impacts: While UFO-RL offers significant efficiency advantages, we also con- sidered potential negative societal impacts. A primary risk is that increasing the efficiency of LLM fine-tuning could lower the cost and technical barrier for malicious actors to fine-tune existing base models for harmful purposes (e.g., generating misinformation, toxic content). if these base models themselves pose such risks. Nevertheless, our work primarily focuses on data selection in the domain of mathematical reasoning. where the direct negative societal impact is typically less severe than in general text generation. Furthermore, our method focuses on optimizing the learning process and does not inherently introduce new harmful capabilities.
A.3 Model Details
For our experiments, we selected range of prevalent language models to evaluate. Specifically, we utilized models from the Qwen2.5 family (0.SB, 1.5B, 3B, 7B) (21). The Qwen series is commonly employed in various RL training contexts. [2][3] Additionally, we included Llama3. 1etmaa..-BB-inne 1-8B-instruct, widely recognized model known for its strong capabilities across many tasks, and Mistral-7B- Instructv0.3, more established architecture which has shown relatively average performance on some mathematical benchmarks. The selection covers models with varying parameter counts and architectural characteristics to assess performance broadly.
A.4
Dataset Details
Our experiments utilized specific datasets for both training and evaluation. For training, we primarily used the training split of GSM8K []] a widely adopted dataset for enhancing models' mathematical abilities through RL, which comprises relatively simple elementary school-level math problems. We complemented this with SAPO-MATH-I7E 23) more recent dataset focused on RL for math, featuring broader range of problem types and generally considered more difficult.
For evaluation, we assessed models on the GSM8K test set, which contains elementary math problems. To gauge performance on more complex mathematical and quantitative reasoning tasks, we included [11. Furthermore, MMLU [5) was used as general-domain text understanding benchmark to evaluate the models' overall capabilities and generalization ability beyond mathematical tasks.
A.5 Training and Evaluation Details
During the training phase, all experiments were conducted using the open-rl framework, executed on a computing cluster equipped with NVIDIA A 110 GPUs. parameters included a learning rate of le-6. e--.ning acceleration was achieved through the use of DeepSpeed Zero-2 optimization
19

---

technology. During the training process, seven responses were generated for each problem, with response generation accelerated using vLLM.
In the model evaluation phase, all our experiments employed a zero-shot evaluation method. To ensure fairness, the temperature was set to 0. Additionally, utilized vL..M to accelerate inference, thereby improving efficiency.
20


Reviews:
[{"review_id": "Q2eX1eZSGi", "reviewer": "Reviewer_3ahk", "summary": "This paper presents UFO-RL, an uncertainty-driven framework for improving the efficiency of reinforcement learning (RL) fine-tuning for large language models (LLMs). Drawing inspiration from the Zone of Proximal Development (ZPD) theory, the authors hypothesize that LLMs learn most effectively from examples that are neither too easy nor too difficult but exhibit moderate uncertainty. This paper first prove this hypothesis and then proposes a lightweight, single-pass confidence estimation based on average token-level log-probabilities, enabling efficient identification of informative training instances.", "questions": "1. The authors are encouraged to explicitly contrast their method with related work in active learning, providing a clearer positioning within the broader literature.\n2. Can the authors provide empirical results comparing UFO-RL’s single-pass confidence estimation to conventional multi-sampling methods in terms of both performance and computational cost?\n3. Are there scenarios where single-pass estimates significantly diverge from more robust multi-sample uncertainty metrics, and what are the practical implications of such divergences?\n4. Token-level log-probabilities may be affected by calibration issues or may fail to capture sequence-level semantic uncertainty. Have the authors evaluated the robustness of their method under varying model calibration qualities?\n5. Beyond mathematical reasoning, have the authors considered evaluating UFO-RL on language modeling, instruction tuning, or other RL fine-tuning tasks to demonstrate broader applicability?\n6. How does the proposed method scale with larger LLMs (e.g., >13B), where uncertainty estimation and data selection efficiency become more critical?", "limitations": "The same as the weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The application of ZPD-inspired learning dynamics to RL-based LLM fine-tuning is intuitively appealing. \n2. The proposed single-pass confidence estimation method provides a simple yet computationally efficient proxy for data difficulty, avoiding expensive multi-sampling or ensemble-based uncertainty estimation commonly used in other works.\n\nWeaknesses:\n1. While the application to LLM RL is novel, the broader concept of leveraging model uncertainty for data selection has been extensively explored in areas such as active learning, coreset selection, and data distillation. The paper does not mention these works in the paper.\n2. The comparison focuses primarily on full-data RL or random sub-sampling baselines. A more rigorous, direct comparison with multi-sampling-based difficulty estimation methods is missing, leaving open questions about the relative trade-offs in computational cost versus uncertainty estimation fidelity.\n3. The evaluation is concentrated on mathematical reasoning tasks with relatively small to mid-sized LLMs (up to 7B). The method’s scalability to larger models or applicability to diverse RL tasks  remains unexplored.", "was_revised": true, "final_justification": "I have discussed with the author and appreciate their detailed rebuttal.\n\nThe proposed single-pass confidence estimation method offers a simple yet computationally efficient proxy for data difficulty, avoiding the expensive multi-sampling or ensemble-based uncertainty estimation commonly used in prior work. In the rebuttal, the authors provided comparisons with multi-sampling methods, clarified the differences from related fields and works, and further evaluated generalization ability. I am satisfied with their responses and believe this paper presents a promising solution for data selection.\n\nTherefore I will raise my score.", "rebuttal": "We sincerely thank you for your comprehensive and insightful review. Your constructive feedback is invaluable for strengthening our work. We were encouraged that you found the application of ZPD theory \"intuitively appealing\" and our single-pass confidence estimation \"a simple yet computationally efficient proxy.\"\n\nWe have carefully considered all your comments and conducted new experiments to address your questions. Below, we address your concerns point-by-point.\n\n### **On Weaknesses and Core Questions**\n\n**1. Positioning with Respect to Related Work**\n\nThank you for encouraging us to clarify our work's positioning. Our framework provides an efficient data selection strategy for RL fine-tuning, which is fundamentally different in its **goals** and **paradigm** from existing data selection fields. We will incorporate this discussion into the camera-ready version of our paper to better delineate our unique contributions.\n\n* **vs. Active Learning**: Active Learning aims to minimize **human labeling costs**. In contrast, UFO-RL is designed to minimize **computational costs** (i.e., GPU hours) in RL training, a distinct optimization dimension.\n* **vs. Coreset Selection**: The goal of Coreset Selection is to find a small, **representative** subset that approximates the full dataset. Conversely, inspired by ZPD theory, UFO-RL intentionally creates a **biased, non-representative** subset that is most **effective** for the model's current learning state.\n* **vs. Data Distillation**: Data Distillation is a model compression technique concerned with knowledge transfer between different models. UFO-RL focuses on the **self-improvement of a single model** by optimizing its own learning process.\n\n**2. Direct Comparison with Multi-Sampling Methods**\n\nThis is a crucial point. To provide the direct comparison you requested and to demonstrate our method's consistency across various model sizes, we have run new experiments and present the complete results for the Qwen2.5 family below.\n\n**Table A: Performance & Cost Comparison on GSM8K (on a single 8xA100 node)**\n| Model | Method | Data Selection Time (s) | Total RL Training Time (s) | GSM8K Acc (%) | Math500 Acc (%) |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| **Qwen2.5-0.5B** | Multi-sample | 422 | 142 | 45.91 | 30.20 |\n| | **UFO-RL (Ours)** | **2** | **140** | **46.27** | **30.60** |\n| | Full Data | 0 | 1815 | 52.66 | 30.80 |\n| **Qwen2.5-1.5B** | Multi-sample | 464 | 411 | 76.34 | 55.00 |\n| | **UFO-RL (Ours)** | **6** | **407** | **76.63** | **55.40** |\n| | Full Data | 0 | 5694 | 76.78 | 53.00 |\n| **Qwen2.5-3B** | Multi-sample | 524 | 742 | 83.48 | 65.20 |\n| | **UFO-RL (Ours)** | **11** | **739** | **84.37** | **67.40** |\n| | Full Data | 0 | 10224 | 83.33 | 66.20 |\n| **Qwen2.5-7B** | Multi-sample | 854 | 1211 | 91.14 | 76.00 |\n| | **UFO-RL (Ours)** | **22** | **1154** | **92.03** | **76.40** |\n| | Full Data | 0 | 12959 | 91.88 | 75.00 |\n| **Qwen2.5-14B (New)** | Multi-sample | 1145 | 1841 | 91.71 | 78.40 |\n| | **UFO-RL (Ours)** | **39** | **1787** | **93.80** | **79.60** |\n| | Full Data | 0 | 22228 | 93.53 | 79.40 |\n\n**Key Findings:**\n* **Performance & Efficiency:** From 0.5B to 14B parameters, UFO-RL consistently achieves performance comparable to or better than the baselines while drastically reducing the total training time. This consistent trend highlights the method's robustness.\n* **Regarding Metric Divergence:** We argue that the difference between metrics stems from the **superior granularity of our method**. Our continuous confidence score provides a more precise proxy for the ZPD, enabling better sample selection.\n\n**3. Robustness to Model Calibration and Sequence-Level Uncertainty**\n\nThis is an excellent question. Our method is inherently robust to calibration issues. To provide empirical proof, we conducted a new experiment using **temperature scaling** (T=0.8 and T=1.2) to simulate different calibration states on Qwen2.5-7B. The resulting 10% data subsets selected by UFO-RL had an **85.8% overlap**, confirming the method's robustness. We plan to add this analysis to the final version of the paper.\n\n**4. Generalization Ability**\n\nTo address your concerns about generalization, we verified our method's effectiveness across both **task types** and **model scales**.\n\n* **Task Generalization:** To demonstrate broader applicability beyond math, we evaluated UFO-RL on the **AI2 Reasoning Challenge (ARC-C)**, a scientific QA task.\n\n    **Table B: Performance & Efficiency on ARC-Challenge (LLaMA3.1-8B)**\n    | Method | Data % | Data Selection Time (s) | Total Training Time (s) | ARC-C Acc (%) |\n    | :--- | :--- |:--- | :--- | :--- |\n    | Full Data | 100% | 0 | 2432 | 87.11 |\n    | Random | 10% | ~1 | 214 | 86.26 |\n    | **UFO-RL (Ours)** | **10%** | **4** | **203** | **88.23** |\n\n    The results confirm that UFO-RL is also effective for general reasoning tasks, outperforming the full-data baseline with ~1/12 of the training time.\n\n* **Scalability to Larger Models:** The results in **Table A** clearly show a consistent and effective scaling path from 0.5B to **14B** models, robustly demonstrating our method's effectiveness and scalability.\n\n**5. Writing Quality**\n\nThank you for the feedback. We have thoroughly reviewed the manuscript and will further improve its clarity, precision, and conciseness in the final version.\n\nWe again thank you for your valuable time and expertise. We believe the new experiments and analyses have significantly strengthened our paper and addressed your concerns. We hope you will re-evaluate our work favorably."}, {"review_id": "j0RG44TrSg", "reviewer": "Reviewer_y4mU", "summary": "The paper introduces UFO-RL, a data-selection framework for RL fine-tuning of LLMs. A single-pass confidence score replaces 16-sample accuracy, making uncertainty estimation up to 185 × faster and letting the authors train on the top “fuzzy” data. In the experiments, model trained on subset matches or exceeds full-data performance.", "questions": "Could you provide further justification for the specific quadratic form $1 - (s_i - \\mu)^2$? Have you experimented with other functions to isolate the \"intermediate uncertainty\" region? \n\nHave you considered a dynamic implementation where confidence scores and the selected data subset are periodically re-calculated during training, may act as ablation study?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n- Lightweight uncertainty estimation: Confidence evaluation is up to 185 × faster than 16-sample accuracy.\n- Broad evaluation: Results span Qwen2.5, Llama-3.1, Mistral families and three test sets, with multiple runs and error bars. \n\nWeaknesses:\n- Heuristic method: The whole method is based on ZPD hypothesis, which has no theoretical basis. The paper did include experiments to verify the hypothesis, but in Fig.1, red line may overstate the clarity of the trend, the raw data (black dots) shows inconsistencies, especially for smaller or weaker models. \n\n- Confidence score: Token-level confidence estimation is not a new concept, and it may not serve as a reliable proxy for semantic uncertainty. The similarity between confidence and semantic uncertainty is sensitive to prompt design; however, the paper does not include any experiments analyzing prompt effects on similarity in Table 2.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your invaluable feedback and insightful comments on our paper. Your review is highly constructive and has been instrumental in helping us refine our work. We are encouraged that you recognize the significant advantages of our method in terms of computational efficiency and breadth of evaluation. In response to the concerns and questions you raised, we have conducted further analysis and supplementary experiments, which we address point-by-point below.\n\n### **Regarding Main Concerns (Weaknesses)**\n**1. On the Theoretical Basis and Heuristic Nature of the Method**\n**Reviewer's Comment:** \"The whole method is based on ZPD hypothesis, which has no theoretical basis. The paper did include experiments to verify the hypothesis, but in Fig.1, red line may overstate the clarity of the trend, the raw data (black dots) shows inconsistencies, especially for smaller or weaker models.\"\n\n**Our Response:**\nWe appreciate this insightful point. We wish to clarify that the motivation for our work is **grounded in the intrinsic mechanisms and efficiency bottlenecks of Reinforcement Learning for LLMs, particularly algorithms like GRPO**.\n\nIn RL algorithms such as GRPO, policy optimization relies on multi-sampling from the same prompt and calculating gradients based on the collective reward signal from the resulting generation group. Through this process, we identified a critical inefficiency when the data is either too easy or too difficult:\n\nIf a problem is **too easy**, two problematic scenarios arise. First, if all sampled generations are correct, the uniform reward of 1 provides no reward variance, leading to a vanishing gradient[1]. Second, if the group contains mostly correct answers and only a few incorrect ones, the group-relative optimization will assign a negative reward signal to the correct answers, incorrectly penalizing good generations.\n\nConversely, if a problem is **too difficult**, similar issues occur. If all generations are incorrect, the uniform reward of 0 again results in a vanishing gradient[1]. If the group consists of mostly incorrect answers and only a few correct ones, the group-relative optimization will not only heavily reward the rare correct answers but may also assign a positive signal to \"less incorrect\" answers, causing the model to learn a suboptimal or flawed policy.\n\nIn both of these extreme cases, the lack of reward variance or the skewed reward signals cause the learning process to become ineffective or unstable. This observation from algorithmic practice forms the direct motivation for our research: **to achieve efficient RL training, it is imperative to select data that can produce a mix of correct and incorrect outcomes**.\n\nIt is precisely from this algorithm-level requirement that we introduce the Zone of Proximal Development (ZPD) theory. We employ it not as a strict mathematical theorem, but as a powerful **conceptual framework** to explain and guide how we can systematically identify and select this \"intermediate difficulty\" (or \"fuzzy\") data, which provides the most effective learning signals for algorithms like GRPO.\n\nRegarding the volatility in Figure 1, we agree with your observation and consider it expected for smaller models. However, the crucial finding is that **a clear non-monotonic trend consistently emerges across all six models.** This cross-model consistency strongly demonstrates that focusing on intermediate-difficulty data is a universally effective optimization strategy that directly addresses the vanishing-gradient problem inherent in the RL algorithm itself.\n\n**2. On the Novelty and Reliability of the Confidence Score**\nReviewer's Comment: \"Token-level confidence estimation is not a new concept, and it may not serve as a reliable proxy for semantic uncertainty. The similarity between confidence and semantic uncertainty is sensitive to prompt design; however, the paper does not include any experiments analyzing prompt effects on similarity in Table 2.\"\n\n**Our Response:**\nWe thank the reviewer for this close examination, which allows us to further clarify our core innovation.\n\nAs mentioned, methods like GRPO require identifying \"non-extreme\" data, but the conventional approach—**evaluating data via actual multi-sampling—imposes a prohibitive computational cost**, which is the central problem we aim to solve.\n\nWe fully agree that \"token-level confidence\" is not a new concept. Our core innovation lies in its application: We are the first to propose using the confidence score from a **single forward pass** as a **computationally efficient proxy** to predict a sample's \"fuzziness\" under multi-sampling. In other words, we use it to rapidly identify data most likely to provide a useful learning signal for GRPO, thereby **completely bypassing the expensive multi-sampling evaluation process**. Our contribution is a practical engineering framework (UFO-RL) designed to enhance RL training efficiency by directly tackling this algorithmic bottleneck.\n\nRegarding its reliability as a proxy, we followed your suggestion and conducted a **prompt-sensitivity ablation study**. The results (table below) show that despite variations in prompt style, the similarity between our confidence score and multi-sample accuracy remains high (>0.8), demonstrating its robustness as an efficient proxy.\n\n**Supplemental Experiment: Effect of Prompt Design on Confidence-Accuracy Similarity**\n| Model | Prompt Template | Similarity |\n| :--- | :--- | :--- |\n| Qwen2.5-7B | Template 1 (Original) | 0.86 |\n| | Template 2 (Concise) | 0.84 |\n| | Template 3 (CoT) | 0.85 |\n| Llama3.1-8B | Template 1 (Original) | 0.82 |\n| | Template 2 (Concise) | 0.81 |\n| | Template 3 (CoT) | 0.83 |\n\nWe will add this experiment to the appendix to further strengthen the case for our method's reliability.\n\n### **Regarding Specific Questions**\n**1. On the Justification for the Quadratic Form of the \"Fuzziness Score\"**\n**Reviewer's Comment**: \"Could you provide further justification for the specific quadratic form $Score(s_i)=1−(s_i−\\mu)^2$? Have you experimented with other functions to isolate the 'intermediate uncertainty' region?\"\n\n**Our Response:**\nThis is an excellent question. We chose the quadratic function because it is the most mathematically simple and computationally efficient way to model our goal of finding \"intermediate difficulty\" data—that is, to assign the highest score to samples with confidence near the mean, which are most likely to produce the mixed-reward signals GRPO requires.\n\nTo prove that our method's success is not contingent on this specific function, we conducted a new ablation study. In this experiment, we used a scoring function based on absolute distance $Score = 1 - |s_i - \\mu|$, which also selects for data near the mean but is simpler and has no additional hyperparameters. The results show that this function performs almost identically to the original quadratic function. This more strongly demonstrates that **the effectiveness of UFO-RL stems from the core principle of selecting intermediate-difficulty data, not from a specific mathematical formula**.\n\nSupplemental Experiment: Performance Comparison of Different \"Fuzziness Score\" Functions\n| Model | \"Fuzziness Score\" Function | GSM8K Accuracy (%) |\n| :--- | :--- | :--- |\n| **Qwen2.5-7B** | Quadratic (Original) | 92.03 |\n| | Absolute Distance | 91.91 |\n| **Llama3.1-8B** | Quadratic (Original) | 88.30 |\n| | Absolute Distance | 88.18 |\n\n**2. On a Dynamic Data Selection Ablation Study**\n**Reviewer's Comment:** \"Have you considered a dynamic implementation where confidence scores and the selected data subset are periodically re-calculated during training, may act as ablation study?\"\n\n**Our Response:**\nThis is a very insightful suggestion. As the model learns via the GRPO algorithm, its capabilities change, and thus the range of \"effective data\" should also be dynamic. We therefore **conducted a new ablation study** to explore a curriculum-based dynamic strategy.\n\nIn this dynamic implementation, we first selected the 5% \"fuzziest\" data from the entire training set and trained the model on this subset for the first half of the training process. Then, using the updated model, we re-evaluated the remaining 95% of the data and selected the new 5% \"fuzziest\" data for the second half of training. This ensures the model is always focusing on the most informative samples relative to its current state.\n\nThe results (as shown in the newly added table below) indicate that this dynamic strategy does yield a small but **consistent performance improvement** (~0.3-0.4%), but at the cost of an additional data selection step. Given that a core goal of our work is to **maximize efficiency while maintaining performance**, the current static method achieves a better trade-off between performance and cost. Nevertheless, we fully recognize the potential of dynamic selection and will discuss it in our future work.\n\n**Supplemental Experiment: Static vs. Dynamic Data Selection Strategy**\n| Model | Method | GSM8K Accuracy (%) | Data Selection Time (s) |\n| :--- | :--- | :--- | :--- |\n| **Qwen2.5-7B** | UFO-RL (Static) | 92.03 | 175 |\n| | UFO-RL (Dynamic 5%+5%) | **92.35** | 175 (initial) + 161 (mid) = 336 |\n| **Llama3.1-8B** | UFO-RL (Static) | 88.30 | 186 |\n| | UFO-RL (Dynamic 5%+5%) | **88.72** | 186 (initial) + 175 (mid) = 361 |\n\nOnce again, we sincerely thank you for your detailed review and insightful feedback. We are confident that the revisions and additions made in response to your comments will significantly strengthen the quality and impact of our paper. We look forward to your further feedback on our revised manuscript.\n\n[1]Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025."}, {"review_id": "N8FyBFSFcY", "reviewer": "Reviewer_fb4U", "summary": "This paper proposes UFO-RL (Uncertainty-Focused Optimization for Reinforcement Learning) which can make the LLM finetune more effectively by intelligently selecting training data. It is inspired by the Zone of Proximal Development (ZPD) theory, which says that model benefits more from data that have not yet mastered but demonstrate the potential to comprehend. So this paper proposes the method to access the difficulty/estimate the uncertainty of data, so that to identify informative training instances. The paper claims that this estimation method can speed up the previous methods for estimation.", "questions": "Please refer to the weakness part.", "limitations": "No.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper introduces an interesting question about how to train LLM more effectively. \n2. The experiment parts is interesting and it is tested for different LLMs that this data selection method is effective.\n\nWeakness:\nI think the major weakness is lack of clear structure in presenting the methods. \n1. The paper initially mentions that this method is inspired by the ZPD theory. It sounds interesting but is there any part about theoretic proof or intuitively how this method can have theoretical basis is introduced? I feel it is more like an experiment guided results without too much theory basis.\n2. The method is not clearly explained in the paper. While most contents of the paper focus on experimental results, what is exactly the algorithm to pick data that have not yet mastered but demonstrate the potential to comprehend?", "was_revised": true, "final_justification": "I think the rebuttal solves my concern about the relationship between theory and experimental result in this paper. Therefore I raised my socre.", "rebuttal": "We sincerely thank you for your valuable time and constructive feedback. We agree that a solid theoretical foundation is crucial. Your insightful comments have been instrumental in helping us to improve the rigor and clarity of our work.\n\nBefore addressing your specific concerns, we would like to emphasize that **our original manuscript had already laid the groundwork for our method's machine learning basis in several key sections**:\n\n1.  **Problem Definition (Introduction)**: We explicitly identified the core challenge in RL fine-tuning as the need for \"robust policy gradient estimation\" and noted the lack of mechanisms to prioritize instances yielding the most impactful \"learning signal\".\n2.  **Sampling-based Difficulty Analysis (Section 3)**: We quantified data difficulty using \"sampling accuracy,\" a statistical estimate based on empirical rewards, and our experiments provided initial evidence that data of intermediate difficulty is most effective for learning.\n3.  **Confidence-based Efficiency Optimization (Section 4)**: We proposed an efficient confidence estimation method based on the model's intrinsic generation probabilities, requiring only a \"single forward pass\" as a more efficient machine learning metric for uncertainty.\n\nYour review has prompted us to connect these analyses more deeply and explicitly with the intrinsic mechanism of the **GRPO (Group-Relative Policy Optimization)** algorithm used in our experiments.\n\n---\n\n**Concern 1: Deepening the Theoretical Foundation — Why Extreme Data is Harmful for GRPO**\n\nWe wish to clarify that our method's efficacy stems from its precise alignment with the intrinsic mechanism of the GRPO algorithm, specifically by avoiding the pitfalls that arise when GRPO processes data of extreme difficulty.\n\nGRPO's core strategy is to learn from the contrast between \"good\" and \"bad\" responses generated for the same prompt. This contrastive mechanism fails—and can even produce harmful signals—when the data is not well-calibrated:\n\n* For **\"easy\" data**, where the model's generated responses are almost all correct, the GRPO algorithm **loses its basis for effective contrast**. In this scenario, its optimization objective becomes ambiguous. Instead of learning the fundamental difference between 'right' and 'wrong', it might be misled into distinguishing between subtle, non-essential variations (e.g., phrasing) among multiple correct answers. This can generate a **perverse learning signal**, which erroneously penalizes a perfectly valid solution simply because it differs slightly from another correct one in the same group.\n\n* Similarly, for **\"hard\" data**, where all responses are incorrect, GRPO **lacks a correct 'positive example' to learn from**. To perform its relative optimization, the algorithm may be forced to reward the \"least incorrect\" response among a group of failures. This creates a **misleading learning signal**, as it is **not teaching the model how to be correct but is instead reinforcing a flawed reasoning path**, merely because it is marginally better than other flawed paths.\n\nTherefore, the data that allows the GRPO algorithm to function most effectively and safely is precisely the **\"intermediate difficulty\" or \"fuzzy\" data** identified by UFO-RL. These instances reliably provide a mix of correct and incorrect answers, offering the clean and true contrast necessary for GRPO to generate a high-quality policy gradient. This demonstrates that UFO-RL is not just a tool for efficiency, but also a **necessary safeguard for the training stability and correctness of contrast-based algorithms like GRPO**.\n\n---\n\n**Concern 2: Clarification of the Algorithm**\n\nWe completely agree that the algorithm's description required more clarity. The process is deterministic and reproducible as follows:\n\n1.  **Confidence Calculation**: For each sample, a single forward pass computes the average log-probability of the generated sequence, which serves as its confidence score.\n2.  **\"Fuzziness\" Score Calculation**: A \"fuzziness\" score, $Score(s_{i})=1-(s_{i}-\\mu)^{2}$, is calculated to identify samples within the ZPD, where $s_i$ is the normalized confidence and $\\mu$ is the dataset's mean confidence. This prioritizes samples with near-average confidence.\n3.  **Ranking and Selection**: Samples are ranked by their \"fuzziness\" score in descending order, and the **top 10% are selected** for the training set.\n\n**Revision Plan**: To ensure maximum clarity, we will add a formal **Algorithm pseudo-code box** to the final version of the paper.\n\n---\n\n**In Summary**\n\nThank you once again for your insightful review. We hope this clarification clearly demonstrates that our work is built on a solid analytical foundation and that our theoretical core is strongly coupled with the intrinsic mechanism of the GRPO algorithm. Our method is designed to select data that yields the most effective learning signal while avoiding harmful, misleading updates. We are confident that by making these points more explicit, the paper will be substantially strengthened.\n\nWe sincerely hope these explanations fully address your concerns and kindly ask you to reconsider your evaluation of our work."}, {"review_id": "dd48jik7ZY", "reviewer": "Reviewer_bKLK", "summary": "RL is one of the core procedures in LLM post-training. Data, constituting the environment for training LLMs with RL, is the key to its success. This paper investigates the important direction of data-centric RL, and proposes an efficient and effective approach to select the most helpful training data. The proposed approach, named UFO-RL, uses an uncertainty-based metric to rank the training data by their closeness to being of intermediate difficulty. Empirically, by only using 10% of the selected data, the LLMs can be trained to reach performance that is competitive to training with the full dataset, also outperforming other proposed baselines.", "questions": "1. Could you elaborate on the similarity metrics used in Table 2?\n2. In lines 168-169, it does not seem to be consistently true that larger models show more robustness when trained on harder examples compared to smaller models in Figure 1.\n3. While the paper demonstrates the effectiveness using 10% of the data, it is unclear why such a choice is made. It would be helpful to have a plot that shows the change in performance if lower/higher percentage of data points are supplied, so that we can have a better understanding on the relationship between the fraction of the data and the model performance, and further examine whether it is possible to outperform training with full data.", "limitations": "As discussed in weakness 1, there are insufficient results to support the claims for other policy optimization algorithms other than GRPO.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well written with intriguing motivation from Zone of Proximal Development. \n2. The pilot study in Section 3 demonstrates that data with intermediate difficulty benefits RL for LLMs the most with great clarity. \n3. The proposed uncertainty-based approach is sound and reasonable in the field of LLMs, well-supported by the pilot study and empirical verifications. \n4. Comprehensive experiments have been performed on a variety of LLMs across different sizes and three datasets. The experimental results also show the efficiency and effectiveness of the proposed method. \n\n**Weaknesses**\n\n1. Dependency on GRPO. The paper seems to only demonstrate that the idea, findings, and results for confidence-based data selection work well with GRPO, which will not update the model efficiently when the samples are too easy or too difficult, due to the design of its advantage function. However, GRPO is not representative of all RL algorithms. It is unclear whether the same conclusion could be drawn for other algorithms such as PPO or REINFORCE. Without such experiments, it is insufficient to prove that the proposed idea works for RL in general, and it might be better to qualify the main theme of the paper to GRPO instead. \n2. Content unclarity. Table 1 and its related content look a bit confusing. It is unclear what numbers in the table represent, as they don’t seem to be valid distributions (not summing up to one). The authors said “A considerable fraction of examples for many models is concentrated in the extreme accuracy bins” on lines 172-173, but the numbers seem to be even higher in (85, 100] for Qwen2.5-7B and Llama3.1. \n3. Missing details in comparison. Computational Cost Analysis in Section 5.3 might not have been fair, because the initial computational overhead incurred during example scoring is not discussed. In Table 6, it is also unclear which dataset is used to measure the speed, and whether the time is for one step or for the entire training (training Qwen2.5-0.5B in 140s seems too fast to be true if it’s for the entire training).", "was_revised": true, "final_justification": "During the rebuttal, the authors have successfully addressed or promised to address most of my concerns. It's an interesting paper with an effective methodology that improves the training efficiency and performance for RLVR for LLM (see strengths). The only concern I still have is that the support for the method, UFO, is not strong enough.\n\nTo claim that UFO is useful for RL, we need strong theoretical and empirical support. The ZPD theory, though intuitive and interesting, is not mathematically justified. Therefore, the requirements for experimental comprehensiveness are typically higher. As acknowledged by the authors, the paper was tied to GRPO, and it is reasonable to provide additional justification for other RL algorithms. The authors additionally ran PPO with a variety of Qwen models and showed that their method is helpful. However, it's still only on one dataset, GSM8k, despite the fact that this concern was raised during the discussion. And also, it's only for one family of models, Qwen. \n\nI am positive about the work, but I cannot give a higher score because of the aforementioned weaknesses.", "rebuttal": "Thank you very much for your constructive and insightful review. We agree with the key weaknesses you identified and have conducted a series of new supplementary experiments to address your concerns. We believe these revisions significantly strengthen our paper. Our responses are as follows:\n\n**1. Regarding Algorithm Dependency: Focused on GRPO, Applicable to PPO**\nWe thank the reviewer for this critical point. We wish to emphasize that while **our work is primarily focused on and validated within the GRPO framework**, the underlying principle of our method—selecting data from the Zone of Proximal Development —is algorithm-agnostic.\n\nTo prove its broader applicability, we **additionally tested UFO-RL on the mainstream Proximal Policy Optimization (PPO) algorithm**.\n\n* **New Experimental Results (Qwen2.5-7B on GSM8K):**\n    | Algorithm | Data Strategy      | Accuracy (%) |\n    | :--- | :--- | :--- |\n    | PPO | Full Data (100%) | 90.48 |\n    | PPO | Random (10%)     | 89.51 |\n    | **PPO** | **UFO-RL (10%)** | **90.77** |\n\n* **Conclusion:** This experiment confirms that UFO-RL's effectiveness is not confined to GRPO and generalizes successfully to PPO. We will clarify in the revised manuscript that GRPO is our primary framework while including this PPO validation to demonstrate the generalizability of our approach.\n\n**2. Regarding Lack of Clarity in Table 1:**\nWe apologize for the confusion regarding Table 1. Your observation is correct, and it highlights a lack of clarity in our presentation, which we will fix.\n\nThe **core purpose** of this table was to illustrate that a significant portion of the data is polarized at the extremes (i.e., consistently wrong or consistently correct). To emphasize this point, the original table **intentionally presented only a subset of the data distribution**—specifically the percentages for data points near the extremes. This is why the percentages shown did not sum to 100%, which understandably caused confusion.\n\nYou correctly noted that for stronger models like Qwen2.5-7B, the `(85, 100]` bin contains a very large fraction of the data. This actually reinforces our central argument: for capable models, the multi-sampling accuracy metric becomes less useful as it classifies most data as \"easy,\" failing to provide the fine-grained difficulty assessment needed for efficient learning. This is precisely the problem our continuous confidence score is designed to solve.\n\nTo resolve this, we will revise Table 1 in the paper to show a complete, granular, and mutually exclusive distribution.\n\n* **Proposed New Format for Table 1:**\n| Accuracy Bin | Description | Qwen2.5-7B (%) | Llama3.1-8B (%) |\n| :--- | :--- | :--- | :--- |\n| `Acc = 0%` | Consistently Wrong | 1.36 | 0.90 |\n| `(0, 15%)` | Mostly Wrong | 1.85 | 1.50 |\n| `[15%, 85%]` | Intermediate | 21.08 | 43.86 |\n| `(85%, 100%)`| Mostly Correct | 26.64 | 32.21 |\n| `Acc = 100%` | Consistently Correct | 49.07 | 21.53 |\n| **Total** | | **100.00** | **100.00** |\n\nThis revised format will transparently show the entire data distribution and more effectively highlight the polarization issue that motivates our work.\n\n**3. Regarding Fairness of the Computational Cost Analysis:**\nYour query regarding the end-to-end cost analysis is entirely correct. To provide a fully transparent and fair comparison, we have updated the **end-to-end cost analysis to cover all Qwen model sizes.** **As we stated in Appendix A.5 of our original manuscript, all training was conducted on a computing cluster equipped with 8 NVIDIA A100 GPUs**. This powerful hardware configuration explains the fast training speeds on smaller data subsets. We will state this configuration more explicitly in the main text of the paper.\n\n* **New End-to-End Performance and Cost Comparison (GSM8K, 8x A100 GPU):**\n| Model | Method | Data Selection Time (s) | RL Training Time (s) | **Total Time (s)** | **Total Speedup** (vs. Full) | GSM8K Acc (%) | Math500 Acc (%) |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Qwen2.5-0.5B | **UFO-RL (Ours)** | 2 | 140 | **142** | **12.8x** | 46.27 | 30.60 |\n| | Full Data | 0 | 1815 | 1815 | 1.0x | 52.66 | 30.80 |\n| Qwen2.5-1.5B | **UFO-RL (Ours)** | 6 | 407 | **413** | **13.8x** | 76.63 | 55.40 |\n| | Full Data | 0 | 5694 | 5694 | 1.0x | 76.78 | 53.00 |\n| Qwen2.5-3B | **UFO-RL (Ours)** | 11 | 739 | **750** | **13.6x** | 84.37 | 67.40 |\n| | Full Data | 0 | 10224 | 10224 | 1.0x | 83.33 | 66.20 |\n| Qwen2.5-7B | **UFO-RL (Ours)** | 22 | 1154 | **1176** | **11.0x** | 92.03 | 76.40 |\n| | Full Data | 0 | 12959 | 12959 | 1.0x | 91.88 | 75.00 |\n\n* **Conclusion:** The comprehensive data show that this efficiency advantage holds across all model sizes. The one-time screening cost of UFO-RL is negligible compared to the massive savings in RL training time, ultimately achieving a **10-14x end-to-end training speedup across the entire model family** while maintaining or even surpassing the performance of full-data training.\n\n---\n#### **Responses to Specific Questions:**\n\n1.  **What is the similarity metric in Table 2?**\n    It is the **Spearman's rank correlation coefficient**. We chose it because we are more concerned with the ranking consistency of data difficulty between the two methods rather than a linear relationship. We will state this explicitly in the revised text.\n\n2.  **Regarding the claim that \"larger models are more robust\"?**\n    Your observation is correct; the original statement was too absolute. We will revise it to a more nuanced \"**a tendency for the performance peak to shift to more difficult data**,\" which is more precise and better reflects the trends shown in Figure 1.\n\n3.  **Why was 10% of the data chosen?**\n    This is an excellent question. We conducted an **ablation study on the data selection percentage (k)** to determine the optimal trade-off between cost and performance.\n\n   * **New Experimental Results (UFO-RL, Qwen2.5-7B):**\n| Data Percentage (k%) | 5% | **10%** | **20%** | 30% | 100% (Full) |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Accuracy (%) | 90.47 | **92.03** | **92.71** | 92.41 | 91.88 |\n\n\n   * **Conclusion:** Performance peaks around 20%.We chose 10% as our main reported number because it **achieves the best trade-off between computational cost** (half that of the 20% run) **and model performance** (over 99% of the peak performance), demonstrating the extreme efficiency of our method.\n\n---\n**Summary**\n\nThank you again for your valuable feedback. We believe these new experiments and revisions substantially strengthen the rigor and generalizability of our work and have fully addressed your concerns. We hope to earn your support for our paper."}]

Author rebuttal:

Meta-review: This sbumission proposes an uncertainty-driven framework UFO-RL for data selection in LLM RL. The idea is inspired by the Zone of Proximal Development theory. It hypothesizes that LLMs learn most effectively from data of intermediate difficulty. UFO-RL estimates sample difficulty using a single-pass confidence metric based on average token-level log-probabilities, which hel0ps avoid the computational cost of traditional multi-sampling strategies. Experiments show that UFO-RL accelerates data evaluation by up to 185× compared to multi-sampling approaches, and training on the top 10% fuzzy subset of data achieves comparable or superior results to training on the full dataset across multiple LLM models and tasks. 

Strengths of this submission: 
1. reviewers R3ahk, y4mU recognize that UFO-RL achieves up to 185× faster uncertainty estimation and over 10× e2e training speedup compared to multi-sampling methods.
2. reviewers bKLK, fb4U like the experimental setup (three LLM families: qwen, mistral, and llama on different tasks and datasets.
3. the idea of adopting zpd into the context of LLM RL with an efficient uncertainty metric is a practically valuable contribution, according to two reviewers.

Weaknesses of this submission as pointed out by the reviewers: 
1. though reviewers enjoy the idea of borrowing from the ZPD hypothesis, it is mostly empirical. 
2. the reviewers argue that while additional PPO experiments were provided, the method remains primarily validated under GRPO, and broader applicability to other RL algorithms remains underexplored --- on which the AC disagrees with the reviewer. Since it works on both PPO and GRPO --- two representative frameworks, it is difficult to require more experiments as long as the authors do not overstate its generality. 
3. some reviewers noted the absence of comparisons with related techniques like active learning or data distillation and requested clearer explanations of certain tables and computational cost analyses (R3ahk, bKLK).



During rebuttal, the authors provided additional experiments comparing UFO-RL with multi-sampling methods across models up to 14B parameters. It helps demonstrate comparable performance with significant efficiency gains. They clarified distinctions from active learning, coreset selection, and data distillation, reinforcing the idea of their approach. To address concerns about generality, they added PPO-based experiments across multiple Qwen models, showing similar benefits, though some reviewers noted that more diverse settings would be preferable. While one reviewer maintained skepticism regarding theoretical justification, most reviewers acknowledged that the new results resolved their primary concerns and (planned to) raise their scores accordingly. Overall, the rebuttal strengthens the case for acceptance.

In summary, the submission addresses an important problem in scaling RL for LLMs. It presents a simple and effective method that substantially reduces computational cost without compromising performance. While theoretical grounding is limited and evaluation breadth could be expanded, the experiments and added analyses during rebuttal strengthen confidence in the method’s practical value. All factors considered, this submission is recommended for acceptance if the conference space allows.
Decision: Accept (poster)