Title: UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Selection

LLMs

Reinforcement Learning
Abstract: A primary impediment to scaling reinforcement learning (RL) for large language model (LLM) training is the substantial computational cost, predominantly arising from the necessity of multi-sampling for policy optimization and evaluation. This underscores the critical yet challenging nature of efficient training data selection. Drawing inspiration from the Zone of Proximal Development (ZPD) theory, which posits that learners acquire knowledge more effectively from tasks of intermediate difficulty, we hypothesize that LLMs exhibit optimal learning from data they have not yet mastered but demonstrate the potential to comprehend. Conventional methodologies for assessing data difficulty or informativeness typically rely on computationally intensive multi-sampling or iterative procedures. To address this limitation, we introduce UFO-RL (**U**ncertainty-**F**ocused **O**ptimization for **R**einforcement **L**earning), a novel framework that employs a computationally efficient single-pass uncertainty estimation technique to identify informative training instances. This method, requiring only a single forward pass and obviating the need for iterative next-token computation, achieves a significant acceleration (up to 185$\times$) in data evaluation compared to multi-sampling approaches. UFO-RL leverages this efficient metric to select data within the model's estimated ZPD for training. Extensive experimentation across diverse LLMs and mathematical benchmarks demonstrates that training with a mere 10\% of the data, carefully selected by UFO-RL, yields performance comparable to or even surpassing that of full-data training. Furthermore, this targeted data selection results in up to a 16$\times$ reduction in overall training time, concurrently enhancing training stability and improving generalization capabilities. Thus, UFO-RL presents a practical and highly efficient strategy for scaling RL fine-tuning of LLMs by focusing learning efforts on the most informative and valuable data, thereby mitigating the computational bottlenecks associated with traditional RL training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRL is one of the core procedures in LLM post-training. Data, constituting the environment for training LLMs with RL, is the key to its success. This paper investigates the important direction of data-centric RL, and proposes an efficient and effective approach to select the most helpful training data. The proposed approach, named UFO-RL, uses an uncertainty-based metric to rank the training data by their closeness to being of intermediate difficulty. Empirically, by only using 10% of the selected data, the LLMs can be trained to reach performance that is competitive to training with the full dataset, also outperforming other proposed baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well written with intriguing motivation from Zone of Proximal Development. \n2. The pilot study in Section 3 demonstrates that data with intermediate difficulty benefits RL for LLMs the most with great clarity. \n3. The proposed uncertainty-based approach is sound and reasonable in the field of LLMs, well-supported by the pilot study and empirical verifications. \n4. Comprehensive experiments have been performed on a variety of LLMs across different sizes and three datasets. The experimental results also show the efficiency and effectiveness of the proposed method. \n\n**Weaknesses**\n\n1. Dependency on GRPO. The paper seems to only demonstrate that the idea, findings, and results for confidence-based data selection work well with GRPO, which will not update the model efficiently when the samples are too easy or too difficult, due to the design of its advantage function. However, GRPO is not representative of all RL algorithms. It is unclear whether the same conclusion could be drawn for other algorithms such as PPO or REINFORCE. Without such experiments, it is insufficient to prove that the proposed idea works for RL in general, and it might be better to qualify the main theme of the paper to GRPO instead. \n2. Content unclarity. Table 1 and its related content look a bit confusing. It is unclear what numbers in the table represent, as they don’t seem to be valid distributions (not summing up to one). The authors said “A considerable fraction of examples for many models is concentrated in the extreme accuracy bins” on lines 172-173, but the numbers seem to be even higher in (85, 100] for Qwen2.5-7B and Llama3.1. \n3. Missing details in comparison. Computational Cost Analysis in Section 5.3 might not have been fair, because the initial computational overhead incurred during example scoring is not discussed. In Table 6, it is also unclear which dataset is used to measure the speed, and whether the time is for one step or for the entire training (training Qwen2.5-0.5B in 140s seems too fast to be true if it’s for the entire training).\n\nQUESTIONS:\n1. Could you elaborate on the similarity metrics used in Table 2?\n2. In lines 168-169, it does not seem to be consistently true that larger models show more robustness when trained on harder examples compared to smaller models in Figure 1.\n3. While the paper demonstrates the effectiveness using 10% of the data, it is unclear why such a choice is made. It would be helpful to have a plot that shows the change in performance if lower/higher percentage of data points are supplied, so that we can have a better understanding on the relationship between the fraction of the data and the model performance, and further examine whether it is possible to outperform training with full data.\n\nLIMITATIONS:\nAs discussed in weakness 1, there are insufficient results to support the claims for other policy optimization algorithms other than GRPO.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes UFO-RL (Uncertainty-Focused Optimization for Reinforcement Learning) which can make the LLM finetune more effectively by intelligently selecting training data. It is inspired by the Zone of Proximal Development (ZPD) theory, which says that model benefits more from data that have not yet mastered but demonstrate the potential to comprehend. So this paper proposes the method to access the difficulty/estimate the uncertainty of data, so that to identify informative training instances. The paper claims that this estimation method can speed up the previous methods for estimation.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper introduces an interesting question about how to train LLM more effectively. 
2. The experiment parts is interesting and it is tested for different LLMs that this data selection method is effective.

Weakness:
I think the major weakness is lack of clear structure in presenting the methods. 
1. The paper initially mentions that this method is inspired by the ZPD theory. It sounds interesting but is there any part about theoretic proof or intuitively how this method can have theoretical basis is introduced? I feel it is more like an experiment guided results without too much theory basis.
2. The method is not clearly explained in the paper. While most contents of the paper focus on experimental results, what is exactly the algorithm to pick data that have not yet mastered but demonstrate the potential to comprehend?

QUESTIONS:
Please refer to the weakness part.

LIMITATIONS:
No.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
The paper introduces UFO-RL, a data-selection framework for RL fine-tuning of LLMs. A single-pass confidence score replaces 16-sample accuracy, making uncertainty estimation up to 185 × faster and letting the authors train on the top “fuzzy” data. In the experiments, model trained on subset matches or exceeds full-data performance.

STRENGTHS AND WEAKNESSES:
Strength:
- Lightweight uncertainty estimation: Confidence evaluation is up to 185 × faster than 16-sample accuracy.
- Broad evaluation: Results span Qwen2.5, Llama-3.1, Mistral families and three test sets, with multiple runs and error bars. 

Weaknesses:
- Heuristic method: The whole method is based on ZPD hypothesis, which has no theoretical basis. The paper did include experiments to verify the hypothesis, but in Fig.1, red line may overstate the clarity of the trend, the raw data (black dots) shows inconsistencies, especially for smaller or weaker models. 

- Confidence score: Token-level confidence estimation is not a new concept, and it may not serve as a reliable proxy for semantic uncertainty. The similarity between confidence and semantic uncertainty is sensitive to prompt design; however, the paper does not include any experiments analyzing prompt effects on similarity in Table 2.

QUESTIONS:
Could you provide further justification for the specific quadratic form $1 - (s_i - \mu)^2$? Have you experimented with other functions to isolate the "intermediate uncertainty" region? 

Have you considered a dynamic implementation where confidence scores and the selected data subset are periodically re-calculated during training, may act as ablation study?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents UFO-RL, an uncertainty-driven framework for improving the efficiency of reinforcement learning (RL) fine-tuning for large language models (LLMs). Drawing inspiration from the Zone of Proximal Development (ZPD) theory, the authors hypothesize that LLMs learn most effectively from examples that are neither too easy nor too difficult but exhibit moderate uncertainty. This paper first prove this hypothesis and then proposes a lightweight, single-pass confidence estimation based on average token-level log-probabilities, enabling efficient identification of informative training instances.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The application of ZPD-inspired learning dynamics to RL-based LLM fine-tuning is intuitively appealing. \n2. The proposed single-pass confidence estimation method provides a simple yet computationally efficient proxy for data difficulty, avoiding expensive multi-sampling or ensemble-based uncertainty estimation commonly used in other works.\n\nWeaknesses:\n1. While the application to LLM RL is novel, the broader concept of leveraging model uncertainty for data selection has been extensively explored in areas such as active learning, coreset selection, and data distillation. The paper does not mention these works in the paper.\n2. The comparison focuses primarily on full-data RL or random sub-sampling baselines. A more rigorous, direct comparison with multi-sampling-based difficulty estimation methods is missing, leaving open questions about the relative trade-offs in computational cost versus uncertainty estimation fidelity.\n3. The evaluation is concentrated on mathematical reasoning tasks with relatively small to mid-sized LLMs (up to 7B). The method’s scalability to larger models or applicability to diverse RL tasks  remains unexplored.\n\nQUESTIONS:\n1. The authors are encouraged to explicitly contrast their method with related work in active learning, providing a clearer positioning within the broader literature.\n2. Can the authors provide empirical results comparing UFO-RL’s single-pass confidence estimation to conventional multi-sampling methods in terms of both performance and computational cost?\n3. Are there scenarios where single-pass estimates significantly diverge from more robust multi-sample uncertainty metrics, and what are the practical implications of such divergences?\n4. Token-level log-probabilities may be affected by calibration issues or may fail to capture sequence-level semantic uncertainty. Have the authors evaluated the robustness of their method under varying model calibration qualities?\n5. Beyond mathematical reasoning, have the authors considered evaluating UFO-RL on language modeling, instruction tuning, or other RL fine-tuning tasks to demonstrate broader applicability?\n6. How does the proposed method scale with larger LLMs (e.g., >13B), where uncertainty estimation and data selection efficiency become more critical?\n\nLIMITATIONS:\nThe same as the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}