Title: Understanding Data Influence in Reinforcement Finetuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Fine-Tuning

Large Language Models

Data-centric AI
Abstract: Reinforcement fine-tuning (RFT) is essential for enhancing the reasoning and generalization capabilities of large language models, but its success heavily relies on the quality of the training data. While data selection has been extensively studied in supervised learning, its role in reinforcement learning, particularly during the RFT stage, remains largely underexplored. In this work, we introduce RFT-Inf, the first influence estimator designed for data in reinforcement learning. RFT-Inf quantifies the importance of each training example by measuring how its removal affects the final training reward, offering a direct estimate of its contribution to model learning. 
To ensure scalability, we propose a first-order approximation of the RFT-Inf score by backtracking through the optimization process and applying temporal differentiation to the sample-wise influence term, along with a first-order Taylor approximation to adjacent time steps. 
This yields a lightweight, gradient-based estimator that evaluates the alignment between an individual sample’s gradient and the average gradient direction of all training samples, where a higher degree of alignment implies greater training utility. Extensive experiments demonstrate that RFT-Inf consistently improves reward performance and accelerates convergence in reinforcement fine-tuning.


Peer reviews:
comments: SUMMARY:
This paper applies the TracIn method to RLF, proposing a framework called RFT-DA. It estimates the contribution of individual training samples through a first-order influence approximation with linear complexity (i.e. TracIn), and uses these scores to filter and prioritize valuable examples. This data selection approach improves training stability and efficiency. Experiments on mathematical reasoning tasks show that using only 20% of selected data achieves comparable or better performance than full-data training, with the 7B model matching or outperforming the o1-mini baseline.

STRENGTHS AND WEAKNESSES:
### Strengths
Despite some minor typos, the paper is overall well written and clearly structured. Applying the TracIn method to RLF is a novel combination. The proposed approach is theoretically simple and well motivated, with a clear derivation and sound design. The experimental evaluation is thorough, and the method shows promising potential for reducing training cost by enabling data-efficient RLF according to principled influence analysis.
### Weaknesses
One main motivation is to make training more efficient, but RLF-DA requires additional computation cost despite several practical strategies are applied. The author should elaborate in the main text regarding whether the benefits offered by RFT-DA justify its increased computational overhead.

QUESTIONS:
1. How do you choose the 90% threshold for the score summation of top-scoring samples? How sensitive are the results to this choice?
2. Can RFT-DA be used to dynamic or continually evolving datasets?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRFT-DA is a data-selection framework that improves the efficiency and effectiveness of reinforcement fine-tuning (RFT). RFT-DA uses a lightweight, gradient-based estimator to evaluate the alignment between an individual sample's gradient and the average gradient direction of all training samples. RFT-DA's 7B model is on par with or outperforms the ol-mini on a number of benchmarks. The RFT-DA framework provides a scalable, first-order, gradient-based estimator for computing RFT-DA scores with linear computational complexity. The framework has been validated across a number of benchmarks, including MATH-500, AIME24, AMC23, and OlympiadBench.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper is written well and easy to understand.\n2. The idea is somewhat novel and justified solidly.\n3. The selected data can generalize towards other models or RL algorithms.\n\nWeaknesses:\n1. Only considering math or code benchmarks is not convinced.\n2.While results on 1.5B and 7B models are strong, it remains an open question whether the findings and the specific data selection ratios (e.g., 20%) would hold for much larger models (e.g., 70B+).\n3. Many typos (e.g., line 346 'dmodel')\n\nQUESTIONS:\nQ1. The experiments consistently select 8,530 samples (~20% of the dataset) based on a rule of removing negative-scoring samples and then taking the top 90% of the remaining score summation. While this procedure is explicit, the rationale for the \"90% of score sum\" threshold seems heuristic. How was this value determined, and how sensitive is the model's performance to this data selection budget?\n\nQ2. What are the characteristics of samples that receive very high positive scores versus those that receive negative scores?\n\nLIMITATIONS:\nI do not see any potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RFT-DA, a data selection framework for reinforcement fine-tuning (RFT) of large language models that identifies valuable training examples by measuring how their removal affects final training rewards. The core innovation is a sample-level ablation analysis that quantifies individual training example importance through their contribution to model learning.\n\n**Main Contributions:**\n- **Data ablation scoring framework**: Defines D(z) as the difference in expected reward when sample z is removed from training, providing a direct measure of sample importance for RFT\n- **Efficient first-order approximation**: Introduces a gradient-based estimator D̂(z) that measures alignment between individual sample gradients and overall training direction, achieving linear computational complexity O(NE + NC)\n- **Theoretical analysis**: Provides worst-case error bounds for the approximation under Lipschitz continuity assumptions, with bounds tighter than many supervised learning influence functions\n- **Empirical validation**: Demonstrates that using only 20% of data selected by RFT-DA scores achieves superior performance compared to full datasets, with trained 7B models matching o1-mini performance\n\nThe method backtracts through the optimization process using temporal differentiation and first-order Taylor approximation, resulting in a lightweight estimator that evaluates gradient alignment across training stages.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Addresses a significant gap in RFT data selection methodology**. The paper tackles an important but underexplored problem of principled data selection for reinforcement fine-tuning, moving beyond heuristic approaches to provide a theoretically grounded framework for quantifying sample importance.\n\n- **Strong empirical validation with comprehensive benchmarks**. The evaluation spans multiple mathematical reasoning benchmarks (AIME24, MATH-500, AMC23, Minerva, OlympiadBench) across different model sizes and RL algorithms, demonstrating consistent improvements with substantial margins over existing methods (9.1 points over difficulty-based, 6.3 over LIMR).\n\n- **Practical efficiency with theoretical guarantees**. The method achieves remarkable data efficiency (20% of data for superior performance) while providing formal error bounds for the first-order approximation, making it both practically valuable and theoretically sound for large-scale applications.\n\n- **Good positioning of the work** in the rapidly evolving field. Well written over-all. \n\n### Weaknesses\n\n- **Limited theoretical justification for the gradient alignment assumption**. The core assumption that gradient alignment with the overall training direction indicates sample importance lacks rigorous theoretical foundation. The connection between local gradient similarity and global contribution to final rewards is not well-established, particularly in the complex optimization landscape of reinforcement learning.\n\n- **Narrow domain evaluation despite RFT's broader applications**. While mathematical reasoning is a primary RFT target, the evaluation lacks validation on other important domains where RFT is commonly applied, such as scientific reasoning (GPQA), conversational AI (ArenaHard), or code generation tasks, limiting confidence in the method's general applicability beyond mathematics.\n\n- ** Separate surrogate training is an expensive overhead**\n\nQUESTIONS:\n1. **How does per-sample gradient computation align with modern batch-based training practices?** The method requires individual sample gradients, but modern RFT typically uses batched updates - how do you reconcile this with practical training pipelines that rely on batch statistics?\n\n2. **Can you provide detailed comparison conditions for the claimed o1-mini performance?** The assertion that a 7B model matches or out-performs o1-mini performance across benchmarks is remarkable, but the tables don't seem to support it. i recommend some revision of this claim.\n\n3. **Why is the separate surrogate training phase necessary rather than computing scores during actual training?** The method requires expensive surrogate training to generate checkpoints for gradient computation, but provides no ablation on whether this two-phase approach is essential - is there possible alternatives?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents data ablation for reinforcement fine-tuning (RFT-DA), a framework that can quantify the importance of individual training examples in RFT. RFT-DA uses an efficient first-order approximation approach to estimate the importance of a sample by evaluating the alignment between its gradient and the gradient over all samples. The samples with the highest alignment scores are kept for RFT training. Experimental results demonstrate the effectiveness of this method.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n    - The paper presents a clear and well-motivated research objective.\n    - The proposed method is straightforward and achieves strong empirical performance.\n- Weaknesses\n    - The analysis of computational complexity lacks rigor.\n        - The authors claim that the total computational complexity for scoring the entire dataset is $O(NE + NC)$, but the definitions of $E$ and $C$ are unclear. From my understanding, both seem to refer to the number of training epochs. If that is the case, the notation may be misleading. The authors may mean to use $NE$ to represent the computational cost of training the model for $E$ epochs and saving checkpoints, and $NC$ to denote the cost of computing the data ablation score for each sample. However, these two components represent different stages of the pipeline and should not be directly summed.\n    - The overall presentation of the paper could be improved. See questions.\n\nQUESTIONS:\n- The inclusion of the “Motivation” and “Our Approach” paragraph titles in the Introduction section seems to be unnecessary.\n- Line 121: It is inappropriate to equate a set with a sample\n- Line 168: The function $\\mathcal{J}(\\cdot)$ should denote the expected advantage rather than the expected reward, consistent with the definition in Equation 1.\n- Line 180: There is no $\\mathcal{Z} / z$ in Equation 5.\n- Line 181: There is no $T$ in Equation 5.\n- It may be more appropriate to relocate Figure 1 and its accompanying description to the Experiments section, as it illustrates empirical behavior rather than methodological design.\n- Line 265: Broken section reference.\n- Line 295: Should be $1e-6$ rather than $1\\times e^{-6}$.\n- Line 387: “good enough” is too informal for the academic context.\n- Line 449: Wrong authorship of Olympiadbench.\n- Supplementary Material Line 82: Should be $\\mathcal{J}^0(z)=0$.\n- How is RFT-DA compared to LIMR in terms of computational efficiency?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}