Understanding Data Influence in Reinforcement Finetuning
Authors: Haoru Tan
Xiuzhe Wu
Sitong Wu
Shaofeng Zhang
Yanfeng Chen
Xingwu Sun
Jeanne Shen
XIAOJUAN QI
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Fine-Tuning
Large Language Models
Data-centric AI
Primary area: optimization

Abstract: Reinforcement fine-tuning (RFT) is essential for enhancing the reasoning and generalization capabilities of large language models, but its success heavily relies on the quality of the training data. While data selection has been extensively studied in supervised learning, its role in reinforcement learning, particularly during the RFT stage, remains largely underexplored. In this work, we introduce RFT-Inf, the first influence estimator designed for data in reinforcement learning. RFT-Inf quantifies the importance of each training example by measuring how its removal affects the final training reward, offering a direct estimate of its contribution to model learning. 
To ensure scalability, we propose a first-order approximation of the RFT-Inf score by backtracking through the optimization process and applying temporal differentiation to the sample-wise influence term, along with a first-order Taylor approximation to adjacent time steps. 
This yields a lightweight, gradient-based estimator that evaluates the alignment between an individual sample’s gradient and the average gradient direction of all training samples, where a higher degree of alignment implies greater training utility. Extensive experiments demonstrate that RFT-Inf consistently improves reward performance and accelerates convergence in reinforcement fine-tuning.

Understanding Data Influence in Reinforcement Finetuning
Haoru Tan'
Xiuzhe Wust
Sitong Wu
Shaofeng Zhang*
Yanfeng Chen
Xingwu Sun
Jeanne Shen
Xiaojuan Qi!
'hh University of Hong Kong
Hunyuan Team, Tencent
The Chinese University of Hong Kong
*University of Science and Technology of China
Stanfordd University
hrt Arteeeeee. hku. hk, stone-wueling cuhk, edu, hk
Abstract
Reinforcement fine-tuning (RFT) is essential for enhancing the reasoning and gen- eralization capabilities of large language models, but its success heavily relies the quality the training data. While data selection has been extensively studied supervised learning, its role in reinforcement learning, particularly during the RFT stage, remains largely underexplored. In this work, we introduce RFT-Inf, the first influence estimator designed for data in reinforcement learning. RFT-Inf quantifies the importance of each training example by measuring how its removal affects the final training reward, offering a direct estimate of its contribution model learn- ing. To ensure scalability, we propose a first-order approximation of the RFT-Inf score by backtracking through the optimization process and applying temporal differentiation to the sample-wisg wise influence term, along with a first-order Taylor approximation adjacent time steps. This yields a lightweight, gradient- based estimator that evaluates the alignment between an individual sample's 's gradient and the average gradient direction of all training samples, where higher degree alignment implies greater training utility. Extensive experiments demonstrate that RFT-Inf consistently improves reward performance and accelerates convergence reinforcement fine-tuning.
Introduction
Reinforcement Fine-Tuning (RFT) [1-4] has emerged as a powerful technique for refining the capabilities of large language models (LLMs) 2, by leveraging reward-driven optimization [6, 7]. Unlike Supervised Fine-Tuning (SFT) [8-10], which guides model behavior through direct supervision [1]]. RFT enables models to learn more robust reasoning strategies and generalize beyond seen data [1, 11, 12]. Recent studies have confirmed the critical role RFT in advancing the reasoning capabilities of LLMss, particularly in complex tasks requiring multi-step inference and decision-making.
Background. A fundamental aspect of successful RFT lies in the construction and utilization of high-quality training data [1, 13, 14]. However, current understanding of data's role RFT remains
!Coresponding Authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

largely heuristic or empirical, often guided by reward trends observed during training [13, 15] or the perceived difficulty of training examples [16. 17]. Despite its importance, there remains a lack of principled, quantitative methods for assessing the impact of individual samples on RFT outcomes. Addressing this gap is essential for designing more efficient and effective training pipelines.
Our Method. In this work, we propose RFT-Inf for quantifying the influence of the individual training sample in reinforcement fine-tuning. The core idea behind RFT-Inf is measure how the removal of a single sample influences the final training reward, providing a direct estimate of its contribution to the model's learning process. While brute-force approach would require re-training the model after removing each sample, such computation is prohibitively expensive. To address this, we introduce scalable, first-order approximation of the RFT-Inf score derived from backtracking through the optimization process and applying temporal differentiation to the sample wise influence term, along with a first-order Taylor approximation to adjacent time steps. This results in a lightweight gradient-based estimator with linear complexities that measures the alignment between a sample's gradient and the average gradient direction of all training samples across different training stages, see (5). A higher alignment corresponds to a higher RFT-Inf score, indicating greater sample importance. Intuitively, if a sample's 's gradient is closely aligned with the overall optimization direction, it is more representative and beneficial to training. Removing such sample would degrade performance, highlighting its importance. We further provide a theoretical worst-case error bound for our first-order approximation, offering insights into its reliability.
We validate RFT-Inf across multiple benchmarks, demonstrating its effectiveness identifying high-impact samples and improving final model performance. Using RFT-Inf for data selection yields significant performance gains over baselines that rely on full datasets or heuristic selection strategies. Notably, in mathematical reasoning tasks, we only require about 20 % data selected by our data influence estimator to achieve more stable training and superior results compared to using the entire dataset. Compared to various heuristic or rule-based data selection methods, our approach significantly outperforms them in both performance and generalization.
2 Related Work
Reinforcement Fine-Tuning Recent breakthroughs in large language models (LLMs) [2, 3] have been significantly driven by Reinforcement Fine-Tuning (RFT) [6, 7, 1, 18, 19], a technique that refines LLM behavior using reward-guided optimization. Unlike Supervised Fine-Tuning (SFT) [8-10], which aligns model outputs with labeled responses [11], RFT employs reinforcement learning to adapt models based on feedback signals. Typically, RFT relies on rule-based reward functions and reinforcement learning algorithms such Proximal Policy Optimization (PPO) [6]. For instance, DeepSeek-R R1 [1] adopts Group Relative Policy Optimization (GRPO) [7], using binary rewards to indicate answer correctness in tasks like mathematics [20] and coding [21], achieving impressive performance. Several studies suggest that RFT enhances cognitive abilities such as reflection and self- correction [22, IJ. while also improving generalization across tasks [1]]. Much of the current RFT research focuses on algorithmic improvements. For example, VinePPO [23] addresses limitations in PPO's value networks for complex reasoning tasks by introducing unbiased Monte Carlo estimates for better credit assignment. Other works aim to simplify GRPO, e.g., by removing the KL-divergence term achieve more robust empirical results [18, 24].
Most recently, some emerging research explores data-centric approaches to improve RFT [13, 14, 18, 15]. For example, LIMR [13] selects samples by analyzing changes in reward trends, while the Historical Variance Score [15] prioritizes data with high reward variability, suggesting that such samples are more impactful. Other works incorporate sample difficulty as selection criterion strike a balance between learning efficiency and model robustness [16, 17]. Despite these advances, existing work on data selection for RFT is largely heuristic and lacks principled methods for quantifying the influence of individual training samples. As result, most methods rely on empirical intuition rather than theoretical insight. This highlights urgent need for data-centric frameworks that enable quantitative analysis of sample importance during RFT, ultimately guiding more effective training.
Data Influence Analysis. Analyzing the impact of individual training samples is a longstanding problem in machine learning [25, 26]. The canonical approach involves removing sample and re-training the model to observe the effect-an idea that is precise but computationally prohibitive [26-28]. To overcome this, many works propose theoretical estimators that approximate sample
2

---

influence without full retraining. A seminal method by Koh and Liang [26] estimates sample influence using the inverse Hessian-gradd graddient product, assuming the model parameters change smoothly under small data perturbations. Building on this, subsequent work has improved the efficiency of Hessian computations through decomposition techniques [29-32], incorporated group-level effects [33, 34], or integrated influence estimation with training procedures [35, 36].
However, these approaches are predominantly designed for supervised learning and often depend on key assumptions: a well-defined, twice-differentiabll objective function, and convexity of the loss landscape [26, 37, 38]. Such conditions facilitate tractable approximation but do not hold in reinforcement learning, where the optimization objective is to maximize expected reward rather than minimize a loss. Consequently, applying these influence functions directly to reinforcement learning is challenging. Moreover, data influence research in supervised learning has been leveraged for tasks such as data selection [26, 39-43] and attribution [29, 44-49), but similar progress in reinforcement learning remains limited. This disconnect motivates the development of new, scalable influence estimation tools tailored specifically for RFT scenarios.
3 Preliminaries
This section introduces the foundational concepts of reinforcement fine-tuning (RFT) [2, 1, 14, 7, 50, 3] applied to large language models (LLMss) [51, 52]. We consider a training dataset 3 2, = (st, ye) composed of structured question guestion-answer answer pairs, where each sample 2; consists of a prompt or question and a corresponding deterministic ground-trute -ruth answer yı, e.g., math problem and correct solution. Let т denote the policy model parameterized by 0, representing the LLM. In this formulation, the input state corresponds to the prompt, and the action a is the model's generated response. A scalar reward measures the quality of the response a for the given prompt 8. The objective of RFT is fine-tune the LLM using reinforcement learning produce correct and high-quality answers [6, 7]. A commonly adopted reward scheme directly compares the model output with the ground truth [1]. assigning a reward of 1 if the output matches the correct answer and r = -1 otherwise.
Objective Function. The standard RFT objective is to optimize the policy no by maximizing the expected advantage:
(I) representing the relative merit of taking action a in state compared to a baseline value v. Optimization proceeds via stochastic
O(0) = B(,,,) A(s,aa)
where the advantage function is defined as A(s, a) r(s,
gradient ascent: 1-+1 0, + (e,), where 7) is the learning rate and the gradient of [(0) is
derived as: VoJ = E(s.a) A(s, a) Ve log TTa (als))
Modern RFT implementations typically enhance this vanilla formulation improve stability and performance. We now briefly review two widely used variants: Proximal Policy Optimization (PPO) [6] and Group Relative Policy Optimization (GRPO) [7, 1], which serve the foundation for many current RFT advancements [18, 12, 50, 53].
Proximal Policy Optimization (PPO) [6] introduces clipping mechanism to stabilize training by constraining policy updates. Its objective is formulated as:
Jrro (0) B(...)
min
Towa A(s, clip
(als))
To(als) (als)
1- 1, A( A(s.
(2)
where is clipping hyperparameter. (a)ss) is the probability of taking action in state according to the current policy with parameters 0, and Tow (s]s) is the probability of taking action a in state
according to the old policy with parameters Bold-
Group Relative Policy Optimization (GRPO) [7, 1] extends PPO by defining advantages through relative comparisons within a group of sampled responses for the same prompt. For each prompt s, group of outputs d1,...,,0
Ag is sampled from the old policy To..- The GRPO objective is:
GRPO
1
G
min
(a:ls) Fa.1()
A(s, A(s, ax), clip
Te (a,ls) 1 c, 1 +c) A(s,aa)) -3KL (No (
To (a:ls)
G
1m1
(3)
3

---

where e is the clipping hyperparameter and 3 is the coefficient controlling the weight of the KL- divergence regularization ensure the optimized model does not deviate excessively from the reference model (e.g,, the initial policy model); and A(s, ai) = ri r4-meun(T1.....7 ra) is the advantage, where corresponds to the reward of each output. GRPO eliminates the need for separate value network, reducing computational cost and simplifying training. For more details, we refer readers to the original GRPO papers [7, l].
4 Method
In this section, we introduce our proposed method. We begin by defining the data influence estimator and presenting first-order approximation for efficient computation, along with theoretical analysis of the approximation error. We then describe how this score can used select high-impact samples to improve the performance of RFT.
4.1 Data Influence in Reinforcement Finetuning
We propose principled approach assess the importance of each training sample by measuring its influence on the final performance of reinforcement fine-tuning through evaluating how model performance changes when the sample is removed from training.
Definition. Let 2 denote the training dataset consisting of samples = (s. y). where is a prompt and y is its deterministic answer (e.g., math problem and its unique solution). Let 2/2 denote the dataset excluding sample 2, and let 8% and 82 denote the parameters obtained by training on Z and 2/z, respectively. Assuming reward function that assigns ground truth and -1 otherwise [1, 14], we define the data influence estimator of a sample 2 as:
if the model output matches the
D(2) J (02) - (02))))))))))
where (-) denotes the expected reward. This score is directional; not only quantifies the importance of a sample but also distinguishes whether the sample is beneficial detrimental to training. A large positive value of D22) indicates that removing the sample significantly reduces the final reward, suggesting that the sample is beneficial. Conversely, negative value implies that the sample is harmful, as removing it would increase the reward. Values near zero indicate negligible influence. Thus, this metric provides direct and interpretable measure of each sample's contribution to RFT.
Efficient First-order Approximation. Precisely Direct computation of D(2) is impractical, as it requires retraining the model for each sample. To address this issue, introduce first-order approximation of the RFT-Inf score by backtracking the score defined Eq. (4) through the optimization process. We apply temporal differentiation and first-order Taylor approximation to adjacent time steps. The detailed derivation is provided the supplementary material. Below the approximation with linear complexities:
D(=) Σ 2nd N
(g g6
where (,:) is the inner-product operator, Z/2 represents the dataset Z excluding sample =: v). and 7), is the learning rate at time step t. T and N denote the maximum number of time steps and the training set size, respectively. g! represents the policy gradient contributed by sample (s, v) at time step l. Typically, for most RL algorithms, we have G! Á(s, a) V log ne (als), where Á(s, a) is the advantage function. In the case of GRPO [7]. > Å(s, a; V log na, (a,ls), where G the group size. The other gradient term 9 is defined as the gradient over all samples, that is, G "xezg!")
This approximation evaluates the alignment between a sample's gradient and the overall training direction. A higher value indicates that the sample's influence aligns with the global training trajectory, marking it as representative and valuable. This can be interpreted as follows: if the sample's gradient g: exhibits a high degree of alignment with the average gradient vector g for all time steps, then

---

Algorithm 1: Data Selection Pipeline
Require: A dataset = (8t ) and selection budget 8; A large language model Ty and a
reinforcement fine-tuning algorithm RFT
1: Train the model Te for E' epochs on Z using RFT and save checkpoints: RFT TTo 2, E))
[0) (0',..., 05)
2: for each sample = (8,, Mi) E 2 do
3: Calculate the data influence estimator (=x)) according to Eq. (5)
4: end for
5: Select the top-ô samples based on their data influence estimators to form the new subset Znew
6: return The subset model Znew
optimizing the network using sample z will have a similar effect optimizing with the full training set. This indicates that the sample a representative and important sample. Therefore, this sample regarded as more representative and significant one.
4.2 Error Analysis
First, we provide worst-case error bound for the proposed approximation, demonstrating its robustness under some mild assumptions:
Under the assumptions that the log-likelihood function log Te (u(())) exhibits t- Lipschitz continuity and that the adventure value is upper-bounded by Amax the approximation error of Eq. (5) is bounded as follows:
Proposition 1.
((()) S {0 Pinax (4N + 4) (2Amax)2 + 20 Amux 2mmxxmmxx {T).
4)
N
where T denotes the maximum number of iterations and N is the number of all training data.
The above proposition formalizes the relationship between the approximation error and influencing factors, particularly the training duration T. To mitigate errors in practice, we adopt a strategy of early stopping during the surrogate training phase, avoiding full convergence. Specifically, we limit the number of update epochs to small value (e.g., two epochs) to ensure the effective execution of surrogate training while also controlling the theoretical maximum error. Remarkably, the resulting error bound remains tighter than many known bounds for influence functions in supervised learning [41, 54], despite the added challenges posed by the reinforcement learning setting.
4.3 Complexity Analysis
Then, we discuss the computational complexity of our estimator. Eq. (5) offers an efficient approx- imation of the data influence estimator with linear computational complexity. Further, rather than summing over all training steps, we adopt a Monte Carlo sampling strategy [55, 56], selecting small number of evenly spaced time steps to estimate the full summation since checkpoints from nearby iterations will be similar. Let the number of selected checkpoints be C. By employing this
strategy, the total computational complexity for scoring the entire dataset is O + ONE++NC) where N is the cost of surrogate training, and N represents the number of training samples and C denotes the number of checkpoints (which also corresponds to the number of sampled checkpoints, as we select one checkpoint for each epoch). This represents substantial efficiency gain over the naive approach, which requires retraining the model for each sample and incurs cost of O ((N - 1)F)
per sample, resulting in O O(N(N- 1)E) overall. Leveraging practical implementations such as PyTorch's efficient gradient operations [57]. or computing only final-layer gradients [55, 58], further enhances scalability.
4.4 Implementation Details
We now describe how to use the proposed data influence estimator to identify and select high-impact samples for reinforcement fine-tuning. The full pipeline is outlined in Algorithm 1. We are given
5

---

structured dataset 2 = {4 (s,, y:) consisting of question answer pairs, where each sample 2; comprises a prompt or question s; and corresponding deterministic answer y;- We are also provided with a pre-trained large language model, serving as the policy model Tp, and reward evaluation mechanism. common and effective approach to reward calculation involves directly comparing the generated answer to the ground truth y: for instance, a reward of r is assigned if the generated answer matches the ground truth, and r = -1 otherwise.
As shown in Algorithm I: the policy model first undergoes small number of reinforcement fine- tuning (RFT) epochs. namely E epochs, on the full training dataset. We refer this training phase as surrogate training. The choice of reinforcement learning algorithm is flexible; for instance, one may use PPO [6], GRPO [7], or any other compatible method. At the end of each epoch, model checkpoint is saved for later use. To reduce training costs, several practical strategies can be employed. These include setting small number of epochs (e.g.. E = 2), or applying serameter-efficiet fine- tuning techniques such LoRA [59] instead of full model fine-tuning, as recommended by recent studies [60, 31]. These techniques have been empirically shown to significantly reduce computational overhead without negatively impacting performance. After surrogate training, the saved checkpoints are used to compute per-sample gradients as well as the overall gradient across the dataset. The data influence estimator for each sample is then computed according to Eq. (5). Using the computed data influence estimators, we perform data selection based on a predefined budget ö, which specifies the size of the target subset. The top-ô samples with the highest data influence estimators are selected to form a refined training set, denoted as Zuow- This curated subset then used for the formal reinforcement fine-tuming (RFT) phase to obtain the final model parameters TT.
5 Experiments
We conducted comprehensive experiments to evaluate the effectiveness of our proposed method. Sec. 5.1 is the main experiment. Then, Sec. 5.2 provides detailed ablation studies to analyze the impact of key components in our approach.
5.1 Main Experiments
Models, Datasets, and Benchmarks. We utilized the dataset released by DeepScaleR [14], which is a comprehensive mathematical dataset compiled from mul- tiple sources, with duplicates removed and data cleaned. This dataset includes AIME problems from 1984 to 2023 and AMC problems before 2023, along with questions from the Omni-MATH [61] and STILL [62] datasets, fea- turing problems from various national and international mathematics competitions.
1.0
066
Training Reward
0.4
Accuracy
0..
FILLE
0.6
254
200
2006
1559
1900
1758
25.00
150
7.00
1000
1208
1990
1756
2009
Training Stops
Training Stops
Figure 1: The reward and accuracy curve during learning the subset selected by RFT-Inf (20% selection ratio). This experiment Bireeccc-----iisill----------
is conducted benchmark is AIME24 [20].
and the selected
This training dataset contains approximately 40,000 math problem-answe pairs. To evaluate the reasoning abilities of the models, we utilize five different mathematics benchmarks: AIME24 [20], MATH-500 [63], AMC23 [64], Minerva [65], and OlympiadBench [66]. Our experiments encompass a variety of model configurations, including DeepSeek-BR Maepseeek-R--iiseee R11-Ditill--wen--.... [1], DeepSeek-Re 1iat---iilll-weeee 7B [1] and -.2mm--..-----mee default algorithm used is GRPO, with group size set to 8.
[67]. Unless otherwise specified, the
Setups. The experiments were conducted using the PyTorch framework on two high-performance computing servers, each equipped with eight NVIDIA H200 GPUs. For our approach, we made the following settings: during the surrogate training phase, performed LoRA training with LoRA

---

Randam
Difficulty
LIMK
HVS
Ours
Queas 1.58 oz AIMB24
Gweea1155 1:58 os MKTH-500
Qwee- 1:58 oLAMC23
Qweni1.5 on Olyagied
112
Pox40t
06-
Selection Bailo
Selection Rato
Selection Ratio
Selection Botio
Queep-TI AIM1124
Owen MAI11-99
Qwen-7llon AMC31
Quim 7I1 Olympisl
15-
130
Paang (50
Paagg (50
-0-
18
15
20
25 Selection Iatio
Seluction Batio
Seiection
Selection Ratio
Figure 2: This figure presents the zero-shot pass @1 1 performance (%) of proposed data selection method ("Ours") against several baselines (Random, Difficulty-basee LIMR, HVS) across four mathematical reasoning benchmarks (AIME24, MATH-500, AMC23, Olympiad) and two model sizes: Qwen-1.5B (DeepSeek-R1-- -eesseee-----istil---P--- -----isiill--eee------- 1. 1.5B) and Qwen-7B SeesSeek-R--iisill---- 7B). The performance plotted function of the data Selection Ratio (from 10% 80%), representing the budget constraint.
rank set to 16 and total of 2 training epochs. We optimized the network using the Adam W optimizer with a constant learning rate of 1 x e- and a weight decay of 0.1.
Baselines. We compare our method against several widely used baselines: (1) Random selection, where a subset of data is chosen uniformly at random. (2) Difficulty-driven selection, where the difficulty of each sample is estimated based on model performance. Specifically, we run inference for each sample with multiple models (DeepSeek-R) (DeepSeet-RL-Distlll-ne ---iitill-Qwen--15B//B- 1.5B/7B/32BB [1]) and use the average pass rate
proxy for difficulty [17]. Based on this, we select those samples with median difficulty [68]. (3) LIMR [13], which determines sample importance by measuring the alignment between sample's 's reward and the overall reward trend during training. (4) Historical Variance Score (HSV) [15], which quantifies sample importance by computing the variance in its reward values over the course of training.
To evaluate our method, we selected two models: DeepSeek Biickk-RIII RI-Distill- Swalll-ween--11.11 1.5B [1] and DeepSeek- RI-Distill Pebistil--Qwe--711 Qwen- 7B [1]. Here, the model we use for scoring is consistent with the final trained model. Both models are derived from the Qwen series and have been distilled using DeepSeek-RR demonstrating a solid foundational reasoning ability. We summarize the relevant experimental results in Fig. 2. Clearly, our method outperforms various baseline approaches under the most constrained budgets. particularly at very low selection ratios. Specifically, at the 20% selection ratio setting: for the 1.5B model type, our method achieves an accuracy of 45.5% in AIME24, outperforming all baseline methods, including LIMR [13] and HVS [15]. On MATH-500, our score of 89.0% is notably higher than the best baseline score of 86.2% from Full Data. In the case of the 20% selection
ratio and the 7B model, method further excels, achieving score of 60.2% AIME24, which is significantly higher than the next best baseline score of 57.3% from HVS [15].
5.2 Further study
In this subsection, we explore several key factors, including the number of checkpoints and the training method (either LoRA rank full-parameter training), also along with the case study.
Number of Checkpoints We employed a Monte Carlo strategy when approximating the estimator in Eq. (5), considering checkpoints from only a few time steps. In practice, we observed that as the
7

---

17.3
67.6
Pact81 Accuracy (%)
472
45.
15.4
460
663
Passe9 Accuracy 0%0
16.5
4,5
45.4
45.3
8.1
66.3
-4
66.2
63.5
430
Number Checkpaints
LoRA Rank Scale)
Figure 3: Ablative experimental results with DeepSeek-RI-Distil---ueee
as the model and
AIME24 as the benchmark. The performance metric zero-shot Pass@1 Accuracy. (a) Pass accuracy versus the number of checkpoints used, showing that increasing checkpoints stabilizes training (decreasing standard deviation). (b) Pass accuracy versus LoRA rank (log scale), demonstrating that performance generally improves with higher LoRA ranks, approaching the Full- Parameter result.
number of checkpoints increased, the final performance improved, see Fig. 3. This is understandable, as it effectively increases the sampling steps, thereby enhancing approximation accuracy. However, this also results in a significant decrease in scoring speed, and the improvement shows trend of stagnation. Therefore, we opted for relatively small number of checkpoints, specifically two, with each corresponding to one epoch.
LoRA-Rande Our method requires training a surrogate model before scoring, using LoRA training with a rank of 16. LoRA (Low-Rank Adaptation) is a technique that efficiently fine-tunes large language models by incorporating low-rank decomposition into the weight updates, enabling ef- fective adaptation with fewer parameters and reduced computational cost. This means that when approximating the estimator E5)( we do not utilize all model parameters. We tested the impact of using LoRA training and the size of the LoRA rank final performance, see the results in Fig. 3. conclusion, we found that using LoRA significantly improves efficiency compared to full-parameter training, without negatively impacting the performance of the data selected based on final data influence estimator. Additionally, we discovered that even with a LoRA rank of 4, final performance remains quite satisfactory.
Case study. In the following, we provide an illustrative examination of several cases, showing their assigned RFT-Inf scores and the resulting importance rankings (in descending order). For benchmarking, we include the data importance rankings generated by the alternative HVS [15] strategy. To establish an objective metric for task complexity, we use the pass rate, measured by the number of correct outputs from eight QwQ-32B [5] samples. Lower pass-rates correlate with higher question difficulty. Our analysis reveals crucial distinction in data valuation: A high degree of concordance is found between RFT-Inf and HVS rankings for data points associated with extreme difficulty levels. For example, in both Case 1 and Case 4, which represent the spectrum's edges, both methods assign significantly low importance ranks, implying general agreement that the marginal utility of such data is negligible. Crucially, the two ranking strategies exhibit substantial disparity when evaluating data points of intermediate or moderate difficulty. This disagreement in identifying the most valuable samples within the middle ground' of difficulty constitutes the fundamental driver behind the varying empirical performance observed in different data selection paradigms.
Case
Each vertex of a regular octagon independently colored either red or blue with equal probability. The probability that the octagon can then be rotated so that all of the blue vertices end up positions where there were originally red vertices is m where m and n are relatively prime positive integers. What is + n??
Problem:
Pass-rate-by-Qaa [5]: 0/8;
Rank-by-RFT-DA 40245:
Rank-by-HVS [15]: 39076.

---

Case
Problem: Among the 900 residents of Aimeville, there are 195 who own a diamond ring. 367 who set of golf clubs, and 562 who residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
garden spade. In addition, each of the 900
byss-r [5]: 5/8;
Rank-by-RFT-DA: 3205:
Rank-by-HVS [15]: 27855.
Case 3
Problem: Let ABC be triangle inseribed in circle w. Let the tangents to w B and C intersect point D, and let AD intersect P. If AB 5, BC 9, and AC 10, AP can be written as the form 12 where m and n are relatively prime integers. Find m + n.
Pass-rate-by-oae [5]: 4/8;
Rank-by-RFT-DA: 1129:
Rank-by-HVS [15]: 30724.
Case 4
Problem: Jen enters a lottery by picking 4 distinct numbers from 5 = 11,33, 2,....... .......9..... 10). numbers randomly chosen from S. She wins prize if at least two of her numbers were of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she a prize is m where m and are relatively prime positive integers. Find m + n.
Pass-rate-by- by [5]: 8/8;
Rank- -F-RT--D--DD 39275:
Rank-by-HVS [15]: 40236.
Conclusion
In this study, we present RFT-Inf, novel approach for data selection in reinforcement fine-tuning, aimed at finding those beneficial training examples while eliminating noisy or harmful ones. Central to RFT-Inf is sample-level influence analysis that assesses the significance of each training example by evaluating how its exclusion impacts the final training reward, providing a clear indication of its contribution to model learning. To enhance scalability, introduce lightweight, gradient- based estimator with theoretical guarantees. Comprehensive experiments demonstrate that RFT-Inf consistently improves reward performance and accelerates convergence in reinforcement fine-tuning,
Limitations. While our method demonstrates strong performance across benchmarks, the experi- mental scale validated in this paper is still limited. In real-world scenarios, both the data and model sizes are significantly larger. the future, we plan to secure additional computational resources to validate performance on a larger experimental scale.
7 Acknowledge
This work has been supported by the National Key R&D Program of China (Grant No. F022YFB3608300, Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422, 17212923, 17215025). The- mebased Research (Grant No. T45-701/22-R/ and Shenzhen Science and Technology Innovation Commission (SGDX2022053011 14050440). Part of the described research work is conducted the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charitics Trust.

---

References
[1] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 2ivi 12948, 2025.
[2] OpenAL. Learning to reason with llms, 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint Xiv:25001 12599, 2025.
[4] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Ye Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint 25:::25002229900 2025.
[5] Qwen Team. Qwq-32b: Embracing the power of reinforvement learning, 2025.
[6] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (rXXvv:707.063320 2017.
[7] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 03300, 2024.
[8] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple rest-time scaling, arXiv preprint 2tt.::2001.199333 2025.
[9] Yinggian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen. Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang. Xiaoxue Cheng. Huatong Song, al. Imitate, explore, and self-improv:: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413, 2024.
[10] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint vxXi::5002.03387 2025.
[11] Tianzhe Chu, Yuexiang Zhai Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Ma. Sit memorizes, rl generalizes: A comparative study of foundation model post-tres post-training. arXiv preprint arXiv:2501.17611, 2025.
[12] Zichen Liu, Changyu Chen, Wenjun Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprini arXiv:2503.2078:: 2025.
[13] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less more for rl scaling. arXiv preprint 1xXi:22502.11886 2025.
[14] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1.5b model by scaling rl. https://prettt-radeee radilo--dddddd-ddooaoeo 075. mot...ooooon.iit... wttpriinnsiteeecpSSeeeeeeee -urullss--uaaasnnnggee -r-e-iewwwiith-aee www-w-hth------Moa 2025, Notion Blog.
by-Scaling RL- 14-- 10000000000000003000000000044440
[15] Yiping Wang, Qing Yang. Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng. Xuehai He. Kuan Wang, Jianfeng Gao. Weizhu Chen, Shuohang Wang. Simon Shaolei Du, and Yelong Shen. Reinforcement learning reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057/ 2025.
[16] Weihao Zeng. Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
[17] Yunjie Ji, Sitong Zhao, Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Yiping Peng. Han Zhao, and Xiangang Li. How difficulty aware staged reinforeement Icarning enhances llms' reasoning capabilities: A preliminary experimental study, arXiv preprint 2xxx:2504400829 2025.
[18] Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source 1lm reinforcement learning system at scale. arXiv preprint Xrxv:2503., 14476, 2025.
[19] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models, 2025.
10

---

[20] AIME. American invitational mathematics examination. http:://wwagpooooeoooo.cec/a///p 2024.
[21] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and lon Stoica. Livecodehench; Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
[22] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self- improving reasoners, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025.
[23] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking potential for IIm reasoning through refined credit assignment. arXiv preprint arXiv:2410.0167,, 2024.
[24] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement lcarning baseline for model reasoning. 2025.
[25] R Dennis Cook. Assessment of local influence. Journal of the Royal Statistical Society Series B: Statistical Methodology, 48(2):133-155 155, 1986.
[26] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017.
[27] Mariya Toneva, Alessandro Sordoni, Remi Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
Tachet des Combes, Adam Trischler, Yoshua Bengio, and
[28] Vitaly Feldman and Zhang Chiyuan. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in neural information processing systems. 2020.
[29] Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin L,, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXhv:2308.033296 2023.
[30] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[31] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned mlms and diffusion models. arXiv preprint arXiv:2310.00902 2023.
[32] Yegor Klochkov and Yang Liu. Revisiting inverse hessian vector products for calculating influence functions, 2024.
[33] Pang Koh, Kai-Siang Ang, Hubert H. K. Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Advances neural information processing systems, 2019.
[34] Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In International Conference on Machine Learning, 2020.
[35] Sung Yub Kim, Kyungsu Kim, and Eunho Yang. Gex: A flexible method for approximating influence via geometric ensemble. Advances in Neural Information Processing Systems, 36, 2024.
[36] Mycongseob Ko, Feiyang Kang, Weiyan Shi, Ming Jin, Zhou Yu, and Ruoxi Jia. The mirrored influence hypothesis: Efficient data influence estimation by hamnsssig forward passes. In CVPR, 2024.
[37] Anna Choromanska, Mikael Henaff, Michael Mathicu, Gerard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. Journal of Machine Learning Research, 38:192-2000 204, 2015.
[38] Yann Dauphin, Razvan Pascanu. Caglar Gulcehre, Kyunghyun Cho. Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advancex in Neural Information P Processing Systems, pages 2933-2941, 2014,
[39] Ruoxi Jia, Fan Wu, Xuchui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In IEEE/ CVF Computer Vision ond Pattern Recognition Conference, 2021.
[40] Siyi Tang, Amirata Ghorbani, Rikiya Yamashita, Sameer Rehman, Jared Dunnmon, James Zou, and Daniel L. Rubin. Data valuation for medical imaging using shapley value and application to large-scale chest x-ray dataset. Science Report, 11:8366, 2021.

---

[41] Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019.
[42] Anshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu, Oudier gradient analysis: Efficiently identifying detrimental training samples for deep learning models, 2024.
[43] Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In International Conference on Computer Vision, 2021.
[44] Hugh Chen, Scott M Lundberg, and Su-In Lee. Explaining a series of models by propagating shapley values. Nature communications, 13( (1512,, 2022.
[45] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leelere, and Aleksander Madry. Trak: Autributing model behavior at scale. arXiv preprint arXiv:2303. 14186, 2023.
[46] Zheng Dai and David Gifford. Training data attribution for diffusion models. arXiv preprint arXiv:2306.022 02174, 2023.
[47] Andrew lyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry, Datamodels: Predicting predictions from training data, arXiv preprint (xXi::2202.00622 2022.
[48] Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang. Fan Wang. and Xiaojuan Qi. Data pruning via Advances neural information processing systems. 2023.
[49] Haoru Tan, Sitong Wu, Xiuzhe Wu, Wang Wang. Bo Zhao, Zeke Xie, Gui-Song Xia, and Xiaojuan Qi. Understanding data influence with differential approximation. 2025.
[50] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofci Zhu, Wenyuan Xu, Jiaze Chen, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint 18 2025.
[51] AI@ Meta Llama Team. The Ilama 3 herd of models. 2024.
[52]
Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[53] Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang. Zheng Lin, Minglei Zhang, Shaojic Wang, Yinghan Cui, Chao Wang. Junyi Peng, Shimiao Jiang, Shiqi Kuang. Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementation of large-scale reinforcement learning Ilm, 2025.
[54] Jiachen T Wang. Dawn Song. James Zou, Prateek Mittal, and Ruoxi Jia. Capturing the temporal dependence of training data influence. arXiv preprint 24X:::4412.09538, 2024.
[55] Garima Pruthi, Frederick Liu, Sundararajan Mukund, and Satyen Kale, Estimating training data influence by tracing gradient descent. arXiv preprint arXiv:2002.08453 2020.
[56] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation data for machine learning. In International conference on machine learning, pages 2242-2251, 2251. PMLR, 2019.
[57] Adam Paszke, Sam Gross, Soumith Chintala, G Chaman, E Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Antiga, A Lerer, and ct.al. Automatic differentiation in pytorch. Advances in neural information processing systems Workshop, 2017.
[58] Shuo Yang. Zeke Xie, Hanyu Peng. Min Xu, Mingming Sun, Ping Li. Dataset pruning: Reducing train- ing data by examining generalization influence. In International Conference on Learning Representations, 2023.
[59] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen, LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[60] Mengzhou Xia, Sadhika Malladi. Suchin Gururangan, Sanjeev Arora, and Dangi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402. 2024.
[61] Bofei Gao, Feifan Song, 7he Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, I Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic henchmark for large language models. arXiv preprini arXiv:2410.07985, 2024.
[62] Jinhao Jiang. Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang. Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, Zheng Liu, Dong Yan, Jian Xie, Zhongyuan Wang, and Ji- Ji-Rong Wen. Enhancing IIm reasoning with seward-guided tree search. arXiv preprint arXiv:2411.11694 2024.
12

---

[63] HuggingFaceil4 Team. Math-500; A subset of the math dataset. http:///gggggaaa..coooeaaaaaaaaa
HuggingFaceel4//ATTE MMATH444AATHT-OO 2024, Accessed: 2025-05-15.
[64] Qwen Team. Ame23 benchmark dataset. http::://gttbbb..mm//eeee/////// www.//ennI//uuen222222 5-Mat,, 2024. Ac- cessed: 2025-05-15.
[65] David Lewkowyez. Ethan Dyer, Guy Gur-Ari, Igor Babuschkin, Jeffrey Wu, Tom Henighan, Jared Kaplan, et al. Solving quantitative reasoning problems with language models. arXiv preprint urXiv2206.14888.
2022.
[66] Yeyu Feng, Yiyang Gu, Ziyang Liu, Jiayi Wang. Qihao Zhu, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint urXiv:2402.14000 2024.
[67] Meta AL Llama 3.2 instruct. https: https://hugginggtee -amaae///eo//aaaaaaaaaaaaaa---- 2- 2.1.llmm----------ntuuut.. 2024. Accessed: 2025-05-15.
[68] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In International Conference on Learning Representations, 2023.
[69] Haoru Tan, Chuang Wang. Sitong Wu, Tieqiang Wang, Ya-Yao matching with proximal matching networks. In The Annual AAAI Conference on Artificial Intelligence (AAAI), 2021.
Yao Zhang, and Cheng- Liu. Proxy graph
[70] Barak A. Pearlmutter. Fast exact multiplication by the bessian. Neural Computation, 6(1):147-160, 1994.
[71] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[72] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[73] Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. In Neural Information Processing Systems (NeurIPS), 2023.
[74] Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, al. Nemotron-4 340b technical report. arXiv preprint 2rXiv:2406. 1/ 17704 2024.
[75] Sitong Wu, Haoru Tan, Yukang Chen, Shaofeng Zhang, Jingyao Li, Bei Yu, Xiaojuan Qi, and Jiaya Jia. Mixture-of-scores: Robust image-text data quality score via three lines code. In International Conference on Computer Vision (ICCV), 2025.
[76] Haoru Tan, Sitong Wu, Zhuotao Tian, Yukang Chen, Xiaojuan Qi, and Jiaya Jia. Saco loss: Sample-wise affinity consistency for vision-language pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[77] Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, 2024.
[78] Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: general vision transformer backbone with pale-shaped attention. In The Annual AAAI Conference on Artificial Intelligence (AAAT), 2022.
[79] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A coreset approach. In International Conference on Learning Representations, 2018
[80] Peizhao Li and Hongfu Liu. Achieving fairesss at no utility cost via data reweighing with influence, 2022.
[81] Haoru Tan, Chuang Wang, Sitong Wu, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Ensemble quadratic assignment network for graph matching. International Journal of Computer Vision (IJCV), 2024.
[82] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matci Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representations, 2019.
[83] Kawin Ethayarajh. Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint 0rXiv:2402.01306, 2024.
13

---

[84] Samyadeep Basu, Phillip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference Learning Representations, 2021.
[85] Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. arXiv preprint arXiv:2212.04011 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the relevant section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Please see the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The code and data will be made publicly available upon acceptance through peer review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips... cc/
guuides/CoeeSumissss oonooicc)))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/ wttpccccpppcc//c.c..cc...ccc.lcl pubppcppppggepep wwwpc//gguie/eeeeeee Sute:/ooeeSSieseeeee ssssssssssssiioooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the csGuidel ines?
NeurIPS Code of Ethics https: htt////eurrpp..pe cc/ccc/pp/cc http:p:////rrppp.pppppbiicccce
Answer: [Yes]
Justification:
We strictly follow the relevant Guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please check the relevant section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The training algorithm involved in this article will not cause the originally safe model to become dangerous.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This paper does not existing assets
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This article only uses LLM to help check for spelling mistakes in very small quantities.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https /tt//eerrss...e/ /ttp///ips.c//oooeeeeeeeeeeee/ee cctc//o/mmerrrccee22///////// for what should or should be described.
21

---

A Impact Statement
This paper introduces novel influence analysis algorithm advance RFT training. It has potential positive societal effects, such as improving understanding of data roles in developing robust systems and possibly reducing data bias. However, there are concerns that the negative impacts could be misused for inhumane social surveillance, which legislative bodies worldwide should address seriously.
B Mathematical Proof: Approximation
Let 6 denote the learned parameter at the t-th iteration on the full dataset and 6" denote the learned parameter at the t-th iteration on the dataset without sample 2, we use (2(2) denote the empirical expected reward at the f-th step,
= - ((-2))
where & < T, and D==) = (z). Note that the network on the full set and that on the subset Z f/z started from the same initialization, hence, (²) = O. Let's start with the identical equation below,
(22) -D((s)DD (z)) + (DT- (D- -2 (D(=)= -D" (2) + ( - DJ 2x)) +... (3) (())) D*
(D'(2) (D'())-D(ss) (z)) (x)
+...
= AD" + AD" + AD
Let's take one single item AD as an example,
AD' = D')
[[(]]]]][-_---)-)))))))>>))) -
By using the first-order Taylor approximation and according to the gradient-ascent algorithm in RFT, we have that,
J(0' - (00=--00-))==00----00-000000-0
where the policy gradient V = (a) (A(a, V log Tg. (als)) Hence, we have the equation for the differential term AD', that is,
AD' -
Given specific sample 2, we present unified empirical expected reward formulation for RFT:
1 Σ A(a..))) J. = N
( +) 2(,
where is a coefficient. Hence, we have the expected reward on the full training set by setting e = 0 and the subset without sample by (approximately) setting 7. which is convention in influence analysis for supervised learning [26, 33, 34, 31, 41, 58, 29, 46]. We can treat the ( + c) A(a, 2) as the new reward for sample 2. Hence, the policy gradient given T. is:
1 VJ.
Σ 3/1
A(a., log (a, + ( +c) 100AA(,,. Vog
1
N
Here, the Taylor approximation again to approximate VJ(("=))) with ( which is
also a convention in influence analysis for supervised learning [26, 33, 34, 31, 41, 58, 29,46],
VO(('=))> = ( +
J.J-
== (cc=
- - (c= ()= 0)) 1 (
N (9)
9 1-1
de
where g-- v log - (as12)A(as,))
22

---

For other variants, the formulation for --- also needs to be adapted, for example, the group relative formulation for GRPO [7]]
1 V log me ((..)))())))))
0 -' G
SSG
where az,i means the i-th answer from the model given the input question/prompt 2, A(az,) ms(a:]=) Fau (ail))
metai]e] Fouu (a:1s) A(s, a
- e, 1 + +e) A(s, & KL (Te))
Hence, we have,
| (=) Aा
~ 1) (/
1 VJ(0"11)
N -
No
N
2n' VJ 00(0011)19-1
(10)
V 1-1
-1=11)
N
N2
24/
where 1,: is the inner-product operator. By substituting the approximation for D' into Eq.(7),
we have that,
2n² VJ
D(=) = AD + AD + ADD ~ Σ
(11)
N
where 0 TTt<..
C
Mathematical Proof: Error Bound
We provide worst-case error bound for the proposed approximation, demonstrating its robustness under some mild assumptions.
Under the assumptions that the log-likelihood function log Te(a(s) exhibits Lipschitz continuity and that the adventure value is upper-bounded by Amaro the approximation error bounded as follows:
Proposition 1.
D(z) s (
{ Vinax Timax(41 (4N + 4) (eAmax)² + 2 }
(12)
N
where T denotes the maximum number of iterations and N is the number of all training data.
Our estimator is based on the time-domain differential operation over D(z) E, AD*(z), and performing Taylor approximation on AD* (z). Hence, the overall error is bounded by
[D(=) - D()) < (2) - -
We set as the upper bound of the norm of the policy gradient. Hence, we have d < Amax,, where f the Lipschitz constant of the log-likelihood function log (als)) with é-Lipschitz continuity.
According to (10), the error comes from the following parts: The first approximation term, (D' = 7 [ gradient-ascent algorithm in RFT, the error is bounded by
J (0'=1) using the first-order Taylor approximation and the
AD'(=) -
= 11 - - -((-))) -
|
|||
S 20yd + 2nd
(13)
23

---

The second approximation term in approximating V=((""))) with VJ(("))). that is, VJ(("=)))
2
VJ((0-1) 1 We have
1 -
1 N
S 2ndd 1
+
2n
d
N
d.
(14)
N
N2
The last error term comes from ignoring Nzz Vot-i.g5 1) in the final approximation in Eq.(10). this error is bounded by d= Hence, the overall error is bounded by
[D()) - D(=) s ()) - (D))))))()))
T 28np 2nd 2nd + 2n
de
1/ d
d
N
N2
N2
2m
2n )dl + 26ndd
T
(417
N
N2
(Ar) +
417
dd + 281yd)
N
Mmnux (4N + 4) d² + 2endd
T
N
Mmax (4N + +4)
(e.Amx))² + 2 T
N
(15)
24


Reviews:
[{"review_id": "s9p1SattIb", "reviewer": "Reviewer_L7kr", "summary": "This paper presents data ablation for reinforcement fine-tuning (RFT-DA), a framework that can quantify the importance of individual training examples in RFT. RFT-DA uses an efficient first-order approximation approach to estimate the importance of a sample by evaluating the alignment between its gradient and the gradient over all samples. The samples with the highest alignment scores are kept for RFT training. Experimental results demonstrate the effectiveness of this method.", "questions": "- The inclusion of the “Motivation” and “Our Approach” paragraph titles in the Introduction section seems to be unnecessary.\n- Line 121: It is inappropriate to equate a set with a sample\n- Line 168: The function $\\mathcal{J}(\\cdot)$ should denote the expected advantage rather than the expected reward, consistent with the definition in Equation 1.\n- Line 180: There is no $\\mathcal{Z} / z$ in Equation 5.\n- Line 181: There is no $T$ in Equation 5.\n- It may be more appropriate to relocate Figure 1 and its accompanying description to the Experiments section, as it illustrates empirical behavior rather than methodological design.\n- Line 265: Broken section reference.\n- Line 295: Should be $1e-6$ rather than $1\\times e^{-6}$.\n- Line 387: “good enough” is too informal for the academic context.\n- Line 449: Wrong authorship of Olympiadbench.\n- Supplementary Material Line 82: Should be $\\mathcal{J}^0(z)=0$.\n- How is RFT-DA compared to LIMR in terms of computational efficiency?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n    - The paper presents a clear and well-motivated research objective.\n    - The proposed method is straightforward and achieves strong empirical performance.\n- Weaknesses\n    - The analysis of computational complexity lacks rigor.\n        - The authors claim that the total computational complexity for scoring the entire dataset is $O(NE + NC)$, but the definitions of $E$ and $C$ are unclear. From my understanding, both seem to refer to the number of training epochs. If that is the case, the notation may be misleading. The authors may mean to use $NE$ to represent the computational cost of training the model for $E$ epochs and saving checkpoints, and $NC$ to denote the cost of computing the data ablation score for each sample. However, these two components represent different stages of the pipeline and should not be directly summed.\n    - The overall presentation of the paper could be improved. See questions.", "was_revised": true, "final_justification": "Most of my concerns have been addressed, except for the issue of time complexity. The presentation issues cannot be fully resolved at the discussion stage. Since this paper offers a clear contribution despite its weaker presentation, I rate it as a borderline accept.", "rebuttal": "Response to Reviewer L7kr's Comments:\n\n---\n\nWe sincerely thank the reviewer for the insightful feedback. We appreciate the constructive criticism and have carefully addressed the concerns raised regarding the computational complexity analysis and the overall presentation of the paper. \n\n---\n\n### **1. On the computational complexity analysis.**\n\nWe will revise this! Regarding the definitions of NE, N, and C, please refer to Line 212 of the manuscript. Following your suggestion, we will clarify the definitions of N, C, and E to improve clarity.\n\nSpecifically: \n\n- \\( N \\) denotes the total number of training samples,\n- \\( E \\) is the number of training epochs,\n- \\( C \\) is the number of checkpoints selected for score computation.\n\n\nAccordingly, the computational complexity of the surrogate training phase is \\( O(NE) \\), while the scoring process incurs a complexity of \\( O(NC) \\). Therefore, the overall computational complexity is \\( O(NE) + O(NC) \\).\n\n---\n\n\n### **2. On the overall presentation:**\n\nWe appreciate the feedback. We have carefully revised the manuscript to improve clarity, flow, and readability. Specifically: \n\n(a). Line 121: This was a typo caused by a compilation error. We have corrected it: the correct expression should be \"Z = {z_i = (x_i, y_i)|_{i=1}^N }\". The {} notation is not properly rendered in math mode.\n\n(b). The function $J(\\cdot)$ denotes the expected advantage, and we will ensure it remains consistent with the definition provided in Equation (1). \n\n(c).Equation (5) indeed does not involve $Z \\setminus z$, and we will remove the reference to $Z \\setminus z$ in Line 180. Additionally, we will explicitly include the time horizon $T$ in the summation notation for clarity. \n\n(d).The incorrect section reference in Line 265 was due to a compilation error, and we will correct it in the revised version. And we have updated the reference to OlympiadBench: Chaoqun He, Renjie Luo, Yuzhuo Bai, et al., OlympiadBench, ACL 2024. \n\n(e).Line 82 in the Supplementary Material should be $D^0(z) = 0$. Thank you for catching this typo, and we will correct it accordingly.\n\n\n---\n\n\n### **3. Speed Comparison with LIMR:** \n\nIn the supplementary material (Table 3), we include a comparison with HVS on the MATH-500 dataset. Notably, HVS incurs virtually no additional time cost during the scoring phase. LIMR demonstrates comparable computational cost, as both rely solely on reward statistics collected during training and do not require any additional computation at inference. However, both methods exhibit significantly lower performance (Pass@1 of 83.6) compared to full-set training (86.2), limiting their practical utility in performance-critical scenarios. \n\nIn contrast, our proposed approach achieves substantially higher performance: 89.0 with RFT-DA and 88.3 with the faster variant, RFT-DA Fast. Despite these gains, the total computation time—including surrogate training, scoring, and subset training—is significantly lower than full-set training (192.2 hours for RFT-DA, 134.2 hours for RFT-DA-Fast vs. 277.6 hours for full-set training). \n\nThese results will be added to the main paper for clarity. It’s worth noting that the scoring time is measured using 8 GPUs, which can be significantly reduced by using more GPUs. \n\n\n---\n\n**RFT-DA Dynamic: A Faster and More Efficient Variant:** \n\nTo further reduce computation time for scoring, we introduce **RFT-DA Dynamic**, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \\in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score: \n\n$$D(z) \\approx \\sum_{t \\in \\{t^z_1, t^z_2, \\ldots\\}} \\frac{2\\eta_t}{N} \\langle G_z^t, G_{B_t}^t \\rangle,$$ \n\nwhere $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation. \n\nThe table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance:\n\n\n\n\nMethod   | $\\~$  Time Cost of Scoring   |  $\\~$ Math-500 Pass@1  \n---|---|--- \nHVS| 4.7 sec $\\approx 0$|83.3\nLIMR| 5.3 sec $\\approx 0$ |83.6\nRFT-DA (ours)|63.1 Hours| 89.0\nRFT-DA-Dynamic (ours) | 32 sec | 88.3\n\nWhile **RFT-DA-Dynamic** yields slightly lower accuracy than the original RFT-DA, it achieves comparable computational efficiency to HVS and LIMR, requiring less than one minute for scoring. Importantly, it still significantly outperforms HVS, LIMR, and full-set training in terms of final model performance, making it a highly effective and scalable solution for data selection under budget constraints."}, {"review_id": "sSesvMrG8R", "reviewer": "Reviewer_s3Rt", "summary": "This paper introduces RFT-DA, a data selection framework for reinforcement fine-tuning (RFT) of large language models that identifies valuable training examples by measuring how their removal affects final training rewards. The core innovation is a sample-level ablation analysis that quantifies individual training example importance through their contribution to model learning.\n\n**Main Contributions:**\n- **Data ablation scoring framework**: Defines D(z) as the difference in expected reward when sample z is removed from training, providing a direct measure of sample importance for RFT\n- **Efficient first-order approximation**: Introduces a gradient-based estimator D̂(z) that measures alignment between individual sample gradients and overall training direction, achieving linear computational complexity O(NE + NC)\n- **Theoretical analysis**: Provides worst-case error bounds for the approximation under Lipschitz continuity assumptions, with bounds tighter than many supervised learning influence functions\n- **Empirical validation**: Demonstrates that using only 20% of data selected by RFT-DA scores achieves superior performance compared to full datasets, with trained 7B models matching o1-mini performance\n\nThe method backtracts through the optimization process using temporal differentiation and first-order Taylor approximation, resulting in a lightweight estimator that evaluates gradient alignment across training stages.", "questions": "1. **How does per-sample gradient computation align with modern batch-based training practices?** The method requires individual sample gradients, but modern RFT typically uses batched updates - how do you reconcile this with practical training pipelines that rely on batch statistics?\n\n2. **Can you provide detailed comparison conditions for the claimed o1-mini performance?** The assertion that a 7B model matches or out-performs o1-mini performance across benchmarks is remarkable, but the tables don't seem to support it. i recommend some revision of this claim.\n\n3. **Why is the separate surrogate training phase necessary rather than computing scores during actual training?** The method requires expensive surrogate training to generate checkpoints for gradient computation, but provides no ablation on whether this two-phase approach is essential - is there possible alternatives?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Addresses a significant gap in RFT data selection methodology**. The paper tackles an important but underexplored problem of principled data selection for reinforcement fine-tuning, moving beyond heuristic approaches to provide a theoretically grounded framework for quantifying sample importance.\n\n- **Strong empirical validation with comprehensive benchmarks**. The evaluation spans multiple mathematical reasoning benchmarks (AIME24, MATH-500, AMC23, Minerva, OlympiadBench) across different model sizes and RL algorithms, demonstrating consistent improvements with substantial margins over existing methods (9.1 points over difficulty-based, 6.3 over LIMR).\n\n- **Practical efficiency with theoretical guarantees**. The method achieves remarkable data efficiency (20% of data for superior performance) while providing formal error bounds for the first-order approximation, making it both practically valuable and theoretically sound for large-scale applications.\n\n- **Good positioning of the work** in the rapidly evolving field. Well written over-all. \n\n### Weaknesses\n\n- **Limited theoretical justification for the gradient alignment assumption**. The core assumption that gradient alignment with the overall training direction indicates sample importance lacks rigorous theoretical foundation. The connection between local gradient similarity and global contribution to final rewards is not well-established, particularly in the complex optimization landscape of reinforcement learning.\n\n- **Narrow domain evaluation despite RFT's broader applications**. While mathematical reasoning is a primary RFT target, the evaluation lacks validation on other important domains where RFT is commonly applied, such as scientific reasoning (GPQA), conversational AI (ArenaHard), or code generation tasks, limiting confidence in the method's general applicability beyond mathematics.\n\n- ** Separate surrogate training is an expensive overhead**", "was_revised": true, "final_justification": "my concerns are addressed by reviewer response, thus updating final rating.", "rebuttal": "Response to Reviewer s3Rt: \n\n\nWe sincerely appreciate your time and feedback! We hope our response addresses your concerns. If you have any further questions, please don't hesitate to let us know. We are more than happy to discuss further.\n\n---\n\n### **1. On the theoretical justification for the gradient alignment.** \n\n\nGreat question! The use of gradient alignment, as formulated in Equation 5, is supported by both theoretical justification and empirical evidence. \n\nFirst, as detailed in Section 3 of the supplementary material, Equation 5 provides a **first-order approximation** of the **leave-one-out influence** of a sample z on the RFT process. This influence, formally defined in Equation 4, quantifies how the removal of a specific sample affects the training outcome, thus enabling us to assess the importance of individual samples. \n\nWe further analyze the estimation error of the gradient alignment score in **Proposition 1**, where we show that our approximation of leave-one-out influence is **more accurate** than many existing influence estimation methods in supervised learning literature. This theoretical result underpins the validity of using gradient alignment as a proxy for sample importance. \n\nIntuitively, in RFT, a sample whose gradient is well-aligned with the aggregate update direction (i.e., the average gradient of other samples) is more representative and contributes more constructively to training. This idea is not only theoretically motivated but also strongly supported by our empirical results. As shown in Tables 1, 2, and 3, our approach consistently outperforms baseline methods, including those using the full dataset—demonstrating **notable performance gains** with significantly reduced training data. \n\nIn summary, both the **theoretical formulation** and the **experimental validation** provide solid support for our gradient alignment assumption as a meaningful and effective criterion for sample selection in reinforcement fine-tuning.\n\n---\n\n### **2. More evaluation results on GPQA and ArenaHard.** \n\nDuring the rebuttal period, we conducted experiments on GPQA and ArenaHard. The model used for the experiments was DeepSeek-R1-Distill-Qwen-7B, and the datasets involved included DeepScaleR and a 50K subset of the Tulu-3-Preference dataset. The RL algorithm employed was Reinforce++, and the reward model used was Tulu-3-8B-RM. To ensure fairness, all methods were filtered to include only the top 90% based on summation rate. \n\nNotably, RFT-DA continues to maintain its leading advantage! We will update these results in the revision.\n\n\nMethod | GPQA | Arena-Hard\n---|---|---\nNo training| 48.4 | 17.3\nHVS|52.8|26.9\nLIMR|51.4|24.2\nRFT-DA (ours)|**53.5**|**28.5**\nFull-set training|51.7|27.4\n\nMoreover, for **larger model settings**, we present the training results on the 32B (DeepSeek-R1-Distill-Qwen-32B) and 70B (DeepSeek-R1-Distill-Llama-70B) models. Here, we directly apply the subsets chosen in the 1.5B model experiments for RFT on these larger-scale models. The performance advantages of RFT-DA remain consistent with those observed on the 1.5B and 7B models, outperforming both HVS and LIMR, as well as the training results on the full set! \n\nMethod|Math-500 Pass@1 for 32B|Math-500 Pass@1 for 70B\n---|---|---\nNo training|94.3|94.5\nHVS|93.7|94.1\nLIMR|92.9|93.2\nRFT-DA (ours)|**96.9**|**97.1**\nFull-set Training|95.1|96.5\n\n---\n\n### **3. Comparison with O1-mini.**\n\n\nThe results presented in Table 1 indicate that, at the 7B model scale, our method achieves performance comparable to that of o1-mini on AMC23 and Olympiad (89.4 vs. 90.0 and 65.4 vs. 67.2). Notably, on AMC23, our method even surpasses the performance of o1-mini (91.0 vs. 90.0). We will update our statement to reflect that our method can match or even exceed the performance of o1-mini on certain benchmarks.\n\n---\n\n### **4. On the necessity of a separate surrogate training and scoring phase.**\n\nThis is a great question! We appreciate your suggestion! We will include this new version with improved speed in our revision. The separate surrogate training and scoring phases are the main contributors to the time consumption in our method when compared to HVS and LIMR. Following your suggestion, we have also implemented a dynamic version of conducting scoring during surrogate training as below.\n\n\n\n**RFT-DA Dynamic: A Faster and More Efficient Variant:** To further reduce computation time for scoring, we introduce **RFT-DA Dynamic**, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \\in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score:\n\n$$D(z) \\approx \\sum_{t \\in \\{t^z_1, t^z_2, \\ldots\\}} \\frac{2\\eta_t}{N} \\langle G_z^t, G_{B_t}^t \\rangle,$$\n\nwhere $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation. \n\nThe table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance:\n\n\n\nMethod   | $\\~$  Time Cost of Scoring   |  $\\~$ Math-500 Pass@1  \n---|---|--- \nHVS| 4.7 sec $\\approx 0$|83.3\nLIMR| 5.3 sec $\\approx 0$ |83.6\nRFT-DA (ours)|63.1 Hours| **89.0**\nRFT-DA-fast (ours)|5 Hours| **88.1**\nRFT-DA-Dynamic (ours) | 32 sec | **88.3**\nFull-set training | - | 86.2\n\n**RFT-DA Dynamic** maintains comparable performance as RFT-DA while being much more efficient, making it a highly effective and scalable solution for data selection under budget constraints."}, {"review_id": "K1awX7RlmI", "reviewer": "Reviewer_wKRB", "summary": "RFT-DA is a data-selection framework that improves the efficiency and effectiveness of reinforcement fine-tuning (RFT). RFT-DA uses a lightweight, gradient-based estimator to evaluate the alignment between an individual sample's gradient and the average gradient direction of all training samples. RFT-DA's 7B model is on par with or outperforms the ol-mini on a number of benchmarks. The RFT-DA framework provides a scalable, first-order, gradient-based estimator for computing RFT-DA scores with linear computational complexity. The framework has been validated across a number of benchmarks, including MATH-500, AIME24, AMC23, and OlympiadBench.", "questions": "Q1. The experiments consistently select 8,530 samples (~20% of the dataset) based on a rule of removing negative-scoring samples and then taking the top 90% of the remaining score summation. While this procedure is explicit, the rationale for the \"90% of score sum\" threshold seems heuristic. How was this value determined, and how sensitive is the model's performance to this data selection budget?\n\nQ2. What are the characteristics of samples that receive very high positive scores versus those that receive negative scores?", "limitations": "I do not see any potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.The paper is written well and easy to understand.\n2. The idea is somewhat novel and justified solidly.\n3. The selected data can generalize towards other models or RL algorithms.\n\nWeaknesses:\n1. Only considering math or code benchmarks is not convinced.\n2.While results on 1.5B and 7B models are strong, it remains an open question whether the findings and the specific data selection ratios (e.g., 20%) would hold for much larger models (e.g., 70B+).\n3. Many typos (e.g., line 346 'dmodel')", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer wKRB:\n\nWe sincerely appreciate your recognition of our work!. We are more than happy to address any questions you may have!\n\n---\n\n\n### **Weakness-1: Evaluation on More Benchmarks and Larger Models.**\n\n\n**Experiments on More Benchmarks.** We further conducted performance tests on GPQA and Arena-Hard! GPQA is a benchmark focused on assessing large models' expert-level knowledge comprehension and complex reasoning abilities, covering fields such as biotech and chemistry. Arena-Hard is an automatic evaluation tool for instruction-tuned LLMs designed to reflect whether the models meet human preferences.\nThe model used for the experiments was DeepSeek-R1-Distill-Qwen-7B, and the datasets involved included DeepScaleR and a 50K subset of the Tulu-3-Preference dataset. The RL algorithm employed was Reinforce++, and the reward model used was Tulu-3-8B-RM. To ensure fairness, all methods were filtered to include only the top 90% based on summation rate. \n\nNotably, RFT-DA continues to maintain its leading advantage! We will update these results in the revision.\n\n\nMethod | GPQA | Arena-Hard\n---|---|---\nNo training| 48.4 | 17.3\nHVS|52.8|26.9\nLIMR|51.4|24.2\nRFT-DA (ours)|**53.5**|**28.5**\nFull-set training|51.7|27.4\n\n\n\n\n**Experiments on Larger Models.** Thank you for your suggestion! We present the training results on the 32B (DeepSeek-R1-Distill-Qwen-32B) and 70B (DeepSeek-R1-Distill-Llama-70B) models. Here, we directly apply the subsets chosen in the 1.5B model experiments for RFT on these larger-scale models. The performance advantages of RFT-DA remain consistent with those observed on the 1.5B and 7B models, outperforming both HVS and LIMR, as well as the training results on the full set! \n\nMethod|Math-500 Pass@1 for 32B|Math-500 Pass@1 for 70B\n---|---|---\nNo training|94.3|94.5\nHVS|93.7|94.1\nLIMR|92.9|93.2\nRFT-DA (ours)|**96.9**|**97.1**\nFull-set Training|95.1|96.5\n\n\n---\n\n\n\n### **Weakness-2: On the typos.**\n\nThank you for pointing that out! We have conducted a thorough proofreading of the paper and corrected the typos in the revision. \n\n---\n\n\n### **Q1: On the Selection of Settings and Sensitivity of the Budget.**\n\n\nThat's a great question. \n\n**How was the 90% budget chosen?** Rather than applying a fixed threshold on the RFT-DA score, which varies significantly across models and datasets, we opted for a more robust, generalizable strategy based on **score summation rate**. Specifically, we found through extensive experiments on GSM8K that selecting samples whose cumulative scores account for **90% of the total positive score sum** yields **the highest data efficiency**, defined as the ratio of performance gain to the amount of data used. Please refer to the table below for detailed results. \n\nThis heuristic proved to be **consistently effective** across different tasks and model sizes, including experiments on MATH-500 and the Puzzle dataset. As a result, we adopted this 90% summation rate as a unified selection criterion throughout all experiments reported in the paper.\n\n**Sensitivity to budget selection.** We also evaluated the sensitivity of our method to different data selection budgets. The results are provided in the supplementary material (Table 2 and the table below), under the following experimental settings: \n\n- **Setting 1 (MATH-500):** We use the DeepSeek-R1-Distill-Qwen-1.5B model and evaluate Pass@1 accuracy on the MATH-500 benchmark.\n- **Setting 2 (GSM8K):** We use Qwen-2.5-1.5B-Instruct as the base model and evaluate Pass@1 accuracy on the GSM8K validation set.\n- **Setting 3 (Puzzle dataset):** We follow the Logic-R framework to generate 5,000 puzzle-style questions varying in difficulty and number of roles (2–5 roles). We then test generalization on 6–8 role puzzles using the same Qwen-2.5-1.5B-Instruct model. \n\nAcross all three settings, we consistently observed that setting the summation threshold to **90%** yielded the **best trade-off between performance and data size**, and generalized well across datasets and tasks. This empirical consistency motivated our use of the 90% threshold in all main results reported in the paper. \n\n\n\n|                     |   Setting-1   |                |                |   Setting-2   |                |                |   Setting-3   |                |                |\n|---------------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------\n|                     | Performance ↑  | budget δ ↓     | data efficiency ↑ | Performance ↑  | budget δ ↓     | data efficiency ↑ | Performance ↑  | budget δ ↓     | data efficiency ↑ |\n| No-training         | 82.8           | 0              | -              | 80.4           | 0              | -              | 0.1            | 0              | -              |\n| Full-data-training   | 86.2           | 40300          | 0.008%         | 94.2           | 7500           | 0.18%          | 82.7           | 5000           | 1.65%          |\n| Discard the negative | 87.9           | 39485          | 0.012%         | 96.8           | 7260           | 0.22%          | **83.5**       | 4832           | 1.72%          |\n| summation-rate = 95% | **89.2**      | 19295          | 0.033%         | **97.8**       | 6721           | 0.26%          | 83.3           | 3933           | 2.11%          |\n| summation-rate = 90% | 89.0          | 8530           | **0.072%**     | 97.4           | 1329           | **1.28%**      | 83.4           | 3805           | **2.18%**      |\n| summation-rate = 80% | 87.2          | 7024           | 0.062%         | 95.8           | 1230           | 1.25%          | 77.2           | 3619           | 2.13%          |\n| summation-rate = 70% | 80.5          | 6575           | -0.034%        | 94.3           | 1132           | 1.23%          | 64.8           | 3133           | 1.94%          |\n\n\n---\n\n\n### **Q2. What are the characteristics of samples that receive very high positive scores versus those that receive negative scores?**\n\nThat's an excellent question! We found that after sorting the data in descending order based on the RFT-Score, the lower-ranked instances (indicating a negative impact) are primarily those where the model is consistently either completely correct or completely incorrect across all eight responses. In contrast, the higher-ranked entries are those where the model occasionally answers correctly but also makes mistakes. \n\nThis underscores the importance of including questions that the model can occasionally answer correctly during the RFT process. By reinforcing learning on such data, the model can thoroughly explore potential reasoning paths, both correct and incorrect. Then, the feedback from rewards helps the model quickly identify and learn these correct reasoning abilities! We showcase several such data cases on Pages 3 and 4 of the original supporting materials."}, {"review_id": "UyClJOqj83", "reviewer": "Reviewer_rGmp", "summary": "This paper applies the TracIn method to RLF, proposing a framework called RFT-DA. It estimates the contribution of individual training samples through a first-order influence approximation with linear complexity (i.e. TracIn), and uses these scores to filter and prioritize valuable examples. This data selection approach improves training stability and efficiency. Experiments on mathematical reasoning tasks show that using only 20% of selected data achieves comparable or better performance than full-data training, with the 7B model matching or outperforming the o1-mini baseline.", "questions": "1. How do you choose the 90% threshold for the score summation of top-scoring samples? How sensitive are the results to this choice?\n2. Can RFT-DA be used to dynamic or continually evolving datasets?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nDespite some minor typos, the paper is overall well written and clearly structured. Applying the TracIn method to RLF is a novel combination. The proposed approach is theoretically simple and well motivated, with a clear derivation and sound design. The experimental evaluation is thorough, and the method shows promising potential for reducing training cost by enabling data-efficient RLF according to principled influence analysis.\n### Weaknesses\nOne main motivation is to make training more efficient, but RLF-DA requires additional computation cost despite several practical strategies are applied. The author should elaborate in the main text regarding whether the benefits offered by RFT-DA justify its increased computational overhead.", "was_revised": true, "final_justification": "My concerns have been addressed. I'm willing to keep my original score, which reflects a positive recommendation for acceptance.", "rebuttal": "Response to Reviewer rGmp\n\nWe truly value your acknowledgment of our efforts! Your time and contributions to the academic community are greatly appreciated. We are more than willing to answer any questions you might have!\n\n---\n\n### **Weakness-1: On the computation cost of RFT-DA**\n\n\nIn the supplementary material (Table 3), we include a comparison with HVS on the MATH-500 dataset. Notably, HVS incurs virtually no additional time cost during the scoring phase. LIMR demonstrates comparable computational cost, as both rely solely on reward statistics collected during training and do not require any additional computation at inference. However, both methods exhibit significantly lower performance (Pass@1 of 83.6) compared to full-set training (86.2), limiting their practical utility in performance-critical scenarios. \n\nIn contrast, our proposed approach achieves substantially higher performance: 89.0 with **RFT-DA** and 88.3 with the faster variant, **RFT-DA Fast**. Despite these gains, the total computation time—including surrogate training, scoring, and subset training—is significantly lower than full-set training (192.2 hours for RFT-DA, 134.2 hours for RFT-DA Fast vs. 277.6 hours for full-set training). These results will be added to the main paper for clarity. It’s worth noting that the scoring time is measured using 8 GPUs which can be significantly reduced by using more GPUs. Next, we will introduce a revised version of RFT-DA that reduces the scoring phase time from several hours to 32 seconds.\n\n**RFT-DA Dynamic: A Faster and More Efficient Variant:** To further reduce computation time for scoring, we introduce **RFT-DA Dynamic**, which eliminates the need for fixed model checkpoints by computing scores dynamically during surrogate training. Specifically, at iteration $t$, given a batch $B_t$ and a sample $z \\in B_t$, we compute and store the inner product between the gradient of $z$ and the average gradient of the batch $B_t$. After training, we aggregate these stored inner products across all iterations in which each sample was involved to estimate its final score:\n\n$$D(z) \\approx \\sum_{t \\in \\{t^z_1, t^z_2, \\ldots\\}} \\frac{2\\eta_t}{N} \\langle G_z^t, G_{B_t}^t \\rangle,$$\n\nwhere $t^z_i$ denotes the iteration index of $i$-th time in which sample $z$ was used for training. This approach allows RFT-DA Dynamic to accumulate all scoring information on-the-fly during training, thus incurring negligible additional computation. The table below summarizes the comparison across methods in terms of scoring time using 8 GPUs and performance: \n\n\n\nMethod   | $\\~$  Time Cost of Scoring   |  $\\~$ Math-500 Pass@1  \n---|---|--- \nHVS| 4.7 sec $\\approx 0$|83.3\nLIMR| 5.3 sec $\\approx 0$ |83.6\nRFT-DA (ours)|63.1 Hours| **89.0**\nRFT-DA-fast (ours)|5 Hours| **88.1**\nRFT-DA-Dynamic (ours) | 32 sec | **88.3**\nFull-set training | - | 86.2\n\nWhile **RFT-DA Dynamic** yields slightly lower accuracy than the original RFT-DA, it achieves comparable computational efficiency to HVS and LIMR, requiring less than one minute for scoring. Importantly, it still significantly outperforms HVS, LIMR and full-set training in terms of final model performance, making it a highly effective and scalable solution for data selection under budget constraints. We will include the above in the paper.\n\n\n---\n\n\n### **Q1: On the Selection of Settings and Sensitivity of the Budget.**\n\n\nThat's a great question. This issue has also been raised by other reviewers. Here, I've restated a unified response, which I hope will address your concerns.\n\n**How was the 90% budget chosen?** Rather than applying a fixed threshold on the RFT-DA score, which varies significantly across models and datasets, we opted for a more robust, generalizable strategy based on **score summation rate**. Specifically, we found through extensive experiments on GSM8K that selecting samples whose cumulative scores account for **90% of the total positive score sum** yields **the highest data efficiency**, defined as the ratio of performance gain to the amount of data used. Please refer to the table below for detailed results. \n\nThis heuristic proved to be **consistently effective** across different tasks and model sizes, including experiments on MATH-500 and the Puzzle dataset. As a result, we adopted this 90% summation rate as a unified selection criterion throughout all experiments reported in the paper.\n\n**Sensitivity to budget selection.** We also evaluated the sensitivity of our method to different data selection budgets. The results are provided in the supplementary material (Table 2 and the table below), under the following experimental settings: \n\n- **Setting 1 (MATH-500):** We use the DeepSeek-R1-Distill-Qwen-1.5B model and evaluate Pass@1 accuracy on the MATH-500 benchmark.\n- **Setting 2 (GSM8K):** We use Qwen-2.5-1.5B-Instruct as the base model and evaluate Pass@1 accuracy on the GSM8K validation set.\n- **Setting 3 (Puzzle dataset):** We follow the Logic-R framework to generate 5,000 puzzle-style questions varying in difficulty and number of roles (2–5 roles). We then test generalization on 6–8 role puzzles using the same Qwen-2.5-1.5B-Instruct model. \n\nAcross all three settings, we consistently observed that setting the summation threshold to **90%** yielded the **best trade-off between performance and data size**, and generalized well across datasets and tasks. This empirical consistency motivated our use of the 90% threshold in all main results reported in the paper. \n\n\n\n|                     |   Setting-1   |                |                |   Setting-2   |                |                |   Setting-3   |                |                |\n|---------------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------\n|                     | Performance ↑  | budget δ ↓     | data efficiency ↑ | Performance ↑  | budget δ ↓     | data efficiency ↑ | Performance ↑  | budget δ ↓     | data efficiency ↑ |\n| No-training         | 82.8           | 0              | -              | 80.4           | 0              | -              | 0.1            | 0              | -              |\n| Full-data-training   | 86.2           | 40300          | 0.008%         | 94.2           | 7500           | 0.18%          | 82.7           | 5000           | 1.65%          |\n| Discard the negative | 87.9           | 39485          | 0.012%         | 96.8           | 7260           | 0.22%          | **83.5**       | 4832           | 1.72%          |\n| summation-rate = 95% | **89.2**      | 19295          | 0.033%         | **97.8**       | 6721           | 0.26%          | 83.3           | 3933           | 2.11%          |\n| summation-rate = 90% | 89.0          | 8530           | **0.072%**     | 97.4           | 1329           | **1.28%**      | 83.4           | 3805           | **2.18%**      |\n| summation-rate = 80% | 87.2          | 7024           | 0.062%         | 95.8           | 1230           | 1.25%          | 77.2           | 3619           | 2.13%          |\n| summation-rate = 70% | 80.5          | 6575           | -0.034%        | 94.3           | 1132           | 1.23%          | 64.8           | 3133           | 1.94%          |\n\n\n---\n\n\n### **Q2: Can RFT-DA be used on dynamic or continually evolving datasets?**\n\nThis is a great suggestion, and thank you for raising it! \n\n**RFT-DA is indeed well-suited for dynamic or continually evolving datasets.** As shown in Equation 5, the RFT-DA score measures the **gradient alignment of a sample with respect to the gradient of the overall data**. This formulation naturally supports incremental updates. When the dataset changes, for example, through the addition of new samples or the removal of existing ones, we can simply **update the aggregate gradient** accordingly by incorporating the gradients of new samples or removing the contributions of those no longer present. This allows the RFT-DA scores to be **recomputed efficiently without retraining from scratch**, making it highly adaptable to evolving datasets."}]

Author rebuttal:

Meta-review: This paper studies a timely problem, data selection for RFT. This paper gives an intuitive method with extensive and strong empirical results. All reviewers like the paper. The AC agrees and recommends acceptance.
Decision: Accept (poster)