A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
Authors: Yuzheng Hu
Fan Wu
Haotian Ye
David Forsyth
James Zou
Nan Jiang
Jiaqi W. Ma
Han Zhao
Venue: NeurIPS 2025 oral ()
Keywords: data attribution
online reinforcement learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose the first framework of data attribution for online RL.

Abstract: Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability. Data attribution provides a principled way to trace model behavior back to training samples, yet existing methods assume fixed datasets, which is violated in online RL where each experience both updates the policy and shapes future data collection.
In this paper, we initiate the study of data attribution for online RL, focusing on the widely used Proximal Policy Optimization (PPO) algorithm. We start by establishing a *local* attribution framework, interpreting model checkpoints with respect to the records in the recent training buffer. We design two target functions, capturing agent action and cumulative return respectively, and measure each record's contribution through gradient similarity between its training loss and these targets. We demonstrate the power of this framework through three concrete applications: diagnosis of learning, temporal analysis of behavior formation, and targeted intervention during training. Leveraging this framework, we further propose an algorithm, iterative influence-based filtering (IIF), for online RL training that iteratively performs experience filtering to refine policy updates. Across standard RL benchmarks (classic control, navigation, locomotion) to RLHF for large language models, IIF reduces sample complexity, speeds up training, and achieves higher returns. Together, these results open a new direction for making online RL more interpretable, efficient, and effective.

A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
Yuzheng Hu" UIUC Urbana, IL 61801 yh460illinois.. edu
Fan Wu" UIUC
Haotian Ye Stanford University Stanford, CA 94305 haotianye@stanford.o edu
Urbana, IL 61801 fanw6@illinois.. edu
David Forsyth UICC Urbana, IL 61801 daf@illinois edu
James Zou Stanford University Stanford, CA 94305 j amesz0stant ord. edu
Nan Jiang UIUC
Urbana, IL. 61801
nanjiang@ill iiniiiiiiiiiiiii.i edu
Jiaqi W. Ma UIUC
Han Zhao UTUC
Urbana, IL 61801
Urbana, IL 61801
j jiaqima@illinois. edu
hanzhao@illinoie edu
Abstract
Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability. Data attribution provides a principled way trace model behavior back to training samples, yet existing methods assume fixed datasets, which is violated in online RL where each experience both updates the policy and shapes future data collection. In this paper, we initiate the study of data attribution for online RL, focusing the widely used Proximal Policy Optimization (PPO) algorithm. We start by establishing a local attribution framework, interpreting model checkpoints with respect the records in the recent training buffer. We design two target functions, capturing agent action and cumulative return respectively, and measure each record's contribution through gradient similarity between training loss and these targets. We demonstrate the power of this framework through three concrete applications: diagnosis of learning, temporal analysis of behavior formation, and further targeted intervention during training. Leveraging this framework, propose an algorithm, iterative influence-based filtering (IIF), for online RL training that iteratively performs experience filtering to refine policy updates. Across standard RL benchmarks (classic control, navigation, locomotion) to RLHF for large language models, IIF reduces sample complexity, speeds up training, and achieves higher returns. Together, these results open a new direction for making online RL more interpretable, efficient, and effective.
Introduction
Reinforcement learning (RL) has achieved remarkable success across wide range of decision-making tasks, from game playing [Mnih al., 2015, Silver et al., 2016] to robotic control [Andrychowicz et al., 2020] and the alignment of large language models (LLMs) [Ouyang et al., 2022], Among its
'Equal contribution 'Equal advising
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

variants, online RL, which continuously alternates between data collection and policy updates (e.g.. A3C [Mnih et al., 2016]. PPO [Schulman al., 2017]), is well-suited to real-time, adaptive, and safety-critical domains such as autonomous driving, as it enables on-the-fly correction of mistakes and rapid adaptation to non-stationary environments [Sallab et al., 2017, Andrychowicz, et al., 2020]. However, modern online RI. faces several challenges, including sample inefficiency, high variance, and training instability, often requiring millions of interactions for convergence and yielding inconsistent performance across runs [Henderson et al., 2018, Yu, 2018, Dulac-Amold et al., 2019].
These challenges, together with their deployment in high-stakes domains, necessitate a deeper understanding of the operational mechanisms of online RL. To this end, prior work has explored various methods for RL interpretability [Milani et al., 2024, Cheng et al., 2025]. While useful, these methods often lack the fine- teee-griied explanations necessary for effective interventions or have limited applicability (see Sec. 6 for a detailed review of related work). Addressing these limitations requires exploring new paradigms.
In recent years, data attribution [Deng et al., 2025] has emerged as a powerful approach for machine learning interpretability. offering a complementary perspective by tracing model behaviors back to training data. This framework further benefits downstream applications such as data selection [Xia et al., 2024], bias mitigation [Wang et al., 2024], fact tracing [Chang et al., 2025], among others. However, applying data attribution to online RL is non-trivien non-trivial. In online RL, agents continuously interact with their environment; each collected experience not only contributes to policy updates but also influences future rollouts collected by the evolving policy. This violates the core assumptions of traditional data attribution methods, which are designed for static datasets and fixed objectives.
In this work, we address this gap by presenting the first study of data attribution for online RL, specifically focusing on the widely used Proximal Policy Optimization (PPO) algorithm [Schulman al., 2017]. Our contributions are threefold:
principled and flexible framework (Sec. 3). We propose a local data attribution framework for online RL, interpreting model checkpoints w.r.t. the records from the recent training buffer. We define the attribution entity as the atomic unit in PPO training. design two target functions that capture agent actions and cumulative returns, and measure each record's 's influence through gradient similarity between its training loss and the target.
2. Fresh insights into learning (Sec. 4). We demonstrate the power of our framework through three applications: a) diagnosis of learning: we show records most harmful for learning feature inac- curate advantage estimates; b) temporal analysis of behavior formation: we reveal an intriguing phase transition of critical records in shaping agent behaviors; c) targeted intervention: we show that removing records with the most negative influences can effectively improve model training.
3. Improved training (Sec. 5). Building on the targeted intervention, we further develop an iterative influence- bhasee filtering algorithm (IIF) that significantly improves standard online RL training. Across standard RL benchmarks modern RLHF for large language models, IIF consistently improves sample efficiency, reduces computational cost, and enhances final performance.
2 Preliminaries
2.1 Online reinforcement learning
We consider the online RL setting, where an agent learns to maximize long-term returns by interacting with the environment. The environment È is modeled as a Markov Decision Process (MDP) defined by the tuple (S, A, P, R, 7, do), where is the state space, AA the action space, P the transition function, R the reward function, 7 E [0, 1] the discount factor, and do E P(S) the initial state distribution. At timestep 6, the agent observes St, takes action ae, receives reward Tr, and transitions to 8t+1-
Online RL typically proceeds in alternating training rounds of data collection and model training (Fig. 1). In round k. the data collection phase involves the agent executing the current policy Toto, sampling experiences over multiple episodes to accumulate n transition records in a rollout buffer ((b), Each record contains the raw transition (st, Qz, r2)) and several computed quantities, including the action log probability log (a, (t)). estimated value vt. and advantage estimate Á.. Model parameters are then updated iteratively starting from e(*) = B(k); at optimization step j. training the mini-batch 13tt (k) drawn from (*)) updates parameters from 0fth) to 0 In this paper,

---

Env
Rollout buffer ((k)
Training round k
(o dng ra. log no, Vi, Ao)
Sampling experiences
upieodos
Alternating learning cycle
by Poy, Py., log, syy Av
Model ((k)
Batching
Batching
Model (
Record batch
Record batch
B
PPO tra ining updates
Measuring influence
Local Data Attribution
Per-racord training gradient
Target functions
VaL (oft) 2.)
Inner product
f faction fretura)
Validation gradient (o
Record influence
Figure 1: An Illustration of the alternating learning cycle in online RL (Sec. 2.1) and our local data attribution framework (Sec. 3.1). Online RI. operates in alternating rounds of data collection and policy updates; our local data attribution framework quantifies how individual records from single round influence different aspects of policy update in that round.
we focus on Proximal Policy Optimization (PPO), widely used, effective algorithm in various applications [Berner et al., 2019, Andrychowicz et al., 2020, Ouyang et al., 2022].
Proximal policy optimization (PPO) [Schulman et al., 2017]. PPO is policy gradient method for online RL that optimizes a clipped surrogate function. The core PPO objective, which is typically combined with value function loss and entropy bonus during optimization, is defined as:
(aass)
To(a|s)
A(s A(s..a)
PPO (0) E (ft))
min
A(s, a). clip
c. 1.1+c)
Tgtt (a|s)
Tgua (als)
where is a hyperparameter that limits policy changes between rounds and promotes stable learning.
2.2 Data attribution
Data attribution, which quantifies the influence of individual training samples on model behavior, has become increasingly important in machine learning [Grosse et al., 2023, Wang et al., 2023, Zheng et al., 2024]. Common techniques include influence functions [Koh and Liang, 20171, Data Shapley [Ghorbani and Zou, 2019], SGD-influence [Hara et al., 2019], TracIn [Pruthi et al., 2020], and TRAK [Park et al., 2023]. We focus on TracIn due its conceptual simplicity, relative efficiency, and widespread use in recent works [Xie et al., 2024, Xia et al., 2024, Lin et al., 2024].
TracIn [Pruthi et al., 2020]. TracIn measures the cumulative change in target function f() resulting from the optimization steps involving a specific training sample 2,. Formally, consider training model parameterized by 0 training set {= by minimizing the empirical loss Σi= f(0, 2,) using stochastic gradient descent (SGD). At step j. with parameters 0,. learning rate Mi and mini-batch Bj, a first-order Taylor expansion of s()) around 0g gives:
f(e,) - f(0++1) = Vof(0,) (0, -00+1) = yi AEB
Vof(O;) Vol(O,, 21).
Accumulating these contributions over the relevant training iterations yields the TracIn score for 2;: TracIn(2;) = Σ Vol(O,) Vol(O,.
J:
3 A Local Data Attribution Framework for Online RL
Online RL presents unique challenges for data attribution, due to the way data interacts with model parameters during learning. To tackle this challenge, we introduce local attribution framework tailored to local policy optimization inherent in online RL.
Challenges. The key feature online RL the circular dependency between data and model- carlier experiences drive policy updates, and updated policies produce new experiences to learn
3

---

from. The dependency of data model (red arrows in Fig. 2) is unique to online RL and cannot be addressed by existing attribution methods. Current data attribution methods include retraining-based (e.g., Ghorbani and Zou [2019]) and gradient-based, with the latter fur- ther divided into static and dynamic [Ham- moudeh and Lowd, 2024]. Retraining-basee methods require training the model once for each of the records being evaluated, which is computationally expensive in any setting and particularly prohibitive in RL. Static methods implicitly assume model parameters are ob- tained from solving an empirical risk minimiza- tion problem over fixed dataset, which is vio- lated in the non-stationary, sequential data set- ting here. While dynamic methods (e.g., TracIn) capture the temporal dependencies of training data influences on model parameters, they still fail to account for this key effect of data-model dependency. If we compute influence scores using the original formulas from standard supervised learning, they capture only the impact on parameter updates. ignoring the extra channel of influences through future data generation. As a result, the scores may deviate significantly from the true influence we seek to measure. Furthermore, quantifying influences through this channel is challenging because sampling stochastic and non-differentiable.
Butter D
Model ((0)
cample
(Au die logt
A.y
g(3)
samplo
Teerina
r B
D
B
Influence from
paramiter updatees
(
data genoration
Figure 2: Twofold data influence: driving policy
updates, shaping future data collection.
3.1 A framework of local data attribution
Our local data attribution framework addresses the circular data-model dependency. Online RL involves a local policy optimization structure, i.c., round & optimizes on a fixed buffer ((k) of on- policy data. Thus, each round serves as natural unit of analysis. Our framework operates at this level, examining how records in (k)) contributes to the updates from ((k) to 0(++1), This circumvents the challenges in tracing influence through the complex, cascading. and non non-differentiable dependencies across the training history. Below, we detail the three key components of our framework.
Entity of attribution. We consider attribution to individual training records in the rollout buffer, 2 (s;, de, rt, log n, U/; A.), collected from the environment using the current policy ((k), These records form the atomic unit used in PPO updates and provide a natural granularity for attribution.
Target functions. Training data influence is usually reflected through the impact on model behaviors. Here we focus on two core aspects of an RL agent: agent action and cumulative return.
Agent action: To identify records influencing the agent's decision to take specific action at state we define a straightforward target function:
faction (0) log [o(a s).
Cumulative return: We aim to understand which experience records contribute positively negatively to the agent's s ability to maximize cumulative return. Formally, the ideal quantity is the expected return ((0) En [R(T),, [T], where R(T) = E T; and trajectories 7 are sampled by executing Tg. However, using (()) directly poses two fundamental challenges. First, unlike supervised learning with fixed validation set, the data distribution in online RL is inherently policy pplicy-dependent. This intertwining of policy and evaluation means no fixed, universal validation set exists. Second, raw returns H(() exhibit high variance, leading to noisy influence estimates.
To address these challenges, we introduce a stable surrogate objective based on a reference policy ref and advantage estimates Áref.
fretum (0) (s.a)< [Log (a s))²a
(s, as.)).
This target function is structurally equivalent to the objective of REINFORCE with bascline [Sutton and Barto, 2018, Section 13.4]. By sampling from The we obtain a fixed evaluation distribution; using advantage estimates significantly reduces variance compared to raw returns. Maximizing fremim (6) encourages increasing the probability of better-than thatter-hha-averaage actions and decreasing worse- than-average ones, capturing the essence of improving expected return while being tractable.
For attribution round k, we set the reference policy _e = T i.e., the policy snapshot at the beginning of the round. This is key design choice of our contextual framework, which enables

---

us to ask: For the agent at its current stage of training. which experiences will be most helpful or harmful for the next update? Unlike fixed, off-distribution reference that may provide misleading signals due to mismatch with the agent's s current state, our dynamic reference evolves with training, providing a stable and relevant basis for meaningful evaluation and attribution. Furthermore, since the training rollout buffer (3k) is collected under Tg(k), we can directly use as the validation dataset. We provide further discussions on this design choice in Sec. 4.3 and Sec. 5.1.
We note that one key contribution in our framework is the design of tractable yet meaningful target functions, particularly f which can be reused in future work with alternative attribution methods. Remark 1 (Use cases of the two target functions). The two target functions have different use cases. paction is mainly for diagnosis: understanding why the agent takes a specific action at specific state (Sec. 4.2). On the other hand, fretum assesses contribution overall performance, which makes it suitable for both analysis (Sec. 4.1) and algorithmic policy improvement (Sec. 5).
Method of attribution. We adapt TracIn to our online RL setting. For record 2; in the rollout buffer B(k), we compute its influence score by summing over the optimization steps j within round k:
Σ (VoI(o P =))), where I' E { { )reum))
1=
Here, (of( is the gradient of the target function evaluated at (//)) and P (A(()) 24) the per-sample gradient of the PPO training objective for record 2, We also discuss two design choices Sec. 5.1 which substantially reduce the computational and storage costs of the vanilla TracIn.
Finally, we clarify how to interpret the computed influence scores, Records with positive influence benefit behavior formation learning, whereas those with negative influence harm it. We refer to records with the most positive influence as top records and those with the most negative influence as bottom records; these terms will be used throughout the remainder of the paper.
Remark 2 (Extension to other online RL algorithms). While we focus on PPO in our study, our frame- work readily extends to other online RL algorithms. For on-polt policylyyy methods such as TRPO [Schulman et al., 2015] and A3C [Mnih et al., 2016], the adaptation only requires modifying the per-sample loss gradient. For offline methods like DQN MMnih al., 2013], we need additionally change the target function to the Bellman error, In all cases, our alrriuuiin framework reveals whether training records help or hinder learning at the agent's 's current state. A key distinction is that, on-policy methods allow direct validation with current data, whereas off-policy methods require sampling fresh rollouts.
4 Applications of Local Data Attribution
We now illustrate the practical value of our framework. The framework delivers fresh insights for RL researchers and practitioners, enabling key applications such as diagnosis of learning, temporal analysis of agent behavior formation, and targeted interventions during training. We demonstrate these capabilities through extensive empirical studies spanning a range of RL environments and tasks.
Experimental setup. We perform evaluation on a diverse suite of RL environmentt--aaigaiiin (FrozenLake and MiniGrid), classic control (Acrobot and LunarLander), driving (Highway), and locomotion (BipedalWalken) covering discrete and continuous state and action spaces with varying complexity and reward structures. We defer descriptions of environments to Appendix A. and PPO training setups to Appendix A.2. Our code is at https: //github....... com/ wtw/ LDAORLLLDA- ORL.
4.1 Diagnosis of learning: what features bottom records?
In this section, we analyze the bottleneck that hinders learning in online RL. Specifically, examine the bottom records for fretum and uncover a consistent pattern across training rounds (additional examples in Appendix B. these bottom records are characterized by inaccurate advantage estimates, echoing observations in the literature [Ilyas et al., 2018].
Fig. 3(a-b) illustrates two examples. In FrozenLake, bottom records include poor actions receiving high positive Å and good actions receiving negative Á. Similarly, in Mini SiniGrid, the agent drifts from the goal but receives positive A. These instances of misleading ad advantage estimates harm the learning.
Forr GRPO [Shao et al., 2024], which uses gelup-relative baseline rather than dulue-funtiion baseline, the target function needs to be adjusted well.
5

---

(c-d) analysis in FrozenLake
(a) harmful records in FrozenLake (b) harmful records in MiniGrid
absii A
0.50
0.00
200
400
600
800
0.0
-0.2
200
Record (sorted by decreasing influence)
Figure 3: (a-b) Examples of bottom records. (a) Bottom 100 records in FrozenLake at k 5. aggregated over (s, a) for demonstration: arrow indicates action, green/red for positive/negative A. (b) Selected records among bottom 20 in MiniGrid at k = 5: -agent, -goal, gray area-the limited egocentric observation, yellow arrows-agent action in (turn left, turn right, forward}; all records shown are of positive Å. (c-d) These records are harmful due their inaccurate advantage estimates. We sort records by decreasing influence (top on the left). (c) y axis | Ål; points with same/opposite signs for A and colored green/red; top/bottom 20% region shaded green/red, and the intermediate in gray. (d) The product versus record rank, showing strong negative correlation.
We conduct quantitative analysis to characterize what constitutes "inaccurate" advantage estimates. We approximate the true advantage A A (s. a) using Monte Carlo (MC) rollouts from each (s, a), averaging over multiple trajeetoriii trajectories (details in Appendix B.4). We refer this as the MC estimate,
denoted by A, and compare with the advantage estimate Á. We perform analysis in FrozenLake.
Our analysis reveals two key aspects of "iiaccuracy": (I) Sign mismatch: A significant proportion of bottom records exhibit opposite signs for the advantage estimate Á and the MC estimate A (marked by red points in Fig. 3(c)). (2) Large magnitude errors: These records also have large | AJ. Together, sign flips and large magnitude errors generate strong but misleading learning signals. Indeed, the Spearman rank correlation [Spearman, 1904] between each record's influence and the product Á strongly negative (Fig. 3(d)), confirming that misaligned advantages drive harmful gradient steps.
4.2 Temporal analysis of behavior formation: phase transition of top records
investigate the reinforcement in T (a)ss). We track the evolution of top records w.r.t. faction across training rounds, which are critical in shaping the agent's 's behavior. Our analysis reveals an intriguing three-stage phase transition (Fig. 4).
specific behavior (a at s), characterized by monotonic increase
Target behavior
semantically similar s, same a, positive A
semantically
negative
Jbwer
15 stewe 918
aull dhew 663
Ple iiee 4--32.4
Bila alc 10:
clear pallerns
same u, positive
Ae
0.4
Roughness
cide 48:377
Idla-ide A
0,3
0.6
0.2
Ibla
0.1
130
lee. 2045
Dulan lite 19.
0.2
0.0
10
20
30
action-advantage association
semantic clustering
Influence saturation
Figure 4: Phase change of top records in Highway, with the target behavior taking the action "slower" when tailing the front vehicle. In the inner plot, the black curve depicts <{ls)); the red curve shows the measured roughness of the graph. =: ego vehicle; =: other vehicle. Three phases: simple action- action-advanteed advantage associations; semantic clustering (tailing states); @ no clear patterns.

---

1. Initial association: Initially, top records highlight patterns based on simple action- adviotaaa association: they manifest target action paired with positive Á, or alternative actions paired with negative A (see Appendix B.2 for examples). The agent's behavior in this phase is reinforced through this naive association, largely ignoring the context of state. This basic association persists throughout training, even as more complex relationships are learned.
Semantic clustering: As learning progresses, the agent develops more nuanced representations. As a result, a pattern of semantic clustering develops alongside the initial action-ad advantage associa- tion. Top records in this phase demonstrate action-advantage association within states semantically similar to the target state, indicating the agent has learned to generalize across similar situations.
Influence saturation: In the final phase where learning approaches convergence, influence scores for most records stabilize near zero and become dominated by noise. Due to this noise, the top records appear less structured, though the
advantage association still persists.
We quantify these phases by amalyzing the roughness (normalized Dirichlet energy) [Von Luxburg, 2007] of a similarity graph, a measure closely related to the graph Laplacian [Chung, 1997]. In this graph, nodes represent records, values are (Lec normalized) influence scores I, edge weights wij capture semantic similarity and decay with embedding distance (details in Appendix B.2). Roughness, computed as Σ w 13)² //- way, is low when semantically similar records have similar influence; this captures the clustering effect. We track roughness across training rounds. As Fig. 4 shows, roughness remains high in Phase 1, indicating influence scores are largely uncorrelated with semantic similarity. It then significantly drops in Phase 2, representing the formation of semantically meaningful clusters of records with similar influences. In Phase 3, roughness remains low due to the settling of clustering, but exhibits minor fluctuations due to influence scores dominated by noise upon convergence.
4.3 Targeted interventions during training: filtering amplifies policy gain
Sec. 4.1 demonstrates that our framework can identify harmful training records, thereby opening possibilities for targeted interventions. As a sanity check, we apply simple intervention procedure within a single training round to verify if removing these records yields performance gains.
Our procedure is straightforward: in round k; we identify records in ((k) with negative influence scores w.r.t. freturm remove them, and re-train the agent on the filtered dataset starting from (k)) Fig. 5 shows that this consistently improves performance throughout learning and across environments.
FrozenLake
Acrobot
return
0.05
100
DA
0
0.00
5
10
15
20
5
10
15
Round
Round
Figure 5: Boxplots of A return for single round interventions in two environments; red dashed line for zero A. We intervene for each round independently. The A return is computed as the difference between the test return of the model trained on the filtered dataset and the original dataset. Results are shown for 3 random sceds. Additional results can be found in Appendix B.3.
A reader may ask: how can fretum be meaningful when it relies on on-policy data with potentially in- accurate advantage estimates, unlike clean validation data used in traditional data attribution for super- vised learning? Despite potential noise in individual records, the aggregated signal from fretum is rea- sonably robust. This arises from the close alignment of fretum with the PPO objective: effective PPO updates on the training buffer implies reliable freturm for attribution, enabling our intervention to clear away misleading records while retaining beneficial ones. This can be seen as purifying the learning signal, thereby amplifying the improvement achieved by PPO. More discussions are in Appendix B.3.
Iterative Influence-Basee Filtering for Online RL Training
Standard online RL algorithms typically treat all collected experiences uniformly. However, as our analysis in Sec. 4.1 has shown, some records can be harmful for learning. This likely contributes to the notorious sample inefficiency of online RI., challenge widely acknowledged (Yu, 2018). Given this, a natural question arises: can leverage the local data attribution framework to tackle this challenge?
7

---

We propose Iterative Influence- Beasse Filtering (IIF), building on the single-roume interventions in Sec. 4.3. IIF filters records based on their computed influence scores, uses the resulting improved pol- icy to sample new data, and repeats the cycle. This creates a loop for iterative refinement. We detail the algorithm below and showcase its effectiveness traditional RI. environments and RI.F for L.Mss.
5.1
Algorithm and designs
Algorithm 1: Iterative Influence-Basee Filtering (IIF) for Online RL
Define: &: environment. records in rollout buffer. p E (0, 1]: percentage of negative records drop.
Function (modeel:
(model):
B> Stage I: sampling
B + CollectTransitiions (E, model, m)
> collect transitions into buffer B
Stage II: Filtering
f ConputeInfluencemmonn
B)
compute influence for each record drop bottom records
Biitered <- DiscardBot:omRecordaa
p)
b Stage III: training
return PPOUpdate (model, Buierea)
for iter to T'
L model < Update(model)
Alg. outlines IIF. Compared to standard PPO, IIF introduces additional step of filtering (in red) between data collection and training. We further highlight the desiderata and IIF's design choices.
Sample efficiency. We aim to reduce the environment interactions required to reach given performance level. To achieve this, IIF reuses the original rollout buffer ((k) as the validation set for influence calculation, incurring no extra sampling overhead. Furthermore, by selectively filtering bottom records, IIF accelerates learning, thus further reducing the total interactions needed.
Computational cost. We aim to keep the overhead of influence calculation small. This is achieved through two design choices. (1) Instead of iterating over all intermediate checkpoints, we compute the influence scores for the entire rollout buffer (k)) in round & via (O(k)),, V (OCk), .z)). using only the initial parameter g(k), This saves a full training pass and excessive forward/backwwar calculations. (2) We implement an efficient "ghost dot product" following Wang et al. [2025a].
Final performance. We aim to improve the policy's final performance compared to standard training. IIF fulfills this through identifying and filtering out harmful records.
employs a hyperparameter, p. which determines the amount of records to discard. We evaluate various p' s and report the best in Fig, 6. We observe that removing all negative-influene records (p = 100%) as Wang et al. [2025a] often suboptimal, likely due to the non-additivity of sample influence [Hu al., 2024]. Full ablation and recommendations for the choice of are in Appendix B.6.
5.2 Experiments in traditional RL environments
Experimental setup. We evaluate IIF on the diverse set of RL environments introduced in Sec. 4.
Baselines: We compare IIF with standard PPO and a random filtering baseline (dropping a similar fraction of records). We additionally investigate advantage based filtering heuristic in Appendix B.4 motivated by the characterization of bottom records in Sec. 4.1, as well as a TD error based heuristic in Appendix B.5 inspired by the Prioritized Experience Replay algorithm [Schaul al., 2016].
Metrics: We quantify sample efficiency by the reduction in training rounds required for IIF to match standard training, For performance level (measured by test return), let maa(v) and mmr(v) u) be the carliest training rounds where standard training and IIF achieve performance at least v, respectively. The reduction at is defined (1 mag (m))m))(e) x 100%. We report two metrics: SE the mean reduction over a list of strictly increasing performance levels reached by standard training, and S Fpaak the reduction at its peak. We measure computational cost by runtime; the reduction of runtime at the performance peak. Model performance is measured by the average test return over multiple episodes. See Appendix A.2 for further details experimental setups.
similarly define peTp as
Results. Fig. 6(a) presents the test returns for each environment; Fig. 6(b) summarizes the efficiency and runtime metrics. We report a detailed breakdown of runtime in Appendix B.9. Our key findings

---

(a) Test returns over training rounds.
IIF (Ours)
Standard
Random
FrozenLake
Acrobot
MiniGrid
-100
0.6
Return
0.4
0.25
0.2-
-200
0.00
20
10
20
30
40
Highway
LunarLander
BipndalWalker
200
200
25
Return
100
20-
0-
15
-200
10
20
20
80
100
150
Round
Round
Round
(b) Improvement in sample efficiency and runtime
FrozenLake
Acrobot
MiniGrid
Highway
LunarLander
B1 Bipeda ManddalWlkees
SEve (1)
34.0% 2.0%
36.7% 1: 6.5%
65.8%
3.3%
37.7% 6.1%
26.0%
1.8%
31.0% 8.78
S.eaa (1)
19.2 5.9%
48.5% 1 0.8%
61.7% 4.1%
55.1%
2.%
39.7%
3.7%
26.2% 1 8.0%
RT yeak (1) 29.5% + 2.9%
55.2% + 1.0%
69.1% 1%
1.7%
59.9% = 0.7%
44.9%
+ 2.5%
29.25 d: 0.75
Figure 6: (a) Test returns over rounds for IIF vs. baselines. IIF speeds up learning and improves performance. Results are averaged over 5 random seeds. For Acrobot, omit early rounds where returns rise from -500 to -200 for better visualization. (b) Sample efficiency and runtime metrics.
are summarized as follows: 1) IIF achieves substantial sample efficiency gains, showing a 20-67% reduction in training rounds required to match the standard training performance across environments. 2) The computational overhead of IIF is negligible, and offset by the reduced optimization time (see Appendix B.9), leading to significant improvement in runtime. 3) IIF's final performance exceeds stan- dard training in almost every environment. These observed gains stem from effective data attribution rather than mere data reduction: random filtering performs significantly worse than original training.
5.3 Extending IIF to RLHF for large language models
As the final part, we apply IIF to improve Reinforcement Learning from Human Feedback (RLHF). Compared to standard PPO, RLHF introduces several key differences. First, the atomic unit shifts from state-action records to prompt-generation pairs,
(a) Training Reward (1)
where each generation is a trajectory (or sequence) of tokens. Second, RLHF incorporates dual reward sources: a reward model evaluating the final generation, and a per-token KL divergence penalty constrain deviation from a reference model.
4.5
4.0
3.5
IIF (Ours)
Standard
3.0
Random
To accommodate these differences, we adapt IIF for RLHF by employing a sequence-level objective:
20
40
60
80
100
Round
(b) Test toxicity (L) a different test set,
(0) = PannDaayddd( [Tog [logTo(y| noly I Ax) (a, (a.
evaluated using different toxicity detector.
where is prompt drawn from the valida- tion set Dval=
the generation, log (y(ax) E, log To To(yi (ar, yo...., the sequence y given x, and A the advantage estimate at the last token. This objective emphasizes the reward model's 's feedback at the last token.
the log-probability of
0.10
0.05
0
20
40
60
80
100
Experimental results: toxicity mitigation. We consider the task of detoxifying L.LMs using RLHF (Hugging Face, 2023), using gpt-neo-2.7B [Black et al., 2021] as our base
Round
Figure 7: IIF improves the efficiency and performance of RLHHE
Another line of work focuses on improving reward modeling in RLHF (the stage before PPO) via preference data selection [Muldrew et al., 2024, Das et al., 2024, Shen et al., 2025]; this is orthogonal to our work.

---

model. Fig. 7 illustrates the effectiveness of our approach. We defer detailed experimental setups to Appendix A.3 and additional results (e.g.. comparisons with using the target function fretum) in Appendix B.
We further highlight IIF' substantial gains in computational efficiency. IIF filters out negative- influence records (~50% of all) effectively halving the optimization time per round. Furthermore, IIF accelerates learning, requiring less than half the number of rounds to surpass standard training, sig- nificantly enhancing sample efficiency. The overhead of influence calculation is minimal. Collectively, these factors result in an ~4x reduction in total runtime (detailed breakdown in Appendix B. B.12).
Related Work
Interpretability in reinforcement learning has become a central research theme because real-world deployment requires agents that are trustworthy and reliable [Arulkumaran et al., 2017, Sutton and Barto, 2018, Milani et al., 2024, Cheng et al., 2025]. Early studies emphasize feature-level explana- tions: they highlight regions of the observation space that most influence an agent's ' decisions, often through saliency maps or attention heatmaps [Zahavy et al., 2016, Greydanus et al., 2018, Mott et al., 2019, Atrey et al., 2020, Puri et al., 2020]. A complementary thread seeks policy-level explanations. These works approximate learned policies with human-interss inttran-tterrrtaalle rules [Verma al., 2018, Soares et al., 2020), design transparent architectures [TTopin et al., 2021, Demircan et al., 2025], or dissect reward functions to clarify action choices [Juozapaitis et al., 2019, Liu and Zhu, 2025]. More recently, researchers have probed how entire training trajectories shape behavior [Deshmukh et al., 2023].
Zooming in further, identifying critical states offers finer-grainet view of decision making. Several approaches address offline settings [Guo et al., 2021, Yu et al., 2023, Liu et al., 2023, Rishav et al., 2025]. Closer our focus are methods that target online RL such as lazy-MDP [Jacq et al., 2022], StateMask [Cheng et al., 2023] and RICE [Cheng ct al., 2024]. Lazy-MDP augments the action space with "lazy" action and penalizes non-lazy choices; states where the agent still acts are interpreted as important. However, this approach requires modifying the training pipeline. StateMask and RICE train an auxiliary mask network alongside the policy, forcing random actions in selected states while keeping returns roughly unchanged; masked states are deemed non-critical. Nevertheless, these methods crucially rely on the policy being sufficiently developed, which limits their applicability when agents are still learning in complex environments.
Moving beyond these constraints, our work introduces data attribution as principled lens for interpretability in online RL. This approach closes a key methodological gap in the literature, delivers fresh insights for RL researchers and practitioners, and informs more efficient and effective training.
Conclusion and Limitations
This work pioneers data attribution for online RL. by introducing a local attribution framework that addresses the circular dependency between data and model. The framework provides fine- grained insights into how training records shape model behaviors and offers principled approach to enhancing the interpretability, efficiency, and effectiveness of online RL. We discuss a few limitations.
Optimizers. Our framework leverages TracIn, which is designed for SGD [Hammoudeh and Lowd, 2024]. However, adaptive optimizers like Adam [Kingma and Ba, 2015] are prevalent in modern RL [Asadi et al., 2023] and LLMs [Zhao et al., 2025]. In this work, we follow Wang et al. [2025b] and employ SGD as a proxy for Adam. While empirically effective, investigating attribution methods specifically tailored for adaptive optimizers [Xia et al., 2024] is valuable direction for future work.
RL algorithms. Extending our framework to other online RL algorithms, particularly those used for LLMs like GRPO [Shao et al., 2024, DeepSeek-AL 2025, Yu et al., 2025], is a promising avenue. Technically, our framework should generalize provided the attribution entity and per-sample gradients are well-defined. On the application side, leveraging attribution as principled tool for improving LLM reasoning offers an intriguing alternative to existing data selection methods [Li et al., 2025, Shi et al., 2025, Xu et al., 2025, Wang al., 2025c] that are largely based on heuristics.
Counterfactual interpretation. Finally, our local attribution framework, while powerful, lacks a clear counterfactual interpretation. This limitation partly stems from TracIn itself, but primarily from the fundamental difficulty of tracking causal effects across the circular data-model dependency inherent in online RL, as discussed in Sec. 3. We encourage future work to tackle this open problem.
10

---

Acknowledgements
We thank the anonymous NeurIPS 2025 reviewers for their constructive feedback. YH thanks Haozhe Si for assistance in setting up an NVIDIA instance. YH and JM thank Huazheng Wang and Kaiqing Zhang for helpful discussions on variance reduction. YH and HZ are partially supported by NSF IIS Grant No. 11.2416699) and the NVIDIA Academic Grant Program. HZ also acknowledges support from Google Research Scholar Award. Nan Jiang acknowledges funding support from NSF CNS-2112471, NSF CAREER IIS-2141781, Google Scholar Award, and Sloan Fellowship.
References
O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath. Deep reinforcement learning:
A brief survey. IEEE Signal Processing Magazine, 34(6):26 38, 2017.
K. Asadi, R. Fakoor, and Sabach. Resetting the optimizer in deep rl: An empirical study, Advances in Neural Information Processing Systems, 36:72284- 72324, 2023.
A. Atrey, K. Clary, and D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In International Conference on Learning Representations, 2020. URL https: /ttp:/::///pp/////eeeeeeeee
S. Bae, Hong, M. Y. Lee, H. Kim, J. Nam, and D. Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504003880, 2025.
C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer,
S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06668 2019.
S. Black, G. Leo. P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with TessorTens,,,,w,, Mar. 2021. URL https: //doii//oiio 2et//0.5888/ zenodo. 5227715.. you use this software, please cite using these metadata.
A. Chang, D. Rajagopal, T. Bolukbasi, Dixon, and Tenney. Scalable influence and fact tracing for large language model pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /estpp://openreriee
w Wwn.
Z. Cheng, X. Wu, Yu, W. Sun, W. Guo, and X. Xing. Statemask: Explaining deep reinforcement learning through state mask. Advances in Neural Information Processing Systems, 36:62457-62287, 2023.
Z. Cheng, X. Wu, J. Yu, S. Yang, G. Wang, and X. Xing. RICE: Breaking through the training bottlenecks of reinforeement learning with explanation. In Forty-first International Conference on Machine Learning, 2024. URL /
Ptt.oommmm...pss....... PKJqsZD5nQ.
Z. Cheng. J. Yu, and X. Xing. A survey on explainable deep reinforcement learning. arXiv preprint arXiv:2502.06669, 2025.
M. Chevalier-Boisvert,, B. Dai, M. Towers, R. de Lazcano, I.. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 0etp/2300.1331,, 2023.
F. R. Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
N. Das, S. Chakraborty, A. Pacchiano, and S. R. Chowdhury. Active preference optimization for sample efficient RLHF In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024. URL http::tttts::/eeeeeehhe
DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https: http:////axxiv.o//a///////// abrg//bsb/51111.. 12948.
C. Demircan, T. Saanum, A. K. Jagadish, M. Binz, and E. Schulz. Sparse autoencoders reveal temporal difference learning large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// //wprr/vie.eeeeeeeeeeeee wttprrewwwieeeettouuuuuuaeeeaeae nett/forumidd-2iyy5aaa...
11

---

J. Deng, Y. Hu, P. Hu, T.-W. Li, S. Liu, J. T. Wang, D. Ley, Q. Dai, B. Huang, J. Huang. C. Jiao, H. A. Just, Y. Pan, Shen, Y. Tu, W. Wang, X. Wang, S. Zhang, Zhang, R. Jia, H. Lakkaraju. H. Peng, W. Tang, C. Xiong. J. Zhao, H. Tong, H. Zhao, and J. W. Ma. A Survey of Data Attribution: Methods, Applications, and Evaluation the Era of Generative AI. Aug. 2025. URL https://hall
S. V. Deshmukh, A. Dasgupta, B. Krishnamurthy, N. Jiang, C. Agarwal, G. Thhochharous and J. Sub- ramanian. Explaining RL decisions with trajectories. In The Eleventh International Conference on Learning Representations, 2023. URL
net net/ffrum? httt:/ooummmddS_gggaaa i
G. Dulac-Arold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning, 2019. URL httttttts//pppeeeeeeee
S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. Realtoxicityprompt::: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. /462, 2020.
A. Ghorbani and J. Zou. Data shapley: Equitable valuation of data for machine learning- In International conference on machine learning, pages 2242-2251. PMLR, 2019.
Greydanus, A. Koul, J. Dodge, and A. Fern. Visualizing and understanding atari agents. In International conference machine learning, pages 1792-1801. PMLR, 2018.
R. Grosse, J. Bac. C. Anil. N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li. E. Durmus, E. Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03299, 2023.
W. Guo, X. Wu, U. Khan, and X. Xing. Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems, 34: 14:12222-11 11222222226,,,, 2021.
7. Hammoudeh and D. Lowd. Training data influence analysis and estimation: A survey. Machine Learning, (15)))351-2403 2024.
S. Hara, A. Nitanda, and T. Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019.
Henderson, R. Islam, Bachman. J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
E. J. Hu, yelong shen, Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Repre- sentations, 2022. URL https: openreviereeeeee wtt./fettrunnnne forum? id=nZeWeeeFYf...
Y. Hu, P. Hu, H. Zhao, and J. Ma. Most influential subset selection: Challenges, promises, and beyond. Advances in Neural Information Processing Systems, 37:111778--118100, 2024.
Hugging Face, Detoxifying language model using ppo. https: httg::///gggggggggeeeeee co/ www./o//////oooooooooooooooo ttrl/ en/ hnn/detoxifying_...21
cttpefaaaeccc//ooooo 2023. TRL documentation (v0.17.0), accessed May 8, 2025.
A. Ilyas, L. Engstrom, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. A closer look at deep policy gradients. arXiv preprint arXiv:1811 02553, 2018.
A. Jacq, J. Ferret, O. Pietquin, and M. Gcist. Lazy-mdps: Towards interpretable rl by learning when to act. In Proceedings of the International Foundation for Autonomous Agents and Multiagent Systems, pages 669-677, 2022.
7. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Veles Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
D. Kingma and Ba. Adam: A method for stochastic optimization. In The Third International Conference on Learning Representations, 2015.
W. Koh and P. Liang. Understanding black- box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
E. Leurent. An environment for autonomous driving decision-making. https://github. com/ Seeurent/highway-eett env, 2018
X. Li, H. Zou, and P. Liu. Limr: Less is for rl scaling. arXiv preprint 1rXiv:2502.11888. 2025.
H. Lin, J. Long, Z. Xu, and W. Zhao. Token-wise influential training data retrieval for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 841-860, 2024.
12

---

H. Liu, M. Zhuge, B. Li, Y. Wang, F. Faccio, B. Ghanem, and J. Schmidhuber. Learning to identify critical states for reinforcement learning from videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1955-1965, 2023.
Liu and M. Zhu. UTILITY: Utilizing explainable reinforcement learning to improve reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025. URL pettps://ppereviii www.eeeeeee/ruoeeeeeeeeeeeeeeeee wtw.ormmmdddta/aa........
Milani, N. Topin, M. Veloso, and Fang. Explainable reinforcement learning: A survey and comparative review. ACM Computing Surveys, 56(7):1-36, 2024.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:
2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):52-533, 2015.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PmLR, 2016.
A. Mott, Zoran, M. Chrzanowski, D. Wierstra, and D. Jimenez Rezende. Towards interpretable re- inforcement learning using attention augmented agents. Advances in neural information processing systems, 32, 2019.
W. Muldrew, P. Hayes, M. Zhang, and D. Barber. Active preference learning for large language models. In Forty-first International Conference on Machine Learning, 2024. URL https: /openreviee seee//oorumid=CTTE6egg3
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777
2022.
S. M. Park, K. Georgiev. A. Ilyas, G. Leclerc, and A. Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning, pages 27074-27113, PMLR, 2023.
G. Pruthi. F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 19900-999999,, 2020.
N. Puri, S. Verma, P. Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and Singh. Explain your Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2020. URL https: http://pppneeeee.eeeeeeee nwtp:ormmfffiiiett SJgzLkBKPB.
move:
A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Emnssts, and N. Dormann. 3asles- Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL ht org/ pttp//apprr////220--222 13641113222222262..22 html.
R. Rishav, S. Nath, V. Michalski, and F. Kahou. Behaviour discovery and attribution for explainable reinforcement learning. arXiv preprint arXiv:2503. /4973, 2025.
A. E. Sallab, M. Abdou, F. Perot, and S. Yogamani. Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv: 0704.02044222.. 2017.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016. http:/// wwtpii..oggr////////eeees 05952.
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707006377 2017.
Z. Shao, P. Wang, Q Zhu, R. Xu, J. Song. X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint 402.v:2402..33888 2024.
13

---

Y. Shen, H. Sun, and J.-F. Ton. Reviving the classics: Active reward modeling in large language model alignment. arXiv preprint 2tti:255220433544 2025.
T. Shi, Y. Wu, L. Song, T. Zhou, and J. Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schritwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587)4844488) 2016.
E. Soares, P. P. Angelov, B. Costa, M. P. G. Castro, S. Nageshrao, and D. Filev. Explaining deep learning models through rule-based approximation and visualization. IEEE Transactions on Fuzzy Systems, (0(8)2339--2401 2020.
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):22-011, 1904.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, second edition, 2018. URL http::/anncompletett
set//ook/ thotthoheobooooo 2nd. html.
N. Topin, S. Milani, Fang, and M. Veloso. Iterative bounding mdps: Learning interpretable policies via non-interpretable methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9923-9931, 2021.
M. Towers, A. Kwiatkowski, J. Terry, U. Balis, G. D. Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, R. Vicent,, A. Pierré, S. Schulhoff, J. J. Tai, H. Tan, and O. G. Younis. Gymnasium: standard interface for reinforcement learning environments, 2024. URL
https://arxiv.. org hts///bs//444 2407. 17032.
A. Verma, Murali, R. Singh, P. Kohli, and S. Chaudhuri. Programmatically interpretable rein- forcement learning. In International conference on machine learning, pages 5045-5054. PMLR, 2018.
B. Vidgen, Thrush, Z. Waseem, and D. Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection. In ACL, 2021.
U. Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17:395-416, 2007.
L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec. Trl: Transformer reinforcement learning. https: /tte:///iittt http://ggguuuuu..ommnaaaae com/huggingface/ trl. 2020.
H. Wang, Z. Wu, and J. He. Fairif: Boosting fairness in deep learning via influence functions with validation set sensitive attributes. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 721-730, 2024.
J. T. Wang, P. Mittal, D. Song, and R. Jia. Data shapley in one training run. In The Thirteenth International Conference on Learning Representations, 2025a. URL HD6DWc
J. T. Wang, D. Song. J. Zou, P. Mittal, and R. Jia. Capturing the temporal dependence training data influence. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreviee
net/ orum? iidddddduufi d...H..DEES.....
S.-Y. Wang, A. A. Efros, J.-Y. Zhu, and R. Zhang. Evaluating data attribution for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7192-7203, 2023.
Y. Wang, Q. Yang, Z. Zeng, L. Ren, L. Liu, B. Peng. H. Cheng. X. He, K. Wang, J. Gao, W. Chen, S. Wang, S. S. Du, and Y. Shen. Reinforcement learning for reasoning in large language models 2025c.
with training example. arXiv preprint arxiv:2504.20511
M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. LESS: Selecting influential data for targeted instruction tuning. Forty-first International Conference on Machine Learning, 2024. URL ht https: //tpp////////eeiee////////////ee wwwprrvviiie.nee//ouirr... het.aeeneeeeetoomoeedmod-- dit.mmmmmmddppPmmmmmmmmmm
T. Xie, H. Li, A. Bai, and C.-J. Hsieh. Data attribution for diffusion models: Timestep-induced bias in influence estimation. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https:/ http::///ppeeeeeeeee
met:/forumiid==PLyumm
Y. F. Xu, Y. Savani, Fang, and Kolter. Not all rollouts are useful: Down-sampling rollouts in Il reinforcement learning. arXiv preprint 2rx.v:250..13818, 2025.
14

---

J. Yu, W. Guo, Q. Qin, G. Wang, T. Wang, and X. Xing. {AIRS): Explanation for deep reinforcement learning based security applications. In 32nd USENIX Security Symposium (USENIX Security 23), pages 7375-7392, 2023.
Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source IIm reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025.
Y. Yu. Towards sample efficient reinforcement learning. In IJCAI, pages 5739-5743, 2018.
T. Zahavy, N. Ben -rihem, and S. Mannor. Graying the black box: Understanding dqns. In International conference on muchine learning, pages 1899-1908. PMLR, 2016.
R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. M. Kakade. Deconstructing what makes good optimizer for autoregressive language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https: httppp:://ppeee http//ppperevveieeeeeee wwwpeeeviee.nee/forttttit fforummiddzzfesocceqp
X. Zheng, T. Pang, C. Du, J. Jiang, and M. Lin. Intriguing properties of data attribution diffusion models. The Twelfth International Conference on Learning Representations, 2024. URL https: / https://openreweee /ttp///ppneeeeeeee///ee/ee wwt./eee/ttooooo
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have tried our best ensure that the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information on the experimental setups in Appendix A. Our code is also also publicly available https: / /ttpt/ggttt//////// com/LDAORL/IDA- ORL.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Our code is publicly available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ wttpc//guiess//oeeeeeeee
wwwssssoooooooooooo onooliccyy olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //wpps.ccpppiiii wtt.cc//pbb. wwwpccccpccc/ceeee.ceeeee.eccele http:/ppblcccp///eeeeeeoo
httpa///ggiieeeeeeeuubbiiiiiiiia http///ooSuuiiiseeeeeee iotbbbmsssiins ieiiionooolily for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of the experiments are discussed in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We use random seeds for all experiments; we include error bars in all reported results (Figs. 5 to 7) in the main paper as well as more results in Appendix B (Figs. 11 to 14 and tables 2, 5 and 6).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the compute resources is provided in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that the research conducted in this paper adheres to its principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper conducts fundamental research aimed understanding the role of data in online RI., and leverages this understanding to improve RI. training. We do not anticipate any immediate societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the RL environments datasets, models, code frameworks, and included their licenses in Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : http://neiipaa cc cc///onfeeeeccc/222/111111 for what should or should be described.
22

---

A Detailed Experimental Setups
A.1 Standard RL environments
We offer a detailed description of the RL environments used in our experiments in Table 1.
Gymnasium and Hi ghway are licensed under MIT license; MiniGrid is licensed under Apache-2.0 license.
Table A summary description RL environments we use in experiments. Besides MiniGrid and Highway, other environments are from Gymnasium [Towers et al.. 2024].
Eny
Eny ID & Args
Goal
State Space
Action Space
Reward Structure
MiniGrid
NiniGrid- Empty-8xde -x8--x000
Navigate to tur-
3x 7 image, repre-
discrete actions:
0.9 (step_co count/max steps) success, 0 other- wise
Sparse:
[Chevalier-
get location
senting egncentric
{turn right, move forward,
Boisvert ul., 2023]
view of the agent's
observation
pickup, drop, toggle, done)
FrozenLake
FrozenLake-v1',
Navigate from start to goal without falling
discrete integer: agent position index on the grid
discrete actions:
Sparse: reach- ing goal, 0 otherwise
map=4x4, alipper==False
Lett, Down, Right, Up)
into holes
Acrobot
Acrabot-vi"
Swing up the link to reach a target height
Ra providing infor-
discrete actions: Dense: -1 step til reaching the target height
mation about the two rotational joint angles and their angular ve-
{-1,0,1) torque (N m)
locities
Hi Highaay [Leurent, 2018]
hi highway-vo', vehicle_ _ocut 10
Drive at high speed while avoiding colli- sions
Kinematic Ohserva- tion: 5 array of and nearhy vehi- cles, including their location and speed
discrete
actions:
Dense:
[LANE, LEFT, IDLE, LANE MANE_RIGHT,, RIGHT. FASTER, SLOWER]
(o oattl//omas thew? enllision at each step
LunarLander
LunarLander-v22
v22
Land safely the pad from flight
R²; the coordinates of the lander, its lin- car velocities, angle, angular velocity, and whether each leg is ground
discrete actions: [do nothing. fire left.
Dense: +10 per leg contact: -0.03 per
main, right)
side-engine step:
main-engine step: +100 on safe landing; -100 crush; dis-
with
fance/velncity/angle terms
BipedalWalker
BipedslNalxer-v3" Traverse rough terrain without falling
g24, hull ungle speed. angular ve- locity, horizental vertical speed, joints positions & angular speed, legs contact with ground, 10 lidar measurements
continuous actions: motor speed values in 1. 1] for joints hips and knees
Dense: per for- ward step: -100 on fall; small penalty proportional to torque magnitude
A.2 Experimental setups for standard RL
Training setups. We adopt Stable-Baselinee [Raffin et al., 2021] (MIT license) our training framework for the standard RL experiments. We use PPO [Schulman et al., 2017] as our RL algorithm and adopt the default training hyperparamters and network architectures for most environments unless otherwise specified.
ststpe://minigrrddffarraa htt.://gg.......aaaaaoooooeeenia http///wrreeeeett/tnniiii///aaaa
hattpe:/gyymmsiumm. farama. www.:/aw........eerrreeeeeeeeeea
"https://gymaasiiun
f.symasaium.arrmaa........ farama.cmm http://aww..a..aeeaeeaaaaaoaoooa
farama. fart::://iilalayym....r......
Dttpp::/ggymassumm.e
lande//
10
"nttps:/sstable ineeleesssellnee.....m.
23

---

Training hyperparameters: We use s_steps=2048 (i.e.,
[B 2048).
fatch_size=64 (i.e., |B55) = 64), m_eeochs=10 (i.e., each rollout buffer will be used for 10 epochs), learning rate=5e-3 with optimizer=SGD in all environments except Bipedalwalker, for which we use 3e-4 with Adam. total timesteps per environment are: 102,400 for FrozenLake (50 rounds), 81,920 for MiniGrid (40 rounds), 102,400 for Acrobot (50 rounds), 204,800 for Highway (100 rounds), 307,200 for LunarLander (150 rounds), 1,024,000 for BipedalWalker (1000 rounds). Other hyperparameters include ent_ cocf=0.0, range=02, gamma=0.99, gac lambda=0.95, vf_coef=0.5, max_ grad_norm=0.35
Network architectures: For FrozenLake, Acrobot, Highway, LunarLander, and BipedalWalker, we use the default KlpPolicy Stable-Baselines.. This policy uses two-layer MLP networks (64 hidden units per layer), taking the flattened observation as input. For MiniGrid with image input, we use an adapted CanPolicy with a custom feature extractor. The extractor comprises two convolutional layers (with 16 and 32 filters respectively, and 3x3 kernels) followed by a linear layer of 64 hidden units.
Evaluation setups. We evaluate the stochastic performance each policy Tyco every training round & by averaging returns over multiple evaluation episodes. Specifically, we run 1000 episodes for LunarLander, Acrobot, MiniGrid, and FrozenLake; and 100 episodes for Highway and BipedalWalker.
A.3 Experimental setups for RLHF
We follow Hugging Face [2023] to set up this experiment. The base model is a 2.7B parameter GPT-Neo model [Black et al., 2021] (MIT license).
Training setups. We adopt TRL 11 [von Werra et al., 2020] (Apache-2.0 license) our training framework to fine-tune the based model via PPO. We employ LoRA [Hu et al., 2022] to perform PEFT fine-tuning, with rank of 16, of 32 and dropout of 0.05. The dataset for PPO training is real-toxici ceexxiiiityyppp prr extract the first 10-15 tokens prompt, generate a 30-token continuation, and score it with the reward model, a toxicity detector LFTW R4 (VVigen et al., 2021]. The reward signal is the raw logits of the label "neutral" of the detector.
[Gehman et al., 2020] (Apache-2.0 license). For each example, we
The naming of the hyperparameters in TRL slightly differs from the ones in Stable-Baselimg Here stick to the naming in TRL to report the hyperparameters and clarfy their meanings using notations. We follow Hugging Face [2023] to use batch, size=256 (i.c., n [a) 256),
mini seri__atch_siisseeee (i.e., [B(*] = 1), ppo_epochs=4 (i.e., each rollout buffer will be used for epochs), learning_rate=le-5 rete= with Adam optimizer, and all other default hyperparameters in TRL. We
train for one epoch over the training dataset, which amounts to 109 rounds in total.
Evaluation setups. We evaluate the performance of each policy тg(N) at every training round Evaluation is performed on Wiki-Toxice which is of different distribution than the training dataset. For each toxic sample, we use the full sample as the prompt (significanity longer than used in training and thus more likely elicit toxic continuations), and generate 30-token continuation (same as the training setup). We then evaluate the toxicity of the generated continuation using another toxicity detector 400 samples, and we report the mean toxicity probability.
Evaluation is conducted over
" hhtp://...aggggae.......aaaeeees
24

---

B Additional Experimental Results
B.1 More demonstrations of harmful records
Harmful records for learning across training rounds. We examine the bottom records w.r.t f fretmm in different training rounds k and present the results in Fig. 8. (Results in the main paper, Fig. 3(a), corresponds to k = 5 here.)
(a) k
(b) k 5
(c) k 10
Figure 8: Bottom records in different training rounds in FrozenLake. Arrow indicates action, green/red indicates positive/negative Á.
Across all three snapshots (A: 2, 5, 10), the bottom records share a clear and consistent pattern: inaccurate advantage estimate, rewarding the agent for poor action (moving away from the goal) and penalizing the agent for a good one (moving towards the goal).
Harmful records in complex environments. We look into two complex environments. In BipedalWakeer (locomotion), our analysis reveals bottom records where the agent incor- rectly penalized with a large negative advantage for executing a successful recovery move (e.g.. applying corrective torque with deeply bent knee (~35%) during landing or push-off). (We omit the visualizations for this environment as does not conveniently support rendering given status vectors; the above analysis is done based on direct analysis of values status vectors.) Pong (Atari), find that bottom records filtered by IIF consist of uninformative transitions (the ball being out of play or already moving away from the agent) that receive (inaccurately) high advantage estimates. By filtering out these samples, IIF achieves significant improvement in training efficiency. These results show that 1) bottom records feature inaccurate advantage estimates; 2) IIF effective, holding generally across different environments. Examples are shown in Fig. 9.
B.2 Quantifying phase change via weighted graph roughness analysis
Measurement protocol. We provide full details of our quantitative investigation.
For each round k, we build the similarity graph Gy using records with positive influence scores in B(k) and their influence scores [Von Luxburg, 2007]. We embed each record 2 as a node in the graph, with the node value being the L normallzzz influence score I, = 11/1//]lo.. k., the node embedding being the record embedding e; extracted by well-trained network (obtained at the end of the PPO training). We set edge weights by Gaussian kernel Wij = exp(-l e - 2 with a chosen via the median-distane heuristic. We retain each node's nearest neighbors when building the similarity graph. This reduces computational cost. In practice, we find that varying from 20 to 100 has little effect on the roughness measure.
With the graph Gg built, we compute the graph roughness as follows:
Roughness(G:) Σie, Wey I,)
Eisy Wij We repeat this process for all rounds & and plot the change of roughness over rounds.
25

---

frame 0
frame
frame 2
frame
frame 0
frame
frame 2
frame
frame 0
frame
frame 2 D]
frame
Figure 9: Bottom records for the Pong. The top and middle figures correspond to the case where the ball it out of play. The bottom figure corresponds to the case where the ball is moving away from the agent. (Note that in Pong, the ego agent is the one on the right.)
2=2
Roughness
0.8
Target behavior
0.04
0.4
0.02
Round
positive
semantically similar same positive A
patterns
1a+2, A+27.11
0m22A-100)
0en2, Av2121
2--2, A-110)
(a-2,
(a- A-2266
Iam2, Aw15.7)
(=:2.
Ian2,
different negative
semantically similar different negative A (m-0.
Am-33
lam Ama17
Au-17)
(mm) Amsta)
Am-Ja
Figure 10: Phase change of top records in Acrobot.
Results more environments. study another environment Acrobot, investigating the phase change and measuring the roughness metric across rounds. The results are presented in Fig. 10. We observe a consistent trend of the three phases, aligned with the findings discussed Sec. 4.2.
In Phase top records include those with the same action and positive Å, and those with alternative actions and negative Á. Roughness is high this phase. In Phase 2, semantically similar records (that consistently show the action-advantal advantage association) emerge top records; roughness decreases significantly in this phase. Phase 3, learning approaches convergence and the semantic clustering stabilizes; influence scores become dominated by noise, causing roughness to show minor fluctuations.
26

---

Influence-guided intervention
Random drop
LrkzenLake
FrozenLake
A return
0.05
0.00
P114
5
10
15
20
10
20
Acrobot
Acrobot
A retur
100
I
-50
10
15
5
Highway
Highway
A return
0
10
15
20
10
MiniGrid
MiniGrid
A retur
0.00
0.00
-0.05
20
40
10
20
30
Round
Round
Figure 11: Boxplots of A return for single rollout interventions in four enviromments, comparing influence-guided intervention (left) with random drop (right). We perform intervention for each iteration independently by removing bottom records and then retrain the model. The A return is calculated as the difference between the return from the model trained on the filtered dataset and the original dataset. Results are shown for three random seeds.
B.3 Additional results for single-round intervention
Fig. 11 (as extension of Fig. 5) presents the results of single-round interventions in four en- vironments, additionally comparing with the random baseline that discards a similar amount of records.
We discuss several key takeaways: (1) selluence-guided intervention mostly leads performance gains, while random drop mostly leads to performance degradation. Hi gigaaay see Fig. 6), the attribution signal can become
(2) When standard PPO fails
to improve (e.g. dip at round À unreliable, producing negative A return (see Fig. 11 at k = 9 in Hi Highway), leading occasionally to interventions that fail bring any improvement. However, as long as PPO's overall trend upward, our intervention can effectively purify the learning and and drive net improvement over the full run.
We also note that while our approach has flavor of variance reduction. in the sense that it removes outlier gradients, it is fundamentally different from standard variance reduction techniques such as Generalized Advantage Estimation [Schulman et al., 2016] or baseline extraction [Schulman et al., 2017, Sutton and Barto, 2018]. In particular, the analysis in Sec. 4.1 shows that our method identifies genuinely harmful rather than useless samples, and thus has a bias-correction effect.
B.4 Advantage-based heuristic
Method. Sec. 4.1 characterizes the properties of the bottom harmful records-sign mismatch and large magnitude errors. Inspired by these findings, we design the following two heuristics for experience filtering:
Heuristic I: We discard records with opposite signs for A and Å. Among these records, we sort them by LA - A and discard the top p% records with the largest error.
27

---

Heuristic 2: We discard records with opposite signs for A and A. Among these records, we sort them by A A and discard the bottom p% records with the smallest product (i.e., the most negative).
Implementation.
These heuristies fundamentally rely on obtaining reliable estimate of the true advantage function, ( a), for each training record. We obtain A using Monte Carlo (MC) estimates, i.c.
(s = Q*s,)))) (s) =E r = 5,a, = a - E Σ Trtellss:
In environments with small, discrete state and action spaces, we can leverage the collected rollout buffer (*) to obtain the estimate AT ((k) itself would include multiple occurrences of (s, a) pairs or visits to state s, allowing for empirical averaging.
However, in environments with large discrete contiumous state/action spaces, specific state-action pairs (s, a) are rarely encountered multiple times in B(k), Accurately estimating AT (s, a) for each record in these more complex settings would require resetting the environment to the specific and then performing numerous independent rollouts under policy Tg(a). This procedure is generally computationally infeasible.
For consideration of computational efficiency, in our study below, we limit to environments with discrete state and action spaces, where we compute using the collected rollout buffer (**), instead of performing additional sampling in the environment.
Results. Fig. 12 compares the two advantage- based heuristics against IIF and standard training in FrozenLake and MiniGrid.
(Ours)
Advantage Heuristic (1)
Advantage Heuristic (2)
Stan
FrozenLake
MiniGrid
0.75
0.6
Retur
0.50
0.4
0.2
0.00
0
10
20
30
0
10
20
30
40
Round
Round
Figure 12: Test returns over training rounds for the two advantage-based heuristics, compared with IIF and standard PPO. Results averaged over three random seeds.
In FrozenLake, a small discrete environment, both heuristics closely match IIF's learning curve and final return, and substantially outperforms standard PPO. This result serves as a validation of our initial findings in Section 4.1, confirming that transitions exhibiting sign mismatch or large advantage estimation errors are indeed key properties of harmful experiences, and that filtering based on these properties can significantly improve training efficiency.
However, in MiniGrid, which features a significantly larger state space, the advantage-based heuris- tics fail to improve upon the standard PPO baseline and in fact even degrade performance. There are two possible reasons. (1) The advantage estimates Á are noisy due to the limited number of repeated visits per (s, a) and in (k), leading to inaccurate filtering, (2) These heuristics rely solely on the relationship between estimated and true advantages; in comparison, IIF's influence score, derived from gradients, captures broader, more nuanced set of characteristies of harmful records. This richer representation allows IIF to perform effective filtering when simple advantage heuristics fail.
In summary, these results validate our core insights: properties like sign mismatch and large estimation errors are indeed indicative of harmful training records. At the same time, their failure in more complex environments highlights the limitations of these simple heuristics. Our IIF framework.
28

---

by contrast, is more generally applicable; its influence scores capture a broader and more nuanced understanding of records' values beyond simple advantage discrepancies, enabling effective filtering even in complex domains.
B.5 TD error based heuristic
Motivation. Prioritized Experience Replay (PER) [Schaul et al., 2016] demonstrate that reweighting transitions in proportion to their comporal--iffereen (TD) error accelerates learning and improves performance in off- methods. TD error serves useful heuristic, indicating how "surprising" or "important" transition is for updating the value function. While PPO is an on-policy method that typically uses smaller, on-policy rollout buffer rather than large replay buffer like those in off-policy algorithms, the core idea of focusing learning on more impactful experiences remains relevant. Inspired by PER, investigate integrating TT error based reweighting mechanism into the PPO training process to prioritize samples within its rollout buffer.
Implementation.
For each transition (s;; az, F., s() collected and stored in the rollout buffer ((k), we first compute its TD error. The TD error for record is defined as:
8, ri + VV*000 (s{) V (st),
where 770(00) denotes the current value function estimate (under the current policy Tgcss).
We then assign a priority to each record using a rank-based approach following Schaul et al. [2016].
We sort all transitions in the buffer (8) descending order based on the absolute value of their TD error, [6g]. The base priority for transition i is set as P. //rank(i). where rank(i) denotes the rank of transition i. Then, the probability of sampling record is
P.
W/i EjEBa) Pp"
where = 0.6 (following Schaul et al. [2016])
This weighting scheme ensures that transitions with larger absolute TD errors receive higher emphasis during the PPO optimization steps.
Results. We evaluate the performance of the TD error based reweighting heuristic by comparing it against IIF and standard PPO on FrozenLake and LunarLander. Fig. 13 presents the test returns over training rounds for these approaches.
(Ours)
TD Error Based Heuristic
Standard
FrozenLake
LunarLander
1.00
200
100
Return
0.50
0-
0.25
100
0.00
200
30
20
40
100
Round
Round
Figure 13: Test returns over training rounds for the TD error based heuristic, compared with IIF and standard PPO. Results are averaged over three random seeds.
In FrozenLake, a simple environment, both TD error and IIF accelcrate convergence, reaching opti- mal return sooner. The TD error heuristic nearly matches IIF's speed, confirming that large TD emrors align well with truly useful transitions when the state-action space is small and reward structure simple.
In contrast, in the more complex LunarLander, the TD error heuristic degrades performance: it learns more slowly than even standard PPO and exhibits greater variance. Although this heuristic succeeds in PER, we comment that there are intrinsic differences in the off-policy scenario where PER was proposed and evaluated, vs. the on-policy scenario (e.g., PPO) we study in this paper vast, diverse buffer. However, in PPO, raw TD emors
(Fig. 1). PER applies the TD error heuristic mix estimator noise with true signal; PPO's small, fresh, on-policy batches exacerbate that noise: Our influence scores, in comparison, appears more robust in such scenarios.
29

---

B.6 IIF performance under various filtering percentages
We evaluate the impact of the filtering percentage hyperparameter p on the performance of our proposed IIF method. The filtering percentage (as introduced in Algorithm 1) determintes the proportion of negative-influene training records to discard from the bottom. We explore wide range of values for p E (100.%. 50.0%, 25.0%, 12.5%, 6. 6.25%), reducing the percentage by half at each level. Note that p 100.0% means discarding all negative-influenc records.
Fig. 14 shows the test returns over training rounds for IIF with varying p's compared to baselines. We additionally quantify their efficiency using two metrics: Lave and S Bpaak (introduced Sec. 5.2). We summarize these efficiency statistics in Table 2.
Standard Random
(p = 12.5%)
(p =p==5 50.0%)
IIF (p 25.0%)
IIF (p 100.0%)
FrozenLake
Acrobot
1.00
100
0.75
Return
0.50
150
0.25
200
10
20
30
5
10
15
20
MiniGrid
LunarLander
Return
0.4
0.2
-200
10
30
40
20
40
60
100
Round
Round
Figure 14: Test returns over training rounds for IIF with a range of filtering percentages p. compared to the baselines. Larger p means more aggressive filtering, Results are averaged over three random seeds.
Table 2: Sample efficiency comparison across varying filtering percentages. Results show the improvement in sample efficiency metrics ESEg and S Speak) for different filtering percentages. across simpler and more complex environments. Bold values indicate the best performing value of p; italicized values show the second best. Results are averaged over three runs.
(a) S (t)
FrozenLake
Acrobot
MiniGrid
LunarLander
p 12.5%
23.5%
1 3.1%
29.2% 1. 0.8%
67.5% 1 5.1%
28.2% 1 1.3%
25.0%
30.5%
1 3.3%
35. 1%
0.6%
60.3%
1. 10.6%
22.7% 1. 5.6%
p 50.0%
33.7%
1 3.49
36.7%
6.5%
67.0%
1. 5.3%
10.2%
1. 6.5%
100.0%
32.7%
+: 1.7%
35.0%
0.5%
75.4%
3.6%
8.9% :: 2.0%
(b) S E (t)
FrozenLake
Acrobot
MiniGrid
LunarLander
12.5%
15.6% + 5.1%
31.5%
2.2%
67.4%
4.4%
41.6% + 5.7%
p 25.0%
22.1%
+ 7.4%
48.5%
+: 0.8%
58.8% + 13.1%
32, 22.9% + 13.1%
50.0%
79.6%
+ 8.4%
48.5%
0.8%
50.6%
20.7%
15.5% 17.1%
p 100.0%
15.9%
+: 5.5%
43.1% + 5.7%
54.9% +: 22.5%
15.8% 4 7.3%
We highlight several key findings:
30

---

Discarding all negative records (p = 100%) is suboptimal. As shown in Figure 14, setting p 100% leads suboptimal final performance, slower learning progress (also reflected in Table 2), and instability in training. This observation aligns with the concept of non-additivity of sample influence [Hu et al., 2024].
Any level of filtering improves performance over standard training. Applying IIF with almost any filtering percentage demonstrates improvement compared to standard training. This underscores the general effectiveness of IIF in mitigating negative influence by removing a portion of identified negative samples.
The optimal filtering percentage varies with environment complexity. simpler envi- ronments (e.g. FrozenLake, Acrobot), removing half of the negative samples (p 50%) yields the best performance overall-simphl environments could involve plenty of redun- dancy; aggressive pruning focuses learning the most informative transitions. In contrast, in more complex environments (MiniGrid, LunarLander) the interplay among records is subtler: overly large filtering discard mnderlineeusefitn transitions, while gentler filtering (p = 12.5%) can achieve better performance.
Based on these findings, for our main experiments (see Sec. 5.2) we choose the specific filter- ing percentages to reflect the optimal configuration per environment. We use p 50% for FrozenLake, Acrobot, Highway; 12.5% for MiniGrid, LunarLander; and = 6.25% for BipedalWalker.
B.7 Evaluating IIF with the Adam optimizer
Our main experiments in traditional RL environments are conducted using the SGD optimizer (see Appendix A.2). Here we additionally apply the Adam optimizer on two environments, MiniGrid and LunarL
We report the test return in Fig. 15, and sample efficiency and runtime metrics in Table 3. One observation is that IIF gains less with Adam compared to SGD in MiniGrid, whereas the trend is reversed for LunarLander (see Fig. 6 for reference). This is partly because Adam significantly speeds up training compared to SGD in MiniGrid (and thus reduces the room of improvement), but less so in LunarLander.
MiniGrid
LunarLander
IIF (Ours)
200
0.8
Standard
0.6
0.4
-100
0.2
200
0
5
10
15
20
0
20
40
60
80
Figure 15: Test returns over rounds for HIF vs. the standard training baseline, when using the Adam optimizer. Results show that IIF delivers clear and substantial benefit regardless of the choice of optimizers or environments.
Table 3: Sample efficiency and runtime comparisons when using the Adam optimizer.
MiniGrid
LunarLander
S (t)
24.1% + 1.4%
46.7%
4.5%
S Speak (t)
13.3% +: 3.1%
62.2% 5.0%
RT RTpeak peak (t)
18.5%
1.0%
65.9% 3.2%
31

---

B.8 Statistical significance of final performance gains
We compute the 95% confidence interval (CI) for the performance gain of IIF over the standard baseline (as shown in Fig. 6(a)). Concretely, we compute half-width = to 93.437.4 x SE = 2.776 x SE. Results in Table 4 confirm a statistically significant improvement in the performance gain.
Table 4:
95% confidence interval (CI) for the performance gain of IIF over the standard baseline across 5 random seeds.
MiniGrid
LunarLander
BipedalWalker
95% CI [0.04, 0.33] [22.54, 130.52]
[24.40, 75.99]
B.9 Runtime for experiments on traditional RL environments
We report the runtime for experiments on traditional RL environments in Table 5.
For per-round runtime, we report the time for the influence calculation step and the optimization step. The overhead of IIF in the influence calculation step is negligible. As IIF discards p% of the negative records, it enjoys reduction in optimization time.
For total runtime, we first report the runtime for all training rounds (labeled as "All rounds"), and then report the runtime corresponding to the (reduced) rounds needed for IIF to match the peak performance of standard PPO (labeled "Matching peak"). IIF's improvement in sample efficiency leads to a further speedup.
Finally, we report peaa (also presented in Fig. 6(b)), calculated as the reduced percentage of wall clock time for IIF to match standard PPO. In summary, IIF presents a 29%-67% % reduction in runtime, effectively speeding up learning.
Table Per-round runtime and total runtime (in seconds), as well as the percentage of overall reduced runtime for experiments on traditional RL environments, Results are averaged over 3 training runs each for IIF and standard training. A dash (-) indicates that measure is not applicable.
FrozenLake
Acrobot
MiniGrid
standard
standard
IIF
standard
Per-round runtime
Influence cale
0.11 1
0.25 = 0.1
:: 0.0
Optimization
1.51 0.0
2.01 +: 0.5
1.42 = 002
2.02 0.00
4.52 :: 0.06
5.02 # 0,07
Totul runtime
rounds
82.15 à: 1.95
93.85 a 268
70.01 0.72
79.87 :: 1.00
365.23 = 2.11
378.41 1:9
Matching peak
64.64 +: 3.98
35.80 0.79
107.43
RTTaa (reduced runtime %) (1)
31.27%
55.16%
71.59% 1.08%
Highway
LunarLander
BipadalWalker
IIF
standard
IIF
standard
IIF
standard
Per-round runtime
Influence cale
0.13 a 0.00
= um
0.12 um
Optimization
2.39
3.29 +
1.85
2.05
3.09 0.20
3.30 023
Total runtime
All rounds
214.41 0:22
233.66 0.36
318.68 1.29
328.79
676.78 4:431
691.28 1333
Matching peak
93.73 18
183.64 1: 6.60
489.55 a: 4.71
RTyen (reduced runtime %) (f)
59.89% # 0.72%
44.11% + 2,99
29.16% # 0.66%
B.10 Difficulty based heuristic
Inspired by the difficulty- filtering (e.g., pass@k primarily used improve LLM Reasoning (RLVR) in GRPO (Yu et al., 2025, Bae et al., 2025], we develop difficulty-based filtering approach for PPO. Concretely, we use reward as a proxy for difficulty and filter records receiving top and bottom rewards. However, this heuristic performs worse than random because it systematically removes data with both highest and lowest influence scores, thereby harming the learning process. This finding aligns with our results in Appendix B.5 for traditional RI., where an analogous heuristic using TD error as a proxy for difficulty also proved ineffective. Therefore, our evidence shows that while valid for GRPO, difficulty-bass based filtering is ineffective heuristic for PPO.
32

---

B.11 Comparing two target functions for RLHF
In the main text (Sec. 5.3), we introduced target functions for RLHF: the standard one fremm, and an adapted sequence-level objective f Here we show the comparison of the two in Fig. 16.
IIF (FeR)
IIF (frmum))
Standard
Random
(a) Training reward (f)
(b) Test toxicity (L)
0.15
4.0
Retumn
3.5
3.0
0.05
20
40
100
20
80
100
Round
Round
Figure 16: Comparing two target functions gee with freturn for RLHF. Results are averaged over random seeds.
Overall, from both the training and testing curves, IIF with fan clearly outperforms the others. Although IIF with fretur initiallly improves faster than standard PPO, it plateaus, eventually converging to the same levels as the standard PPO baseline. This highlights that, the adapted sequence- level objective is more effective in RLHF's trajectory-centrie setting with dual reward signals.
B.12 A breakdown of runtime for the RLHF experiments
Table breaks down the wall-clock time (in seconds) for each component of one RLHF training round, under standard PPO and our IIF. The overhead of influence calculation in IIF is significantly offset by reduced optimization time, leading 2x speedup per round.
Beyond this per-round saving, IIF requires fewer rounds to achieve comparable performance with standard PPO (requiring 32.75% + of training rounds, taking up 16.82% 1.32% of runtime combined with per-round speedup). Furthermore, IIF reaches convergence to a higher reward faster as well (requiring 48.51% % + 2.44% of training rounds, taking up 24.90% 0.80% of wall-clock time). This marks a 4x overall speedup plus performance improvement compared to standard PPO.
Table 6: Per-round runtime (in seconds) for RLHF with IIF vs. standard PPO. IIF halves optimization time by pruning 50% of the data each round, while the overhead of influence calculation is negligible. Reported results are averaged over all 109 training rounds in 3 training runs (using 3 random seeds). A dash (-) indicates that measure is not applicable.
IIF
Standard PPO
%
Response generation & scoring
1,71 +
1.59 +
Forward
1.03 + 0.04
0.99 + 0.00
Influence calculation
2.15 + 40.39 + 0.35
Optimization
85.56 + 0.17
Total per-round runtime
45.28 +: 0.47
88.15 4 0.22
51.37%
C Compute resources
All experiments were conducted two Linux servers:
Machine 1: Dual Intel Xeon Silver 4314 CPUs (16 cores/socket, 64 threads total), 251 GiB
RAM, 4 NVIDIA RTX A6000 GPUs (48 GiB VRAM each).
Machine 2: Dual AMD EPYC 7J13 CPUs (64 cores/socket, threads total), TiB RAM, GPUs (80 GiB VRAM each).
NVVIDI A
33

---

For experiments on standard RL benchmarks, we use both Machine and 2; for experiments on RLHF, we use Machine only.
All runtime results reported in Appendix B.9 were measured on Machine 1; all runtime results in Appendix 1.12 were measured on Machine 2.
34


Reviews:
[{"review_id": "i2m0H0SVCs", "reviewer": "Reviewer_AVXm", "summary": "This paper studies computing data attribution in online RL: essentially finding \"importance/influence\" of a datapoint to the overall performance in online RL. The task of data attribution has been studied in offline learning tasks while this paper extends that to online RL. The main challenge is that in online RL there is dependency between data and policy/model which breaks assumptions in previous data attribution tasks. The solution that this paper proposes is simple: it studies PPO and simply performs data attribution in each batch of learning, viewing the entire online learning as an iteration of offline learning tasks.\n\nThe paper uses the TracIn algorithm (Pruthi et al., 2020), which measures the difference in a target function using first-order Taylor approximation and using gradient updates of a loss function (Line 94). The results are quite broad: firstly, results are presented on studying which examples have a positive and negative influence on the target function. It then proposes a filtering approach where negative influence (bottom records) are removed, and this leads to better RL performance. Finally, results are presented on a LLM task of detoxifying LLMs. \n\nOverall, I liked this paper, and while it would have been great to have more realistic LLM experiments and more ablations, I am advocating for acceptance.", "questions": "- Can authors compare how the proposed filtering will compare with other approaches such as filtering prompts/inputs where the pass@k performance is too high (too hard) or too low (too easy)?\n\n- How can learning influence of different inputs be useful for a person doing RLHF for an LLM task, which is a timely application? A discussion on this can be useful. Authors have already mentioned how they can filter out bad records, but is there any other use?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- Important problem of detecting data attribution in online RL\n- Simple proposal directly extending TracIn (simplicity is better than unnecessary novelty and complexity!)\n- Broad range of results from analysis, to proposing algorithms that use it to generate better results, and results on timely LLM experiments\n\n**Weakness**\n\n- On the improvement side, it is not fully clear whether the gains in filtering can come due to a reason other than what is proposed. E.g., some RLHF proposals filter out \"easy prompts\" and this leads to improvement in RL gains. The paper compares with some other filtering approaches but these are in the Appendix (C.4 and C.5) and should be discussed more in the paper.\n\n- On the analysis side, it is not clear what the impact of finding bottom records will be besides filtering them in the algorithm.", "was_revised": true, "final_justification": "Reviewers have provided helpful clarification. I will keep my score.", "rebuttal": "Thanks for the positive review and insightful questions! We are grateful that the reviewer appreciates the contributions of our work, and address their comments in what follows.\n\n> **Explanation of the performance gain**\n\nFiltering based on data attribution scores allows us to remove samples with harmful learning signals. As summarized by Reviewer 6kkY: *“The gradient direction of the value/policy function is the direction for improvements. If a sample loss function gradient doesn't align well with the above direction, this sample will hinder optimization… Thus, it should be removed.”* This principle is powerful because attribution scores are directly tied to the target function we aim to optimize (e.g., expected return), leading to a highly targeted filtering process.\n\nBeyond this general insight, our work develops a **RL-specific** interpretation of this effect. In particular, Sec 4.1 shows that the bottom records typically feature **inaccurate advantage estimates**. Removing them thus purifies the learning signal. This gives a more concrete, though partial (as noted in Appendix C.4) explanation to why filtering in RL is helpful. \n\n\n> **Difficulty-based filtering**\n\nThanks for this insightful point. To the best of our knowledge, difficulty-based filtering (e.g., pass@k) is primarily used to improve LLM Reasoning (RLVR), with GRPO being the most commonly adopted RL algorithm. In GRPO, advantages are normalized across multiple responses for the same prompt. Consequently, for an extremely easy or hard prompt where all responses receive similar rewards, the resulting advantage estimates are near-zero, providing little learning signal.\n\nHowever, this logic does not apply to PPO. We demonstrate this with **a new experiment in RLHF** where we used *reward as a proxy for difficulty* and filtered records receiving top and bottom rewards. This heuristic performed worse than random because it systematically removed data with *both* highest and lowest influence scores, thereby harming the learning process. This finding aligns with our results in Appendix C.5 for traditional RL, where an analogous heuristic using *TD error as a proxy for difficulty* also proved ineffective. Therefore, our evidence shows that while valid for GRPO, difficulty-based filtering is an ineffective heuristic for PPO. We will add this discussion to the revision.\n\nExtending our framework to RLVR with GRPO is a promising direction that we are actively working on. However, due to the many technical challenges involved, we believe this is best suitable for a dedicated follow-up work.\n\n> **More applications of our framework**\n\nThanks for this thoughtful question. In our paper, we have demonstrated that IIF achieves a significant improvement in training efficiency (4x reduction in wall-clock time) and final reward, which we believe is of sufficient interest to RLHF practitioners. Additionally, we believe our framework holds the potential in a few other applications:\n\n1. **Debugging and improving the reward model**: A core challenge in RLHF is that the reward model is often imperfect. Our framework can identify training records that are “harmful” to the RLHF objective and trace this back to the flaws in the reward model—e.g., over-rewarding sycophantic or verbose but unhelpful responses. Practitioners can then use this set of records to debug and recalibrate the reward model, creating a cycle of iterative improvement.\n\n2. **Active prompt curation for efficient alignment**: The efficiency of RLHF depends heavily on the quality of prompts used for generation. Our framework can be used to identify the set of prompts that provide the most value for policy improvement at any given stage of training. This would allow practitioners to strategically curate or up-sample the most informative prompts, leading to more targeted and efficient alignment in subsequent training.\n\n> **Final note**\n\nWe are happy to engage in any follow-up questions. Thanks again for your time and effort in improving our work!"}, {"review_id": "Tvhk9e4QkG", "reviewer": "Reviewer_6kkY", "summary": "This paper proposes a local attribution framework that explains how each data sample in the replay buffer affects the learning dynamics of the target functions (reward to go, policy) based on TraceIn. This local attribution method solves the loopy data dependency issue specifically pertaining to RL. The paper also proposes a simple filtering algorithm based on the attribution method that greatly enhances RL training performance.", "questions": "1. Can one swap TracIn with other attribution algorithms? Will the same framework still apply?\n2. Is theta the whole set of network parameter? If so, could you also analyze the per layer gradient? If only one layer's gradient suffices for attribution, it can greatly reduce the computation/storage cost.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1. Well written and easy to follow.\n2. The proposed method is well motivated and justified by both formal explanations and experiments. The TraceIn method, though not proposed by the Author, is adapted well to the RL context. Intuitively, the gradient direction of the value/policy function is the direction for improvements. If a sample loss function gradient doesn't align well with the above direction, this sample will certainly hinders the optimization of the value/policy function. Thus, it should be removed.\n3. The experiment results demonstrate great practical potential of the proposed simple training algorithm. The sample efficiency gain demonstrated in those smaller enviornments are significant. Though as I request in below, more enviornments could be helpful for understanding the limit of the proposed method.\n\n**Weakness**\n1. Having more experiments in complex games (like Atari) and robotics environments would be better. For example, in the Pong game, during early stage training, agents may fall into local minima where it freezes on one side of the screen after a few moves. Could you demonstrate that your IIF method can boost sample efficiency in Atari games like this as well? And also demonstrate empirically that those transitions filtered out are reasonable. Similar case study on robotics environment could also be helpful like in hopper, will there be falling state mistakenly valued higher during training?", "was_revised": true, "final_justification": "I recommend 5 because the proposed method is easy to understand and use. The paper itself is also joyful to read. As I requested, they also added case studies for more complex environment like Atari and BipedalWalkers. The results make sense to me. I believe the proposed method is indeed useful for debugging and accelerating real world reinforcement learning agents training.", "rebuttal": "Thanks for the positive review and insightful questions! We are grateful that the reviewer appreciates the contributions of our work, and address their comments in what follows.\n\n> **Results and analyses for other games** \n\nFollowing the reviewer’s suggestion, we conducted **new case studies** on two complex environments in the early-to-mid stage of training.\n1. In **Pong** (Atari), we found that bottom records filtered by IIF consist of uninformative transitions (the ball being out of play or already moving away from the agent) that receive (inaccurately) high advantage estimates. By filtering out these samples, IIF achieves significant improvement in training efficiency. \n2. In **BipedalWalker** (Robotics), our analysis revealed bottom records where the agent was incorrectly penalized with a large negative advantage for executing a successful recovery move (e.g., applying corrective torque with a deeply bent knee (~35°) during landing or push‑off).\n\nThese results show that 1) bottom records feature inaccurate advantage estimates; 2) IIF is effective, holding generally across different environments. We will include these new analyses and results in the revision.\n\n> **Extension to other attribution algorithms** \n\nYes, our framework is general and compatible with any gradient-based data attribution algorithm. The three core components of our framework (entity of attribution, target function, attribution method) are modular and can be designed independently. This allows for the straightforward integration of other attribution methods like influence functions [1] or SGD-influence [2].\n\nWe selected TracIn for this study primarily for its conceptual simplicity and empirical efficiency. We believe investigating the interplay between our framework and other attribution algorithms is a promising direction for future work.\n\n> **Gradients of selected layers** \n\nThanks for the thoughtful suggestion. In our work, influence is computed using the gradient with respect to the full set of model parameters.\n\nWhile using gradients from selected layers can reduce computational costs, as explored in prior work [3,4], this approximation can introduce bias to the attribution scores (e.g., see [5]). Our work instead leverages the *ghost dot product* technique from [5]. It enables highly efficient influence calculation using the full gradient (and thus avoiding the bias of layer selection), while adding negligible computational overhead to standard training. A detailed runtime analysis is in Appendix C.7 (Table 3), where we show that the overhead from our full-gradient influence calculation is minimal.\n\n> **Final note**\n\nWe are happy to engage in any follow-up questions. Thanks again for your time and effort in improving our work!\n\n\n**References**\n\n[1] Koh, Pang Wei, and Percy Liang. \"Understanding black-box predictions via influence functions.\" International conference on machine learning. PMLR, 2017.\n\n[2] Hara, Satoshi, Atsushi Nitanda, and Takanori Maehara. \"Data cleansing for models trained with SGD.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[3] Pruthi, Garima, et al. \"Estimating training data influence by tracing gradient descent.\" Advances in Neural Information Processing Systems 33 (2020): 19920-19930.\n\n[4] Yeh, Chih-Kuan, et al. \"First is better than last for language data influence.\" Advances in Neural Information Processing Systems 35 (2022): 32285-32298.\n\n[5] Wang, Jiachen T., et al. \"Data shapley in one training run.\" ICLR 2025."}, {"review_id": "BxG168C6VI", "reviewer": "Reviewer_G83y", "summary": "This paper operates in the on-policy reinforcement learning setting and studies local data attribution. The work proposes a gradient attribution framework that allows to identify harmful instances in a single PPO batch. It is demonstrated what structure these instances have and why they might be harmful. Then, a new algorithm that filters our harmful data points is presented and experimentally evaluated.", "questions": "Q1: When should I use the action or the return target function?  Why are we not using the action function for the algorithm?  \nQ2: Why do we need a target function that differs from the original loss. Why can’t we use the PPO objective?", "limitations": "The work has an explicit limitations section that mentions various concerns. The text is straight-up about the limitation to gradient descent. Overall, I think this section is sufficient.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**Motivation**\nThe problem statement is interesting and quite relevant to recent times.\n\n**Clarity**  \n* The text is well structured and well written. It is easy to follow and provides sufficient detail in the method and experiment section.\n\n**Claims and evidence**  \n* The claims about diagnosing learned functions and behavior formation are sufficiently well supported by sections 5.1 and 5.2.\n\n**Method**  \nThe method seems simple to implement and potentially quite useful.\n\n**Novelty**  \n* I’m not familiar with the attribution line of work but from what I understand reading the text, the TracIn method already captures what the manuscript sets out to do. The main contribution as written in line 155 is the design of the target functions which effectively compute reference losses at the beginning of the round. This seems like a prudent choice but I believe the text here could be simplified.\n\n**Experimental Design and Analyses**\n* The structure of the experiments is well done, going from understanding the identified data points to removing the algorithmically.\n* The experiments cover a sufficiently large set of domain properties.\n* The qualitative experiment is appropriate for the claim of identifying and interpreting bad instances and provides nice insights into some of the learned behaviors.\n* The experiments for return, sample efficiency and runtime efficiency claims are well designed, however they may lack statistical significance (see weaknesses).\n\n__________ \n**Neutral Points**\n\n**Other suggestions**  \nThis work provides a new angle on data selection. It might make sense to reference related work in the RL area that has dealt with played with similar ideas such as experience replay techniques. These could be used as a motivation for this paper as they are generally not easily applicable to PPO.\n__________ \n\n**Weaknesses**\n\n**Clarity**\n* The figures are difficult to understand. I’m not sure what direction to read them from and what they are supposed to convey. They only started making more sense once the framework was described in the text. I was confused when I first saw Figure 1 which is the opposite effect I would hope to gain from a Figure. Similarly, Figure 3 and 4 are difficult to understand, the caption is too brief and does not provide the takeaways from the plot.\n* The description of online RL in section 2.1 is specific to PPO but stated as general online RL. SAC does not store log probabilities or advantage estimates. In general, the paper talks a lot about on-policy RL but calls it online RL. Some of the assumptions that lead to the algorithmic choice are specific to this wording. For instance, off-policy methods are still online but they have a data buffer on which I might be able to compute attribution scores. I believe a clearer distinction would benefit the text.\n* There are a couple of statements that ought to describe the algorithm choices that are not well formulated and might need more support\n  * L111: “Retraining-based methods are too costly and thus inapplicable.\" What makes them costly and why are they inapplicable in RL when this costlyness was not an issue in other domains?\n  * L116:  “Technically ,one can compute influence scores ignoring this extra channel of influence, but these scores would deviate significantly from the ideal effect.” It is not clear to me why that is the case or how we would know this.\n* It is unclear to me why the target function for agent action attribution does not have an expectation and why the validation set issue is only relevant for the return.\n* It is unclear to me when I should use which target function. See Q1.\n**Claims and evidence**  \n* The work claims to provide a principled and flexible framework but is only applied to PPO and objective seems tied to PPO.\n* L137: “unlike supervised learning with a fixed validation set, the data distribution in online RL is inherently policy dependent.”  This issue is still an issue because the reference distribution is per run and differs across algorithm executions. It is not clear to me how this would be remedied by looking at a local reference policy. It is also not clear to me how an action is sampled from a policy in the equation following L142.\n* The text claims that an evolving reference is crucial for meaningful evaluation. There is no evidence provided for this statement. Why can’t I use an offline dataset that is shared across runs on which I compute my estimates?\n\n**Experimental Design and Analyses**\n* 3 random seeds are generally insufficient to draw definite conclusions especially in RL [2, 3].\n* The measure of variance is not reported in Figures 5 and 6.\n\n**Relation To Scientific Literature**  \n* The fact that PPO learns poor advantage values has long been known, the work should probably cite [1].\n* Other work on identifying samples to train on such as prioritized experience replay have long histories and many relevant works that might be useful to cite.\n\n[1] Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? Ilyas et al.\n[2] Deep Reinforcement Learning that Matters. Henderson et al.\n[3] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas et al.\n__________ \n**Summary**\n\nOverall, I think this paper is well written and proposes an interesting and useful idea. The method is simple and follows related work. The experiments demonstrate some benefits but reporting 3 seeds is not sufficient. There are also various points where the text could improve in clarity that I highlighted; especially the distinction between on-policy and online RL should be made clear. That all being said, I will recommend \"accept with minor adjustments for now\". Especially concerns about statistical validity should be remedied for me to recommend acceptance.", "was_revised": true, "final_justification": "I think this paper is an interesting attempt to solve an interesting problem and the community will benefit from knowing about it. After the discussion phase, I believe that my concerns around clarity and experimental evaluation have been addressed, and I recommend acceptance.", "rebuttal": "Thanks for the detailed, thoughtful and constructive feedback! We are grateful that the reviewer appreciates the contributions of our work, and address their comments in what follows.\n\n> **Statistical significance of experimental results** \n\nWe appreciate this critical feedback, and have performed **additional experiments** in three environments to address the concern. Specifically, we follow [1] to use **5 random seeds** and report the mean $\\pm$ standard error. As the rebuttal format is text-only, we present the sample efficiency and runtime results in the table below, following Fig. 6(b). \n\n*Table*: Improvement of IIF compared to standard training in sample efficiency and runtime. Experimental configs are the same as in Fig. 6(b).\n\n|                    | FrozenLake      | MiniGrid       | LunarLander     |\n|:-------------------|----------------:|---------------:|----------------:|\n| *SE*_ave (↑)   | 34.0 ± 2.0      | 65.8 ± 3.3     | 26.0 ± 1.8      |\n| *SE*_peak (↑)  | 19.2 ± 5.9      | 61.7 ± 4.1     | 39.7 ± 3.7      |\n| *RT*_peak (↑)  | 29.5 ± 2.9      | 69.1 ± 1.7     | 44.9 ± 2.5      |\n\nThe results show that IIF provides substantial improvements in sample efficiency and runtime, and these gains are **statistically robust**. We will update all figures and tables in the revision using 5 random seeds. \n\n> **Q1: Role of target functions** \n\nThe two target functions have different use cases. $f^{\\text{action}}$ is mainly for **diagnosis**—understanding why the agent takes a specific action (e.g., abnormal / harmful) at a specific state. On the other hand, $f^{\\text{return}}$ assesses contribution to **overall performance**. This makes it suitable for both analysis (as in Sec. 4.1) and algorithmic policy improvement. We believe it is not suitable to use $f^{\\text{action}}$ to guide RL training because it is not tied to a *universally desirable outcome*. Using it would require an external \"oracle\" to determine which targeted behaviors to reinforce at each specific state, which, however, is the very problem that RL is designed to solve.\n\n> **Q2: Target function $f^{\\text{return}}$**\n\nThanks for this excellent question! Our goal is to measure a sample's influence on the agent's **true objective** (maximizing return), so we use $f^{\\text{return}}$ as a clean and direct target. The PPO loss, in contrast, is a complex surrogate designed for stable training—it combines policy, value, and entropy terms. Attributing influence to this composite loss would be difficult to interpret; it is unclear whether a sample is deemed influential because it improves the policy, refines the value estimate, or simply affects exploration via the entropy term. By decoupling the optimization algorithm from the evaluation objective, our approach allows us to ask a much more direct and meaningful question: *\"Does this sample help the policy achieve higher expected returns?\"*\n\n> **Clarity**\n\n1. **Figs. 1, 3, 4**: We appreciate the comment and will revise the figures and their captions for better readability. \n\n2. **Online RL vs. On-policy RL**: Thanks for highlighting the important distinction. In the revision, we will 1) add a paragraph in Sec. 3 to explicitly state that the core challenge we address—the circular dependency between data and model—is a fundamental characteristic of the entire *online RL* paradigm, while the solution we present and evaluation we perform is specific to PPO, an on-policy algorithm; 2) discuss how our framework can generalize to other on-policy and off-policy RL algorithms (see “Generalizability / extensions”)\n\n3. **Statements (L111, L116)**:\n\n- L111: Retraining-based methods require training the model once for each of the $N$ records being evaluated. This is computationally expensive in any setting and particularly prohibitive in RL, where a single training run involves a long sequential process that can take hours, making $N$ separate runs infeasible. \n\n- L116: As illustrated in Figure 2, a training record has two influence pathways: direct parameter updates (green arrows) and subsequent data generation (red arrows). Any attribution method that ignores the second channel captures only a partial effect, yielding scores that likely deviate significantly from the true effects. \n\n4. **The action target function**: This target function does not require an expectation because it is designed as a diagnostic tool for a specific behavior of interest (taking an action $a$ at a state $s$). In contrast, $f^{\\text{return}}$ concerns maximizing the expected return over *a distribution of states*, thus requiring a validation dataset.\n\n5. **Role of target functions**: See “Q1: Role of target functions”\n\n6. **Generalizability / extensions**: By design, our framework can generalize to other on-policy and off-policy algorithms with straightforward adaptations. The core principle of measuring influence via gradient similarity is universal. For on-policy methods, this primarily involves replacing the PPO gradient (the $\\mathcal{L}^{\\text{PPO}}$ term under line 156) with the loss gradient of the alternative algorithm. For value-based off-policy algorithms like DQN, we additionally need to change the target function to be the Bellman error. We will add these discussions in the revision. \n\n7. **Clarifications of L137, L142**:\n\n- L137: Using reference distributions that vary across runs is a direct consequence of our **contextual, local** framework. The goal is *not* to identify universally \"good\" data, but rather to answer a more immediate question: \"For the agent *at its current stage of training*, which of its recent experiences were most helpful or harmful for the next training update?\"\n\n- L142: We sample trajectories using the reference policy $\\pi^{\\text{ref}}$, i.e., $a \\sim \\pi^{ref}(s)$. \n\n8. **Benefit of dynamic reference**: This is related to the previous question (L137). Using a dynamic reference is a deliberate choice to measure progress that is contextual to the agent's current state. A fixed, off-distribution reference can be misleading due to distribution mismatch; for instance, a novice agent gains little from a dataset sampled from an expert policy which covers states it never reaches. \n\n    To demonstrate this empirically, we performed **additional experiments** of single-round interventions (as in Sec 4.3, lines 221-223) using a *fixed* reference dataset from a well-trained policy. The results show that our intervention consistently improves the target function ($\\Delta f^{\\text{return}} > 0$) but degrades actual test performance ($\\Delta \\text{return} < 0$) except for the few final rounds. This validates our claim that an evolving, on-policy reference is crucial to ensure effective intervention. \n\n\n> **Experimental design and analyses**\n\n1. **Random seeds**: See “Statistical significance of experimental results”\n\n2. **Measure of variance**: Thanks for the thoughtful comment; we will update the figure captions to improve clarity. The boxplots in Fig. 5 show the full data distribution (median, IQR, and extent) over 3 random seeds. The shaded areas in Fig. 6 represent the standard error, used to visualize confidence and assess the **statistical significance** of the differences between the methods [1].\n\n\n> **Related work** \n\nThanks for pointing out the relevant literature! We would like to note that we have already compared our method with an adaptation of Prioritized Experience Replay [2] in the on-policy setup (see Appendix C.5). In the revision, we will include a more thorough discussion of this line of work, and also discuss [3] in Sec. 4.1.\n\n\n> **Final note**\n\nIf you find our response satisfactory, we kindly ask you to increase your rating to support our work. We are happy to engage in any follow-up questions. Thanks again for your time and effort in improving our work!\n\n\n**References**\n\n[1] Henderson, Peter, et al. \"Deep reinforcement learning that matters.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[2] Schaul, Tom, et al. \"Prioritized experience replay.\" arXiv preprint arXiv:1511.05952 (2015).\n\n[3] Ilyas, Andrew, et al. \"Are deep policy gradient algorithms truly policy gradient algorithms.\" arXiv preprint arXiv:1811.02553 (2018)."}, {"review_id": "sIiNysUriD", "reviewer": "Reviewer_1ZdL", "summary": "The paper adapts a data attribution framework (TracIn) for supervised learning for the on-policy RL algorithm PPO. The authors motivate why traditional attribution methods would fail in the RL setting, namely, because they are designed for static datasets. They adapt TracIn to on-policy RL by i) treating the rollouts collected for one update epoch as a “local” dataset, and (ii) adapting TracIn-style gradient similarity for RL by using action log-probabilities and advantage-weighted action log-probabilities. The paper highlights three use cases for data attribution in RL: diagnosing learning pathologies (wrong advantage estimates), identifying temporal phases of learning, and single-round data filtering. Building on the last use case, the authors propose **Iterative Influence-based Filtering (IIF)**: After every round they drop some records with negative influence and train PPO on a filtered buffer. The experiments show improvements in sample efficiency and wall clock time across a range of RL tasks (discrete/continuous control, dense/sparse rewards), as well as improved toxicity mitigation in RLHF.", "questions": "- Can the authors comment on possible extensions to off-policy algorithms? What could an extension look like?\n- I'm a bit confused about the TracIn calculation, e.g., here \n$\\eta_j \\sum_{i \\in \\mathcal B_j} \\nabla_\\theta f(\\theta_j) \\cdot \\nabla_\\theta \\ell(\\theta_j, z_i)~.$ \nWhy does the first multiplication term not depend on a transition $z_i$?\n- Is the baseline in Figure 6 using Adam? If not, how does your method compare to PPO+Adam? I think this is relevant because it is a limitation of your method and not PPO in general that SGD has to be used.\n- It seems that IIF is most helpful in sparse-reward settings (Minigrid). Can you comment on the mechanism of why this is the case?\n- $f^\\text{action}$ is only used in Section 4.2. Could it also be used to speed up training? If so, how?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, the paper provides a nice extension of a data attribution method to PPO. It is well-motivated and provides a number of insights into PPO. I liked Section 4, in particular, and believe these findings to be relevant to the general RL community. If the authors address some of the mentioned weaknesses and questions satisfactorily (especially the benchmark against PPO+Adam), I am willing to raise my score.\n\n### Strengths\n\n- Well motivated.\n- Interesting insights. I really enjoyed reading Section 4.\n- Diverse set of benchmark environments.\n- The authors provide code, ensuring reproducibility.\n- Good presentation.\n\n### Weaknesses\n\n- There seems to be some confusion between online RL and on-policy RL in the paper, and these two terms seem to be used interchangeably. In my opinion, this is not correct. For example, the description under Figure 1 states that \"Online reinforcement learning operates in alternating rounds of data collection and policy updates\", which is a description of on-policy RL. However, DQN is an off-policy algorithm, but it is also online RL. From my perspective, the authors should replace \"online RL\" with \"on-policy RL\" to avoid confusion.\n- In a similar vein, this is an RL paper, so it could use RL terminology: A record seems to be a transition, so it would be helpful to name it as such.\n- The evaluation uses only a single RL algorithm (PPO), making it difficult to judge how well the framework can be adapted to other (on-policy) algorithms.\n- IIF introduces a new crucial hyperparameter $p$. New hyperparameters in themselves are not a problem as long as an algorithm is reasonably robust. However, $p$ apparently must be tuned individually for each environment, which is suboptimal in practice.", "was_revised": true, "final_justification": "The paper proposes an interesting line of work and provides valuable insights into PPO training. My initial concerns focused on three points: clarity, the optimizer selection, and the exclusive reliance on PPO to validate IIF. The authors’ rebuttal satisfactorily clarified my questions. Their additional experiments showed robust performance gains with a different optimizer choice, so I raised my score accordingly. Although evaluating IIF only with PPO remains a concern, PPO’s widespread adoption makes this limitation by itself insufficient to warrant rejection.", "rebuttal": "Thanks for the detailed, thoughtful and constructive feedback! We are grateful that the reviewer appreciates the contributions of our work, and address their comments in what follows.\n\n> **PPO+Adam**\n\nThanks for raising this important point! We would like to clarify that while our IIF framework is developed for SGD, it can be used to accelerate training when the optimizer is Adam. In particular, we already used Adam in **BipedalWalker** (Fig. 6, last environment) and **RLHF** (Fig. 7) for both standard training and IIF (details in Appendices B.2 and B.3). In these cases, following [1], we train the model with Adam but compute the influence scores **as if the model is trained with SGD**, i.e., using the same formula below line 156. This approach is not perfect (we have explicitly listed it as a limitation & future work in Sec. 6), but it has already achieved substantial gain!\n\nFor the first 5 environments in Fig. 6, we used SGD for both standard and IIF (details in Appendix B.2), since it already performs well in these environments. Following the reviewer’s comment, we conducted **additional experiments** on two of these environments using Adam. As the rebuttal format is text-only, we present the sample efficiency and runtime results in the table below, following Fig. 6(b).\n\n*Table*: Improvement of IIF compared to standard training in sample efficiency and runtime. Optimizer = Adam; learning rate = 3e-4, 1e-3 for MiniGrid and LunarLander, respectively. \n\n|                        | MiniGrid       | LunarLander    |\n|:-----------------------|---------------:|---------------:|\n| *SE*_ave (↑)     | 24.1 ± 1.4     | 46.7 ± 4.5     |\n| *SE*_peak (↑)    | 13.3 ± 3.1     | 62.2 ± 5.0     |\n| *RT*_peak (↑)    | 18.5 ± 1.0     | 65.9 ± 3.2     |\n\nThe results show that IIF delivers a clear and substantial benefit regardless of the choice of optimizers or environments, confirming our framework’s generality. One observation is that IIF gains less with Adam compared to SGD in MiniGrid, whereas the trend is reversed for LunarLander. This is partly because Adam significantly speeds up training compared to SGD in MiniGrid (and thus reduces the room of improvement), but less so in LunarLander. We will include these new results as well as discussions on the optimizers in the revision.\n\n> **RL terminologies** \n\n1. **Online RL vs. On-policy RL**: Thanks for highlighting the important distinction. In the revision, we will 1) add a paragraph in Sec. 3 to explicitly state that the core challenge we address—the circular dependency between data and model—is a fundamental characteristic of the entire *online RL* paradigm, while the solution we present and evaluation we perform is specific to PPO, an on-policy algorithm; 2) discuss how our framework can generalize to other on-policy and off-policy RL algorithms (see “Generalizability / extensions”)\n\n2. **Record vs. transition**: We chose the term “record” because it more accurately reflects the unit of our data attribution method. In our framework, a record is defined as $z_i = (s_i,a_i,r_i,\\log \\pi_i, v_i,\\hat A_i)$, which encompasses not only a raw transition but also computed values like the log probability and the advantage estimate, which are essential for influence calculation. Moreover, “record” more accurately captures the atomic unit in RLHF, which is a (prompt, response) pair. That said, we appreciate the reviewer’s suggestion and will clarify how a “record” is related to a “transition” in the revision.\n\n\n> **Generalizability / extensions**\n\nBy design, our framework can generalize to other on-policy and off-policy algorithms with straightforward adaptations. The core principle of measuring influence via gradient similarity is universal. For on-policy methods, this primarily involves replacing the PPO gradient (the $\\mathcal{L}^{\\text{PPO}}$ term under line 156) with the loss gradient of the alternative algorithm. For value-based off-policy algorithms like DQN, we additionally need to change the target function to be the Bellman error. We will add these discussions in the revision. \n\n\n> **Clarification questions**\n\n1. **Hyperparameter $p$**: We agree with the reviewer that an additional hyperparameter entails additional tuning; however, from our ablations in Appendix C.6, we have demonstrated that *any* level of filtering can improve performance over standard training, and that a value between 12.5% and 50% works well across settings.\n\n2. **TracIn calculation**: We use $f^{\\text{return}}$ to explain the two gradient terms in TracIn. The first is the **target gradient** which does not depend on $z_i$’s ($\\nabla_\\theta f^{\\text{return}}(\\theta_j)$), computed on a validation set and represents the desired direction for policy improvement. The second is the **per-record gradient** ($\\nabla_\\theta \\mathcal{L}^{\\text{PPO}}(\\theta_j, z_i)$), computed for a single training record $z_i$ and captures its learning signal. A positive dot product between these two vectors indicates the record is helpful for policy improvement, whereas a negative score indicates it is harmful.\n\n3. **Role of target functions**: The two target functions have different use cases. $f^{\\text{action}}$ is mainly for **diagnosis**—understanding why the agent takes a specific action (e.g., abnormal / harmful) at a specific state. On the other hand, $f^{\\text{return}}$ assesses contribution to **overall performance**. This makes it suitable for both analysis (as in Sec. 4.1) and algorithmic policy improvement. We believe it is not suitable to use $f^{\\text{action}}$ to guide RL training because it is not tied to a *universally desirable outcome*. Using it would require an external \"oracle\" to determine which targeted behaviors to reinforce at each specific state, which, however, is the very problem that RL is designed to solve.\n\n> **IIF in sparse-reward settings** \n\nOur new results on MiniGrid show IIF's benefit is smaller with Adam, suggesting part of the gain may stem from compensating for SGD's inefficiencies. Nevertheless, we hypothesize that IIF could potentially be more useful in sparse-reward settings. The noisy and misleading advantage estimates common in such environments are precisely what our method identifies as \"harmful records\", making them ideal for filtering via IIF. Investigating the joint effects of environment and optimizer on IIF’s performance is an important direction for future work.\n\n> **Final note**\n\nIf you find our response satisfactory, we kindly ask you to increase your rating to support our work. Thanks again for your time and effort in improving our work!\n\n\n**References**\n\n[1] Wang, Jiachen T., et al. \"Data shapley in one training run.\" ICLR 2025."}]

Author rebuttal:

Meta-review: This paper introduces the first data attribution framework for online reinforcement learning, adapting existing gradient-based methods to reinforcement learning, in particular the PPO algorithm. The work was unanimously praised by reviewers for its novelty, clear motivation, and significant empirical results. The paper is very well-written, and particularly in the context of the relative dearth of diagnostic tools for understanding and resolving learning difficulties in reinforcement learning, is poised to contribute a valuable tool to the RL researcher’s toolbox. The proposed filtering algorithm, IIF, demonstrates substantial improvements in sample efficiency and performance across a diverse range of task. Reviewers expressed concerns regarding statistical robustness, the choice of optimizer, and comparisons to other heuristics; however, these concerns were addressed by the authors during a highly productive rebuttal period. The authors are encouraged to include the additional evaluations run during the rebuttal period, in particular the additional random seeds and study of the adam optimizer, in their camera-ready version.
Decision: Accept (oral)