Title: A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: data attribution

online reinforcement learning
Abstract: Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability. Data attribution provides a principled way to trace model behavior back to training samples, yet existing methods assume fixed datasets, which is violated in online RL where each experience both updates the policy and shapes future data collection.
In this paper, we initiate the study of data attribution for online RL, focusing on the widely used Proximal Policy Optimization (PPO) algorithm. We start by establishing a *local* attribution framework, interpreting model checkpoints with respect to the records in the recent training buffer. We design two target functions, capturing agent action and cumulative return respectively, and measure each record's contribution through gradient similarity between its training loss and these targets. We demonstrate the power of this framework through three concrete applications: diagnosis of learning, temporal analysis of behavior formation, and targeted intervention during training. Leveraging this framework, we further propose an algorithm, iterative influence-based filtering (IIF), for online RL training that iteratively performs experience filtering to refine policy updates. Across standard RL benchmarks (classic control, navigation, locomotion) to RLHF for large language models, IIF reduces sample complexity, speeds up training, and achieves higher returns. Together, these results open a new direction for making online RL more interpretable, efficient, and effective.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper adapts a data attribution framework (TracIn) for supervised learning for the on-policy RL algorithm PPO. The authors motivate why traditional attribution methods would fail in the RL setting, namely, because they are designed for static datasets. They adapt TracIn to on-policy RL by i) treating the rollouts collected for one update epoch as a “local” dataset, and (ii) adapting TracIn-style gradient similarity for RL by using action log-probabilities and advantage-weighted action log-probabilities. The paper highlights three use cases for data attribution in RL: diagnosing learning pathologies (wrong advantage estimates), identifying temporal phases of learning, and single-round data filtering. Building on the last use case, the authors propose **Iterative Influence-based Filtering (IIF)**: After every round they drop some records with negative influence and train PPO on a filtered buffer. The experiments show improvements in sample efficiency and wall clock time across a range of RL tasks (discrete/continuous control, dense/sparse rewards), as well as improved toxicity mitigation in RLHF.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper provides a nice extension of a data attribution method to PPO. It is well-motivated and provides a number of insights into PPO. I liked Section 4, in particular, and believe these findings to be relevant to the general RL community. If the authors address some of the mentioned weaknesses and questions satisfactorily (especially the benchmark against PPO+Adam), I am willing to raise my score.\n\n### Strengths\n\n- Well motivated.\n- Interesting insights. I really enjoyed reading Section 4.\n- Diverse set of benchmark environments.\n- The authors provide code, ensuring reproducibility.\n- Good presentation.\n\n### Weaknesses\n\n- There seems to be some confusion between online RL and on-policy RL in the paper, and these two terms seem to be used interchangeably. In my opinion, this is not correct. For example, the description under Figure 1 states that \"Online reinforcement learning operates in alternating rounds of data collection and policy updates\", which is a description of on-policy RL. However, DQN is an off-policy algorithm, but it is also online RL. From my perspective, the authors should replace \"online RL\" with \"on-policy RL\" to avoid confusion.\n- In a similar vein, this is an RL paper, so it could use RL terminology: A record seems to be a transition, so it would be helpful to name it as such.\n- The evaluation uses only a single RL algorithm (PPO), making it difficult to judge how well the framework can be adapted to other (on-policy) algorithms.\n- IIF introduces a new crucial hyperparameter $p$. New hyperparameters in themselves are not a problem as long as an algorithm is reasonably robust. However, $p$ apparently must be tuned individually for each environment, which is suboptimal in practice.\n\nQUESTIONS:\n- Can the authors comment on possible extensions to off-policy algorithms? What could an extension look like?\n- I'm a bit confused about the TracIn calculation, e.g., here \n$\\eta_j \\sum_{i \\in \\mathcal B_j} \\nabla_\\theta f(\\theta_j) \\cdot \\nabla_\\theta \\ell(\\theta_j, z_i)~.$ \nWhy does the first multiplication term not depend on a transition $z_i$?\n- Is the baseline in Figure 6 using Adam? If not, how does your method compare to PPO+Adam? I think this is relevant because it is a limitation of your method and not PPO in general that SGD has to be used.\n- It seems that IIF is most helpful in sparse-reward settings (Minigrid). Can you comment on the mechanism of why this is the case?\n- $f^\\text{action}$ is only used in Section 4.2. Could it also be used to speed up training? If so, how?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper operates in the on-policy reinforcement learning setting and studies local data attribution. The work proposes a gradient attribution framework that allows to identify harmful instances in a single PPO batch. It is demonstrated what structure these instances have and why they might be harmful. Then, a new algorithm that filters our harmful data points is presented and experimentally evaluated.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**Motivation**\nThe problem statement is interesting and quite relevant to recent times.\n\n**Clarity**  \n* The text is well structured and well written. It is easy to follow and provides sufficient detail in the method and experiment section.\n\n**Claims and evidence**  \n* The claims about diagnosing learned functions and behavior formation are sufficiently well supported by sections 5.1 and 5.2.\n\n**Method**  \nThe method seems simple to implement and potentially quite useful.\n\n**Novelty**  \n* I’m not familiar with the attribution line of work but from what I understand reading the text, the TracIn method already captures what the manuscript sets out to do. The main contribution as written in line 155 is the design of the target functions which effectively compute reference losses at the beginning of the round. This seems like a prudent choice but I believe the text here could be simplified.\n\n**Experimental Design and Analyses**\n* The structure of the experiments is well done, going from understanding the identified data points to removing the algorithmically.\n* The experiments cover a sufficiently large set of domain properties.\n* The qualitative experiment is appropriate for the claim of identifying and interpreting bad instances and provides nice insights into some of the learned behaviors.\n* The experiments for return, sample efficiency and runtime efficiency claims are well designed, however they may lack statistical significance (see weaknesses).\n\n__________ \n**Neutral Points**\n\n**Other suggestions**  \nThis work provides a new angle on data selection. It might make sense to reference related work in the RL area that has dealt with played with similar ideas such as experience replay techniques. These could be used as a motivation for this paper as they are generally not easily applicable to PPO.\n__________ \n\n**Weaknesses**\n\n**Clarity**\n* The figures are difficult to understand. I’m not sure what direction to read them from and what they are supposed to convey. They only started making more sense once the framework was described in the text. I was confused when I first saw Figure 1 which is the opposite effect I would hope to gain from a Figure. Similarly, Figure 3 and 4 are difficult to understand, the caption is too brief and does not provide the takeaways from the plot.\n* The description of online RL in section 2.1 is specific to PPO but stated as general online RL. SAC does not store log probabilities or advantage estimates. In general, the paper talks a lot about on-policy RL but calls it online RL. Some of the assumptions that lead to the algorithmic choice are specific to this wording. For instance, off-policy methods are still online but they have a data buffer on which I might be able to compute attribution scores. I believe a clearer distinction would benefit the text.\n* There are a couple of statements that ought to describe the algorithm choices that are not well formulated and might need more support\n  * L111: “Retraining-based methods are too costly and thus inapplicable.\" What makes them costly and why are they inapplicable in RL when this costlyness was not an issue in other domains?\n  * L116:  “Technically ,one can compute influence scores ignoring this extra channel of influence, but these scores would deviate significantly from the ideal effect.” It is not clear to me why that is the case or how we would know this.\n* It is unclear to me why the target function for agent action attribution does not have an expectation and why the validation set issue is only relevant for the return.\n* It is unclear to me when I should use which target function. See Q1.\n**Claims and evidence**  \n* The work claims to provide a principled and flexible framework but is only applied to PPO and objective seems tied to PPO.\n* L137: “unlike supervised learning with a fixed validation set, the data distribution in online RL is inherently policy dependent.”  This issue is still an issue because the reference distribution is per run and differs across algorithm executions. It is not clear to me how this would be remedied by looking at a local reference policy. It is also not clear to me how an action is sampled from a policy in the equation following L142.\n* The text claims that an evolving reference is crucial for meaningful evaluation. There is no evidence provided for this statement. Why can’t I use an offline dataset that is shared across runs on which I compute my estimates?\n\n**Experimental Design and Analyses**\n* 3 random seeds are generally insufficient to draw definite conclusions especially in RL [2, 3].\n* The measure of variance is not reported in Figures 5 and 6.\n\n**Relation To Scientific Literature**  \n* The fact that PPO learns poor advantage values has long been known, the work should probably cite [1].\n* Other work on identifying samples to train on such as prioritized experience replay have long histories and many relevant works that might be useful to cite.\n\n[1] Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? Ilyas et al.\n[2] Deep Reinforcement Learning that Matters. Henderson et al.\n[3] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas et al.\n__________ \n**Summary**\n\nOverall, I think this paper is well written and proposes an interesting and useful idea. The method is simple and follows related work. The experiments demonstrate some benefits but reporting 3 seeds is not sufficient. There are also various points where the text could improve in clarity that I highlighted; especially the distinction between on-policy and online RL should be made clear. That all being said, I will recommend \"accept with minor adjustments for now\". Especially concerns about statistical validity should be remedied for me to recommend acceptance.\n\nQUESTIONS:\nQ1: When should I use the action or the return target function?  Why are we not using the action function for the algorithm?  \nQ2: Why do we need a target function that differs from the original loss. Why can’t we use the PPO objective?\n\nLIMITATIONS:\nThe work has an explicit limitations section that mentions various concerns. The text is straight-up about the limitation to gradient descent. Overall, I think this section is sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a local attribution framework that explains how each data sample in the replay buffer affects the learning dynamics of the target functions (reward to go, policy) based on TraceIn. This local attribution method solves the loopy data dependency issue specifically pertaining to RL. The paper also proposes a simple filtering algorithm based on the attribution method that greatly enhances RL training performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. Well written and easy to follow.\n2. The proposed method is well motivated and justified by both formal explanations and experiments. The TraceIn method, though not proposed by the Author, is adapted well to the RL context. Intuitively, the gradient direction of the value/policy function is the direction for improvements. If a sample loss function gradient doesn't align well with the above direction, this sample will certainly hinders the optimization of the value/policy function. Thus, it should be removed.\n3. The experiment results demonstrate great practical potential of the proposed simple training algorithm. The sample efficiency gain demonstrated in those smaller enviornments are significant. Though as I request in below, more enviornments could be helpful for understanding the limit of the proposed method.\n\n**Weakness**\n1. Having more experiments in complex games (like Atari) and robotics environments would be better. For example, in the Pong game, during early stage training, agents may fall into local minima where it freezes on one side of the screen after a few moves. Could you demonstrate that your IIF method can boost sample efficiency in Atari games like this as well? And also demonstrate empirically that those transitions filtered out are reasonable. Similar case study on robotics environment could also be helpful like in hopper, will there be falling state mistakenly valued higher during training?\n\nQUESTIONS:\n1. Can one swap TracIn with other attribution algorithms? Will the same framework still apply?\n2. Is theta the whole set of network parameter? If so, could you also analyze the per layer gradient? If only one layer's gradient suffices for attribution, it can greatly reduce the computation/storage cost.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies computing data attribution in online RL: essentially finding \"importance/influence\" of a datapoint to the overall performance in online RL. The task of data attribution has been studied in offline learning tasks while this paper extends that to online RL. The main challenge is that in online RL there is dependency between data and policy/model which breaks assumptions in previous data attribution tasks. The solution that this paper proposes is simple: it studies PPO and simply performs data attribution in each batch of learning, viewing the entire online learning as an iteration of offline learning tasks.\n\nThe paper uses the TracIn algorithm (Pruthi et al., 2020), which measures the difference in a target function using first-order Taylor approximation and using gradient updates of a loss function (Line 94). The results are quite broad: firstly, results are presented on studying which examples have a positive and negative influence on the target function. It then proposes a filtering approach where negative influence (bottom records) are removed, and this leads to better RL performance. Finally, results are presented on a LLM task of detoxifying LLMs. \n\nOverall, I liked this paper, and while it would have been great to have more realistic LLM experiments and more ablations, I am advocating for acceptance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Important problem of detecting data attribution in online RL\n- Simple proposal directly extending TracIn (simplicity is better than unnecessary novelty and complexity!)\n- Broad range of results from analysis, to proposing algorithms that use it to generate better results, and results on timely LLM experiments\n\n**Weakness**\n\n- On the improvement side, it is not fully clear whether the gains in filtering can come due to a reason other than what is proposed. E.g., some RLHF proposals filter out \"easy prompts\" and this leads to improvement in RL gains. The paper compares with some other filtering approaches but these are in the Appendix (C.4 and C.5) and should be discussed more in the paper.\n\n- On the analysis side, it is not clear what the impact of finding bottom records will be besides filtering them in the algorithm.\n\nQUESTIONS:\n- Can authors compare how the proposed filtering will compare with other approaches such as filtering prompts/inputs where the pass@k performance is too high (too hard) or too low (too easy)?\n\n- How can learning influence of different inputs be useful for a person doing RLHF for an LLM task, which is a timely application? A discussion on this can be useful. Authors have already mentioned how they can filter out bad records, but is there any other use?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}