Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences

## Abstract

Abstract In reinforcement learning (RL) with experience replay, experiences stored in a replay buffer influence the RL agent’s performance.
Information about the influence of these experiences is valuable for various purposes, such as identifying experiences that negatively influence poorly performing RL agents.
One method for estimating the influence of experiences is the leave-one-out (LOO) method.
However, this method is usually computationally prohibitive.
In this paper, we present Policy Iteration with Turn-over Dropout (PIToD), which efficiently estimates the influence of experiences.
We evaluate how accurately PIToD estimates the influence of experiences and its efficiency compared to LOO.
We then apply PIToD to amend poorly performing RL agents, i.e., we use PIToD to estimate negatively influential experiences for the RL agents and to delete the influence of these experiences.
We show that RL agents’ performance is significantly improved via amendments with PIToD.

## 1 Introduction

In reinforcement learning (RL) with experience replay, the performance of an RL agent is influenced by its experiences.
Experience replay (Lin, ) is a data-generation mechanism indispensable in modern off-policy RL methods (Mnih et al., ; Hessel et al., ; Haarnoja et al., ; Kumar et al., ).
It replays past experiences for an RL agent during training.
These experiences influence the performance (e.g., cumulative rewards) of an RL agent (Fedus et al., ).
This observation naturally leads us to pose a critical research question:
What if we could quantify the influence of the experiences on the RL agent’s performance?
Such information could be useful for many purposes.
For example, we could improve the RL agent’s performance by deleting negatively influential experiences.

However, estimating the influence of experiences with feasible computational cost is not trivial.
One might consider estimating it by the leave-one-out (LOO) method (left part of Figure [1](#S1.F1)), in which we retrain an RL agent for each possible experience deletion.
As we will discuss in Section [3](#S3), this method has quadratic time complexity and quickly becomes intractable due to retraining.

In this paper, we present PIToD, a policy iteration (PI) method that efficiently estimates the influence of experiences (right part of Figure [1](#S1.F1)).
PI is a method that covers a wide range of RL methods (Section [2](#S2)).
PIToD is PI augmented with turn-over dropout (ToD) (Kobayashi et al., ) to enable it to efficiently estimate the influence of experiences without retraining an RL agent (Section [4](#S4)).
We evaluate how accurately PIToD estimates the influence of experiences and its efficiency compared to the LOO method (Section [5](#S5)).
We then apply PIToD to amend poorly performing RL agents by deleting negatively influential experiences (Section [6](#S6)).
To the best of our knowledge, our work is the first to: (i) estimate the influence of experiences on the performance of RL agents within a feasible computational cost, and (ii) modify RL agents’ performance simply by deleting influential experiences.

Figure: Figure 1: Leave-one-out (LOO) influence estimation method (left part) and our method (right part). LOO estimates the influence of experiences by retraining an RL agent for each experience deletion. In contrast, our method estimates the influence of experiences without retraining.
Refer to caption: /html/2405.14629/assets/x1.png

## 2 Preliminaries

In Section [4](#S4), we will introduce our PI method for estimating the influence of experiences in the RL problem.
As preliminaries for this, we explain the RL problem, PI, and influence estimation.

Reinforcement learning (RL).
RL addresses the problem of an agent learning to act in an environment.
The environment provides the agent with a state $s$. The agent responds by selecting an action $a$, and then the environment provides a reward $r$ and the next state $s^{\prime}$.
This interaction between the agent and environment continues until the state reaches termination.
The agent aims to find a policy $\pi:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}_{+}$ maximizing cumulative rewards (return).
Q-function $Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is generally used to estimate the return.

Policy iteration (PI).
PI is a method for solving RL problems.
In PI, the policy and Q-functions are updated by iteratively performing policy evaluation and improvement.
Many implementations of policy improvement and evaluation have been proposed (e.g., Lillicrap et al. (); Fujimoto et al. (); Haarnoja et al. ()).
In the main content of this paper, for simplicity, we base our discussion on the policy evaluation and iteration of the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., ).
In policy evaluation of DDPG, the Q-function $Q_{\phi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, parameterised by $\phi$, is updated as:

$$ $\phi\leftarrow\phi-\nabla_{\phi}\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{B},~{}a^{\prime}\sim\pi_{\theta}(\cdot|s^{\prime})}\left[\left(r+\gamma Q_{\bar{\phi}}(s^{\prime},a^{\prime})-Q_{\phi}(s,a)\right)^{2}\right],$ (1) $$

where $\mathcal{B}$ is a replay buffer containing the collected experiences, and $Q_{\bar{\phi}}$ is a target Q-function.
In policy improvement of DDPG, policy $\pi_{\theta}$, parameterised by $\theta$, is updated as:

$$ $\theta\leftarrow\theta+\nabla_{\theta}\mathbb{E}_{s\sim\mathcal{B},~{}a_{\theta}\sim\pi_{\theta}(\cdot|s)}\left[Q_{\phi}(s,a_{\theta})\right].$ (2) $$

Estimating influence of experiences.
Given the policy and Q-functions updated by PI, we aim to estimate the influence of experiences on performance.
Formally, letting $e_{i}$ be the $i$-th experience contained in the replay buffer $\mathcal{B}$, we evaluate the influence of $e_{i}$ as

$$ $L\left(Q_{\phi,\mathcal{B}\backslash{\{e_{i}\}}},\pi_{\theta,\mathcal{B}\backslash{\{e_{i}\}}}\right)-L\left(Q_{\phi,\mathcal{B}},\pi_{\theta,\mathcal{B}}\right),$ (3) $$

where $L$ is a metric for evaluating the performance of the Q-function and policy,
$Q_{\phi,\mathcal{B}}$ and $\pi_{\theta,\mathcal{B}}$ are the Q-function and policy for which the parameters are updated with all experiences contained in $\mathcal{B}$,
and $Q_{\phi,\mathcal{B}\backslash{\{e_{i}\}}}$ and $\pi_{\theta,\mathcal{B}\backslash{\{e_{i}\}}}$ are the ones for which the parameters are updated with $\mathcal{B}$ other than $e_{i}$.
$L$ is defined to align with our specific interests.
In this paper, we define $L$ in terms of policy and Q-function loss for the experiments described in Section [5](#S5), and in terms of empirical return and Q-estimation bias for the applications detailed in Section [6](#S6).

## 3 Leave-one-out (LOO) influence estimation

What method could be used for estimating the influence of experiences?
One straightforward method is using the LOO algorithm (Algorithm [1](#alg1)).
This algorithm estimates the influence of experiences by retraining the RL agent’s components (i.e., policy and Q-functions) for each possible deletion of an experience.
Specifically, it retrains the policy $\pi_{\theta^{\prime}}$ and Q-function $Q_{\phi^{\prime}}$ using $\mathcal{B}\backslash\{e_{i}\}$, via $I$ policy iterations (lines 4–6).
Here, $I$ is equal to the number of policy iterations required for training the original policy $\pi_{\theta}$ and Q-function $Q_{\phi}$.
After retraining the components, the influence of $e_{i}$ is evaluated using Eq. [7](#S3.Ex1) with $\pi_{\theta^{\prime}}$, $Q_{\phi^{\prime}}$ and $\pi_{\theta}$, $Q_{\phi}$ (line 7).

Figure: Algorithm 1 Leave-one-out influence estimation for policy iteration

However, in typical settings, Algorithm [1](#alg1) becomes computationally prohibitive due to retraining.
In typical settings (e.g., Fujimoto et al. (); Haarnoja et al. ()), the size of the buffer $\mathcal{B}$ is small at the beginning of policy iteration and increases by one with each iteration.
Consequently, the size of $\mathcal{B}$ is almost equal to the number of iterations $I$ (i.e., $|\mathcal{B}|\approx I$).
Since Algorithm [1](#alg1) retrains the RL agent’s components via $I$ policy iterations for each $e_{i}$ in $\mathcal{B}$, the total number of policy iterations across the entire algorithm becomes $I^{2}$.
The value of $I$ typically ranges from $10^{3}$ to $10^{6}$ (e.g., Chen et al. (); Haarnoja et al. ()), which makes it hard to complete all policy iterations in a realistic timeframe.

In the next section, we will introduce a method to estimate the influence of experiences without retraining the RL agent’s components.

## 4 Policy iteration with turn-over dropout (PIToD)

In this section, we present Policy Iteration with Turn-over Dropout (PIToD), which estimates the influence of experiences without retraining.
The concept of PIToD is shown in Figure [2](#S4.F2), and an algorithmic description of PIToD is shown in Algorithm [2](#alg2).
Inspired by ToD (Kobayashi et al., ), PIToD uses masks and flipped masks to dropout parameters of the policy and Q-function.
Further details will be elaborated in the following paragraphs.

Figure: Figure 2: Concept of PIToD. PIToD uses mask $\mathbf{m}_{i}$ and flipped mask $\mathbf{w}_{i}$. It applies $\mathbf{m}_{i}$ to policy and Q-function for PI with $e_{i}$. Additionally, it applies $\mathbf{w}_{i}$ to policy and Q-function for estimating the influence of $e_{i}$.
Refer to caption: /html/2405.14629/assets/x2.png

Masks and flipped masks.
PIToD uses mask $\mathbf{m}_{i}$ and flipped mask $\mathbf{w}_{i}$, which are binary vectors uniquely associated with experience $e_{i}$.
The mask $\mathbf{m}_{i}$ consists of elements randomly initialized to either 0 0 or $1$.
$\mathbf{m}_{i}$ is used to dropout the policy and Q-function’s parameters during PI with $e_{i}$.
Additionally, the flipped mask $\mathbf{w}_{i}$ is the negation of $\mathbf{m}_{i}$, i.e., $\mathbf{w}_{i}=\mathbf{1}-\mathbf{m}_{i}$.
$\mathbf{w}_{i}$ is used for the dropout when estimating the influence of $e_{i}$.

Policy iteration with mask (lines 5–6 in Algorithm [2](#alg2)).
PIToD applies $\mathbf{m}_{i}$ to the policy and Q-functions at PI with $e_{i}$.
It executes PI with variants of policy evaluation (Eq. [1](#S2.E1)) and improvement (Eq. [2](#S2.E2)) where masks are applied to policy and Q-function parameters.
The policy evaluation for PIToD is

$$ $\phi\leftarrow\phi-\nabla_{\phi}\mathbb{E}_{e_{i}=(s,a,r,s^{\prime},i)\sim\mathcal{B},~{}a^{\prime}\sim\pi_{\theta,\mathbf{m}_{i}}(\cdot|s^{\prime})}\left[\left(r+\gamma Q_{\bar{\phi},\mathbf{m}_{i}}(s^{\prime},a^{\prime})-Q_{\phi,\mathbf{m}_{i}}(s,a)\right)^{2}\right].$ (4) $$

The policy improvement for PIToD is

$$ $\theta\leftarrow\theta+\nabla_{\theta}\mathbb{E}_{e_{i}=(s,i)\sim\mathcal{B},~{}a_{\theta,\mathbf{m}_{i}}\sim\pi_{\theta,\mathbf{m}_{i}}(\cdot|s)}\left[Q_{\phi,\mathbf{m}_{i}}(s,a_{\theta,\mathbf{m}_{i}})\right].$ (5) $$

Here, $Q_{\phi,\mathbf{m}_{i}}$, and $\pi_{\theta,\mathbf{m}_{i}}$ are the Q-function and policy to which the mask $\mathbf{m}_{i}$ is applied.
$Q_{\phi,\mathbf{m}_{i}}$ and $\pi_{\theta,\mathbf{m}_{i}}$ compute their outputs using the parameters that are not dropped out by $\mathbf{m}_{i}$, for inputs comming from $e_{i}$.
Thus, some readers may suspect that, in Eq. [4](#S4.E4) and Eq. [5](#S4.E5), the parameters that are dropped out by $\mathbf{m}_{i}$, which can be obtained by applying $\mathbf{w}_{i}$, are never influenced by $e_{i}$.
Theoretically, if $Q_{\phi,\mathbf{m}_{i}}$ and $\pi_{\theta,\mathbf{m}_{i}}$ are dominantly influenced by $e_{i}$, the parameters obtained by $\mathbf{w}_{i}$ are provably not influenced by $e_{i}$ (see Appendix [A](#A1) for details).
Based on this theoretical property, we evaluate the influence of $e_{i}$ by applying $\mathbf{w}_{i}$ to policy and Q-functions (see the next paragraph for details).

Figure: Algorithm 2 Policy iteration with turn-over dropout (PIToD)

Estimating the influence of experience with flipped mask (lines 7–8 in Algorithm [2](#alg2)).
PIToD periodically estimates the influence of $e_{i}$ by applying $\mathbf{w}_{i}$ to the policy and Q-function.
Specifically, it estimates the influence of $e_{i}$ (Eq. [3](#S2.E3)) as

$$ $L\left(Q_{\phi,\mathbf{w}_{i}},\pi_{\theta,\mathbf{w}_{i}}\right)-L\left(Q_{\phi},\pi_{\theta}\right),$ (6) $$

where the first term is the performance in the case of deletion of $e_{i}$, and the second term is the performance with all experiences.
$Q_{\phi,\mathbf{w}_{i}}$ and $\pi_{\theta,\mathbf{w}_{i}}$ are the Q-functions and policy with dropout based on $\mathbf{w}_{i}$.
$Q_{\phi}$ and $\pi_{\theta}$ are the Q-function and policy without dropout.
For the second term, if we want to highlight the influence of $e_{i}$ more significantly, the term can be evaluated by alternatively using the masked policy and Q-functions: $L\left(Q_{\phi,\mathbf{m}_{i}},\pi_{\theta,\mathbf{m}_{i}}\right)$.
In our experiments (Sections [5](#S5) and [6](#S6)), we perform influence estimation every $I_{\text{ie}}$ iterations (line 7 in Algorithm [2](#alg2)).
Specifically, $I_{\text{ie}}=5000$ for Section [5](#S5), and $I_{\text{ie}}=50000$ for Section [6](#S6).
These influence estimations by PIToD do not require retraining the policy and Q-function for each case of experience deletion, unlike LOO method.

## 5 Evaluations for PIToD

In the previous section, we introduced PIToD, a method that efficiently estimates the influence of experiences.
In this section, we evaluate how accurately PIToD can learn the influence of experiences (Section [5.1](#S5.SS1)) and how efficiently PIToD can estimate the influence (Section [5.2](#S5.SS2)).

### 5.1 How accurately does PIToD learn the influence of experiences? Evaluations with self-influence

In this section, we evaluate how accurately PIToD can learn the influence of experiences, by focusing on their “self-influence.”
Self-influence is the influence of an experience on the prediction performance based on the experience itself.
We define self-influence on policy evaluation and policy improvement.
The self-influence of an experience $e_{i}:=(s,a,r,s^{\prime},i)$ on policy evaluation is defined as follows.

$$ $\displaystyle L_{pe,i}(Q_{\phi,\mathbf{w}_{i}},\pi_{\theta,\mathbf{w}_{i}})-L_{pe,i}(Q_{\phi,\mathbf{m}_{i}},\pi_{\theta,\mathbf{m}_{i}}),$ (7) $\displaystyle L_{pe,i}(Q,\pi)=\left(r+\gamma Q(s^{\prime},a^{\prime})-Q_{\phi}(s,a)\right)^{2},~{}~{}a^{\prime}\sim\pi(\cdot|s^{\prime}).$ $$

Here, $L_{pe,i}$ represents the temporal difference error based on $e_{i}$ (i.e., the term based on $e_{i}$ in Eq. [4](#S4.E4)).
The self-influence of $e_{i}$ on policy improvement is defined as follows.

$$ $\displaystyle L_{pi,i}(Q_{\phi,\mathbf{w}_{i}},\pi_{\theta,\mathbf{w}_{i}})-L_{pi,i}(Q_{\phi,\mathbf{m}_{i}},\pi_{\theta,\mathbf{m}_{i}}),$ (8) $\displaystyle L_{pi,i}(Q,\pi)=Q(s,a^{\prime}),~{}~{}~{}~{}a^{\prime}\sim\pi(\cdot|s).$ $$

Here, $L_{pe,i}$ represents the Q-estimation based on $e_{i}$ (i.e., the term based on $e_{i}$ in Eq. [5](#S4.E5)).
We evaluate whether PIToD has correctly learned the influence of experiences, based on the signs of the values of Eq. [7](#S5.E7) and Eq. [8](#S5.E8).
$Q_{\phi,\mathbf{m}_{i}}$ and $\pi_{\theta,\mathbf{m}_{i}}$ are learned by PIToD to minimize $L_{pe,i}$ and maximize $L_{pi,i}$, respectively (lines 5 and 6 in Algorithm [2](#alg2)).
In contrast, $Q_{\phi,\mathbf{w}_{i}}$ and $\pi_{\theta,\mathbf{w}_{i}}$ are not.
Thus, if PIToD has correctly learned the influence of experiences, Eq. [7](#S5.E7) should be positive, and Eq. [8](#S5.E8) should be negative.

We periodically evaluate the ratio of experiences for which PIToD has correctly estimated self-influence in the MuJoCo tasks (Todorov et al., ).
The MuJoCo tasks for this evaluation are Hopper, Walker2d, Ant, and Humanoid.
In this evaluation, 5000 policy iterations (i.e., lines 3–6 of Algorithm [2](#alg2)) constitute one epoch, with 125 epochs allocated for Hopper and 300 epochs for the others.
At each epoch, we (i) calculate the self-influence (Eq. [7](#S5.E7) and Eq. [8](#S5.E8)) of experiences stored in the replay buffer and (ii) record the ratio of experiences for which PIToD has correctly estimated self-influence.

Evaluation results (Figure [3](#S5.F3)) show that the ratio of experiences for which the self-influence (Eq. [7](#S5.E7) and Eq. [8](#S5.E8)) is correctly estimated exceeds the chance rate of 0.5.
For self-influence on policy evaluation (Eq. [7](#S5.E7)), the ratio of correctly estimated experiences is higher than 0.9 across all environments.
Furthermore, for self-influence on policy improvement (Eq. [8](#S5.E8)), the ratio of correctly estimated experiences surpasses 0.7 in Hopper, 0.8 in Walker2d and Ant, and 0.9 in Humanoid.
These results suggest that PIToD more accurately learns the influence of experiences than random estimation.

Supplementary analysis.
How are experiences with significant self-influence distributed?
Figure [4](#S5.F4) shows the distribution of self-influence across experiences.
From the figure, we see that in policy evaluation, the self-influence of older experiences (with smaller normalized experience indexes) becomes more significant as the epoch progresses.
This tendency can be seen as a primacy bias (Nikishin et al., ), suggesting that the RL agent is overfitting more significantly to the experiences of the early stages of learning.
Conversely, in the case of policy improvement, we do not see a clear tendency for primacy bias.
These observations may indicate that policy improvement is less prone to causing overfit to older experiences than policy evaluation.

Figure: Figure 3: The ratio of experiences for which PIToD correctly estimated self-influence. The left-hand figure displays this ratio in policy evaluation cases, where a positive self-influence value (i.e., Eq. [7](#S5.E7) $>0$) is correct. The right-hand figure displays the ratio in policy improvement cases, where a negative self-influence value (i.e., Eq. [8](#S5.E8) $<0$) is correct. In both figures, the vertical axis represents the ratio of correctly estimated experiences, and the horizontal axis shows the number of epochs. Notably, in both cases, the ratio of correctly estimated experiences surpasses the chance rate of 0.5.
Refer to caption: /html/2405.14629/assets/x3.png

Figure: (a) Distribution of self-influence on policy evaluation (Eq. [7](#S5.E7)).
Refer to caption: /html/2405.14629/assets/x5.png

### 5.2 How efficiently does PIToD estimate the influence of experiences? Evaluation for computational time

We evaluate the computational time required for influence estimation with PIToD and compare it to the estimated time of LOO.
For PIToD’s computational time, we run the method under the same settings as in the previous section and record its wall-clock time.
For comparison, we also evaluate the estimated time required for influence estimation using LOO (Section [3](#S3)).
To estimate the time for LOO, we record the average time required for one policy iteration with PIToD and multiply this by the total number of policy iterations required for LOO (^1^11The total number of policy iterations for LOO is $I^{2}$, as discussed in Section [3](#S3). However, in the practical implementation of PIToD used in our experiments, we divide the experiences in the buffer into groups of 5000 and estimate the influence of each group (Appendix [B](#A2)). For a fair comparison with this implementation, we use $\frac{I^{2}}{5000}$ instead of $I^{2}$ as the total number of policy iterations for LOO.).

The evaluation results (Figure [5](#S5.F5)) show that PIToD significantly reduces computational time compared to LOO.
The time required for LOO increases quadratically as epochs progress, taking, for example, more than about $4\cdot 10^{7}$ seconds ($\approx 462$ days) up to 300 epochs in Humanoid.
In contrast, the time required for PIToD increases linearly, taking about $1.4\cdot 10^{5}$ seconds ($\approx$ one day) for 300 epochs in Humanoid.

Figure: Figure 5: Wall-clock time required for influence estimation by PIToD and LOO. The solid line represents the time for PIToD, and the dashed line represents the estimated time for LOO. The results show that while the time required for LOO increases quadratically with the number of epochs, it increases linearly for PIToD.
Refer to caption: /html/2405.14629/assets/x13.png

## 6 Application of PIToD: amending policies and Q-functions by deleting negatively influential experiences

In the previous section, we demonstrated that PIToD can accurately and efficiently estimate the influence of experiences.
What scenarios might benefit from this capability?
In this section, we demonstrate how PIToD can be used to amend underperforming policies and Q-functions.

We amend policies and Q-functions by deleting experiences that negatively influence performance (^2^22The algorithmic description of our amendment process is presented in Algorithm [4](#alg4) in Appendix [C](#A3).).
We evaluate the performance of policies and Q-functions on the basis of returns and Q-estimation biases (Fujimoto et al., ; Chen et al., ), respectively.
The influence of an experience $e_{i}$ on the return, $L_{\text{ret}}$, is evaluated as follows:

$$ $\displaystyle L_{\text{ret}}(\pi_{\theta,\mathbf{w}_{i}})-L_{\text{ret}}(\pi_{\theta}),~{}~{}\text{where}~{}L_{\text{ret}}(\pi)=\mathbb{E}_{a_{t}\sim\pi(\cdot|s_{t})}\mathbb{E}\left[\sum_{t=0}\gamma^{t}r(s_{t},a_{t})\right].$ (9) $$

Here, $s_{t}$ is sampled from an environment.
In our setup, $L_{\text{ret}}$ is estimated using Monte Carlo returns collected by rolling out policies $\pi_{\theta,\mathbf{w}_{i}}$ and $\pi_{\theta}$.
The influence of $e_{i}$ on Q-estimation bias, $L_{\text{bias}}$, is evaluated as follows:

$$ $L_{\text{bias}}(Q_{\phi,\mathbf{w}_{i}})-L_{\text{bias}}(Q_{\phi}),\\ ~{}~{}\text{where}~{}L_{\text{bias}}(Q)=\mathbb{E}_{a_{t}\sim\pi_{\theta}(\cdot|s_{t}),a_{t^{\prime}}\sim\pi_{\theta}(\cdot|s_{t^{\prime}})}\mathbb{E}\left[\sum_{t=0}\frac{\left|Q(s_{t},a_{t})-\sum_{t^{\prime}=t}\gamma^{t^{\prime}}r(s_{t^{\prime}},a_{t^{\prime}})\right|}{\left|\sum_{t^{\prime}=t}\gamma^{t^{\prime}}r(s_{t^{\prime}},a_{t^{\prime}})\right|}\right].$ (10) $$

Here, $L_{\text{bias}}$ quantifies the discrepancy between the estimated and true Q-values using their L1 distance.
When $L_{\text{ret}}(\pi_{\theta})$ and $L_{\text{bias}}(Q_{\phi})$ indicate poor performance, we identify and delete the experience $e_{*}$ that has the strongest negative influence on them.
We apply $\mathbf{w}_{*}$, which maximizes Eq. [9](#S6.E9), to the policy to delete $e_{*}$.
Additionally, we apply $\mathbf{w}_{*}$, which minimizes Eq. [10](#S6.E10) to the Q-function to delete $e_{*}$.

The results of the policy and Q-function amendments are shown in Figure [6](#S6.F6) (^3^33To prevent cherry-picking of incidentally high scores, we re-evaluated scores for Figure [6](#S6.F6) instead of reusing those used to determine $\mathbf{w}_{*}$.).
The figure shows that performance significantly enhances through the amendments.
From the policy amendment results (left part of Figure [6](#S6.F6)), it is clear that the performance (returns; $L_{\text{ret}}$) significantly improves, especially in Hopper, Walker, and Ant.
For example, in Hopper, the return before the amendment (the blue dashed line) was approximately 1000, but after the amendment (the blue solid line), it exceeded 3000.
Additionally, from the Q-function amendments results (right part of Figure [6](#S6.F6)), it is apparent that the performance (Q-estimation bias; $L_{\text{bias}}$) is notably improved, particularly in Ant and Humanoid.
For instance, in Humanoid, the estimation bias of the Q-function before the amendment (the red dashed line) was approximately 20 for 200–300 epochs, but after the amendment, it was reduced to about 10 (the red solid line).

Figure: Figure 6: Results of policy amendments (left) and Q-function amendments (right). The solid lines represent the post-amendment performances: return for the policy (left; i.e., $L_{\text{ret}}(\pi_{\theta,\mathbf{w}_{*}})$) and bias for the Q-function (right; i.e., $L_{\text{bias}}(Q_{\phi,\mathbf{w}_{*}})$). The dashed lines show the pre-amendment performances: return (left; i.e., $L_{\text{ret}}(\pi_{\theta})$) and bias (right; i.e., $L_{\text{bias}}(Q_{\phi})$). These figures demonstrate that the amendments significantly improve returns and reduce biases.
Refer to caption: /html/2405.14629/assets/x15.png

## 7 Related work

Influence estimation in supervised learning.
Our research builds upon prior studies that estimate the influence of data within the supervised learning (SL) regime.
In Section [4](#S4), we introduced our method that estimates the influence of data, (i.e., experiences) in RL settings.
Methods that estimate the influence of data have been well-studied in the SL research community.
Typically, these methods require SL loss functions that are differentiable with respect to model parameters (e.g., Koh & Liang (); Yeh et al. (); Hara et al. (); Koh et al. (); Guo et al. (); Chen et al. (); Schioppa et al. ()).
However, these methods are not directly applicable to our RL setting, as the requisite SL loss functions are unavailable.
In contrast, turn-over dropout (ToD) (Kobayashi et al., ) estimates the influence without requiring differentiable SL loss functions.
We have successfully applied ToD in RL settings (Sections [4](#S4), [5](#S5), and [6](#S6)).
For this successful application of ToD, we provided a theoretical justification (see Appendix [A](#A1)) and considered practical implementations (see Appendix [B](#A2)).

Influence estimation in RL.
Previous studies in the RL research community have estimated the influence of the replay buffer (or overall experiences) properties by comparing RL agents.
These properties include the buffer size (Zhang & Sutton, ; Liu & Zou, ; Yarats et al., ), on-policyness (Fedus et al., ; Kumar et al., ; Di-Castro et al., ; Liu et al., ; Nikishin et al., ), non-stationarity (Igl et al., ; Di-Castro et al., ), and optimality (Fu et al., ).
To assess the influence of these properties, studies typically compare the performance of agents trained with varying properties.
However, this agent comparison method necessitates training multiple RL agents under each property, leading to high computational costs as the number of properties increases.
Thus, as with the LOO method (Section [3](#S3)), it becomes computationally infeasible when applied to estimate the influence of many experiences.

Interpretable RL.
Our method (Section [4](#S4)) estimates the influence of experiences, providing interpretability (^4^44According to the taxonomy of Arrieta et al. (), our method can be classified as a post-hoc explainability method.).
Previous studies in the RL community have proposed interpretable methods based on symbolic (or relational) representation (Džeroski et al., ; Yang et al., ; Lyu et al., ; Garnelo et al., ; Andersen & Konidaris, ; Konidaris et al., ), interpretable proxy models (e.g., decision trees) (Degris et al., ; Liu et al., ; Coppens et al., ), and saliency explanation (Zahavy et al., ; Greydanus et al., ; Mott et al., ; Wang et al., ; Anderson et al., ).
Unlike these studies, our study proposes a method to estimate the influence of individual experiences on RL agent performance.
This method helps us, for example, identify influential experiences when RL agents perform poorly, as demonstrated in Section [6](#S6).

## 8 Conclusion and future work

In this paper, we proposed PIToD, a policy iteration (PI) method that efficiently estimates the influence of experiences (Section [4](#S4)).
We demonstrated that PIToD (i) accurately estimates the influence of experiences (Section [5.1](#S5.SS1)), and (ii) significantly reduces the time required for influence estimation compared to the leave-one-out (LOO) method (Section [5.2](#S5.SS2)).
Moreover, we applied PIToD to identify and delete negatively influential experiences, which substantially improved the performance of policies and Q-functions that had not been learned successfully (Section [6](#S6)).

While we believe that PIToD provides a solid foundation for understanding the relationship between experiences and RL agent performance, it also highlights several areas for further work:
Refining implementation decisions for PIToD.
PIToD employs dropout with a high dropout rate, which often leads to instability in learning.
This necessitates meticulous design decisions in the implementation of PIToD.
Although we have considered several design choices to stabilize learning (Appendix [B](#A2)), further refinement may be necessary to enhance PIToD’s practicality.
Exploring broader applications of PIToD.
In this paper, we applied PIToD to amendments of RL agents in a single-task RL setting (Section [6](#S6)).
However, we believe that the potential applications of PIToD extend to various RL settings.
For instance, it could be applied to multi-task RL (Vithayathil Varghese & Mahmoud, ) (including multi-goal RL (Liu et al., ) or meta RL (Beck et al., )), continual RL (Khetarpal et al., ), safe RL (Gu et al., ), or offline RL (Levine et al., ), to identify experiences significantly influencing RL agents and amend them in failure scenarios.
Investigating the broader applicability of PIToD in these settings presents an exciting avenue for future work.

Broader impact. Our work may contribute to the development of advanced autonomous agents. However, there is a potential risk that these agents could be misused for harmful purposes.

## Appendix A Important theoretical property of PIToD

In this section, we theoretically prove the following property of PIToD:
“Given that the policy $\pi_{\theta}$ and the Q-function $Q_{\phi}$ are updated according to Algorithm [2](#alg2), the versions of these functions to which the flipped mask $\mathbf{w}_{i}$ is applied, namely $Q_{\phi,\mathbf{w}_{i}}$ and $\pi_{\theta,\mathbf{w}_{i}}$, are not influenced by the gradients pertaining to the experience $e_{i}$.”
This property is important as it supports the validity of employing the flipped mask $\mathbf{w}_{i}$ to estimate the influence of $e_{i}$ in PIToD.

First, we define the terms used in our theoretical proof:
Experience:
We define an experience $e_{i}$ as $e_{i}=(s,a,r,s^{\prime},i)$, where $s$ is the state, $a$ is the action, $r$ is the reward, $s^{\prime}$ is the next state, and $i$ is a unique identifier.
We also define another experience as $e_{i^{\prime}}$, where $i^{\prime}$ is a unique identifier.
Parameters:
At the $j$-th iteration of Algorithm [2](#alg2) (lines 3–6), we define the parameters of the Q-function and policy that are not dropped by the mask $\mathbf{m}_{i^{\prime}}$ as $\phi_{j,\mathbf{m}_{i^{\prime}}}$ and $\theta_{j,\mathbf{m}_{i^{\prime}}}$, respectively.
Conversely, we define parameters that are dropped by $\mathbf{m}_{i^{\prime}}$ as $\phi_{j,\mathbf{w}_{i^{\prime}}}$ and $\theta_{j,\mathbf{w}_{i^{\prime}}}$.
Policy and Q-function: We define the policy and Q-function, where all parameters except for $\phi_{j,\mathbf{m}_{i^{\prime}}}$ and $\theta_{j,\mathbf{m}_{i^{\prime}}}$ are set to zero (i.e., dropped), as $Q_{\phi_{j,\mathbf{m}_{i^{\prime}}}}$ and $\pi_{\theta_{j,\mathbf{m}_{i^{\prime}}}}$. Similarly, the policy and Q-function, where all parameters except for $\phi_{j,\mathbf{w}_{i^{\prime}}}$ and $\theta_{j,\mathbf{w}_{i^{\prime}}}$ are zero, are defined as $Q_{\phi_{j,\mathbf{w}_{i^{\prime}}}}$ and $\pi_{\theta_{j,\mathbf{w}_{i^{\prime}}}}$.

Next, we introduce two assumptions required for our proof.
The first assumption is for the policy and Q-function with masks.

###### Assumption 1 .

$Q_{\phi_{j,\mathbf{m}_{i^{\prime}}}}$ and $\pi_{\theta_{j,\mathbf{m}_{i^{\prime}}}}$ can be replaced with $Q_{\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}$ and $\pi_{\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}$, whose parameters $\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ and $\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ satisfy the following gradient properties:

The property of $\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ is as follows:

$$ $\displaystyle\nabla_{\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}\left(r+\gamma Q_{\bar{\phi}_{j,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\right)^{2},~{}~{}a^{\prime}\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ $\displaystyle=$ $\displaystyle\nabla_{\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}\left(r+\gamma Q_{\bar{\phi}_{j,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\right)^{2}\cdot\mathbb{I}(i=i^{\prime}),~{}~{}a^{\prime}\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime}).$ $$

Here, $\mathbb{I}$ is an indicator function, which returns $1$ if the specified condition (i.e., $i=i^{\prime}$) is true and 0 otherwise.

The property of $\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ is as follows:

$$ $\displaystyle\nabla_{\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}Q_{\phi_{j+1,\mathbf{m}_{i}}^{\prime}}(s,a),~{}~{}~{}a\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s)$ $\displaystyle=$ $\displaystyle\nabla_{\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}Q_{\phi_{j+1,\mathbf{m}_{i}}^{\prime}}(s,a)\cdot\mathbb{I}(i=i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s).$ $$

Intuitively, this assumption can be interpreted as follows: “$Q_{\phi_{j,\mathbf{m}_{i^{\prime}}}}$ and $\pi_{\theta_{j,\mathbf{m}_{i^{\prime}}}}$ are dominantly influenced by the experience $e_{i^{\prime}}$ (i.e., the influence of the other experiences is negligible).”
This assumption likely holds, for instance, when the network for policy and Q-function is sufficiently large so that the overlap of their subnetworks corresponding to each experience is negligibly small.

The second assumption is for $\phi_{j,\mathbf{w}_{i^{\prime}}}$ and $\theta_{j,\mathbf{w}_{i^{\prime}}}$:

###### Assumption 2 .

For the gradient with respect to $\phi_{j,\mathbf{w}_{i^{\prime}}}$, the following holds:

$$ $\displaystyle\nabla_{\phi_{j,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma Q_{\bar{\phi}_{j,\mathbf{m}_{i}}}(s^{\prime},a^{\prime})-Q_{\phi_{j,\mathbf{m}_{i}}}(s,a)\right)^{2},~{}~{}a^{\prime}\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ (11) $\displaystyle=$ $\displaystyle\nabla_{\phi_{j,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma Q_{\bar{\phi}_{j,\mathbf{m}_{i}}}(s^{\prime},a^{\prime})-Q_{\phi_{j,\mathbf{m}_{i}}}(s,a)\right)^{2}\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}a^{\prime}\sim\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime}).$ $$

For the gradient with respect to $\theta_{j,\mathbf{w}_{i^{\prime}}}$, the following holds:

$$ $\displaystyle\nabla_{\theta_{j,\mathbf{w}_{i^{\prime}}}}Q_{\phi_{j+1,\mathbf{m}_{i}}}(s,a),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s)$ (12) $\displaystyle=$ $\displaystyle\nabla_{\theta_{j,\mathbf{w}_{i^{\prime}}}}Q_{\phi_{j+1,\mathbf{m}_{i}}}(s,a)\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j,\mathbf{m}_{i}}}(\cdot|s).$ $$

Intuitively, this assumption can be interpreted as follows: “When updating parameters using $e_{i}$, the parameters dropped out (i.e., $\phi_{j,\mathbf{w}_{i}}$ and $\theta_{j,\mathbf{w}_{i}}$) are not influenced by the gradient evaluated with $e_{i}$.”

Based on the above assumptions, we will derive the property of PIToD described at the beginning of this section (^5^55“Given that the policy $\pi_{\theta}$ and the Q-function $Q_{\phi}$ are updated according to Algorithm [2](#alg2), the versions of these functions to which the flipped mask $\mathbf{w}_{i}$ is applied, namely $Q_{\phi,\mathbf{w}_{i}}$ and $\pi_{\theta,\mathbf{w}_{i}}$, are not influenced by the gradients pertaining to the experience $e_{i}$.”).
Some readers may think that Assumption [2](#Thmassumption2) corresponds to this property.
However, in addition to assuming Assumption [2](#Thmassumption2), it is necessary to guarantee that the components used to create target signals for Eq. [11](#A1.E11) and Eq. [12](#A1.E12) (i.e., the components highlighted in red below) are also not influenced by $e_{i}$ when $i\neq i^{\prime}$.
Otherwise, $\phi_{j,\mathbf{w}_{i}}$ and $\theta_{j,\mathbf{w}_{i}}$ could be updated using components influenced by $e_{i}$ even when $i\neq i^{\prime}$.

$$ $\displaystyle\nabla_{\phi_{j,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma{\color[rgb]{1,0,0}Q_{\bar{\phi}_{j,\mathbf{m}_{i}}}(s^{\prime},a^{\prime})}-Q_{\phi_{j,\mathbf{m}_{i}}}(s,a)\right)^{2}\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}a^{\prime}\sim{\color[rgb]{1,0,0}\pi_{\theta_{j,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})}.$ $$

$$ $\displaystyle\nabla_{\theta_{j,\mathbf{w}_{i^{\prime}}}}{\color[rgb]{1,0,0}Q_{\phi_{j+1,\mathbf{m}_{i}}}(s,a)}\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j,\mathbf{m}_{i}}}(\cdot|s).$ $$

Thus, we will guarantee based on Assumption [1](#Thmassumption1) that these red-highlighted components are not influenced by $e_{i}$ when $i\neq i^{\prime}$.

Based on Assumption [1](#Thmassumption1), the following theorem holds:

###### Theorem 1 .

Given that, for $j>0$, the parameters $\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ and $\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ are updated similarly to the original parameters $\phi_{j,\mathbf{m}_{i^{\prime}}}$ and $\theta_{j,\mathbf{m}_{i^{\prime}}}$, according to Eq. [4](#S4.E4) and Eq. [5](#S4.E5), the following holds:

$$ $\displaystyle\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ $\displaystyle\leftarrow$ $\displaystyle\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}^{\prime}}(s,a)\right)^{2}\cdot\mathbb{I}(i=i^{\prime}),$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime}).$ $$

$$ $\displaystyle\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}\leftarrow\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\cdot\mathbb{I}(i=i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s).$ $$

###### Proof.

$$ $\displaystyle\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ $\displaystyle\leftarrow$ $\displaystyle\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\nabla_{\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}\sum_{(s,a,r,s^{\prime},i)}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}^{\prime}}(s,a)\right)^{2},$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}$ $\displaystyle\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\phi_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}^{\prime}}(s,a)\right)^{2}\cdot\mathbb{I}(i=i^{\prime}),$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ $$

$$ $\displaystyle\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}$ $\displaystyle\leftarrow$ $\displaystyle\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\nabla_{\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}\sum_{(s,a,r,s^{\prime},i)}Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s)$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}$ $\displaystyle\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\theta_{j-1,\mathbf{m}_{i^{\prime}}}^{\prime}}Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\cdot\mathbb{I}(i=i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s)$ $$

(1) Apply Assumption [1](#Thmassumption1).
∎

This theorem implies that $Q_{\phi_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}$ and $\pi_{\theta_{j,\mathbf{m}_{i^{\prime}}}^{\prime}}$ are influenced dominantly by the experience $e_{i^{\prime}}$ for $j>0$.
Thus, if the red-highlighted components above can be replaced with these components, it can be said that $\phi_{j,\mathbf{w}_{i}}$ and $\theta_{j,\mathbf{w}_{i}}$ are not influenced by gradients depending on $e_{i}$ in both cases of $i=i^{\prime}$ and $i\neq i^{\prime}$.
Below, we will show that such a replacement is doable.

Based on Assumptions [1](#Thmassumption1) and [2](#Thmassumption2), the following theorem holds:

###### Theorem 2 .

For $j>0$, the parameters $\phi_{j,\mathbf{w}_{i^{\prime}}}$ and $\theta_{j,\mathbf{w}_{i^{\prime}}}$ in Algorithm [2](#alg2) are updated as follows:

$$ $\displaystyle\phi_{j,\mathbf{w}_{i^{\prime}}}$ $\displaystyle\leftarrow$ $\displaystyle\phi_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\phi_{j-1,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}}(s,a)\right)^{2}\cdot\mathbb{I}(i\neq i^{\prime}),$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ $$

$$ $\displaystyle\theta_{j,\mathbf{w}_{i^{\prime}}}$ $\displaystyle\leftarrow$ $\displaystyle\theta_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\theta_{j-1,\mathbf{w}_{i^{\prime}}}}Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s)$ $$

###### Proof.

For $\phi_{j,\mathbf{w}_{i^{\prime}}}$,

$$ $\displaystyle\phi_{j,\mathbf{w}_{i^{\prime}}}$ $\displaystyle\leftarrow$ $\displaystyle\phi_{j-1,\mathbf{w}_{i^{\prime}}}-\nabla_{\phi_{j-1,\mathbf{w}_{i^{\prime}}}}\sum_{(s,a,r,s^{\prime},i)}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}}(s,a)\right)^{2},$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s^{\prime})$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}$ $\displaystyle\phi_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\phi_{j-1,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}}(s,a)\right)^{2}\cdot\mathbb{I}(i\neq i^{\prime}),$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s^{\prime})$ $\displaystyle\stackrel{{\scriptstyle(2)}}{{=}}$ $\displaystyle\phi_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\phi_{j-1,\mathbf{w}_{i^{\prime}}}}\left(r+\gamma Q_{\bar{\phi}_{j-1,\mathbf{m}_{i}}^{\prime}}(s^{\prime},a^{\prime})-Q_{\phi_{j-1,\mathbf{m}_{i}}}(s,a)\right)^{2}\cdot\mathbb{I}(i\neq i^{\prime}),$ $\displaystyle~{}~{}a^{\prime}\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}^{\prime}}(\cdot|s^{\prime})$ $$

(1) Apply Assumption [2](#Thmassumption2).
(2) Apply Assumption [1](#Thmassumption1).

Similarly, for $\theta_{j,\mathbf{w}_{i^{\prime}}}$,

$$ $\displaystyle\theta_{j,\mathbf{w}_{i^{\prime}}}$ $\displaystyle\leftarrow$ $\displaystyle\theta_{j-1,\mathbf{w}_{i^{\prime}}}-\nabla_{\theta_{j-1,\mathbf{w}_{i^{\prime}}}}\sum_{(s,a,r,s^{\prime},i)}Q_{\phi_{j,\mathbf{m}_{i}}}(s,a),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s)$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}$ $\displaystyle\theta_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\theta_{j-1,\mathbf{w}_{i^{\prime}}}}Q_{\phi_{j,\mathbf{m}_{i}}}(s,a)\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s)$ $\displaystyle\stackrel{{\scriptstyle(2)}}{{=}}$ $\displaystyle\theta_{j-1,\mathbf{w}_{i^{\prime}}}-\sum_{(s,a,r,s^{\prime},i)}\nabla_{\theta_{j-1,\mathbf{w}_{i^{\prime}}}}Q_{\phi_{j,\mathbf{m}_{i}}^{\prime}}(s,a)\cdot\mathbb{I}(i\neq i^{\prime}),~{}~{}~{}a\sim\pi_{\theta_{j-1,\mathbf{m}_{i}}}(\cdot|s)$ $$

∎

This theorem implies that:

- (i)
When $i=i^{\prime}$, neither $\theta_{j,\mathbf{w}_{i^{\prime}}}$ nor $\phi_{j,\mathbf{w}_{i^{\prime}}}$ are influenced by gradients dependent on experience $e_{i^{\prime}}$.
- (ii)
When $i\neq i^{\prime}$, $\theta_{j,\mathbf{w}_{i^{\prime}}}$ and $\phi_{j,\mathbf{w}_{i^{\prime}}}$ are updated without depending on the components that might be influenced by $e_{i^{\prime}}$.

Thus, we conclude that “$Q_{\phi,\mathbf{w}_{i^{\prime}}}$ and $\pi_{\theta,\mathbf{w}_{i^{\prime}}}$, and consequently $Q_{\phi,\mathbf{w}_{i}}$ and $\pi_{\theta,\mathbf{w}_{i}}$, are not influenced by the gradients related to the experiences $e_{i^{\prime}}$ and $e_{i}$, respectively.”

###### Assumption 1 .

###### Assumption 2 .

###### Theorem 1 .

###### Proof.

###### Theorem 2 .

###### Proof.

## Appendix B Practical implementation of PIToD

In this section, we explain the practical implementation of PIToD.
Specifically, we explain (i) the soft actor-critic (SAC) (Haarnoja et al., ) version of PI with mask, (ii) group mask, and (iii) key implementation decisions for stabilization.
This practical implementation is used in our experiments (Section [5](#S5) and Section [6](#S6)).

SAC version of PI with mask.
The SAC version of PI with masks is presented in Algorithm [3](#alg3).
The mask is applied to the policy and Q-functions during policy evaluation (lines 5–8) and policy improvement (line 9).
For the policy evaluation, it updates two Q-functions $Q_{\phi_{j}}$, where $j\in\{1,2\}$, as:

$$ $\phi_{j}\leftarrow\phi_{j}-\nabla_{\phi_{j}}\mathbb{E}_{e_{i}=(s,a,r,s^{\prime},i)\sim\mathcal{B},~{}a^{\prime}\sim\pi_{\theta,\mathbf{m}_{i}}(\cdot|s^{\prime})}\left[\left(r+\gamma\left(\min_{j^{\prime}=1,2}Q_{\bar{\phi}_{j^{\prime}},\mathbf{m}_{i}}(s^{\prime},a^{\prime})-\alpha\log\pi_{\theta,\mathbf{m}_{i}}(a^{\prime}|s^{\prime})\right)\\ ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}-Q_{\phi_{j},\mathbf{m}_{i}}(s,a)\right)^{2}\right].$ $$

This is a variant of Eq. [1](#S2.E1) that uses clipped double Q-learning with two target Q-functions $Q_{\bar{\phi}_{j^{\prime}},\mathbf{m}_{i}}$ and entropy bonus $\alpha\log\pi_{\theta,\mathbf{m}_{i}}(a^{\prime}|s^{\prime})$.
Additionally, for policy improvement, it updates policy $\pi_{\theta}$ as

$$ $\theta\leftarrow\theta+\nabla_{\theta}\mathbb{E}_{e_{i}=(s,i)\sim\mathcal{B},~{}a_{\theta,\mathbf{m}_{i}},a\sim\pi_{\theta,\mathbf{m}_{i}}(\cdot|s)}\left[\left(\frac{1}{2}\sum_{j=1}^{2}Q_{\phi_{j},\mathbf{m}_{i}}(s,a_{\theta,\mathbf{m}_{i}})-\alpha\log\pi_{\theta,\mathbf{m}_{i}}(a|s)\right)\right].$ (13) $$

This is a variant of Eq. [2](#S2.E2) that uses the entropy bonus.

Figure: Algorithm 3 SAC version of PI with group mask in PIToD

Group Mask.
In our preliminary experiments, we observed that the influence of a single experience on performance was negligibly small.
To examine more substantial influences, we shifted our focus from estimating the influence of each experience to that of grouped experiences.
To estimate the influence of grouped experiences, we organize experiences into groups and assign a mask to each group.
Specifically, we treated every 5000 experiences as a single group.
This grouping process was implemented by assigning a group identifier to each experience, calculated as $i^{\prime\prime}\leftarrow\lfloor i^{\prime}/5000\rfloor$ (line 3 of Algorithm [3](#alg3)).

Key implementation decisions for stabilization.
Our preliminary experiments revealed that directly applying masks and flipped masks to dropout the parameters of the policy and Q-function destabilizes training.
To address this issue, we implemented macro-block dropout and layer normalization (Figure [7](#A2.F7)).
Macro-block dropout.
Instead of applying dropout (with the masks) to individual parameters, we apply dropout at the block level.
Specifically, we group several parameters into a “block” and apply dropout to these blocks.
In our experiment, we use an ensemble of 20 multi-layer perceptrons (MLPs) for the policy and Q-function, and treat each MLP’s parameters as a single block.
This approach significantly improved training stability.
Layer Normalization.
We apply layer normalization (Ba et al., ) followed by each activation (ReLU) layer.
Recent works show layer normalization stabilizes training in a wide range of RL settings (e.g., Hiraoka et al. (); Ball et al. ()).
This further enhanced learning stability in our experiment.

Figure: Figure 7: Network architectures for policy and Q-function. The policy network takes states as inputs and outputs the parameters of the policy distribution (mean and variance for a Gaussian distribution). The Q-function network takes state-action pairs as inputs and outputs Q-estimates. These networks incorporate macro-block dropout and layer normalization. Macro-block dropout. Our architecture utilizes an ensemble of 20 multi-layer perceptrons (MLPs), applying dropout with masks (and flipped masks) to each output of MLPs. Layer normalization. Each MLP applies layer normalization after every activation (ReLU) layer.
Refer to caption: /html/2405.14629/assets/x17.png

To evaluate the effect of our key implementation decisions, we compare four implementations of Algorithm [3](#alg3):

applies vanilla dropout with masks to each parameter of the policy and Q-function. This implementation corresponds to a naive application of ToD (Kobayashi et al., ).

applies layer normalization to the policy and Q-function.

applies macro-block dropout to the policy and Q-function.

applies layer normalization and macro-block dropout to the policy and Q-function.

These implementations are compared based on the empirical returns in test episodes.

The comparison results (Figure [8](#A2.F8)) indicate that the implementation with our key decisions (PIToD+LN+MD) achieves the highest returns in each environment.

Figure: Figure 8: Ablation study results. The vertical axis represents the return, and the horizontal axis represents the number of epochs. In each environment, the implementation with our key decisions (PIToD+LN+MD) achieves the highest returns.
Refer to caption: /html/2405.14629/assets/x18.png

## Appendix C Algorithm for amending policy and Q-function used in Section 6

Figure: Algorithm 4 Amendment of policy and Q-function using influence estimates

## Appendix D Additional experimental results

Figure: (a) Distribution of influence on return (Eq. [9](#S6.E9)).
Refer to caption: /html/2405.14629/assets/x22.png

## Appendix E Computational resources used in experiments

For our experiments in Section [5.2](#S5.SS2), we used a machine equipped with two Intel(R) Xeon(R) CPUs E5-2667 v4 and five NVIDIA Tesla K80 GPUs.
We assigned one GPU to each trial of an experiment.

## Appendix F Hyperparameter settings

The hyperparameter setting for our experiments (Sections [5](#S5) and [6](#S6)) is described in Table [1](#A6.T1).

**Table 1: Hyperparameter settings**
| Parameter | Value |
| --- | --- |
| optimizer | Adam (Kingma & Ba, ) |
| learning rate | $0.0003$ |
| discount rate $\gamma$ | 0.99 |
| target-smoothing coefficient $\rho$ | 0.005 |
| replay buffer size | $2\cdot 10^{6}$ |
| number of hidden layers for all networks | 2 |
| number of hidden units per layer | 128 |
| mini-batch size | 256 |
| random starting data | 5000 |
| replay (update-to-data) ratio | 4 |
| masking (dropout) rate | 0.5 |

## Appendix G Our Source Code

Our source code is available at [https://github.com/TakuyaHiraoka/Which-Experiences-Are-Influential-for-RL-Agents](https://github.com/TakuyaHiraoka/Which-Experiences-Are-Influential-for-RL-Agents)

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: How does sample rejection based on gradient relate to hessian?
soundness: 3
strengths: - This paper focus on experience replay, that is, sampling distribution manipulation, which I think is a under-represented direction in RL research.
- I like the the fact that ToD being applied in the sample consideration, the usage of ToD feels natural and justified for this use case.
summary: This paper aims to study the influence individual experience sample have in training RL agents. The authors used identify masks to differentiate individual samples and observe the difference in the resulting Q values.
weaknesses: - The paper do not have a theoretical underpinning for their approach though some reader may find the idea intuitive, that said, personally, I'm not a fan of removing samples from experience buffers;
- because the way I see it, deleting "negatively influential experiences" seems to be a lenient/hysteresis update commonly seen in optimistic approaches, it may be useful in same case but should be used with caution since it may hinder learning and cause bias; a comparison with similar approaches is not included. I think instead of removing samples, the safer approach is to use gradient clipping or learning rate schedules to stabilize training. Techniques like gradient clipping cap the maximum value of gradients to prevent instability without losing any data --
- Data removal in RL may massively hinder exploration, especially in non-linear environments, speaking of which --
- the paper only evaluate on classic mujoco, and lacks variety of evaluation task, speaking of evaluation --
- The evaluation also lacks other sota methods for comparison, since the promise was to have better performance, rather than better theoretical understanding, which goes back to my first point.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. **[About Weakness 1]** You mentioned that the PI and PE metrics indicate whether a specific experience has a positive or negative effect. Could you clarify this further? The current metrics seem to only reflect whether or not learning utilized $e_i$.
2. **[About Weakness 2]** Is this approach fundamentally different from simply creating a policy ensemble through dropout and selecting the best-performing one? If the truly optimal  $w^*$  is selected, does it necessarily mean that the experience has a negative impact?
3. **[About Weakness 3]** Could you summarize the experiments in the appendix and explain what each aims to demonstrate? It would be more beneficial if these results were integrated into the main paper.
4. **[About Weakness 4]** Could you also present the experiments mentioned above?
soundness: 1
strengths: 1. A novel approach is presented for estimating the influence of specific experiences in RL by using Turn-over Dropout (ToD).
2. It is theoretically demonstrated that the complement mask $w\_i$ for experience $e\_i$ indicates an absence of influence from $e\_i$.
summary: This paper proposes the PIToD method to efficiently estimate which experiences positively or negatively impact RL learning. The proposed approach sets a drop-out mask for each experience and estimates the influence of each experience based on this mask and its complement, allowing for significantly more efficient computation compared to traditional Leave-One-Out (LOO) methods. The authors demonstrate that their method accurately estimates the influence of experiences using various metrics and further show, through experiments, that applying it to SAC improves learning performance.
weaknesses: 1. The metric used to show self-influence appears inappropriate. As noted in L246,  $Q\_{w\_i}$ is in a state where it has not been trained on $e\_i$, so $L\_{pe,i}(Q\_{w\_i}) − L\_{pe,i}(Q\_{m\_i})$ is expected to be greater than zero in most cases, regardless of whether $e\_i$ is beneficial for learning. Additionally, since $\pi\_{m\_i} = \arg\max\_{\pi} L\_{pi,i}(\pi)$, $L\_{pi,i}(\pi\_{w\_i}) − L\_{pi,i}(\pi\_{m\_i})$ is likely always less than zero. Thus, these metrics do not directly indicate whether the experience has a positive or negative influence on RL learning.
2. The evaluation shown in Figure 6 also seems flawed. In Algorithm 4 of Appendix D,  $w^*$ is already defined as  $\arg\max\_w L\_{ret}(\pi, w)$ , so it is unsurprising that high returns are achieved.
3. The main paper contains too few experimental results. While it seems that several experiments were conducted in the appendix, summarizing the purpose and outcomes of these experiments in the main paper would enhance clarity.
4. It would be beneficial to include a direct comparison of mean performance between the original SAC method and the SAC method with PIToD in the main paper.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - For section G, Figure 16 shows very little change in the results when removing adversarial experiences, why is this? Figure 17 shows big differences in the estimations before and after amendments, but it would be good to clarify that your method can identify the adversarial experiences explicitly (as opposed to showing the affect of removing identified experiences which indirectly provides some evidence for this).

- Figure 10 shows huge changes in the results across some architectural choices, please comment more on these. 

(There are also some questions sprinkled throughout the above section)
soundness: 2
strengths: Being able to efficiently estimate the affect a particular data point has on a network used to estimate $\pi$ or $Q$ is a very powerful tool, and to my knowledge this machinery has not been applied to a Deep RL setting before. I like that the authors have aimed to evaluate it for a variety of different purposes. 
However, it seems that only Section G in the appendix, and a short paragraph at the end of section 6 actually attempt to answer the question in the title of the paper.
summary: This paper presents Policy Iteration with Turn-over Dropout as a method for estimating the affect a state-action-reward-next state experience has on a policy or Q function in the area of off-policy RL. The method employs masks to dropout parameters, so that the affect of not training on an experience can be estimated without having to retrain the policy or Q values from scratch. The paper then investigates estimating various quantities including td-error and episode return.
weaknesses: # Major

More explanation in Section 4 of the PIToD method would be very helpful for a reader's understanding. 
"Thus, some readers may suspect that the parameters dropped out by $m_i$ (i.e., the parameters obtained by applying $w_i$) are not influenced by $e_i$." - This sentence caused a lot of confusion to me when reading the paper. The phrasing seems to suggest that the parameters dropped out are indeed affected, but this doesn't seem to be the case (as the reader would suspect). I am also struggling to understand Appendix Section A. Assumption 1 seems extremely strong and appears to be *almost* exactly defining the property you are looking to prove. Looking at the equality, doesn't it mean that the gradient for $i'$ is 0 for everything but $i'$ due to the indicator?

Section C in the appendix contains a lot of interesting content.
The findings from the Group Mask preliminary experiments in the appendix seem quite important - "In our preliminary experiments, we found that the influence of a single experience on performance was negligibly small". This should be mentioned at the very least in the main paper. 
"Key implementation decisions to improve learning."  is also very important. The implementation details, and the architectural experiments you conducted should not be relegated to the appendix in this manner. Especially since Figure 10 shows they are critical to your method working.
Additionally, why are you using an ensemble of 20 MLPs in the architecture in this manner? Is it important to performance? Does it interact positively with your method? More discussion on why these design decisions were made is needed, or at the very least a comment stating it was the first architectural starting point used.

More explanation is needed on Section 5.1 to establish the importance of the ratios being considered. What is the importance/significance of the differences in the ratios between policy eval/improvement? For Eqn 8 I can understand that the td error if higher for an experience you haven't trained on, but for Eqn 9 I do not quite see why the action chosen by $\pi_{\theta, w_i}$ should have a lower estimated Q.

Section 5.2 is very misleading since you're estimating the time it would take LOO. Given that important sections that have been pushed into the appendix, I would advise replacing these with other more relevant/concrete content.  

"In our setup, L_ret is estimated using Monte Carlo returns collected by rolling out policies" - How many rollouts are used for each estimation?
Additionally, how do you utilise Eq. 10 to identify experiences? Are you rolling out the agent multiple times to collect MC returns? Is this factored into your training budget and reflected in Figure 6?

I don't understand why you're not showing Figure 11 in place of Figure 6? What is the particular thing you want to highlight by showing a specially picked subset of the lines in Figure 6?

Much more analysis and results need to be presented on characterising which experiences are harmful (or beneficial). The paper begins this kind of investigation but it feels like an afterthought. To me, this is one of the most exciting parts of the paper, utilising the tools outlined to identify what experiences are harmful for performance (with potential links to 'Ray Interference: a Source of Plateaus in Deep Reinforcement Learning'). This would be of much interest to the community.

# Minor

- No references for RL, and an MDP is never mentioned in the paper.
- No reference for the LOO estimator.
- CQL cited under off-policy RL in the introduction feels unnecessary.
- "In the previous section, we demonstrated that PIToD can accurately and efficiently estimate the influence of experiences." - This is too broad a claim. In section 5 you estimated the influence an experience can have on self-influence.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could I use PIToD and PER together?
- Your algorithm as many loops. Can these ToD iterations be efficiently batched?
- it is unclear why equations 8 and 9 indicate correct influence calculations if they are positive and negative, respectively. Why is this the case?
- why is the "correct experience ratio" of hopper for in policy improvement so much worse than the others, and why does the ratio for walker and ant decrease throughout the epochs? The authors should discuss potential reasons for these differences and what implications they might have for the applicability of PIToD across different environments.
- why is bias mainly an issue with the humanoid environment and not the others? Is this consistent with previous findings in the literature? What specific characteristics does the humanoid environment have that might contribute to this bias issue?
soundness: 3
strengths: - The method is simple, novel, and original.
- Comparison to the leave-one-out naive approach emphasizes the significance of PIToD.
- Diagrams are clear and self-explanatory.
- Results showcase the efficiency advantages of PIToD very well
- the paper is generally well written with a clear narrative and a comfortable flow.
summary: The paper describes a novel method called Policy Iteration with Turn-over Dropout (PIToD)  for excluding experiences that negatively affect the performance of RL agents when used for training via policy iteration. This includes a how to calculate the influence of a single experience on the agent's performance and how to amend the policy given this calculated influence. This is done efficiently through a parameter masking technique called turn-over dropout. The authors provide theoretical justification as to why this masking technique is similar to leaving out a specific experience. PIToD is tested on known four MuJoCo environments and shows improvement in performance for some of the environments while remaining computationally efficient.
weaknesses: - the theoretical justification is lacking because the assumptions (1 and 2) don't seem realistic. Basically, the authors assume that the masks enforce some sort of leave-out rule. It makes sense since they try to minimize overlap (App B), but it is the assumptions are unjustified. The authors should provide empirical evidence supporting assumptions 1 and 2, and discuss the implications if these assumptions don't fully hold in practice.
- missing analysis of results, mostly why the results in figures 3, 4 and 6 look the way they do. The authors should explain why the plots for some environments show lower ratios / performance than other, the reasons for instability in some cases (including shaded confidence intervals). Figure 4 is explained but this explanation is unclear. Specifically, it is unclear why this figure suggests that the agent is overfitting to older experiences.
- figure 4 heatmap scales are all different, making it very difficult to read and compare the graphs. Either use consistent scales for all plots or at least keep colors consistent regardless of the scale, i.e., if 1 is yellow and -1 is blue inn one scale, then in another scale -1 is still blue, and -4 can be a new color, e.g., green.
- does not consider what happens if the buffer is full, something that will eventually happen if training persists. The authors should provide a more intuitive explanation or a simple example that illustrates why the signs of these equations indicate correct influence calculations.
- no comparison to other methods. E.g., PER is definitely comparable.
- redundant mini-paragraphs (start of sections) and parentheses (introduction) that map out the content of the paper are distracting and ruin the flow of reading.

Meta-review: This paper studies how to estimate the influence of a single entry in the experience replay. The naive leave-one-out approach is too expensive and this paper proposes a novel method based on masking the network parameters. That being said, reviewers are concerned about the theoretical justification of the proposed approach. In particular, it is not clear how removing experience would affect exploration of RL algorithms (e.g., Algorithm 3). To better justify the removal of experience, it is necessary to theoretically study how different experiences contribute to exploration and how the removal of experience affects credit assignment. Those effects cannot be characterized by the scalar metric used in the paper. Moreover, to demonstrate the usefulness of the proposed technique, it is worth considering more benchmark environments and more variants of using experience replays.

Decision: Reject