Experience Replay More When It’s a Key Transition in Deep Reinforcement Learning
Year: 2021

Abstract:
We proposed an experience replay mechanism in Deep Reinforcement Learning based on Add Noise to Noise (AN2N), which requires agent to replay more experience containing key state, abbreviated as Experience Replay More (ERM). In the AN2N algorithm, we refer to the states explored more as the key states. We found that how the transitions containing the key state participates in updating the policy and Q networks has a significant impact on the performance improvement of the deep reinforcement learning agent, and the problem of catastrophic forgetting in neural networks is further magnified in the AN2N algorithm. Therefore, We sample the transition used for experience replay according to whether the transition contains key states and whether it is the most recently generated, rather than sampling uniformly, which is the core idea of the ERM algorithm. The experimental results show that this algorithm improves the performance of the agent by a wide margin. We combine the ERM algorithm with Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic policy gradient (TD3) and Soft Actor-Critic (SAC), and evaluate algorithm on the suite of OpenAI gym tasks, SAC with ERM achieves a new state of the art, and DDPG with ERM can even exceed the average performance of SAC under certain random seeds, which is incredible.

heading: 1 Introduction
text: Deep reinforcement learning (RL) has shown its promising future for decision-making in various computer games, such as atari (Mnih et al., 2013; 2015), go (Schrittwieser et al., 2020) and starcraft (Vinyals et al., 2019). However, most successes have been exclusively in simulation due to poor sample efficiency of typical Deep RL algorithm and other challenges. Reinforcement learning can be divided into model-based RL and model-free RL in the light of its data efficiency. Model-free RL is usually subdivided into off-policy RL and on-policy RL. Although model-based RL requires less sampled data, it needs to build a world model and predict the next state based on a lot of prior work, demonstrated in Hafner et al. (2019). Although the on-policy RL algorithm do without establishing a world model and has good stability, the performance improves slowly due to limiting the step of updating the policy, a large number of sampled trajectory data are required in the training process (Schulman et al., 2015; 2017). Off-policy RL is between model-based RL and on-policy RL in terms of sampling efficiency, and the research in this field is enduring (Watkins & Dayan, 1992; Hessel et al., 2018; Barth-Maron et al., 2018) on account of its relatively high data efficiency and world model free. Experience Replay (Lin, 1992) is an important part of improving the data efficiency of Offpolicy RL, which stores experience in a replay buffer and break the temporal correlations by mixing data, therefore, the experience can be used multiple times to update the networks. However, most of the current work is to uniformly sample transitions from the buffer, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015), Soft Actor-Critic (SAC) (Haarnoja et al., 2018), Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018) and many other algorithms (Van Hasselt et al., 2016; Mnih et al., 2016; Andrychowicz et al., 2017; Dabney et al., 2018; Liu et al., 2020). However, these approaches
replay experience transitions at the same frequency, regardless of their meaning. Schaul et al. (2016) develops a framework for prioritizing experience to replay important transitions more frequently, therefore agent learns more efficiently. Prioritized experience replay (PER) method samples transitions with the magnitude of their temporal-difference (TD) error. However, prioritization introduces bias, which needs to be corrected with importance sampling. Meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience (Rothfuss et al., 2018; Mishra et al., 2018), Rakelly et al. (2019) develops an off-policy meta-RL algorithm that disentangles the inference and control of the task, but its performance is still far behind the mainstream off-policy RL algorithms. In this paper, We first analyzed how the state changes from beginning of agent interacting with the environment to the policy converged, and found that the state of the agent is different at different stages, agent will rarely transfer to those states where the policy is far from converging when the agent’s policy already converged. If a large number of initialstates1 are used to train Q-value and policy networks, the network weights that have been learned well will be gradually updated. Therefore, the sampling proportion of the most recently generated transitions should be appropriately increased, so that the agent pays more attention to learning the recent transitions, In this way, the state of the agent is gradually transferred from a poor state to a better state, similar to updating from a poor policy to a better policy stabily in Trust region policy optimization (TRPO) (Schulman et al., 2015) or Proximal policy optimization (PPO) (Schulman et al., 2017). Inspired by the Add Noise to Noise (AN2N) Algorithm (Guo & Gao, 2021), We divide the states into two categories. These states explored with added noise are called key states2, and the rest are called non-key states. For the sake of improving the performance in key states in time, we have increased the probability of sampling new key states, making them more likely to participate in updating policy, we call this process as ERM (Experience Replay More), and combine it with commonly used off-policy RL algorithms in continuous control tasks, such as SAC, which obtained faster convergence and state-of-the-art performance.

heading: 2 Preliminaries
text: We consider a reinforcement learning setup consisting of agent learning policies to maximize the expected reward when interacting with the environment (Sutton & Barto, 2018). At each timestep t, the agent receives an observation ot ∈ O, selects action at ∈ A with respect to its policy π: O → A. After taking the action at in environment E, agent receives a reward rt and the next observation ot+1. The practical problem is usually a partial Markov decision process (POMDP), only part of the observation information could be obtained. To simplify the problem, we assumed the environment is fully-observed, so st = ot,S = O. In reinforcement learning, the action-value function Qπ (s, a) is uesed to approximate the expected sum reward of the ation a in state s, defined as following:
Qπ (s, a) = Est∼pπ,at∼π [ +∞∑ t=0 γtR (st, at) ] (1)
Where γ ∈ [0, 1] is the discount factor, Est∼pπ,at∼π is the expectation over the distribution of the trajectories (s0, a0, s1, a1, . . . ). The mean value of Qπ in the same state s called the value function V π, defined as V π(s) = Ea∼π(·|s) [Qπ(s, a)]. We express the action-value function Qπ in the form of Bellman equation (Bellman & Kalaba, 1965):
Qπ (st, at) = Est+1∼pπ [r(st, at) + γEat+1∼π [Qπ(st+1, at+1)]] (2) 1The initial-state is the state in which agent do not perform well when policy has not converged,
agent almost no longer transferred to this kind of state after policy converged. 2The key state is a state where the agent has not performed well in the past
In this paper, we need to be familiar with DDPG, TD3 and SAC algorithms, here, we mainly introduce DDPG as the basis. DDPG applied two different fully connected neural networks to approximate the action-value function Q(s, a|θQ) and policy function µ(s|θµ), DDPG introduces action-value target network θQ ′ and policy target network θµ ′
, so as to Stable the policy update. Consequently, gradient descent is used to optimize the network weights by minimizing the loss:
L(θQ) = Est∼pµ(st|θµ),at∼µ(st|θµ)[(Q(st, at|θ Q)− yt)2] (3)
Where yt = r(st, at) + γQ ′ (st+1, µ ′ (st+1|θµ ′ )|θQ ′ ) (4)
∇θµJ ≈ Es∼p(st|θµ) [ ∇θµQ(s, a|θQ)|s=st,a=µ(st|θµ) ] = Es∼p(st|θµ) [∇aQ(s, a|θ Q)|s=st,a=µ(st)∇θµµ(st|θ µ)|s = st]
(5)
Where equation 4 derived from equation 2, the weights of target networks are updated periodically to slowly track the learned networks: θ′ ← τθ+(1− τ)θ′ with τ ≪ 1, which alleviates the fluctuation in the agent’s learning process. The policy is updated by equation 5, following the chain rule to the expected sum return Q(s, a|θQ) with respect to parameters θµ. TD3 addresses the problem that DDPG is prone to overestimating the Q function. An additional Q-function is added, and the predicted smaller Q value is used as the calculation of TD-error. At the same time, which also reduces the update frequency of the policy function. The most significant difference between SAC and DDPG is the introduction of policy entropy H(π(·|st)), so the objective function equation 1 is rewritten as:
Qπ (s, a) = Est∼pπ,at∼π [ +∞∑ t=0 γtR (st, at)− α log(π(at|st)) ] (6)
Where α is temperature parameter, which adjusts the optimization target, agent pays more attention to exploration if increase the coefficient α.

heading: 3 Experience Replay More When It’s a Key Transition
text: The addition of experience replay improves the sample efficiency of off-policy RL. Many off-policy algorithms usually sample transitions uniformly from the experience buffer, which
potentially considers the transitions generated at different times to be of the same importance, we will discuss this in section 3.1. Prioritized Experience Replay (PER) is an optimization of the uniform sampling method based on the TD-error value of transitions so as to learn the samples more efficiently, however, PER essentially still does not consider whether the transitions generated at different times are equally important to the current agent’s policy.
3.1 States are Different at Different Stages
Taking the HalfCheetah-v2 simulation environment as an example, we recorded the state information of an agent at different stages based on DDPG algorithm, collecting attitude information at the specific position and the corresponding Q-value starting from 2e4 to 5.6e5, and then we uniformly sample 100 sets of data from the collected data for drawing, as shown in the Fig. 1. The legend in the upper left corner represents the Q value, which is discretized to reasonably reduce the number of Legends. Subgraph (a) is discreted in units of 20, and subgraph b is discreted in units of 100. It can be found when the agent interacts with the environment, not only the policy is gradually improving3, but the state is also changing accordingly. After policy converges and the agent is in a state set different from the previous one, more experience similar to the agent’s recent states should be collected to train the neural network to strengthen the current policy. If the sampling experience for training policy network and the action state network are quite different from the recently generated states, it will not only help improve the policy to a small extent, but will also even forget the newly learned network weights due to the catastrophic forgetting of the neural networks. Therefore, to further improving the sample efficiency, experience replay should be switched from uniform sampling to targeted sampling.
3.2 Replaying More Key Transitions
The AN2N algorithm will record the dilemma state sd during the evaluation process, so when agent interacts with the environment, if the current state sc and sd are considered to be highly similar, an additional exploration noise Na will be added. In this paper, the state sc with additional noise is called the key state sk, and the transition containing sk is called the key transition trank. The agent’s policy is relatively fragile in sk, if agent has not learned how to get out of this dilemma before, agent is more likely to fall into a series of bad
3Better decision-making ability usually gets a higher Q value
states after sk, which greatly reduces the agent’s overall performance. Therefore, whether the agent can learn a good policy in sk is very important for improving the performance. It can be seen from equation 2 that reward of the key state sk will affect the Q value of the state at the previous moment through the iteration of the dynamic equation, nevertheless, if we uniformly sample experience for training the networks, there will be the following three problems:
• Since most of the experience generated by AN2N are non-key states, the probability of sampling key states sk in the experience buffer is small.
• The key state sk is time-sensitive. The new key state needs to participate in training the network as quickly as possible. Otherwise, if agent’s state undergoes a relatively change, its role in improving the policy will be declined.
• Since the agent’s state is gradually changing, more recent the sampled experience, more similar its distribution is to the distribution of the latest experience generated currently, and the more it satisfies the assumption of independent and identical distribution (iid).
In response to the first question, we designed two experience buffers to store key transition and non-key transition respectively, and used min(PrtAN2N ,Kt) to adjust the proportion of sampling key transition, where PrtAN2N is the proportion of the number of key transitions generated by AN2N, Kt is linearly related to the simulation times of the agent. The above ensures that key transitions can be sampled strictly according to the proportion from the experience buffer. For the second and third questions, we will linearly increase the probability of new transition being sampled. Two sampling functions will sample 100 samples in [0, 20] to compare the difference of the linear distribution sampling function and the uniform distribution more vividly, the result shown in Fig. 2. The pseudo code of ERM algorithm is shown in algorithm 1.
Algorithm 1: ERM Input: Sampling ratio PrtAN2N , Kt, Replay buffer Rnon−key, Rkey, batch size bs1, bs2, bssum and AN2N parameters Randomly initialize critic network Q(s, a|θQ) and actor µ(s|θµ) with weights θQ and θµ Initialize target network Q′ and µ′ with weights θQ ′ ← θQ, θµ ′
← θµ for episode e ∈ {1,...,M} do
Initialize a random process N for action exploration Receive initial observation state s1 for t ∈ {1,...,T} do
Execute AN2N action at and observe reward rt and observe new state st+1 if (AN2N exploring more) then
Store key transitio (st, at, rt, st+1) in Rkey else
Store transition (st, at, rt, st+1) in Rnon−key end bs1 = min(PrtAN2N ,Kt) bs2 = bssum − bs1 Sample bs1 and bs2 transitions with linear distribution in Rkey and Rnon−key for training
Run DDPG, SAC or TD3 etc. Algorithm end
end

heading: 4 Experiments
text: In this section, we test the performance of the combinations of Experience Replay More algorithm (ERM) with different off-policy RL algorithms acrossing a variety of continuous control tasks. As is shown in Fig. 3, consistent with the benchmark, we use the mainstream Mujoco physics engine (Todorov et al., 2012) as the simulation environment to test
the performance of the algorithm in the HalfCheetah-v2, Swimmer-v2, Walker2d-v2, and Hopper-v2 tasks, as Mujoco offers a unique combination of speed, accuracy and modeling power, and it is also the first full-featured simulator designed from the ground up for the purpose of motion control. For the specific introduction of the task environment is shown in Table 2 in Appendix A.
We present the ERM by classifying key transitions and sampling them with linear distribution functions for the purpose of increasing the stability and performance with sampling efficiency, described in section 3. In each task, we run our algorithm and test it without exploration noise. The goal of our experimental evaluation is to understand how good the sample efficiency and stability of our method are, compared with prior off-policy RL algorithms. In all tasks, we run experiments for five times, which fix random seeds in 0, 5, 10, 15, 20 respectively. The results of ERM combined with different off policy RL algorithms are analyzed below.
4.1 DDPG with ERM
For the implementation of DDPG with ERM, we use a two layer fully connected network consists of 256× 256 hidden nodes respectively, with rectified linear units (ReLU) between each layer for both the policy and action state networks, and a final tanh unit following the output of the policy network for limiting amplitude. After a certain number of steps, the networks are trained with a mini-batch of 100 transitions repeatedly, sampled from a replay buffer containing the entire history of the agent. See Appendix A for more experimental details.
DDPG with ERM uses two actor-networks and critic-networks respectively to approximate the policy and action-state value. In the test stage, we record the reward of each state of the agent and then calculate all the action state values of the trajectory when the episode finished, save those transitions whose total rewards are minimal. When the agent interacts with the environment, it uses the policy superimposed a small disturbance noise, if the current state is similar to key states, add a disturbance noise on small noise, otherwise, only use the small noise. Besides, we store the key tansitions and non-key transitions separatly, which is convenient for sampling two kinds of transitions in a appropriate proportion with
linear distribution sampling function. The pseudo code of DDPG with ERM is shown in algorithm 2 in Appendix B. We compare the DDPG with ERM algorithm with DDPG baseline, illustrated in Fig. 4, among the four tasks, the performance of DDPG with ERM is higher. In the HalfCheetah task, the method combined with ERM has significantly lower variance while maintaining higher performance, indicating that the algorithm we proposed with DDPG has better stability and sample efficiency.
4.2 TD3 with ERM
TD3 is an optimized version on the basis of DDPG, and the structure is very similar to the DDPG algorithm. The main three improvements are for the problems of DDPG in engineering practice: 1.Clipped Double-Q Learning. TD3 uses two Q networks with the same structure to predict the state action value at the same time, but when calculating TD-error, only the Q value with the smallest prediction participates in the calculation, so as to alleviate the overestimation of DDPG. 2.Delayed Policy Updates. Since updating the policy frequently is prone to make the policy unstable, therefore, TD3 updates the policy with a lower frequency. 3.Target Policy Smoothing. When calculating target Q, a small range of noise is added to the policy to make the calculated target Q value more robust. Therefore, TD3 with ERM can maintain the same network structure and hyper-parameters with DDPG with ERM. See Appendix A for more details.
In the same tasks with above, we tested the performance of TD3 and TD3 with ERM. As illustrated in Fig. 5. In all the test tasks, the performance improvement of TD3 with ERM is very obvious compared to TD3, especially in HalfCheetah and Walker2d tasks. At the same time, the method of combining TD3 and ERM has significantly lower variance while maintaining higher performance in the HalfCheetah and Hopper tasks, indicating that the algorithm we proposed is very stable and sample efficient on TD3.
4.3 SAC with ERM
Compared with TD3, SAC is more different from DDPG, but it is still an Actor-Critic structure. The main differences are as follows: 1.The policy entropy is added to the objective function, which can more fully explore the action space, but there is also an additional temperature coefficient that adjusts the entropy proportion of the policy. 2.SAC does not directly outputting the deterministic policy, but a mean value and variance of a policy’s distribution, so it is necessary to adjust the range of the variance to achieve a noise-like addition. See Appendix A for the specific hyper-parameter settings of SAC. The SAC algorithm is currently one of the most commonly used off-policy RL methods in academic and industry owing to its good performance, stability and sample efficiency. We tested the performance of SAC and SAC with ERM in four tasks, as shown in Fig. 6, we found that in half of the tasks, the performance of SAC with ERM still has a performance improvement compared to sac, while the performance of the remaining tasks was flat, indicating that the algorithm we proposed also owns a better stability and sample efficiency on SAC.
In addition, we summarize the results of the above several algorithms in the HalfCheetah simulation environment. As shown in the sub-graph (a) in Fig. 7. We run agent with each algorithm for 1 million time steps with evaluations every 4000 time steps, where each evaluation reports the average reward over 10 episodes with no exploration noise. Our results are reported over 5 random seeds of the Gym simulator. It can be seen that the performance of SAC with ERM is the best, surpassing state of the art (sac), and its convergence speed is also the fastest, indicating that it has higher sample efficiency compared with other off-policy RL algorithms.
We select the result in DDPG with ERM with the random seed 5 in HalfCheetah-v2 task, we compare it with the average performance of SAC, as shown in the subgraph (b) in Fig. 7. It can be found that although the convergence speed of ddpg with ERM is slower in the early stage, its performance has approached or even exceeded the average performance of SAC in the later stage. This improvement is very incredible for DDPG. We display the statistical data of all the experimental results in Table 1. The first column indicates the algorithm or policy, among which random indicates that the agent uses a random policy to interact with the environment. The numbers in the table represent the average cumulative rewards obtained by the corresponding algorithm or policy in the environment in a episode with 4000 steps. The numbers in bold are the highest performance scores. It can be seen that the SAC with ERM algorithm has the highest score, followed by the DDPG with ERM algorithm, and ERM has the greatest effect on improving the performance of TD3. The statistical average cumulative reward results show that ERM is very helpful to improve the performance of DDPG, TD3 and SAC.
In all the experiments, the most time-consuming part of the algorithm is calculating the similarity between states. To solve this problem, we accelerate the process by using matrix operation, which expands the dimension of current state Sc to be consistent with the dimension of key states buffer, so we calculate the similarity with the matrix Sk composed of all key states quickly. Therefore, in the case of a small increase in time consumption, the performance of the algorithm is significantly improved, especially on the HalfCheetah task.

heading: 5 Conclusion
text: This work divides the states of the agent into key states and non-key states, and analyzes the reasons why the recent generated key states need to be sampled and trained as soos as possible. For the purpose of sampling key transitions more accurately, we introduce two experiences buffers to store the key transitions and non-key transitions respectively, and set a adjustable coefficient to determine the proportion of transitions sampled from two experience buffers. Besides, we analyze the advatages of sampling the key transitions if we make use of linear distribution function from two perspectives: 1.The distribution of transitions sampled recently are more similar to the distribution of latest experience generated, making transitions more satisfie the assumption of independent and identical distribution (iid); 2.The recent generated experience can be used for the training of the Q value network and policy network of the agent more quickly, so as to make up for the lack of the policy in time. Finally, on the basis of AN2N, the combination of ERM method and DDPG, TD3 or SAC algorithm has a very obvious performance improvement on tasks such as HalfCheetah. The performance of the DDPGERM algorithm with some random seeds can even exceed the average performance of SAC, and SACERM has also outperforms the current state of the art.
Acknowledgments
We would like to thank Feng Pan, Weixing Li, Xiaoxue Feng, Yan Gao and many others at Institute of Pattern Recognition and Intelligent System of BIT for insightful discussions and feedback.

heading: A Experiment Details
text: The experiment uses a unified parameter configuration for all algorithms, the two hidden layers are composed of 256 nodes and the neural networks use the rectified non-linearity (Glorot et al., 2011) for all hidden layers. The activation function of the actor output layer uses a tanh for bounding the actions. Adam (Kingma & Ba, 2014) optimizes the weights of the neural networks with a learning rate of 103 for the both of actor and critic networks, discount factor set to 0.99. For the soft target updates, we set τ = 0.005. At the beginning, the agent uses a random policy to generate 104 transitions for the initial training of the network, the proportion of sampling key transitions is fixed by min(PrtAN2N ,Kt), where PrtAN2N = 0.4− 0.2 ∗ ttotaltime , Kt = 50 ∗ t totaltime . We train the networks with minibatch sizes of 100 for all of the tasks, and use a replay buffer size of 106. For DDPG or TD3 with ERM, we set two kind noise Nbig = 0.4 and Nsmall = 0.05, For SAC with ERM, different from the policy generation method of DDPG and TD3, SAC generates the mean and variance of the policy, and then sample a specific action. Therefore, we set the expansion and reduction coefficients of the variance to 1.5 and 0.5. We test our algorithm in the Mujoco task environment. The state dimension, observation dimension, and action dimension of the tasks are shown in Table 2, and the detailed description of the tasks are shown in Table 3.

heading: B Algorithm
text: Appendix B includes two parts: the first part is the overall structure diagram 8 of the Experience Replay More (ERM) algorithm. The core of the entire algorithm is to classify and store the data stream, and sample the experience with a linear distribution function. The second part is the pseudo code of the algorithm: algorithm pseudo code of DDPG with ERM in Algorithm 2, algorithm pseudo code of TD3 with ERM in Algorithm 3 and algorithm pseudo code of SAC with ERM in Algorithm 4.
Algorithm 2: DDPG with ERM Input: Sampling ratio PrtAN2N , Kt, Replay buffer Rnon−key, Rkey, batch size bs1, bs2, bssum and AN2N parameters Randomly initialize critic network Q(s, a|θQ) and actor µ(s|θµ) with weights θQ and θµ Initialize target network Q′ and µ′ with weights θQ ′ ← θQ, θµ ′
← θµ for episode e ∈ {1,...,M} do
Initialize a random process N for action exploration Receive initial observation state s1 for t ∈ {1,...,T} do
Execute AN2N action at and observe reward rt and observe new state st+1 if (AN2N exploring more) then
Store key transitio (st, at, rt, st+1) in Rkey else
Store transition (st, at, rt, st+1) in Rnon−key end bs1 = min(PrtAN2N ,Kt) bs2 = bssum − bs1 Sample bs1 and bs2 transitions with linear distribution in Rkey and Rnon−key for training if t mod u then Sample a random minibatch of N transitions (si, ai, ri, si+1) from R Set yi = ri + γQ ′ (si+1, µ ′ (si+1|θµ ′ )|θQ ′
) Update critic by minimizing the loss:L = 1N ∑ i(yi −Q(si, ai|θQ))2 Update the actor policy using the sampled policy gradient: ∇θµJ ≈ 1N ∑ i ∇aQ(s, a|θQ)|s=si,a=µ(si)∇θµµ(s|θµ)|si
Update the target networks:
θQ ′ ← τθQ + (1− τ)θQ ′
θµ ′ ← τθµ + (1− τ)θQµ ′
end end
end
Algorithm 3: TD3 with ERM Input: Sampling ratio PrtAN2N , Kt, Replay buffer Rnon−key, Rkey, batch size bs1, bs2, bssum and AN2N parameters Randomly initialize critic networks Q(s, a|θQ1 ), Q(s, a|θ Q 2 ), and actor µ(s|θµ) with
weights θQ1 , θ Q 2 and θµ
Initialize target network Q′ and µ′ with weights θQ ′
1 ← θ Q 1 , θ
Q ′ 2 ← θ Q 2 , θµ ′
← θµ for episode e ∈ {1,...,M} do
Initialize a random process N for action exploration Receive initial observation state s1 for t ∈ {1,...,T} do
Execute AN2N action at and observe reward rt and observe new state st+1 if (AN2N exploring more) then
Store key transitio (st, at, rt, st+1) in Rkey else
Store transition (st, at, rt, st+1) in Rnon−key end bs1 = min(PrtAN2N ,Kt) bs2 = bssum − bs1 Sample bs1 and bs2 transitions with linear distribution in Rkey and Rnon−key for training if t mod u then Sample a random minibatch of N transitions (si, ai, ri, si+1) from R Get ã← µ(s|θµ ′
) + ϵ, ϵ ∼ clip (N (0, σ̃),−c, c) Set yi = ri + γminj=1,2 Q ′ (si+1, µ ′ (si+1|ã)|θQ ′ j )
Update critic by minimizing the loss:L = minθj 1N ∑ i(yi −Q(si, ai|θ Q j )) 2 if t mod (d× u) then Update the actor policy µ(s|θµ) using the sampled policy gradient:
∇θµJ ≈ 1N ∑ i ∇aQ(s, a|θQ1 )|s=si,a=µ(si)∇θµµ(s|θµ)|si
Update the target networks:
θQ ′
j ← τθ Q j + (1− τ)θ
Q ′ j
θµ ′ ← τθµ + (1− τ)θQµ ′
end end
end end
Algorithm 4: SAC with ERM Input: Sampling ratio PrtAN2N , Kt, Replay buffer Rnon−key, Rkey, batch size bs1, bs2, bssum and AN2N parameters, Temperature parameter α Randomly initialize critic networks Q(s, a|θQ1 ), Q(s, a|θ Q 2 ), and actor µ(s|θµ) with
weights θQ1 , θ Q 2 and θµ
Initialize target network Q′ and µ′ with weights θQ ′
1 ← θ Q 1 , θ
Q ′ 2 ← θ Q 2
for episode e ∈ {1,...,M} do Initialize a random process N for action exploration Receive initial observation state s1 for t ∈ {1,...,T} do
Execute AN2N action at and observe reward rt and observe new state st+1 if (AN2N exploring more) then
Store key transitio (st, at, rt, st+1) in Rkey else
Store transition (st, at, rt, st+1) in Rnon−key end bs1 = min(PrtAN2N ,Kt) bs2 = bssum − bs1 Sample bs1 and bs2 transitions with linear distribution in Rkey and Rnon−key for training if t mod u then Sample a random minibatch of N transitions (si, ai, ri, si+1) from R Set yi = ri + γ(minj=1,2 Q ′ (si+1, µ(si+1|θµ)|θQ ′
j )− α log µ(st+1|θµ)) Update critic (soft Q-function) by minimizing the loss:
L = 1N ∑ i ∑ j(yi −Q(si, ai|θ Q j )) 2
Update the actor policy using the sampled policy gradient: ∇θµJ ≈ 1N ∑ i
((∇aQ(s, a|θQ)|s=si,a=µ(si)∇θµµ(s|θµ)|si −∇a log µ(st|θµ))−∇θµ log µ(st|θµ)|si)
Update the target networks:
θQ ′
j ← τθ Q j + (1− τ)θ
Q ′ j
end end
end

Review summaries:
The paper proposes to set up experience replay such that transitions are emphasized based on whether the AN2N algorithm decides to explore more, and how recent the transition was experienced. It is based on observations that the state distribution can dramatically drift over the course of policy improvement, and the intuition that it might be better to perform updates on states that the current policy is more likely to actually visit (recent transitions). They apply the proposed sampling method, *experience replay more* (ERM), to a variety of replay-based deep RL algorithms and evaluate them empirically.

The paper proposes a new method (ERM) to bias the decision over which transitions should be replayed more often. In particular, using ERM, some states are deemed as key and they are replayed more often (following also a recency bias). This new method is then tested in 4 environments on the DeepMind control suite by using 3 different RL algorithms (DDPG, SAC, TD3).