Experience Replay More When It's a Key Transition in Deep Reinforcement Learning
Authors: Youtian Guo
Qi Gao
Venue: ICLR 2022 Submitted (2022)
Keywords: Experience Replay More
Key Transitions
Sampling
Add Noise to Noise
Deep Reinforcement Learning

Abstract: We propose a experience replay mechanism in Deep Reinforcement Learning based on Add Noise to Noise (AN2N), which requires agent to replay more experience containing key state, abbreviated as Experience Replay More (ERM). In the AN2N algorithm, we refer to the states where exploring more as the key states. We found that how the transitions containing the key state participates in updating the policy and Q networks has a significant impact on the performance improvement of the deep reinforcement learning agent, and the problem of catastrophic forgetting in neural networks is further magnified in the AN2N algorithm. Therefore, we change the previous strategy of uniform sampling of experience transitions. We sample the transition used for experience replay according to whether the transition contains key states and whether it is the most recently generated, which is the core idea of the ERM algorithm. The experimental results show that this algorithm can significantly improve the performance of the agent. We combine the ERM algorithm with Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic policy gradient (TD3) and Soft Actor-Critic (SAC), and evaluate algorithm on the suite of OpenAI gym tasks, SAC with ERM achieves a new state of the art, and DDPG with ERM can even exceed the average performance of SAC under certain random seeds, which is incredible.

Under review as a conference paper at ICLR 2022
Experience Replay More When It's a Key Transition in Deep Reinforcement Learning
Anonymous authors Paper under double-blind review
Abstract
We proposed an experience replay mechanism in Deep Reinforcement Learning based on Add Noise to Noise (AN2N), which requires agent to replay more experience containing key state, abbreviated as Experience Re- play More (ERM). In the AN2N algorithm, we refer to the states explored more the key states. We found that how the transitions containing the key state participates in updating the policy and Q networks has signif- icant impact on the performance improvement of the deep reinforcement learning agent, and the problem of catastrophic forgetting in neural net- works is further magnified
the AN2N algorithm. Therefore, We sample the transition used for experience replay according to whether the transition contains key states and whether is the most recently generated, rather than sampling uniformly, which is the core idea of the ERM algorithm. The experimental results show that this algorithn hm improves the performance of the agent by a wide margin. We combine the ERM algorithm with Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic policy gradient (TD3) and Soft Eetor-Critic (SAC), and evaluate algorithm on the suite of OpenAI gym tasks, SAC with ERM achieves a new state of the art, and DDPG with ERM can even exceed the average performance of SAC under certain random seeds, which is incredible.
Introduction
learning (RL) has shown promising futture for makisi in various computer games, such as atari (Miih et al,, 2013 2015), go (Schrittwicsor et al 2020) and starcraft (Vinyals et al., 2019). However, most successes have boen exclusively in simulation due to poor sample efficiency of typical Deep RL algorithm and other challenges. Reinforcement learning can be divided into model-based RL and model-free RL in the light of its data efficiency. Model-free RL is usually subdivided into off-policy RL and on- policy RL. Although model-basee RL requires less sampled data, it needs to build a world model and predict the next state based a lot of prior work, demonstrated in Hafner et al (2019). Although the on-policy RL algorithm do without establishing a world model and has good stability, the performance improves slowly due to limiting the step of updating the policy, a large number of sampled trajectory data are required in the training process (Schulman et al. 2015; 2017). Off-policy RL is between model-based RL and on-policy R.L in terms of sampling efficiency, and the rescarch in this field is enduring (Watkins & Dayan, 1992. Hessel al, 2018 Barth-Maror et al., 2018) on account of its relatively high data efficiency and world model free.
Deep reinforcement
Experience Replay (Lin, 1992) is an important part of improving the data efficiency of Off- policy RL, which stores experience in a replay buffer and break the temporal correlations by mixing data, therefore, the experience can be used multiple times to update the networks. However, most of the current work is to uniformly sample transitions from the buffer, such as Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015). Soft Actor-Critic (SAC) (Haarpoja_ct al. 2018). Twin Delayed Deep Deterministic policy gradient (TD3) (u, Simato et al. 2018) and many other algorithms (Van Hasselt et al., 2016: Mnih et al., 2016 Andrychowicz al. 2017: Dabney et al. 2018
Liu al., 2020 However, these approaches

---

Under review as a conference paper at ICLR 2022
replay experience transitions at the same frequency, regardless of their meaning. Schaul et al, (2016) develops a framework for prioritizing experience to replay important transi- tions more frequently, therefore agent learns more efficiently. Prioritized experience replay (PER) method samples transitions with the magnitude of their temporal-differnne (TD) error. However, prioritization introduces bias, which needs to be corrected with importance sampling. Meta-reinforcement learning (meta-RL) algorithms weaechhmsseeabbleee agents to, learn new skills, from amounts of experience (Rothfuss all, 2018: Mishra et al 2018), Rakelly et al (2019) develops an off- off-policy meta-RL algorithm that disentangles the inference and control of the task, but its performance still far behind the mainstream off-policy RL algorithms.
In this paper, We first analyzed how the state changes from beginning of agent interacting with the environment to the policy converged, and found that the state of the agent is different at different stages, agent will rarely transfer to those states where the policy far from gonverging when the agent's 's policy already converged. If a large number of initial- are used to train Q-value and policy networks, the network weights that have been
states are learned well will be gradually updated. Therefore, the sampling proportion of the most recently generated transitions should be appropriately increased, so that the agent pays more attention to learning the recent. transitions, In this way, the state of the agent is gradually transferred from a poor state to a better state, similar to updating from a poor policy ro a better policy stabily in Trust region policy_optimiiee optimization (TRPÓ) (Schulman et al.. 2015) or Proximal policy optimization (PPO) (Schulman et al. 2017).
Inspired by the Add Noise to Noise (AN2N) Algorithm (Guo & Gao, 2021), We divide the states into two categories. These states explored with added noise are called key states and the rest are called non-key states. For the sake of improving the performance in key states in time, we have increased the probability of sampling new key states, making them more likely to participate in updating policy, we call this process as ERM (Experience Replay More), and combine it with commonly used off-policy RI algorithms in continuous control tasks, such as SAC, which obtained faster convergence and state-of-the-an performance.
Preliminaries
We consider a reinforcement learning setup consisting of agent learning policies to maximize the expected reward when interacting with the environment (Sutton & Barto, 2018). At each timestep the agent receives an observation or E O, selects action a; E A with respect to its policy T: O + A. After taking the action Or in environment E, agent receives reward r, and the next observation 0441- The practical problem is usually a partial Markov decision process (POMDP), only part of the observation information could be obtained. To simplify the problem, we assumed the environment is fully-observed, so s = o,, O.
In reinforcement learning, the eetion-value function Q" (s, a) is uesed to approximate the expected sum reward of the ation in state 8, defined as following:
Q (8, = Escxpa Sap
Σ y R
Where E 10 is the discount factor, Eg is the expectation over the distribution of the trajectories (so, ap, $1, ....
The mean value of Q in the same state called the value function V= defined as (=(s) = Ex 1Q (s. a)]. Weexsssss the action-value function Q in the form of Bellman equation (Bellman & Kalaba, 1965):
Q (St, 04) Ee (((s)) - (8141,
(2)
'The initial-state the state which agent not perform well when policy has not converged, agent almost no longer transferred to this kind of state after policy converged. "The key state state where the agent has not performed well in the past
2

---

Under review as a conference paper at ICLR 2022
In this paper, we need to be familiar with DDPG, TD3 and SAC algorithms, here, mainly introduce DDPG as the basis. DDPG applied two different fully connected neural networks to approximate the action-value function Q(s, Q(s, al09) and policy function pus)0"),
DDPG introduces action-value target network 0º and policy target network 0 so as to Stable the policy update. Consequently, gradient descent. used to optimize the network weights by minimizing the loss:
L(0 Es Pacopet ae (atp a.)0) -2a)²
(3)
Where
y1 ar) +7 1 (8 (00+1 )108¹
(4)
V Vau J E Bo-Pientany Qe, al00) apps
(5)
V. Q(s, a (=)))))((s)) Voru(s)))")|s st]
Where equation derived from equation E the weights of target networks are updated periodically to slowly track the learned networks: 8' + T0 (1-7)0 with < which al- leviates the fluctuation in the agent's s learning process. The policy is updated by equation & following the chain rule the expected sum return Q(s, al89) with respect to parameters 0". TD3 addresses the problem that DDPG is prone to overostimating the Q function. An additional Q-function added, and the predicted smaller Q value used as the calculation of TD-error. At the same time, which also reduces the update frequency of the policy func- tion. The most significant difference between SAC and entropy (T((ss))) the objective function equation is rewritten
the introduction of policy
Q (s BapppaaN aiNT
Σ (( a log(T((()ss))
(6)
Where is temperature parameter, which adjusts the optimization target, agent pays more attention to exploration if increase the coefficient a.
60.0
2000.0
500.0
20
20
160.0
100.0
10
10
-10
-30
w22 x14
(a) States between 2ed and 2.4e4 steps
(b) States between 5.6e5 and 5.64e5 steps
Figure 1: In the HalfChectah-v2 environment, the state of the agent in different training stages is compared. The abscissa means the different positions on the agent, and the ordinate means the collected attitude information the specific position, the difference is obvious at position sl, 88, s12 and s14. Different colors of legend indicate different Q values.(a) Start collecting at 2b4.(b) Start collecting 5.6e5.
3 Experience Replay More When It's Key Transition
The addition of experience replay improves the sample efficiency of off-policy RL. Many off-policy algorithms usually sample transitions uniformly from the experience buffer, which
3

---

Under review as a conference paper at ICLR 2022
potentially considers the transitions generated at different times to be of the same im- portance, we will discuss this in section 3.1. Prioritized Experience Replay (PER) an optimization of the uniform sampling method based on the TD-error value of transitions so as to learn the samples more efficiently, however, PER essentially still does not consider whether the transitions generated at different times are equally important to the current. agent's policy.
3.1 States are Different at Different Stages
Taking the HalfCheetah-v2 simulation environment as an example, we recorded the state information of agent at different stages based on DDPG algorithm, collecting attitude information at the specific position and the corresponding Q-value starting from 2c4 to 5.6c5, and then uniformly sample 100 sets of data from the collected data for drawing, as shown in the Fig. D The legend in the upper left corner represents the Q value, which discretized
reasonably reduce the number of Legends. Subgraph (a) is discreted in units of 20, and subgraph is discreted in units of 100. It can be found when the agent interacts with the environment, is also changing accordingly.
only the policy gradually improving but the state
After policy converges and the agent is in a state set different from the previous one, more experience similar to the agent's 's recent states should be collected to train the neural network to strengthen the current policy. If the sampling experience for training policy network and the action stale network are quite different from the recently generated states, it will not only help improve the policy to a small exlent, but will also even forget the newly learned network weights due to the catastrophic forgetting of the neural networks. Therefore, to further improving the sample efficiency, experience replay should be switched from uniform sampling to targeted sampling.
16
14
12
8-
2-
17.5
15
Figure 2: Compare the sampling results by using the linear distribution sampling function and the uniform distribution sampling function. The left picture shows the sampling result by linear distribution, and the right picture shows another. We sampled 100 samples in [0, 20).
3.2 Replaying More Key Transitions
The AN2N algorithm will record the dilemma state sa during the evaluation process, when agent interacts with the environment, the current state &, and $a are considered to be highly similar, an additional exploration noise N₂ will be added. In this paper, the state 8, with additional noise is called the key state 8k, and the transition containing called the key transition trang. The agent's policy is relatively fragile in sk: if agent has not learned how to get out of this dilemma before, agent is more likely to fall into series of bad
is
Better decision-making ability usually gets a higher Q value

---

Under review as a conference paper at ICLR 2022
states after Sk, which greatly reduces the agent's overall performance. Therefore, whether the agent can learn a good policy in S/ is very important for improving the performance.
It can be seen from equation that reward of the key state sk will affect the Q value of the state at the previous moment through the iteration of the dynamic equation, nevertheless, uniformly sample experience for training the networks, there will be the following three problems:
Since most of the experience generated by AN2N are non-key states, the probability of sampling key states se in the experience buffer is small.
The key state sk is time-sensitive. The new key state needs to participate in training the network as quickly possible. Otherwise, if agent's 's state undergoes relatively change, its role in improving the policy will be declined.
Since the agent's state is gradually changing, more similar its distribution is to the distribution of the latest experience generated currently, and the more it satisfies the assumption of independent and identical distribution (iid).
recent the sampled experience,
In response to the first question, we designed two experience buffers to store key transition and non-key transition respectively, and used sampling key transition, where PrtAN2N is the proportion of the mumber key transitions generated by AN2N, K, is linearly related to the simulation times of the agent. The above ensures that key transitions can be sampled strictly according to the proportion from the experience buffer. For the second and third questions, will linearly increase the proba- bility of new transition being sampled. Two sampling functions will sample 100 samples in [0, 20] to compare the difference of the linear distribution sampling function and the uniform distribution more vividly, the result shown in Fig. 2. The pseudo code of ERM algorithm shown in algorithm L.
K₂) to adjust the proportion of
Algorithm 1: ERM
Input: Sampling ratio PrtAN2N, KG, Replay buffer Rnon-keys Rkeys batch size bs1, b82, b8sum and AN2N paramcters
Randomly initialize critic network Q(s, al02) and actor u(s]0n) with weights 0 and o Initialize target network Q' and μ with weights o² 82, Au for episode [l.....M} do Initialize a random process N for action exploration
gu
Receive initial observation state for t E {1. do
Execute AN2N action a, and observe reward rt and observe new state 81+1
(AN2N exploring more) then
Store key transitio (st, Ct, Pt, 8t+1) in Rkey
else
Store transition (St, at, at, Tt, 8641) in Rnon
end
bs = min Amin(Prtet (PrtAN2N,
bs2 bs pspum bsq
Sample b81 and b82 transitions with linear distribution in Rkey and Rnon key for training
Run DDPG, SAC or TD3 etc. Algorithm
end
end
4 Experiments
In this section, we test the performance of the combinations of Experience Replay More algorithm (ERM) with different off poliy RL algorithms acrossing variety of continuous control tasks. As shown in Fig. & consistent. with the benchmark, we use the main- stream Mujoco physics engine (Todorov et al. 2012) as the simulation environment to test.
5

---

Under review as a conference paper at ICLR 2022
the performance of the algorithm in the HaliCheetah-v2. Swimmer-v2, Walker2d-v2, and Hopper-v2 tasks, as Mujoco offers a unique combination of speed, accuracy and modeling power, and it is also the first full-featured simulator designed from the ground up for the purpose of motion control. For the specific introduction of the task environment shown in Table in Appendix A
Figure 3: Samples of Mujoco tasks. In order from the left: HallCheetah-v2, Swimmer-v2, Walker2d-v2, Hopper-v2.
We present the ERM by classifying key transitions and sampling them with linear distri- bution functions for the purpose of increasing the stability and performance with sampling efficiency, described in section g In each task, we run our algorithm and test it without exploration noise. The goal of experimental evaluation is to understand how good the sample efficiency and stability of our method are, compared with prior off-policy RI. algo- rithms. In all tasks, we run experiments for five times, which fix random seeds in 0, 5, 10, 15, 20 respectively. The results of ERM combined with different off policy RL algorithms are analyzed below.
4.1 DDPG with ERM
For the implementation of DDPG with ERM, we use two layer fully connected network consists of 256 x 256 hidden nodes respectively, with rectified linear units (ReLU) between each layer for both the policy and action state networks, and final tanh unit following the output the policy network for limiting amplitude. After a certain number of steps, the networks are trained with mini-batch of 100 transitions repeatedly, sampled from a replay buffer containing the entire history of the agent. See Appendix A for more experimental details.
2000
300.0
324
2540
3000
2530
2000
500
Tote noteeooodd
(a) H2
(b) Swimmer-v2
(c) Walker2d-v2
(d) Hopper-v2
Figure 4: Performance curves of the Agents using DDPG and DDPG with ERM: DDPG
(red), DDPG with ERM (green).
DDPG with ERM uses two actor-networks and critic-networks respectively approximate the policy and action-state value. In the test stage, we record the reward of each state of the agent and then calculate all the action state values of the trajectory when the episode finished, save those transitions whose total rewards are minimal. When the agent interacts with the environment, it uses the policy superimposed a small disturbance noise, the current state is similar to key states, add a disturbance noise small noise, otherwise, only use the small noise. Besides, we store the key tansitions and non-key transitions separatly, which is convenient for sampling two kinds of transitions in appropriate proportion with

---

Under review as a conference paper at ICLR 2022
linear distribution sampling function. The pscudo code of DDPG with ERM is shown in algorithm 2 in Appendix B.
We compare the DDPG with ERM algorithm with DDPG baseline, illustrated in Fig. among the four tasks, the performance of DDPG with ERM is higher. In the HalfCheetah task, the method combined with ERM has significantly lower variance while maintaining higher performance, indicating that the algorithm we proposed with DDPG has better stability and sample efficiency.
4.2 TD3 with ERM
TD3 is an optimized version on the basis of DDPG, and the structure is very similar to the DDPG algorithm. The main three improvements are for the problems of DDPG in engincering practice: 1.Clippdd Double-Q Learning. TD3 uses two Q networks with the same structure predict the state action value the same time, but when calculating TD-error, only the Q value with the smallest prediction participates in the calculation, so as to alleviate the overestimation of DDPG. 2.Delayed Policy Updates. Since updating the policy frequently is prone to make the policy unstable, therefore, TD3 updates the policy with a lower frequency. 3. Target Policy Smoothing. When calculating target Q, a small range of noise is added to the policy to make the caleulated target Q value more robust. Therefore, TD3 with ERM can maintain the same network structure and hyper-parameters with DDPG with ERM. See Appendix A for more details.
30000
8000
M
Techloneeeeeee 30
Tom EtAmmmee
35
Toms Envrneennts
36
Tor Enetsweet
30
(a) HallCheetah-v2
Swimmer-v2
(c) Walker2d-v2
(d) Hopper-v2
Figure 5: Performance curves for a selection of domains using TD3 and TD3 with ERM: TD3 (cyan), TD3 with ERM (yellow).
In the same tasks with above, we tested the performance of TD3 and TD3 with ERM. As illustrated in Fig. & In all the test tasks, the performance improvement of TD3 with ERM very obvious compared to TD3, especially in HalfCheetah and Walker2d tasks. At the same time, the method of combining TD3 and ERM has significantly lower variance while maintaining higher performance in the HalfChoctah and Hopper tasks, indicating that the algorithm we proposed is very stable and sample efficient on TD3.
4.3 SAC with ERM
Compared with TD3, SAC is more different from DDPG, but is still an Critic structure. The main differences are as follows: 1.The policy entropy is added to the objective function, which can more fully explore the action space, but there is also an additional temperature coefficient that adjusts the entropy proportion of the policy. 2.SAC does not directly outputting the deterministic policy, but mean value and variance of a policy's distribution, so it is necessary to adjust the range of the variance to achieve noise-like addition. See Appendix A for the specific hyper-parameter settings of SAC.
The SAC algorithm is currently one of the most commonly used off-policy RI. methods in academic and industry owing to its good performance, stability and sample efficiency. We tested the performance of SAC and SAC with ERM in four tasks, shown in Fig. B we found that in half of the tasks, the performance SAC with ERM still has a performance improvement compared to sac, while the performance of the remaining tasks was flat indi- cating that the algorithm we proposed also owns a better stability and sample efficiency on SAC.

---

Under review as a conference paper at ICLR 2022
2000
8000
(a) 2 (b) Swimmer-v2
(c) Walker2d-v2
Hopper-v2
Figure 6: Performance curves for a selection of domains using SAC and SAC with ERM: SAC (purple), SAC with ERM (blue).
In addition, we summarize the results of the above several algorithuss in the HalfCheetah simulation environment. As shown in the sub-gras -ib-graph (a) in Fig. 0. We run agent with each algorithm for 1 million time steps with evaluations every 4000 time steps, where each evaluation reports the average reward over 10 episodes with no exploration noise. Our results are reported over 5 random seeds the Gym simulator. It can be seen that the performance of SAC with ERM is the best, surpassing state of the art (sac), and its convergence speed is also the fastest, indicating that it has higher sample efficiency compared with other off-policy RL algorithms.
12000
12000
ddpgERM
10000
Performance
8000
Performance
8000
6000
ddpy sacERM ddpgFFM
4000
4000
2000
sac
2000
Id3ERM
0
td3
TotalEnvinteracts
1e5
TotalEnvinteracts
(a) HalfCheetah-v2
(b) HalfCheetah-v2
Figure 7: Performance curves for HallCheetah-v2 task using different algorithms: (a)DDPG (red), DDPG with ERM (blue), TD3 (cyan), TD3 with ERM (yellow), SAC (purple), SAC with ERM (green) (b) SAC (green), DDPG with ERM (red).
We select the result in DDPG with ERM with the random seed 5 in HalfCheetah-v2 task, we compare it with the average performance of SAC, as shown in the subgraph (b) in Fig. 7 It can be found that although the convergence speed of ddpg with ERM is slower in the early stage, its performance has approached or even exceeded the average performance of SAC in the later stage. This improvement is very incredible for DDPG.
We display the statistical data of all the experimental results in Table I. The first column indicates the algorithm or policy, among which random indicates that the agent uses random policy to interact with the environment. The numbers in the table represent the average cumulative rewards obtained by the corresponding algorithm or policy in the envi- romment in episode with 4000 steps. The nunbers in bold are the highest performance scores. It can be seen that the SAC with ERM algorithm has the highest score, followed by the DDPG with ERM algorithm, and ERM has the greatest effect improving the performance of TD3. The statistical average cumulative reward results show that ERM is very helpful to improve the performance of DDPG, TD3 and SAC.

---

Under review as a conference paper at ICLR 2022
Table 1: Mean value of the total reward of agent in different tasks
Environment
HaliChoctah-v2
Swimmer-v2
Walker2d-v2
Hopper-v2
Random
-283+22
1 + 4
2 + 2
19 + 6
DDPG
7790=2058
+ 26
920 550
1313 + 867
DDPGERM
8415+1161
94 + 24
933 + 578
1548 + 861
SAC
9452+1984
41 40 3
3163 + 951
2856 + 502
SACERM
10219 645
3564 = 1216
2883 543
TD3
1240=1455
58 31
2568 = 733
1939 1442
TD3ERM
8795+1023
48 +
13
3305 + 966
2553 + 852
In all the experiments, the most time-consumingg part of the algorithm is calculating the similarity between states. To solve this problem, we accelerate the process by using matrix operation, which expands the dimension of current state S. to be consistent with the di- mension of key states buffer, we calculate the similarity with the matrix Sş composed of all key states quickly. Therefore, in the case of small increase in time consumption, the performance of the algorithm is significantly improved, especially the HalfCheetah task.
Conclusion
This work divides the states of the agent into key states and non-key states, and analyzes the reasons why the recent generated key states need to be sampled and trained soos as possible. For the purpose sampling key transitions more accurately, we introduce two experiences buffers to store the key transitions and non-key transitions respcctively, and set a adjustable cocfficient to determine the proportion of transitions sampled from two experience buffers. Besides, we analyze the advatages of sampling the key transitions if we make use of linear distribution function from two perspectives: 1.The distribution of transitions sampled recently are more similar the distribution of latest experience generated, making transitions more satisfie the assumption of independent and identical distribution (iid); 2. The recent generated experience can be used for the training of the Q value network and policy network of the agent more quickly, of the policy time. Finally, on the basis of AN2N, the combination ERM method and DDPG, TD3 or SAC algorithm has very obvious performance improvement tasks such as HalfCheetah. The performance of the DDPGERM algorithm with some random seeds can exceed the average performance of SAC, and SACERM has also outperforms the current state of the art.
to make for the lack
Acknowledgments
We would like to thank Feng Pan, Weixing Li, Xiaoxue Feng, Yan Gao and many others at Institute of Pattern Recognition and Intelligent System of BIT for insightful discussions and feedback.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welin- der, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experi- ence replay. In Proceedings the 31st International Conference on Neural Information Processing Systems, pp. 5055-5065, 2017.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Hooss, and Timothy Lillicrap. Distributed distribu- tional deterministic policy gradients. In International Conference on Learning Represen- lations, 2018.
Richard Bellman and Robert E Kalaba. Dynamic programming and modern control theory, volume 81. Citeseer, 1965.

---

Under review as a conference paper at ICLR 2022
Rémi Coulom. Reinforcement learning using neural networks, with applications to motor control. PhD thesis, Institut National Polytechnique de Grenoble-INPG, 2002.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Tom Erez, Yuval Tassa, and Emanuel Todorov. Infinite horizon model predictive control for nonlinear periodic tasks. Manuscript under review, 4, 2011.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 1587 1596. PMLR, 10 15 Jul 2018. URL http:://rroceedings mil. mlr,
press press/v80 /ttss//800/fjimooo88..
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 123. JMLR Workshop and Conference Proceedings, 2011.
Youtian Guo and Qi Gao. Exploring more when it needs in deep reinforcement learning. arXiv preprint arXiv:2109.13477 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Inter- national conference machine learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to con- trol: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Com- bining improvements deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.00 022971, 2015.
Long-Ji Ji Lin. Self-ipprvvig reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293 321, 1992.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances Neural Informa- tion Processing Systems, 33:1264-1274 2020.
Nikhil Mishra, Mostafa. Rohaninejad, Xi Chen, and Picter Abbeel. simple neural attentive meta-learner. In International Conference Learning Represcntations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wicrstra, and Martin Riodmiller. Playing atari with deep reinforcement lcarning. arXiv preprint arXiv:1312.5602 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski. et al. Human-level control through deep reinforcement learning. nature, 518(7540):5299555) 2015.
10

---

Under review as a conference paper at ICLR 2022
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-19377 PMLR, 2016.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy rentrreenforeement learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In International Conference on Learning Representations, 2018.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR (Poster), 2016.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839): 604 609, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pp. 1889- 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0623 2017.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems, IEFE, 2012.
pp. 5026 5033.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double g-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-355) 354, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning, Machine learning, 8(3-4):279 292, 1992.
Pawel Wawrzyíski and Ajay Kumar Tanwani. Autonomous reinforcement learning with experience replay. Neural Networks, 41:156-167, 2013.
11

---

Under review as a conference paper at ICLR 2022
A Experiment Details
The experiment uses a unified parameter configuration for all algorithms, the two hidden lay- composed of 256 nodes and the neural networks use the rectified non-lincarity (Glorot et al. 2011) for all hidden layers. The activation function of the actor output layer uses tanh for bounding the actions. Adam (Kingma & Ba, 2014) optimizes the weights of the neural networks with a learning rate of 10" for the both of actor and critic net works, discount factor set 0.99. For the soft target updates, we set T 0.005. At the beginning, the agent uses a random policy to generate 10t transitions for the initial training of the network, the proportion of sampling key transitions fixed by
Kt), where PrIAN2N = 0.4 - 0.2 totaltime + K, 50 totallime" We train the networks with minibatch sizes of 100 for all of the tasks, and use replay buffer size of 106 For DDPG TD3 with ERM, we set two kind noise Noig = 0.4 and N = 0.05, For SAC with ERM, different from the policy generation method of DDPG and TD3, SAC generates the mean and vari- ance of the policy, and then sample a specific action. Therefore, we set the expansion and reduction coefficients of the variance to 1.5 and 0.5.
We test our algorithm in the Mujoco task environment. The state dimension, observa- tion dimension, and action dimension of the tasks are shown in Table and the detailed description of the tasks are shown in Table g
Table 2: Dimensionality of the MuJoCo tasks environment: the dimensionality of the under- lying physics model dim(s), number of action dimensions dim(a) and observation dimensions dim(o).
Task name
HalfCheetah-v2
Swimmer-v2
Walker2d-v2
Hopper-v2
dim(s)
18
27
18
14
dim aa)
6
8
6
4
dim
17
111
41
14
Table 3: The description of the Mujoco tasks environment
Task name HalfCheetah-v2
Brief description
The agent should move forward as quickly possiile with choethh like body that is constrained to the plane (Wawrzyússi & Tanwani 2013).
Swimmer-v2
This lask involves 3-link swimming robot in a viscous fluid, where the goal is to make swim forward as fast as possible (Coulom, 2002). The agent's goal is making twn-dimensional bipedal robot walk forward as fast as possible (Erez et al., 2011).
Walker2d-v2
Hopper-v2
The agent's goal is making a two-dimension one-leggod robot hop
forward as fast as possible (Erez et al., 2011).
B Algorithm
Appendix B includes two parts: the first part is the overall structure diagram of the Experience Replay More (ERM) algorithm. The core of the entire algorithm is to classify and store the data stream, and sample the experience with linear distribution function. The second part is the pseudo code of the algorithm: algorithm pscudo code of DDPG with ERM in Algorithm 2. algorithm pscudo code of TD3 with ERM in Algorithm 3 and algorithm pseudo code of SAC with ERM in Algorithm
12

---

Under review as a conference paper at ICLR 2022
Off policy RL
Data flow
DDPG
TD3
SAC
ERM
AN2N block
Non-key states
Key states
Figure 8: The overall structure diagram of the Experience Replay More (ERM) algorithm.
Algorithm 2: DDPG with ERM
Input:
Sampling ratio PrtAN2N, K,, Replay buffer Rnon-keys Rkeys batch size bs1, bs2, bssum and AN2N parameters
Randomly initialize critic network Q(s, al09) and actor u(s|0r) with weights 09 and 0n Initialize target network Q' and μ with weights 0²¹ + 09, + g for episode E (1....M} do Initialize random process N for action exploration Receive initial observation state 81 for I E (1....T) do
Execute AN2N action a, and observe reward ri and observe new state 8141
if (AN2N exploring more) then
Store key transitio (3t, ae, Te, Te, s1+1) Rkey else
Store transition (st, at Te, s6+1) in Rnon-key
end
bsi min PPtANNN, K,) bsg = bs pumum - bsg
Sample 681 and bsg transitions with linear distribution in Rkey and Rnon-key for training
if I mod u then Sample random minibatch of N transitions (si, 0i, ri, 8241) from R
Set Yi -; + 7 (s (st41100) }009) Update critic by minimizing the loss: N Σ,(u - Q(s a, 02))²
Update the actor policy using the sampled policy gradient:
V = VaQ(s, al09) (=m V {))
Update the target networks:
0 <- 70 + (1 --000
6u <- 70" (1 0"
end end
end
13

---

Under review as a conference paper at ICLR 2022
Algorithm 3: TD3 with ERM
Input:
Sampling ratio PRtANZN, Kr, Replay buffer Rnon-kep Rkeys batch size bs1, bs2, bssum and AN2N parameters
Randomly initialize critic networks a(s,a)09), Q(s, ((s)) and actor m)s(0r)) with weights 87, o and A"
Initialize target network Q' and μ² with weights of 09, og 0 gr A" for episode E (1....MM} do
Initialize random process N for action exploration Receive initial observation state 81 for t {I....T)) do
Execute AN2N action a, and observe reward ra and observe new state 8141 if (AN2N exploring more) then
Store key transitio (3t, ae Te, 86+1) in Rkey
else
Store transition (St, ae Te, 8641) Reon-Key
end
bs1 = min
bs2 bseum bs1
Sample bs1 and b82 transitions with linear distribution in Rkey and Rnon- key for training if t mod u then Sample random minibatch of N transitions (si, ai ri si+1) from Get + p(s(0"))))))))))))))) + e~ clip (N)
Set y4 - 1,n Q' (si+1, (=)11)))09))
Update critic by minimizing the loss:L = mino, 1/ E, Q(si ai 1/109)) if t mod (d x u) then
Update the actor policy u(s(0") using the sampled policy gradient:
Vou
9 {s))
Update the target networks:
09 -09 7 7)09'
0r' 70" + (1 0)0
end end end
end
14

---

Under review as a conference paper at ICLR 2022
Algorithm 4: SAC with ERM
Input:
Sampling ratio PrtAN2N, Kr, Replay buffer Rnon-keys Rkeys batch size bs1, b82. bssum and AN2N paramcters, Temperature parameter a Randomly initialize critic networks a(x,al09); Q(s, al (l08), and actor ()s|0r) with weights 8 o and 6
Initialize target network Q' and μ² with weights of 09, 0g 0 for episode E {1....MM} do
Initialize random process N for action exploration Receive initial observation state for t E {I....T)) do
Execute AN2N action a, and observe reward rt and observe new state 8t41 if (AN2N exploring more) then
Store key transitio (8t, at Te, 86+1) in Rkey
clsc
Store transition (st, at, Tt, S141) in Rnon-key
end
bsi min K2) bsg bs pspum 0sq
Sample and b82 transitions with linear distribution in Rkey and Rnon key for training
if t mod u then
Sample random minibatch of N transitions (s ai Ti, si+1) from R
Set y14 = r; (ming==1,2 Q' (5:+1, (0=)) ) - log u(se+1100) 104)
Update critic (soft Q-function) by minimizing the loss:
L= # E. E, (v. - Q(s a.(0))²
Update the actor policy using the sampled policy gradient:
Ve J 1/ E((VaQ(s, ) Vou u(s
-a. log u(s; (*)) V Vonlog p {(sr(0m)sss)
Update the target networks:
09 t T + (1 -009
end
end
end
15


Reviews:
[{"review_id": "JwdedW05xmH", "reviewer": "Reviewer_Jn1b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is very difficult to read as it contains many typos and fragmented sentences. This problem is so acute that some paragraphs are almost incomprehensible. \n- It is also very unclear how the similarity between states is calculated. For instance in paragraph 3.1.1 this is referred to as a key step to define what transitions are considered ‘key’, but little details are given. This leaving aside that s_c and s_d are not defined in formal terms.\n- A clear downside of the paper is that only a single domain is used (DM Control suite) and even worse ERM was tested in only 4 environments, thus questioning the generality of the method.\n- Figure 4 highlights the results of DDPG with and without ERM. The only environment where there seems to be a small advantage of DDPG+ERM is HalfCheetah, although the error of the baseline is overlapping. So I strongly disagree with the statements of the last paragraph of section 4.1. Also, visually there is no sign of better data efficiency, and if the authors want to make a claim in this direction they should provide more quantitative measures (e.g. area under the curve)\n- Figure 6 highlights the results of SAC with and without ERM. Again, the authors claim that SAC+ERM is better than SAC in half of the task, but this conclusion is not supported by the plots.\n- Figure 7b is particularly weird as, in deep reinforcement learning, it is very unscientific to put draw any conclusions based on a single seed. \n- Table 1 reports some numbers in bold even though the variance is so high that determining what is the best method is a bit arbitrary. Authors should perform proper statistical analysis before highlighting scores in bold.\n- Some baselines are completely missing from the experiments, e.g. baseline+prioritise replay (Schaul et al. 2016)\n", "summary_of_the_paper": "The paper proposes a new method (ERM) to bias the decision over which transitions should be replayed more often. In particular, using ERM, some states are deemed as key and they are replayed more often (following also a recency bias). This new method is then tested in 4 environments on the DeepMind control suite by using 3 different RL algorithms (DDPG, SAC, TD3).\n", "summary_of_the_review": "None of the claims in the abstract, introduction and conclusion are actually supported by the results. Also the paper is not well written, with lots of typos and fragmented sentences.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HmCZlc12UR", "reviewer": "Reviewer_cBYQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The idea is simple, and has intuitive reasons for why it could help. With sufficiently large replay buffers and enough changes made to the policy, it de-emphasizes updates on really old/stale transitions that the optimal policy would rarely visit.\n\n## Weaknesses\n\n1. The paper suffers many spelling, grammatical, and clarity issues. It can be difficult to follow, and greatly detracts from the quality of the paper.\n\n2. The paper acknowledges the PER algorithm and criticizes how such prioritization introduces bias which must be corrected by importance sampling. However, the proposed ERM algorithm (or any replay method in general) is similarly biased in that after each update, the data distribution gets increasingly off-policy. Further, despite acknowledgement of this uncorrected off-policyness, many works simply don't correct the data distribution and find that things still tend to work well empirically. Along this note, it seems like PER is a pretty important baseline to compare to in the space of methods which select which transitions to sample from the replay buffer- can the authors comment on why PER was not compared to, and whether the claims of being \"state of the art\" are reasonable when only comparing to vanilla experience replay?\n\n3. Only 5 runs were used, and across most of the figures, the differences generally don't seem statistically significant. Can the authors comment on whether enough runs were performed, what the shaded regions represent, and whether conclusions can be drawn from the results with reasonable confidence? Similarly, the following line in the abstract is concerning with regards to empirical methodology: \"DDPG with ERM can even exceed the average performance of SAC under certain random seeds\"- it's not a particularly fair comparison to suggest that one algorithm on specific random seeds can outperform the *average* performance of another algorithm. That said, the Figure showing this comparison (7b) doesn't appear statistically significant and it's unclear whether this conclusion can be drawn from it.", "summary_of_the_paper": "The paper proposes to set up experience replay such that transitions are emphasized based on whether the AN2N algorithm decides to explore more, and how recent the transition was experienced. It is based on observations that the state distribution can dramatically drift over the course of policy improvement, and the intuition that it might be better to perform updates on states that the current policy is more likely to actually visit (recent transitions). They apply the proposed sampling method, *experience replay more* (ERM), to a variety of replay-based deep RL algorithms and evaluate them empirically.", "summary_of_the_review": "In light of the above concerns surrounding writing clarity, missing notable baselines, and questionable empirical methodology, I'm recommending strong rejection at this time. I believe a substantial rewrite and significantly more evaluation is needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8W092kZvNHp", "reviewer": "Reviewer_Hemq", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Experience replay is a really important component of many RL algorithms right now. The importance of the distribution of replay states has been convincingly demonstrated in prior work. So the topic of this paper is definitely relevant to the ICLR audience.\n\nOne of my big concerns is that the paper is poorly written. The grammar, spelling, and word choice are all sufficiently flaws to, at times, obscure meaning. Even beyond that, the substance of the paper is not communicated clearly. While I understand that the idea of the paper is to more frequently replay states that are more important in some way, the paper does not communicate how those states are identified or what makes them more important. While the paper does a reasonable job of motivating prioritized replay in general, it does not clearly motivate or justify its specific approach.\n\nThe biggest clarity problem comes from the fact that ERM relies heavily on the AN2N algorithm, which has not been peer-reviewed and has only been available on arXiv for literally a few days at time of reviewing. Given this, the AN2N algorithm needs to be far more precisely described and justified. From the text of this paper, I have no idea how AN2N works or why I should believe that it is a good base to build on. This is critical, because the ERM algorithm makes its decisions based on the output of the AN2N algorithm. Without a clear description and justification of AN2N, the description and justification of ERM is also incomplete. Even if the AN2N paper were more clearly written and its hypotheses clearly evaluated (from a brief skim this does not seem to be the case), it is important that *this* paper be sufficiently self-contained that the reader can understand the significance of the contributions.\n\nHere is a non-exhausted list of some other clarity concerns I encountered:\n- Why is Hafner et al. the citation for model-based RL? MBRL predates 2019 by a long shot.\n- I don't think it is correct to claim that off-policy model-free learning is categorically more sample efficient than on-policy learning. There are things you can do with off-policy learning that you can't really (justifiable) do with on-policy learning, like experience replay, that can improve sample efficiency. However, the same is true in the reverse; for instance eligibility traces are far more sensible in the on-policy case.\n- Page 2: I didn't understand the logical step between discussing (prioritized) experience replay and meta-RL.\n- Page 2: I don't understand what the footnotes are meant to communicate.\n- Page 3/4: I don't understand Figure 1.\n  - Isn't HalfCheetah continuous in state? So what do s0, s1, etc. represent?\n  - Each line on the plot is associated with a particular Q-value (I think??), so what does the y-axis represent? The caption says that it is \"the specific value of the state.\"\n  - The caption says that \"the difference is obvious at states s1, s8, and s12.\" I don't know what I'm supposed to be comparing in order to see a difference. And I don't see why those three states are special compared to any other states.\n  - The main text concludes that \"the state is also changing.\" I don't know what \"the state\"  refers to here, or what change is being discussed.\n\nMy other major concern is that the conclusions are far too strong for the experimental evidence presented. Only five independent trials are run for each algorithm and the error bars are quite large. I couldn't find the explanation of what the error bars actually represent (maybe I missed it), but assuming that it is something like the sample standard deviation the differences in performance are well within the noise in all but a couple of experiments. It is true that across all of the experiments there is a tendency for the ERM curve to be slightly above the non-ERM curve, which lends some credibility to the hypothesis that ERM is an improvement in at least some of these experiments, but even then, the improvement is *very* modest.\n\nFurthermore, the empirical results do not test the central hypotheses of the paper. The paper claims that ERM's (modest) success is a result of specific effects, namely replaying more important/relevant states. It's important to evaluate this hypothesis, for instance by performing ERM but without some of its key features (e.g. without prioritization of recent experience or without the prioritization of \"key\" states). I also note that the comparison is always between ERM and the base RL algorithm. Since ERM is built directly on top of AN2N, it seems critical to include results using AN2N without the additional enhancements of ERM in order to understand the significance of the contributions of both algorithms.", "summary_of_the_paper": "The paper presents Experience Replay More, an algorithm that prioritizes certain transitions in a replay buffer for replay. The approach is combined with multiple base RL algorithms in four MuJoCo domains.", "summary_of_the_review": "While investigating an important problem, the paper is poorly written, the main contribution is not adequately described or evaluated, and the empirical findings that are presented to not convincingly support the conclusions. I recommend that this paper be rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ejhJDQFAIL", "reviewer": "Reviewer_VS76", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The paper motivates why the \"key\" states should be sampled more frequently.\n\n* The proposed sampling scheme is simple to implement, and can be incorporated into many existing offline RL algorithms.\n\nWeaknesses:\n\n* This paper proposes a different sampling scheme than uniform sampling. However, in the experiments, no other sampling schemes are compared at all. The only baseline is uniform sampling. Even though the paper mentioned prioritized experience replay in the Introduction, it is not considered a baseline in the paper. \n* The proposed sampling scheme (ERM) is built upon AN2N (Add Noise to Noise). Therefore, there are at least two variants compared to normal sampling schemes: (1) add more noise to actions following AN2N during exploration (2) sample key states and recent states more frequently. In the experiments, we only see the comparison of normal RL algorithms and RL+ERM (for example, DDPG vs DDPG+ERM, SAC vs SAC+ERM). It is unclear whether the performance improvement (if any) comes from AN2N or AN2N + ERM. To answer this, the authors should include the learning curves of DDGP+AN2N, TD3+AN2N, SAC+AN2N.\n* From Figure 4,5,6, we can see the benefits of using ERM are minor. The proposed sampling scheme does not lead to better performance in most cases.\n* The paper lacks the discussions on how sensitive the proposed sampling scheme is to the extra hyperparameters that it introduces. For example, does $Prt_{AN2N}$ and $K_t$ need to be tuned a lot when the environments change? The paper should include some ablation studies to show how the performance changes as one changes these hyperparameters.\n* The most challenging task in the paper is `Walker2D`. How about other more difficult benchmarking tasks such as Humanoid?\n* The writing of the paper needs to be improved substantially. Grammar issues should at least be fixed.", "summary_of_the_paper": "The paper proposes to change the sampling order in the replay buffer by prioritizing sampling the \"key\" states and most recently generated states. \"Key\" states are the states where the agent performs poorly, measured by the return value.", "summary_of_the_review": "The paper needs more baselines, ablations, benchmarking tasks to justify the effectiveness of the proposed sampling scheme.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting way of prioritizing samples in replay that is compatible with many RL methods. It is evaluated experimentally on different tasks and with different RL algorithms.

The reviewers highly appreciated the revised paper and the detailed replies and discussions.
While this iteration improved the paper substantially, it is still not ready for publication in its current form. In particular:
- The paper is still not self-contained enough
- The reviewers are still not convinced about the statistical significance
- More tasks should be added
- PER needs to be added as a baseline
The authors promised those changes for the final version, but those are so substantive that the paper will need to go thorough another complete review cycle. Hence, we'd like to encourage the authors to re-submit at a different venue.

P.S.: Careful with double-blind submissions, acknowledgements should not be included.
Decision: Reject