Learning to Sample with Local and Global Contexts  in Experience Replay Buffer
Authors: Youngmin Oh
Kimin Lee
Jinwoo Shin
Eunho Yang
Sung Ju Hwang
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
experience replay buffer
off-policy RL

Abstract: Experience replay, which enables the agents to remember and reuse experience from the past, has played a significant role in the success of off-policy reinforcement learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metrics (e.g. TD-error). However, they may result in sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, we aim to address the issue by proposing a new learning-based sampling method that can compute the relative importance of transition. To this end, we design a novel permutation-equivariant neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) as inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS), on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due to sampling diverse and meaningful transitions by NERS that considers both local and global contexts. 

Published as a conference paper at ICLR 2021
LEARNING TO SAMPLE WITH LOCAL AND GLOBAL CONTEXTS FROM EXPERIENCE REPLAY BUFFERS
Youngmin Oh', Kimin Jinwoo Shin', Eunho Yang Sung Ju Hwang
1 Samsung Advanced Institute of Technology
2 University of California, Berkeley
3 Korea Advanced Institute of Science and Technology
4 AITRICS
youngminO oh sasuugg. com, www.:./wwiinnnneeeeeeee..ee.eeee [ji [wios,,, eunhoy, s sJhwang32)@kaisttaarm
ABSTRACT
Experience replay, which enables the agents remember and reuse experience from the past, has played significant role in the success of learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metries (e.g. TD-error). However, they may result sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, aim to address the issue by proposing a new learning-basee sampling method that can compute the relative importance of transition. To this end, we design novel permutation--qqiiaatie neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS)', on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due sampling diverse and meaningful transitions by NERS that considers both local and global contexts.
reinforcement
INTRODUCTION
Experience replay (Mnih et al., 2015), which is memory that stores the past experiences to reuse them, has become a popular mechanism for reinforcement learning (RL), since it stabilizes training and improves the sample efficiency. The success of various off off-policy RL algorithms largely attributes to the use of experience replay (Fujimoto et al., 2018; Haarnoja et al., 8a;b; Lillicrap et al., 2016; Mnih et al., 2015). However, most off off-policy RL algorithms usually adopt unique random sampling (Fujimoto et al., 2018; Haarnoja et al., 201 Sa; Mnih et al., 2015), which treats all past experiences equally, so it is questionable whether this simple strategy would always sample the most effective experiences for the agents to learn.
Several sampling policies have been proposed address this issue. One of the popular directions is to develop rule-based methods, which prioritize the experiences with pre-defined metrics (Isele & Cosgun, 2018; Jaderberg al., 2016; Novati & Koumoutsakos, 2019; Schaul et al., 2016). Notably, since TD-error based sampling has improved the performance of various off-policy RL algorithms (Hessel et al., 2018; Schaul et al., 2016) by prioritizing more meaningful samples, i.e., high TD-error, it is one of the most frequently used rule-based methods. Here, TD-error measures how unexpected the returns are from the current value estimates (Schaul al., 2016).
However, such rule-based sampling strategies can lead to sampling highly biased experiences. For instance, Figure shows randomly selected 10 transitions among 64 transitions sampled using certain
Code available http://ghhhh...ommyyyggmnoooo

---

Published as a conference paper at ICLR 2021
(a) TD-error
(b) Q-value
(c) RANDOM
Figure 1: Sampled transitions on Pendulum-vO from various sampling strategies: (a) Sampling by TD-error, (b) Sampling by Q-value, (c) Sampling uniformly at random. Samples highlighted in black, orange, and cyan boxes denote that their state has the rod in downward, upright, and horizontal positions with appropriate amount of actions, respectively, Samples in red boxes have excessively large actions.
metrics/rules under a policy-based learning, soft actor critic (SAC) (Haarnoja et al., 2018a), on Pendulum-v0 after 30,000 timesteps, which goal is to balance the pendulum to make it stay in the upright position. We observe that sampling by the TD-error alone mostly selects initial transitions (see Figure (()),, where the rods are in the downward position, since is difficult to estimate Q-value on them. Conversely, the sampled transitions by Q-value describe rods in the upright position (see Figure (b)), which will provide high returns to agents. Both can largely contribute to the update of the actor and critic since the advantage term and mean-square of TD-errors are large. Yet, due the bias, the agent trained in such manner will mostly learn what to do in a specific state, but will not learn about others that should be experienced for proper learning of the agent. Therefore, such biased (and redundant) transitions may not lead to increased sample efficiency, even though each sampled transition may be individually meaningful.
On the other hand, focusing only on the diversity of samples also has an issue. For instance, sampling uniformly at random is able to select out diverse transitions including intermediate states such as those in the red boxes of Figure Ic)), where the rods are in the horizontal positions which are necessary for training the agents as they provide the trajectory between the two types of states. However, the transitions occasionally irrelevant for training both the policy and the Q networks. Indeed, states in the red boxes of Figure 1(c) possess both low Q-values and TD-errors. Their low TD-errors suggest that they are not meaningful for the update of Q networks. Similarly, low Q-values cannot be used to train the policy what good actions are.
Motivated by the aforementioned observations, we aim develop method sample both diverse and meaningful transitions. To cache both of them, it is crucial to measure the relative importance among sampled transitions since the diversity should be considered in them, not all in the buffer. To this end, we propose a novel neural sampling policy, which we refer to Neural Experience Replay Sampler (NERS). Our method learns measure the relative importance among sampled transitions by extracting local and global contexts from each of them and all sampled ones, respectively. In particular, NERS is designed to take set of each experience's 's features as input and compute its outputs in an equivariant manner with respect to the permutation of the set. Here, we consider various features of transition such as TD-error, Q-value and the raw transition, e.g., expecting to sample intermediate transitions as those in blue boxes of Figure I(c) efficiently.
To verify the effectiveness of NERS, we validate the experience replay with various off-policy RL algorithms such as soft actor-critic (SAC) (Haarnoja et al., 2018a) and twin delayed deep deterministic (TD3) (Fujimoto et al., 2018) for continuous control tasks (Brockman et al., 2016; Todorov et al., 2012), and Rainbow (Hessel et al., 2018) for discontinuous control tasks (Bellemare et al., 2013). Our experimental results show that NERS consistently (and often significantly for complex tasks having high-dimensional state and action spaces) outperforms both the existing the rule-based (Schaul et al., 2016) and learning-basse based (Zha et al., 2019) sampling methods for experience replay.
In summary, contribution is threefold:
To the best of our knowledge, we first investigate the relative importance of sampled transitions for the efficient design of experience replays.
For the purpose, we design a novel permutation -euttiioin-uvvviiintt neural sampling architecture that utilizes contexts from the individual (local) and the collective (global) transitions with various features to sample not only meaningful but also diverse experiences.
We validate the effectiveness of our neural experience replay diverse continuous and discrete control tasks with various off policy RL algorithms, on which it consistently outperforms both existing rule- and learning-based sampling methods.
2

---

Published as a conference paper at ICLR 2021
2 NEURAL EXPERIENCE REPLAY SAMPLER
We consider a standard reinforcement learning (RL) framework, where an agent interacts with an environment over discrete timesteps. Formally, at each timestep t, the agent receives state 3; from the environment and selects an action a, based on its policy л. Then, the environment returns a reward rt, and the agent transitions to the next state St+1- The goal of the agent is to learn the policy ı that maximizes the return R. = >k=n 7 PTrtkk which is the discounted cumulative reward from the timestep with discount factor y (0, 1), at each state 8t- Throughout this section, we focus on off-policy actor-critic RL algorithms with a buffer B, which consist of the policy (a|s))) (s) (i.e., actor) and Q Q-function (s a) (i.e., critic) with parameters to and 0, respectively.
2.1 OVERVIEW OF NERS
We propose novel neural sampling policy f with parameter ø, called Neural Experience Replay Sampler (NERS). It trained for learning select important transitions from the experience replay buffer for maximizing the actual cumulative rewards. Specifically, at each timestep, NERS receives a set of off- off-policy transitions' features, which are proportionally sampled in the buffer B based on priorities evaluated in previous timesteps. Then it outputs set of new scores from the set, in order for the priorities to be updated. Further, both the sampled transitions and scores are used to optimize the off-policy policy (als) and action-value function Qa a.). Note that the output of NERS should be equivariant of the permutation of the set, so we design its neural architecture to satisfy the property. Next, define the reward yT as the actual performance gain, which is defined as the difference of the expectation of the sum of rewards between the current and previous evaluation policies, respectively. Figure 2 shows overview of the proposed framework, which learns sample from the experience replay. In the following section, we describe our method of learning the sampling policy for experience replay and the proposed network architecture in detail.
NERS
Enviroment:
Local Contist
Conratonate
Calculate
Sampled tions
Festurts
Weights
State fura Reward +
Action a
ft
Peatures
ft
A gent
Training Batch
Fealures
Store
Score
Festanee
Network
Buffer Ø
Sample
Context
Global
Global Network
Score Network
Figure 2: An overview of our neural experience replay sampler (NERS) framework. We first sample transitions proportionally to scores previously calculated. Then, our neural sampling policy evaluates them. Specifically, NERS consists three networks fi, fg and fa. The first two networks obtain local global contexts by considering various features, respectively. Then last network evaluates relative importance (score) by f.- The importance set is used when to sample transitions later and train the agent. This design satisfies the permutation-cquuivaraantt property.
2.2 DETAILED COMPONENTS OF NERS
... n) by [n] for positive integer n. Without loss of generality, suppose that the replay buffer B stores the following information as its i-th transition B, = (*x())) (x((), (x(()) 8x(()+1) where (i) is function from the index of B to corresponding timestep. We
Input observations. Throughout this paper, we denote the set (1.
set of priorities PB { o [B) } that is updated whenever sampling transitions for training the actor and critic. One can sample an index set in [[]]] with the probability pa of i-th transition as follows:
P:
ExE(IBI of
(1)
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Training NERS: batch size and sample size
Initialize NERS parameters Ø, a replay buffer B t- 0. priority set Pg +- and index set +- for each timestep do
Choose de from the actor and collect a sample (st, az, Tt, St41) from the environment
Update replay buffer <- B U {(St, (t41)) and priority set Ps +- Ps U {1.0}
for each gradient step do
Sample an index / by the given set Pa and Eq. (1) with |/|
Calculate a score set {o ker and weights (waJeer by Eq. (4) and Eq. (5), respectively Train the actor and critic using batch (B, Jier C B and corresponding weights {ws) her
Collect I + ZUI I and update Ps (I) by the score set {ow)ker
end for
for the end of an episode do
Choose subset Irrai from I uniformly at random such that [uriinl
Calculate yTe as in Eq. (6)
Update sampling policy using the gradient (7) with respect Itrain
Empty I, i.e., I t-
end for end for
with hyper- syper-parameter a > 0. Then, we define the following sequence of features for (Bi)ier:
D (B, 1) = (x(s)) ax(s); Tw(s)) #x(0)+1; xd(i,, Sx()) Tw(() + max Qa (#x(s) + a) tEx"
(2)
where y is discount factor, 0 is the target network parameter, and (x(() is the TD-error defined as follows:
5x(1) (m(() + max Q3 (8x(0)1, Qo (Sw(0) (n())
The TD-erron indicates how 'surprising' or 'unexpected" the transition is (Schaul et al., 2016). Note that the imput D (B, I) contains various features including both exact values (i.e., states, actions, rewards, next states, and timesteps) and predicted values in the long-term perspective (i.e., TD-emrrs and Q-values). We abbreviate the notation D (B, I) = D (7) for simplicity. Utilizing various information crucial in selecting diverse and important transitions (see Section 3).
Architecture and action spaces. Now we explain the neural network structure of NERS f. Basically, takes D (I) as an input and generate their scores, where these values are used sample transitions proportionally. Specifically, f consists of fi, fg. and fa called learnable local, global and score networks with output dimensions di, dg. and The local network is used to capture attributes in each transition by fi (D (T)) {f, (D (T)), ... fr, (D (T))} E R where fi, (D (T)) E R4 (Ie E [( The global network is used to aggregate collective information of transitions by taking fy avg (D (1)) = EL.(D())
!!| E Igl,d, where fy (D (4)) E gl4xx,, Then by concatenating them,
one can make input for the score network Sa as follows:
D cat (I) = (f., (T) @ fg ave (D()), fun (D( fg 27g (D (())) e R [lx((++))
(3)
where @ denotes concatenation. Finally, the score network generates score set:
fa (D)at(t)) = (ather E R!!
(4)
One can easily observe that f. is permutation-equivanee used to update the priorities set P for transitions corresponding to / by Eq. (1) and to compute importance-sampling weights for updating the critic, compensating the bias of probabilities (Schaul al., 2016)):
with respect to input D (T). The set (o:)er
Blp(t)
where > 0 is hyper- parem-earameter. Then the agent and critic receive training batch D (I) and corresponding weights {w: [weher for training, i.c., the learning rate for training sample B, is set to be proportional to wi. Due to this structure satisfying the permutation-equiiarian property, one
4

---

Published as a conference paper at ICLR 2021
evaluate the relative importance of each transition by observing not only itself but also other transitions.
Reward function and optimizing sampling policy. We update NERS each evaluation step. To optimize our sampling policy, we define the replay reward , the current evaluation as follows: for policies and x' used in the current and previous evaluations in (Zha et al., 2019),
Er
Σ
Tt
Ex
Σ
Tt
(6)
AEE (timestteps in an episode)
AE (timesteps in an episode)
The replay reward is interpreted as measuring how much actions of the sampling policy help the learning the agent for each episode. Notice that y*a only observes the difference of the mean of cumulative rewards between the current and previous evaluation policies since NERS needs to choose transitions without knowing which samples will be added and how well agents will be trained in the future. To maximize the sample efficiency for learning the agent's policy, we propose to train the sampling policy to selects past transitions in order to maximize - To train NERS, one can choose Irain that is subset of index set I for totally sampled transitions in the current episode. Then we use the following formula by REINFORCE (Williams, 1992):
Bi Sraiio
R
Jreain
Σ iE
V lo (D (Terasn))
where p; defined in Eq. (1). The detailed description is provided in Algorithm 1.
While ERO Zha et al. (2019) uses similar replay-reward (Eq. 6), there number of fundamental differences between and our method. First all, ERO does not consider the relative importance between the transitions NERS does, but rather learns an individual sampling rate for each transition. Moreover, they consider only three types of features, namely TD-error, reward, and the timestep, while NERS considers larger set of features by considering more informative features that are not used by ERO, such as raw features, Q-values, and actions. However, the most important difference between the two that ERO performs two-stage sampling, where they first sample with the individually learned Bernoulli sampling probability for each transition, and further perform random sampling from the subset of sampled transitions. However, with such strategy, the first-stage sampling is highly inefficient even with moderate size experience replays, since it should compute the sampling rate for each individual instance. Accordingly, its time complexity of the first-stage sampling depends finally on the capacity of the buffer B3 i.e., O (lB)). On the contrary, NERS uses sum-tree structure in (Schaul et al., 2016) to sample transitions with priorities, so that its time complexity for sampling depends highly on O (log) B.). Secondly, since the number of experiences selected from the first stage sampling is large, it may have little or no effect, making it to behave similarly to random sampling. Moreover, ERO updates its network with the replay reward and experiences that are not sampled from two-stage samplings but sampled by the uniform sampling at random (see Algorithm 2 in Zha al. (2019)). In other words, samples that are never selected affect the training of ERO, while NERS updates its network solely based on the transitions that are actually selected by itself.
EXPERIMENTS
In this section, we conduct experiments to answer the following questions:
Can the proposed sampling method improve the performances of various off-policy RL algorithms for both continuous and discrete control tasks?
Is it really effective to sample diverse and meaningful samples by considering the relative importance with various contexts?
3.1 EXPERIMENTAL SETUP
Environments. In this section, we measure the performances of off-policy RL algorithms opti- mized with various sampling methods the following standard continuous control environments
5

---

Published as a conference paper at ICLR 2021
200
400
400
NERS
175
RANDOM
300
300
200
Total Rweerdss
125
ERO
Total flawerds
100
Total Rewards
200
100-
100
-100
-300
0
0.03
0.02
0.03
0.04
0.05
-400
0.025
0:05
0.025 0.1
0.12
0.15
TimeStops(M)
SimeSteps(M)
TimeStoas(K)
(a) Pendulum (TD3)
(b) Lumarl.nderr (TD3)
(c) Bipeda/ (TD3)
175
300
150-
200
Total Rewards
Total Rewards
75
-100
50
200
25
-100
0
0.01
0.02
0.03
0.04
0.05
0.025
0.15
-200
0.05
0.1
0.15
0.2
0.25
0.3
limeSteos(M))
TimeStepsiMi
TImeStepx(M0)
(d) Pendulum (SAC)
(e) LunarLander-v2 (SAC)
(l) Bipedal (SAC)
Figure 3: Learning curves of off-policy RL algorithms with various sampling methods classical and Box2D continuous control tasks. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs with random seeds.
with simulated robots (e.g., Ant-v3, Walker2D-v3, and Hopper-v3) from the MuJoCo physics en- gine (Todorov et al., 2012) and classical and Box2D continuous control tasks (i.c., Pendulum*2 Lunarl.anderContiuuuus v2, and SipedalWallerrs from OpenAl Gym (Brockman al., 2016). We also consider subset of the Atari games (Bellemare et al., 2013) to validate the effect of our experi- ence sampler on the discrete control tasks (see Table 2). The detailed description for environments explained in supplementary material.
Off-policy RL algorithms. We apply our sampling policy to state-of-the-art off-policy RL algo- rithms, such Twin delayed deep deterministic (TD3) (Fujimoto et al., 2018), and soft actor-critic (SAC) (Haarnoja et al., 20), for continuous control tasks. For discrete control tasks, instead the canonical Rainbow (Hessel al., 2018), Hasselt et al., 2019). Notice that Rainbow already adopts PER. To compare sampling methods, we replaced it by NERS, RANDOM, and ERO in Rainbow, respectively. Due to space limitation, provide more experimental details in the supplementary material.
data-efficient variant of it introduced in (van
Baselines. We compare our neural experience replay sampler (NERS) with the following baselines:
RANDOM: Sampling transitions uniformly at random.
PER (Prioritized Experience Replay): Rule-based sampling of the transitions with high temporal difference errors (TD-errors) (Schaul et al., 2016)
ERO (Experience Replay Optimization): Learning-baat based sampling method (Zha et al., 2019), which computes the sampling score for each transition independently. using TD-error, timestep, and reward as features.
3.2 COMPARATIVE EVALUATION
Figure 3 shows learning curves of each off-policy RL algorithm during training on classical and Box2D continuous control tasks, respectively. Furthermore, Table I and Table 2 show the mean of cumulative rewards on MuJoCo and Atari environments after 500,000 and 100,000 training steps,
PPenduuum" We slightly modify the original Pendulum that openAl Gym supports to distinguishing performances of sampling methods more clearly by making rewards sparser. Its detailed description is given in the supplementary material.

---

Published as a conference paper at ICLR 2021
Environments
NERS
RANDOM
PER
ERO
TD3
Ant-v3
4193.69 (+677.81)
2824.42
2723.57
3515.88
Walker2D-v3 3938.59 (+378.62)
3559.97
2797.86
3394.7 74
Hopper-v3
3062.32 (+451.92)
2152.82
1693.32
2610.40
SAC
Ant-v3
2913.54 (+678.20)
2235.34
1402.67
1844.99
Walker2D-v3
3720.11 (+323.12)
3035.31
3396.99
1057.61
Hopper-v3
2763.69 (+354.18)
2409.51
2223.08
2255.67
Table I: Average of cumulative rewards under SAC and TD3 on MuJoCo Environments after 500,000 training steps across five instances with random seeds. Bold values represent the highest results, and the number in a bracket indicates the improvement due to NERS, compared to that of the best baseline on each environment.
Environments
NERS
RANDOM
PER
ERO
Alien
1125.18 18 (+281.16)
787.49
844.02
810.41
Amidar
167.50 (+18.85)
148.65
125.23
118.30
Assualt
516.52 (+26.46)
490.06
466.06
463.39
Asterix
679.00 (+80.90)
598.10
587.20
534.89
BattleZone
18584.99 (+1500.19)
14643.43
13870.98
17084.80
Boxing
2.51 (+4.83)
-3.07
-2.32
-3.25
ChopperCommand
878.73 (+125.87)
696.34
752.86
727.45
Freeway
28.96 (+0.12)
28.09
28.37
28.84
Frosthite
1707.10 (+280.88)
794.50
1426.22
832.45
KungFuMaster
10925.59 (+2971.23)
7215.50
7527.96
7954.36
MsPacman
1579.27 (+432.45)
1070.19
1146.82
1001.87
Pong
-1836 (+0.26)
18.62
-1.08
-18.76
PrivateEye
91.68 (+13.16)
69.84
78.52
56.13
Qbert
1037.58 (+136.34)
824.15
901.24
895.49
RoadRunner
9689.30 (+2596.17)
6382.82
7093.13
6199.88
Seaquest
386.80 (+30.52)
356.28
343.97
338.79
Table 2: Average of cumulative rewards under Rainbow each Atari environments after 100,000 training steps across five instances. Bold values represent the highest results, and the number in bracket indicates the improvement due to NERS, compared that of the best baseline each environment.
respectively, over five runs with random seeds, respectively." We observe that NERS consistently outperforms baseline sampling methods in all tested cases. In particular, It significantly improves the performance of all off-policy RL algorithms on various tasks, which come with high-dimensional state and action spaces. These results imply that sampling good off- data is crucial in improving the performance of off-policy RL algorithms. Furthermore, they demonstrate the effectiveness of our method for both continuous and discrete control tasks, it obtains significant performance gains on both types of tasks. On the other hand, we observe that PER, which is the rule-based sampling method, often shows on these continuous control tasks, similarly as observed in (Zha et al., 2019). We suspect that this is because PER more appropriate for Q-learning based algorithms than for policy-based learning. since TD-errors are used to update the Q network. Moreover, even though ERO is learning- based sampling method, its performance and sampling behavior is close that RANDOM, due two reasons. First, it considers the importance of each transition individually by assuming the Bernoulli distribution, which may result in sampling of redundant transitions. Second, ERO performs two-stage sampling, where the transitions are first sampled due to their individual importance, and then further randomly sampled to construct batch. However, since too many transitions are sampled in the first stage, the second-stage random sampling is similar to random sampling from the entire experience replay.
performance than uniform random sampling (i.e., RANDOM)
'1. learning curves for each environment are provided in the supplementary material.
7

---

Published as a conference paper at ICLR 2021
600
400
400
NERS
NERS
500
NERS with Few Features
300
KANDOM PFR
300
400
NFRS without Global Context
200
Total Rewards
NERS
-100
-00
-100
-100
-200
-400
0.05
6.1
0.15
0.2
0.25
0.3
0.05
0.1
0.15
0.2
0.25
0.3
TimeStepa(50)
TimeStepatMM)
limeSteps(Mi
(a) BipedalWalker -v3 (SAC)
(b) BipedalWalker-v3 (SAC)
LunarLander--2 (SAC)
Figure 4: (a): Comparison of NERS and variants of NERS only with few features (reward, TD-error, and timestep) and without global context across five instances with random seeds, respectively. (b)-(c): Learning curves under SAC over five instances with random seeds Here, NERS* denotes that variant NERS, where it trained by difference cumulative rewards from each training episode. Any significant difference between NERS and NERS not observable.
five instances with random seeds, respectively.
60
500
400
NERS
30
RANDOM
350
40
PER
Average of Cooelves
300
ERO
30
230
20
200
Average
200
150
100
50
0
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
TimeStepo(M)
TimeStepsiMi
TimeSteps(M)
(a) Bipedal (SAC)
(b) Ant-v3 (SAC)
(c) Walker2D-v3 (SAC)
NERS
RANDOM
Averate of LoR(To
ERO
4.0
3.55
3.0
2.5
0.2
%
0.4
0.5
2.0
0.2
TimeSteps(M)
TimeStepstMil
Timestepps(M)
(d) Bipedal (SAC)
(c) Ant-v3 (SAC)
(f) Walker2D-v3 (SAC)
Figure Curves of the average Q-values (a/b/c) and Log(TDDerrors (d/e/f) of sampled transitions across five instances with random seeds, respectively. One can show that NERS basically selects transitions with high TD-errors in the beginning and both high TD-errors and Q-values finally.
3.3 ANALYSIS OF OUR FRAMEWORK
In this subsection, we first show that each component of NERS is crucial to improve sample efficiency (Figure 4). Next, we show that NERS really samples not only diverse but also meaningful transitions to update both actor and critic (Figure 5).
Contribution by each component. We analyze NERS to better understand the effect of each com- ponent. Figure 4(a) validates the contributions of our suggested techniques, where one observe that the performance of NERS is significantly improved when using the full set of features. This implies that essential transitions for training be sampled only by considering various aspects of the past experiences. Using only few features such as reward, TD-error, and timestep does not result in sampling transitions that yield high expected returns in the future. Figure 4(a) also shows the effect
8

---

Published as a conference paper at ICLR 2021
Method
STDEV of TD-errors
STDEV of Q-values
AVG errrrss
AVG of Q-values
NERS 723,01
65.22
87.54
-104.13
RANDOM 528.76
60.46
62.43
-120.15
PER
1256.71
49.78
139.49
-138.05
ERO
560.56
59.16
65.44
-119.03
Table 3: Sampled transitions' statistical values for Q-values and TD-errors on Pendulum-vD under SAC at 10,000 training steps with initially 1,000 random actions. Here, STDEV and AVG mean the standard deviation and the average, respectively. PER has the highest STDEV of TD-errors but lowest STDEV of Qerrros. NERS has higher STDEV of both TD-errors and Qvalues than RANDOM and ERO.
of the relative importance by comparing NERS with and without considering the global context; we found that the sample efficiency is significantly improved due to consideration of the relative impor- tance among sampled transitions, via learning the global context. Furthermore, although we have considered standard environments where evaluations are free, if there exists an environment where the total number of evaluations is restricted, it may be hard to calculate the replay reward in (6) since cumulative rewards at each evaluation should be computed. Due to this reason, we consider variance of NERS (NERS*) which computes the difference of cumulative rewards in not evaluations but training episodes. Figure 4(b) and Figure 4(c) show the performance of NERS* compared to NERS and other sampling methods under BipedalWalker-v3 and LunearlaaderCooninuuus respectively. These figures show that the performance between the two types of replay rewards is not significantly different.
Analysis statisties of sampled transitions. We now check if NERS samples both meaningful and diverse transitions by examining how its sampling behavior changes during the training. To this end, we plot the TD-errors and Q-values for the sampled transitions during training BipedalWalker-v3, Ant-v3, and Walker2D-v3 under SAC in Figure 5. We can observe that NERS learns to focus on sampling transitions with high TD-errors in the early training steps, while it samples transitions with both high TD-errors and Q-values (diverse) at later training iterations. In the early training steps, the critic network for value estimation may not be well trained, rendering the excessive learning of the agent to be harmful, and thus it is reasonable that NERS selects transitions with high TD-errors to focus on updating critic networks (Figure 5(d-f)), while it focuses both on transitions with both high Q-values and TD-errors since both the critic and the actor will be reliable in the later stage (Figure 5(a-c)). Such an adaptive sampling strategy is a unique trait of NERS that contributes to its success, while other sampling methods, such PER and ERO, cannot do so. Table 3 denotes the statistical values for sampled transitions' TD-errors and Q-values on Pendulum-v3 under SAC at 10,000 steps (with initially 1, 1,000 random actions). It is observable that NERS has higher standard deviation of Q-values and TD-errors than RANDOM and ERO. Although PER has the highest standard deviation of TD-errors than other sampling methods, it has the lowest standard deviation of Q-values instead. Figure and Table 3 show that NERS learns to sample diverse, which means the NERS's ability to sample transitions with different criteria, and meaningful experiences for agents.
4 RELATED WORK
Off-policy algorithms. One of the well-known off-policy algorithms is deep Q-network (DQN) learning with replay buffer (Mnih et al., 2015). There are various variants of the DQN learning, e.g.. (Hasselt, 2010; Wang et al.. 2015: Hessel et al., 2018). Especially, Rainbow (Hessel al.. 2018), which is one of the -tate-of-the-ar Q-learming algorithms, was proposed by combining various techniques to extend the original DQN learning. Moreover, DQN was combined with a policy-based learning, so that various actor-critic algorithms have appeared. For instance, an actor-critic algorithm, which is called deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016), specialized for continuous control tasks was proposed by combination of DPG (Silver et al., 2014) and deep Q-learning (Mnih et al., 2015). Since DDPG is easy to brittle for paper-parameters setting, various algorithms have been proposed to overcome this issue. For instance, to reduce the the overestimation of the Q- Q-value in DDPG, twin delayed DDPG (TD3) was proposed (Fujimoto et al., 2018), which extended DDPG by applying double nentwwrks, target policy smoothing, and different frequencies

---

Published as a conference paper at ICLR 2021
to update policy and Q-networks, respectively. Moreover, another actor-critic algorithm called soft actor-critic (SAC) (Haarnoja et al., 2018a;b) was developed by adding the entropy measure of an agent policy to the reward in the actor-critic algorithm to encourage the exploration of the agent.
Sampling method. Due to the case of applying random sampling, it has been used to various off -oolpyy algorithms until now. However, it is known that it cannot guarantee optimal results, that a prioritized experience replay (PER) (Schaul et al., 2016) that samples transitions proportionally to the TD-error in DQN learning was proposed. As result, it showed performance improvements in Atari environments. Applying PER is also easily applicable to various policy- algorithms, so is one of the most frequently used rule-based sampling methods (Hessel et al., 2018; Hou et al., 2017; Schaul et al., 2016; Wang & Ross, 2019). Furthermore, since it is reported that the newest experiences are significant for efficient Q-learning (Zhang & Sutton, 2015), PER generally imposes the maximum priority on recent transitions to sample them frequently. Based on PER, imposing weights for recent transitions was also suggested (Brittain et al., 2019) to increase priorities for them. Instead of TD-error, different metric can be also used to PER, e.g., the expected return (Isele & Cosgun, 2018; Jaderberg et al., 2016). Meanwhile, different approaches from PER have been proposed. For instance, to update the policy in trust region, computing the importance weight of each transitions was proposed (Novati & Koumoutsakos, 2019), so far-policy experiences were ignored when computing the gradient. Another example is backward updating of transitions from whole episode (Lee et al., 2019) for deep Q-learning. Although the rule-based methods have shown their effectiveness on some tasks, they sometimes derive sub-optimal results on other tasks. To overcome this issue, a neural network for replay buffer sampling was adopted (Zha et al., 2019) and it showed the validness of their method on some continuous control tasks in the DDPG algorithm. However, its effectiveness is arguable in other tasks and algorithms (see Section 3), as it only considers transitions independently and regard few features as timesteps, rewards, and TD-errors (unlike ours). Recently, Fedus et al. (2020) showed that increasing replay capacity and downweighting the oldest transition in the buffer generally improves the performance of Q-learning agents on Atari tasks. How to sample prior experiences is also crucial issue to model-based RL algorithms, e.g., Dyna Sutton (1991) which is a classical architecture. There are variants of Dyna that study strategies for search-control, to selects which states to simulate. For instance, inspired by the fact that tigh-freeuency space requires many samples to learn, Dyna-Vala Pan et al. (2019) and Dyna-Frequency Pan et al. (2020) select states with high-frequency hill climbing value function, and gradient and hessian norm of it, respectively for generating more samples by the models. In other words, how to prioritize transitions when sampling is nontrivial, and learning the optimal sampling strategy is critical for the chaple-efficienyy of the target off off-policy algorithm.
CONCLUSION
We proposed NERS, a neural policy network that learns how to select transitions in the replay buffer to maximize the return of the agent. It predicts the importance of each transition in relation to others in the memory, while utilizing local and global contexts from various features in the sampled transitions as inputs. We experimentally validate NERS on benchmark tasks for continuous and discrete control with various off-policy RI. methods, whose results show that it significantly improves the performance of existing off-policy algorithms, with significant gains over prior rule-based and based sampling policies. We further show through ablation studies that this success is indeed due to modeling relative importance with considcration of local and global contexts.
REFERENCES
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare Brittain, Josh Bertram, Xuxi Yang, and Peng Wei. Prioritized sequence experience replay. arXiv preprint 1905 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arxiv: 2016
10

---

Published as a conference paper at ICLR 2021
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals experience replay. arXiv preprint arXiv:2007.06700, 2020.
Scott Fujimoto, Herke van Hoof, David Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.0590,, 2018b.
Hado V Hasselt. Double q-learning. In NeurIPS, 2010.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen. A novel ddpg method with prioritized experience replay. In SMC, 2017.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI, 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6908 2014.
Su Young Lee, Choi Sungik, and Sae Sau-Young Chung. Sample-efficient deep reinforcement learning via episodic backward update. In NeurlPS, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Hrez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. In SMC, 2019.
Yangchen Pan, Hengshuai Yao, Amir-Massoud Farahmand, and Martha White. Hill climbing on value estimates for search-control in dyna. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp. 3209-3215. AAAI Press, 2019.
Yangchen Pan, Jincheng Mei, and Amir-massoud Farahmand. Frequency-hased search-control in dyna. arXiv preprint arXlv:2002.05822. 2020.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
Richard Sutton. Integrated modeling and control based on reinforcement learning and dynamic programming. Advances in neural information processing systems, pp. 471-478, 1991.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
11

---

Published as a conference paper at ICLR 2021
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When parametric models in reinforcement learning? In Advances Neural Information Processing Systems, pp. 14322-14333, 2019.
Che Wang and Keith Ross. Boosting soft actor-critic: Emphasizing recent experience without forgetting the past. arXiv preprint arXiv: M0Xiv:19060040000000 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 1..055, 2015.
Ronald J Williams. Simple statistical graddenttfollloinng algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 226, 1992.
Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. In IJCAI, 2019.
Shangtong Zhang and Richard Sutton. deeper look experience replay. In ICMR, 2015.
12

---

Published as a conference paper at ICLR 2021
Supplementary Material:
Learning to Sample with Local and Global Contexts in Experience Replay Buffer
A ENVIRONMENT DESCRIPTION
A.1 MUJOCO ENVIRONMENTS
Multi-Joint Dynamics with Contact (MuJoCo) Todorov et al. (2012) is a physics engine for robot simulations supported by openAI gym'. MuJoCo environments provide a robot with multiple joints and reinforcement learning (RI.) agents should control the joints (action) to achieve given goal. The observation each environment basically includes information about the angular velocity and position for those joints. In this paper, we consider the following environments belonging to MuJoCo.
Hopper(-v3) a environment to control a one-legged robot. The robot receives a high return if hops forward as soon as possible without failure.
Walker2d(-v3) is an environment make two-dimensional bipedal legs to walk. Learning to quick walking without failure ensures high return.
Ant(-v3) is an environment to control a creature robot with four legs used to move. RL agents should to learn how use four legs for moving forward quickly to get a high return.
(a) Hopper
(b) Walker2d
(c) Ant
Figure A.1: MuloCo environments
A.2 OTHER CONTINUOUS CONTROL ENVIRONMENTS
Although MuJoCo environments are popular to evaluate RL algorithms, openAl gym also supports additional continuous control environments which belong classic or Box2D simulators. We conduct experiments on the following environments among them.
Pendulum" is an environment which objective to balance pendulum in the upright position to get a high return. Each observation represents the angle and angular velocity. An action is joint effort which range [-2, 2]. Pendulum" is slightly modified from the original (Pendulum-vol -0)) which openAl supports. The only difference from the original is that agents receive reward 1.0 only if the rod sufficiently upright position (between the angle [-n n n/3, n x/3,, where the zero angle means that the rod is in completely upright position) at least more than 20 steps.
LunarLander Contdduuusn)))) is an environment to control a lander. The objective of the lander is landing pad, which is located at coordinates (0, 0), with safety and coming to rest as soon as possible. There penalty if the lander crashes or goes out of the screen. An action is about parameters to control engines of the lander.
"hans//ggymoppemaiccom
13

---

Published as a conference paper at ICLR 2021
3ipedal\allker(<x3) is environment control robot. The objective is to make the robot move forward far from the initial state as far as possible. An observation is information about hull angle speed, angular velocity, vertical speed, horizontal speed, and so An action consists of torque velocity control for two hips and two knees.
(a) Pendulum
(b) LunarLander
(c) BipedalWalke
Figure A.2: Other continuous control environments
Table A.1 describes the observation and action spaces and the maximum steps for each episode (horizon) in MuJoCo and other continuous control environments. Here, and [-1, 1] denote sets of real numbers and those between 0 and 1, respectively.
Environment
Observation space
Action space
Horizon
Hopper
IR
[-1, 1, 1]
1000
Walker2d
IR
[-1, 1, 1
1000
Ant
111
[-1, 1
1000
Pendulum
RR
[ 1, 1]
200
LunarL
RR
1, 1
1000
Bipeda/Walker TR
1, 1600
Table A.I: Dimensions observation and action spaces for continuous control environments
14

---

Published as a conference paper at ICLR 2021
A.3 DISCRETE CONTROL ENVIRONMENT
To evaluate sampling methods under Rainbow Hessel et al. (2018), we consider the following Atari environments. RL agents should learn their policy by observing the RGB screen acheive high scores for each game.
(laees/ooramsssip--4 is game where player should destroy all alien eggs in the RGB screen with escaping three aliens. The player has a weapon which paralyzes aliens.
is a game which format is similar MsPacman. RL agents control monkey in a fixed rectilinear lattice to eat pellets as much as possible while avoiding chasing masters. The monkey loses one life it contacts with monsters. The agents can go to the next stage by visiting certain location in the screen.
Minddr(NoFFrmeskiiin
game where RL agents control spaceship. The spaceship is able move on the bottom of the screen and shoot motherships which deploy smaller ships attack the agents. The objective is to eliminate the enemies.
ExverixNNoFrmmsskippe is game to control tornado. The objective of RL agents is to eat hamburgers the screen with avoiding dynamites.
100
(a) Alien
(b) Amidar
Assault
(d) Asterix
a tank combat game. This game provides first- first-person person perspective view. RL agents control tank to destroy other tanks. The agent should avoid other tanks missile attacks. It also possible to hide from various obstacles and avoid enemy attacks.
is game about the sport of boxing. There are boxers with top- down view and RL agents should control one of them. They get one point if their punches hit from a long distance and two points if their punches hit from a close range. match finished after two minues or 100 punches hitted to the opponent.
Doxinn(NoFrammskipa
is game control helicopter in desert. The helicopter should destroy all enemy aircrafts and helicopters while protecting a convoy of trucks.
ChopperCommande (oFFaaesiii
1:50
000
REEEK
(e) BattleZone
(f) Boxing
(g) ChopperCommand
SowaaaayNoFFaaaeekiei is game where RL agents control chickens to ten-lane highway with traffic. They are only allowed to move up or down. The objective to get across as possible as they until two minutes.
15

---

Published as a conference paper at ICLR 2021
is game control who should collect ice blocks make his igloo. The bottom two thirds of the screen consists of four rows of horizontal ice blocks. He can move from the current row to another and obtain ice block by jumping. RL agents required to collect 15 ice blocks while avoiding some opponents, e.g., crabs and birds.
is game to control fighter save his girl friend. He can two types of attacks (punch and kick) and move/crunch/jump actions.
Sainguu.asterrNNoFrmess
is a game where RL agents control paeman in given mazes for eatting pellets as much as possible while avoiding chasing masters. The pacman loses one life if contacts with monsters.
45'
PLIYER
10
15
(h) Freeway
(i) Frostbite
KungFuMaster
(k) MsPacman
Pong(Noorrameskip-re is game about table tennis. RL agents control in-game paddle to hit ball back and forth. The objective is to gain 11 points before the opponent. The agents earn each point when the opponent fails return the ball.
is a game mixing action, adventure, and memorizationm which control private eye. To solve five cases, the private eye should find and return items to suitable places.
( is game where RL agents control character under pyramid made of 28 cubes. The character should change the color of all cubes while avoiding obstacles and enemies.
a game control a roadrunner (chaparral bird). The roadrunner runs to the left on the road. RL agents should pick bird seeds while avoiding chasing coyote and obstacles such as cars.
Seaquest NoF NaFamessk)))r4) is game to control submarine to rescue divers. It can also attack enemies by missiles.
1000
Polle
Canco
(1) Pong
(m) PrivateEye
(n) Qbert
(o) RoadRunner
(p) Seaquest
16

---

Published as a conference paper at ICLR 2021
B TRAINING DETAILS
Parameter
Valuc
Shared
Batch size (continuous control environments)
128
Batch size (discrete control environments)
32
Buffer size
10
Target smoothing coefficient (r) for soft update
5 10-3
Initial prioritized experience replay buffer exponents (a, 8)
(0.5, 0.4)
Discount factor for the agent reward (7)
0.99
Number of initial random actions (continuous control environments)
10a
Number of initial random actions (discrete control environments)
6,0
Optimizer
Adam Kingma & Ba (2014)
Nonlincarity
ReLU
Observation down-same sampling for Atari RGB
84 84 with grey-scaling
CNN channels for Atari environments
32, 64
CNN filter size for Atari environments
5 x 5, 5 x 5,
CNN stride for Atari environments
ERO
Hidden units per layer Learning rate
64, 64 10-4
NERS
Hidden units per layer after flattening the output from CNNs Hidden units per layer in the local and global networks (fi and fg) Hidden units per layer in the score network (f.)
256, 64, 32 256, 512, 256, 128, 256, 128, 64 128 10-4
Sampling size to update NERS (n))
Learning rate
TD3
Hidden units per layer Learning rate
256, 256 5 10 2 0.1,0.2 0.5 Soft update with interval
Policy update frequency
Gaussian action and target noises
Target noise clip
Target network update
SAC
Hidden units per layer
256, 256
Learning rate
3 10-4
Target entropy
dim A (A is action space) Soft update with interval
Target network update
Rainbow
Action repetitions and Frame stack
4
Reward clipping
True ((-1, 1)) True 1.08 x 10
Terminal on loss of life
Max frames per episode
Target network update
Hard update (every 2,000 updates) 51
Support of Q-distribution for Adam optimizer
1.5 x 10
Learning Rate
10
Max gradient norm
10
Noisy nets parameter
0.1
Replay period every
1
Multi-step return length
20
Q-network's hidden units per layer
256
Table B.l: Hyper parameess
$ increases 1.0 the rule / 0.4n + 1.0(1 11), where 17 the current step/the maximum steps.
17

---

Published as a conference paper at ICLR 2021
5000
5000
4000
NERS
4000
RANDOM
4000
3500
Total Total Rewards
Total Total Rewards
Total Rewards
3000
3000
ERO
3000
2500
7000
2000
1500
1000
1000
500
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
(imeSteos(M)
TimeSteps(M)
TimeSteps(M)
(a) Ant (TD3)
(b) Walker2D (TD3)
(c) Hopper (TD3)
4000
5000
4000
3500
3500
4000
3000
3000
Total Rowards
Total Rowards
lotal Rewards
3000
2500
2000
2000
1500
2000
1500
1000
1000
500
0
0.1
0.2
0.3
0.4
0.5
0
0.1
0.2
0.3
0.4
0.5
0
0.1
0.2
0.3
0.4
0.5
SteostM)
Timeeteos(M)
TimeSteps(M)
(d) Ant (SAC)
(c) Walker2D (SAC)
(f) Hopper (SAC)
Figure C.I: Learning curves of off policy RL algorithms with various sampling methods on MuJoCo tasks. The solid line and shaded regions represent the mean and standard deviation, respectively, across five instances.
Table B provides paraeeeass which we used. We basically adopt parameters for Twin delayed DDPG (TD3) Fujimoto al. (2018) and Soft actor critic (SAC) Haarnoja et al. (2018a;b) in openAI baselines 6, Furthermore, we adopt parameters Rainbow as in van Hasselt et al. (2019) to make data efficient Rainbow for Atari environments. In the case of continuous control environments, we train five instances of TD3 and SAC, where they perform one evaluation rollout per the maximum steps. In the case of discrete control environments, we trained five instances of Rainbow, where they perform 10 evaluation rollouts per 1000 steps. During evaluations, we collect cumulative rewards to compute the replay reward yT=.
We follow the hyper- hyper-paramette parameters in van Hasselt et al. (2019) for prioritized experience replay (PER). We also use the typer-paramett Since NERS is interpreted as an extension of PER, it basically shares e.g., a and B. NERS uses various features, e.g., TD-errors and Q-values, but the newest samples have unknown Q-values and TD-errors before sampling them to update agents policy. Accordingly, we normalize Q-values and TD-errors by taking the hyperbolic tangent function and set 1.0 for the newest samples' TD-errors and Q-values. Furthermore, notice that NERS uses both current and next states in a transition as features, so that we adopt CNN-layers in NERS for Atari environments as in van Hasselt al. (2019). After flattening and reducing the output of the CNN-layers by FC-layers (256-64-32), we make a vector by concatenating the reduced output with the other features. Then the vector is input of both local and global networks fi and fg. In the case of ERO, it does not use states as features, so that CNN-layers are unneccesary.
for experience replay optimization (ERO) used in Zha et al. (2019).
in PER,
Our objective is not to achieve maximal performance but compare sampling methods. Accordingly. to evaluate sampling methods on Atari environments, we conduct experiments until 100,000 steps as in van Hasselt et al. (2019) although there room for better performance if more learning. In the case of continuous control environments, we conduct experiments until 500,000 steps.
C ADDITIONAL EXPERIMENTAL RESULTS
Figure C.I shows additional continuous control environments: Ant, Walker2d, and Hopper under TD3 and SAC, respectively. All tasks possess have high-dimensional observation and action spaces (see Table A. One can show that NERS outperforms other sampling methods at most cases. Moreover, one can observe that RANDOM and ERO have almost similar performance and PER could not show
18

---

Published as a conference paper at ICLR 2021
1003
200
100
500
1403
100
PER CRO
Benner
Tels Romands
200
200
200
100
0
TineStoosMM
TimeS0apsIM)
EmmaStppsMM)
TimeStaporM)
(a) Alien
(b) Amidar
(c) Assault
(d) Asterix
20000
Theel Secverde
Rewerds
2000
325
Timastapili
608
TemeStopaak0
202
0.02
5.00
0.03
0.2
0.22
0.04
0.56
0.00
(c) BattleZone
(f) Boxing
(g) ChopperCommand
(h) Freeway
2033-
15000
1.00
Tetel Dbwoorch
1932
Tooe
-20
3000
0.22
0.00
0.38
-22
TimeStepeh)
Teme5lapx201
Frostbite
KungFuMaster
(k) MsPacman
(I) Pong
2040
14000
1202
12000
400
Told Honuudss
1002
Totel Brwannss
002
Tale
200
-202
2000
-358
202
-2000
0.22
U.D4
0.50
0.5
0.52
(m) PrivateEye
(n) Qbert
(0) RoadRunner
(p) Seaquest
Figure C.2: Learning curves additional Atari environments under Rainbow
better performance to pasec RL algorithms compared to other sampling methods. Detailed learning curves of Rainbow for each environment observable in Figure C.2.
We believe that in spite of the effectiveness of PER under Rainbow, the poor performance of PER under policy-based RL algorithms results from that it is specialized update Qnnwwtorks, that the actor networks cannot be efficiently trained.
One observe that there high variances in some environments, Indeed, it is known that learning more about environments in Figure C. and Figure C.2 improves performance of algorithms. However, focus is not obtain the high performance but compare the speed of learning according the sampling methods under the same off-policy algorithms, so we will not spend more timesteps.
19


Reviews:
[{"review_id": "oWLlXXhr0u", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: The statements about ERO clarify the contribution considerably. 6-->7\n\nThe authors propose an adaptively sampling mechanism optimized for policy improvement (NERS). By incorporating minibatch-wide information into the sampling score (while maintaining permutation invariance), they are able to out-perform reasonable baselines on a wide range of tasks.\n\nWhile NERS rarely beats other methods decisively, it has a strong showing across continuous and discrete action tasks and with a variety of off-policy learners. However, a few things would strengthen the empirical results. Some notation of spread should be reported on all of the Tables (e.g. standard error). Number of random seeds should also be mentioned. The reasoning behind the task selection should also be made explicit. The Atari subset used here is a bit unusual, particularly the choice to not use frame-skip. Investigating the sampling decisions of NERS is attempted in Figure 4, but further work should be done to provide evidence to the 'diversity of samples' claim. Ideally, NERS wouldn't just trade off TD error and Q-value over time, but also within each batch. Reporting something like the average minibatch Q-value/ TD-error standard deviation on NERS vs other methods would be nice. A qualitative evaluation akin to Figure 1 would also help guide intuition.\n\nThe off-policy RL related works section is a bit over-long, discussing things like the dueling architecture which don't seem to be overtly related apart from coming from the same sub-field. On the other side of things, having skimmed the ERO paper it is definitely the most-closely related, and as such deserves a bit more time spent on discussing the differences. For example, it is a bit unclear if the two sampling reward functions are different. An uncharitable reading of this paper would be that it is just an architecture tweak on top of ERO, and while the empirical results help dispel this idea, I think a more explicit comparison would still be useful.\n\nA related point is that the reward function for the sampler is quite unclear (Equation 6). How are these expectation evaluated in practice? I'd assume it'd just be the difference of value functions before and after the update, but the appendix suggest a more involved computation that doesn't appear to have been made explicit anywhere.\n\nFinal small point, towards the end a bi-GRU is mentioned as being used and I can't see where that'd come into play. Perhaps just a typo?\n\nOverall, I like this paper. Evaluating an idea across a variety of learning algorithms, observation and action spaces is no small feat, and the results are very solid. With a few tweaks and explanations this would be a very strong paper.\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_h8jJJnxMjm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\nThis paper proposes to improve sampling from the experience replay buffers that weights samples by their \"relatively usefulness\".  \n\nThis paper proposes to use two encoders - one global that encodes across the current batch of experience replay sample, and one local that encodes each selected timestep. Using the encodings, a scorer scores the experiences and weights the actor and critic losses in proportion to that score. The sampler is trained to maximize the probability that it chooses high reward timesteps on average. \n\n\n \n##########################################################################\n\nPros: \n\nExperience Replay is a widely used technique and improving on the naive random sample method makes sense. The analysis on other sampling methods is insightful. \nThe idea to have a global feature pooled from the sample transitions is unique. \nReweighing each sample's loss at training time is simple and effective.\n\n##########################################################################\n\nCons: \n\nThe paper's idea could be flawed. The sampler is trained so that it prioritizes high reward timesteps as in eq 6.\nBut this is dubious. What if we need to sample the N timesteps right before a high-reward timestep, even though those preceding timesteps do not have high return themselves? And how do you explain why\"NERS focuses on sampling transitions with high TD-errors in the beginning, ... as the timestep progresses, it samples transitions with both high TD-errors and Q-values (diverse)\", given that it's trained with a single objective to maximize sampled reward? \"using various features in an advanced manner.\" is not a satisfactory explaination. \n\n\nThe experiment section is not convincing. \nThe model has not been trained till convergence. e.g. SAC and ERO has all been trained with 1e6+ steps at least. In addition, what is the reason to do Hopper from Mujoco instead of Hopper-V2 from the OpenAI gym? With the latter, you can compare with the numbers published in ERO.\n\n##########################################################################\n\nQuestions during rebuttal period: \nPlease address and clarify the cons above \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XgmjEC5m9H5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Observing that the existed ER-based sampling method may introduce bias or redundancy in sampled transitions, the paper proposes a new sampling method in the ER learning setting. The idea is to take into consideration the context, i.e. many visited transitions, rather than a single one, based on which one can measure the relative importance of each transition. Specifically, the weights of transitions are also learned through a Reinforce agent and hence the sampling distribution is learned to directly improve sample efficiency. \n\nClarity. The presentation is clear in most places. But I do feel the core part of updating the sampling policy needs clarification. \n\nQuality. Please see the questions below.\n\nOriginality/Significance. The method is novel and is potentially interesting to the RL research community. \n\nIn the original prioritized ER paper, they use a ratio to mitigate the biased sampling issue, did the authors ever visualize what the result would be (say in Figure 1) if you use that ratio? \n\nEq (2) the input feature can be highly non-stationary/unstable. For example, some of the variables may decrease all the time, and some others may increase all the time. Intuitively, training with such data should be very challenging. It looks like that in order to resolve the issue of biased sampling, the authors introduce an even more difficult task. Do the authors have some comments about this? \n\nI am confused about how the sampling network is updated. In Algorithm 1, my understanding is that if the current time step is the end of an episode, then update the sampling network. Is it correct? Note that Algorithm 1 indicates that the actor, critic are updated at each time step. But the paper also says that the NERS is updated at each evaluation step and this means that throughout the evaluation episode the policy should be fixed to estimate (6). Can the author further explain how the network is updated? \n\nUsing evaluation to learn parameters seems unrealistic, the evaluation may happen only one time in practice. By evaluation, my understanding is that how much we can gain if we deploy such a policy. If the updating NERS requires to use evaluation data, this largely limited usability. \n\nI am also confused by the statement “The replay reward is interpreted as measuring how much actions of the sampling policy help the learning of the agent for each episode.” Eq (6) says that the reward is actually how much it improved from the current evaluation to the previous one. No matter you use the special sampling method or not, it should make an improvement. So this difference does not indicate “how much the sampling distribution can help.”\n\nIn the empirical study, NERS does not show a clear benefit from the learning curves. I believe it is better to average over a smoothing window before averaging over random seeds. Or do more runs. Doing more runs should not be that computationally expensive at least on Pendulum and LunarLander. And one question, how do you implement the Prioritized ER? Do you also use the importance ratio to anneal the bias as described in section 3.4 in that paper (https://arxiv.org/pdf/1511.05952.pdf)? Do you ever tune a bit the parameter beta in that formulae? \n\nAn additional note about related work. \nThe sampling distribution is an important problem in RL and is not well/rigorously studied. I really think it worths a more complete discussion of related work. For example, the author might also discuss the Langevin dynamics Monte Carlo sampling method in RL (Frequency-based search control in Dyna by Pan et al.), as their sampling distribution is supported by intuition and suggestive theoretical evidence and they show their method is better than prioritized ER and ER. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper is worth publishing. It investigates a novel idea on how to adaptively prioritise experiences from replay based on relative (within-batch) importance. The empirical investigation is thorough, and while the performance improvements are not stunning, the benefit is surprisingly consistent across many environments.
Decision: Accept (Poster)