Model-augmented Prioritized Experience Replay
Authors: Youngmin Oh
Jinwoo Shin
Eunho Yang
Sung Ju Hwang
Venue: ICLR 2022 Poster (2022)
Keywords: RL
Reinforcement Learning
Replay Buffer

Abstract: Experience replay is an essential component in off-policy model-free reinforcement learning (MfRL). Due to its effectiveness, various methods for calculating priority scores on experiences have been proposed for sampling. Since critic networks are crucial to policy learning, TD-error, directly correlated to $Q$-values, is one of the most frequently used features to compute the scores. However, critic networks often under- or overestimate $Q$-values, so it is often ineffective to learn to predict $Q$-values by sampled experiences based heavily on TD-error. Accordingly, it is valuable to find auxiliary features, which positively support TD-error in calculating the scores for efficient sampling. Motivated by this, we propose a novel experience replay method, which we call model-augmented prioritized experience replay (MaPER), that employs new learnable features driven from components in model-based RL (MbRL) to calculate the scores on experiences. The proposed MaPER brings the effect of curriculum learning for predicting $Q$-values better by the critic network with negligible memory and computational overhead compared to the vanilla PER. Indeed, our experimental results on various tasks demonstrate that MaPER can significantly improve the performance of the state-of-the-art off-policy MfRL and MbRL which includes off-policy MfRL algorithms in its policy optimization procedure.

Published as a conference paper at ICLR 2022
MODEL-AUGMENTED
PRIORITIZED EXPERIENCE REPLAY
Youngmin Oh', Jinwoo Shin?, Eunho Yang Sung Ju Hwang?:3
1 Samsung Advanced Institute of Technology
2 Korea Advanced Institute of Science and Technology
3 AITRICS
youngminO oh@ aasuugg. com j iinooss, eunhoy, sjhwang82) @kaist...... ac.
ABSTRACT
Experience replay an essential component off-policy model-free reinforcement learning (MfRL). Due its effectiveness, various methods for calculating priority scores on experiences have been proposed for sampling. Since critic networks are crucial to policy learning. TD-error, directly correlated Q-values, is of the most frequently used features to compute the scores. However, critic networks often under- or overestimate Q-values, it is often ineffective to learn for predicting Q-values by sampled experiences based heavily on TD-error. Accordingly, it is valuable find auxiliary features, which positively support TD-error in calculating the scores for efficient sampling. Motivated by this, we propose a novel experience replay method, which we call augmened prioritized experience replay (MaPER), that employs new learnable features driven from components in model- based RL (MbRL) to calculate the scores on experiences. The proposed MaPER brings the effect of curriculum learning for predicting Q-values better by the critic network with negligible memory and computational overhead compared to the vanilla PER. Indeed, experimental results on various tasks demonstrate that MaPER can significantly improve the performance of the state-of-the-ar off- policy MfRL and MbRL which includes off- off-policy MfRL algorithms in its policy optimization procedure.
INTRODUCTION
Experience replay (Lin, 1992; Mnih et al., 2015), which provides experiences that different policies may collect, is an essential component of policy training in reinforcement learning (RL). By utilizing many past experiences a large buffer, experience replay can stabilize and improve policy training. As result, experience replay with the buffer has significantly contributed to the success of RL on various tasks (Fujimoto et al., 2018; Haarnoja et al., 201 Sa;b; Mnih et al., 2015).
Due its importance, various methods for calculating priority scores of experiences in the buffer have been proposed to sample experiences efficiently (Schaul et al., 2016; Zha et al., 2019; Sinha et al., 2020). Since well-trained critic networks lead effective policy learning, one the most popular methods is to utilize pre-defined metrics for prioritizing experiences based on the temporal difference error (TD-error), directly related the loss of critic networks (Schaul et al., 2016). The prioritized experience replay based on TD-error (PER) has indeed proved its effectiveness in Q-learning (Hessel al., 2018; Schaul et al., 2016).
However. measuring Q-values requires to predict the expectation of returns, which can be obtained after multi-steps, so learning to predict Q-values generally needs a lot of interactions with an environment. Due to the difficulty of the multi-step estimation, sampling based only on high TD- errors may be often ineffective or even degrade the sample-efficiency of the RL framework on some tasks, compared to the uniform sampling, e.g., see Figure 6 in (Hessel et al., 2018). Moreover, Zha et al. (2019); Wang & Ross (2019); Sinha et al. (2020) show that the effectiveness of PER is questionable under some settings other than Q-leanning e.g., policy- based methods. In particular, Zha et al. (2019) reported that transitions with low TD-errors could be suitable for training a policy for some tasks. To overcome this issue, several works attempt design learning-based prioritizing

---

Published as a conference paper at ICLR 2022
Moli-Tak Laute
HumanoidPyyBune (SAC)
HumandidPyBullet (SAC)
Update
600
Looi 1D emor under SAC en Martt
Retur of S&C wi MaPER
Logi eD-eno under
500
G G-valus of SAC wt MarER
Average Value
Retum of SAC
Edération
G-aauue of SAC
MuChi with
300
ID-Emes
200
14
14
PriertySioore
Samp
1.0
1.3
86
0.0
Prierity
...
Updats Prietit Seare by
0.1
0.2
0.3
0.6
0.5
-100
0.1
0.2
0.3
0.4
0.5
Moddeeagmented Privetized Exparinne Poplay (M&PER)
Environment Steps (x<10%)
Environment Steps (x100)
(a) MaPER with MaCN
(b) Model- and TD-errors
(c) Q-value vs Retum
Figure 1: (a) A high-level illustration of Model-augmente Prioritized Experience Replay (MaPER) with Model-augmented Critic Network (MaCN). MaCN obtains more accurate estimation of Q-values via MaPER across the Q-value estimator and the model estimator. (b) Curves of TD-errors and model errors from sampled experiences under SAC. (Haarnoja al., 2018a). MaPER leads much faster decay of TD-errors. (c) Curves of Q-value estimations and returns evaluated by policy of SAC. MaPER leads to a much faster convergence of Q-values to returns. HumanoidPy is a task belonging to Pybullet Gymporium (Ellenberger, The solid lines and shaded regions denote the mean and standard deviations over the five random runs.
strategies (Zha et al., 2019; Sinha et al., 2020; Oh et al., 2021) utilizing various features, including transition's 's elements and quantities driven by critic networks. Here, our motivating observation is that all learnable features used in the existing schemes are from critic networks. For instance, although Zha al. (2019) utilizes multiple features, e.g., reward, TD-error, and timestep, TD-error is only learnable and is obtained from critic networks.
In this paper, we aim for developing a new prioritizing strategy by employing learnable auxiliary features outside critic networks, but can support TD-errors for efficient critics training. To this end, we consider components in Model-based RL (MbRL). To be specific, we first modify the critic network by additionally predicting the reward and the transition with shared weights, which we call Model- auggeeted Critic Network (MaCN). Then, we propose a modified experience replay Prioritized Experience Replay (MaPER), which prioritizes the past
scheme, coined experiences having both high model estimation errors and the TD-errors (see Figure (a))). Namely, MaPER encourages MaCN to predict both the Q-value and the model (i.e., reward and transition) well. Here, our intuition is two-fold:
We expect that prioritizing samples additionally with high model estimation errors helps decrease the model errors at the early stage of training, even if MaCN's estimation of the Q-value is poor.
Once MaCN has small model errors, MaPER prefers samples of high TD-error. Here, the learned representation to predict the model well is also useful to predict Q-value, as Q-value is decided by the model (Sutton& & Barto, 2018).
One may suggest to train MaCN without using MaPER, but this is not effective (see Figure 4.())). This is because MaPER brings type of curriculum learning effect (Bengio et al., 2009) to learn Q-value well by MaCN; gradually increases the complexity of samples for training, from high model estimation emrrr to high TD-errors. Indeed, Figure 1.(b) shows that TD-errors start to abruptly reduce (around 0.05 x 10t steps) after model errors sufficiently reduce. Figure 1.(b)((c) also shows that MaPER improves both the decay rate of TD-errors and the convergence rate of Q-values to returns.
To summarize, our contributions are as follows:
We propose slodel-augmented Prioritized Experience Replay (MaPER), which computes the priority scores of the past experiences additionally considering the model estimation error. MaPER helps reduce TD-errors much faster than learning the Q-values alone, as shown in Figure 1.(b)-(c). It results in large improvement in the return with the comparable number of parameters of the original critic networks.
MaPER can be seamlessly integrated into any modern off-policy RL frameworks with critic networks, including both MfRL. (e.g. SAC (Haarnjj al., 2018a), TD3 (Fujimoto et al., 2018)
2

---

Published as a conference paper at ICLR 2022
and Rainbow (van Hasselt et al., 2019)) and MbRL (e.g. MBPO (Janner et al., 2019)) methods with negligible memory or computational overhead.
We remark that to the best of our knowledge, none of the existing methods improve RL's experience replay by employing model learning although there exist some relevant works for different purposes, e.g., to augment rewards for improving the exploration (Pathak et al., 2017; Shelhamer et al., 2016) or to enhance representation learning for pixel-basee based observations (Lin et al., 2019; Agarwal et al.,
2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner al., 2020b; Srinivas al., 2020; Guo et al., 2020; Lee al., 2019).
METHOD
2.1 PRELIMINARY
In an RL framework, an agent interacts with an environment which consists of reward and transition distributions, R S x A + A (R) and T S x A - A (S), for state and action spaces S and A, where (-.) denotes a set of distributions. The agent selects an action at ~ T(-[s;) by its policy
the current state 8t S receive the next state 81+1 T(s at) and reward T1. ~ R(s,, at) over discrete timesteps t. The agent's objective is to learn a policy æ that maximizes Exp=a. |_ r where p is distribution of initial state 8p at each episode and Ek=o * the discounted cumulative rewards with a discount factor y E [0, 1).
Although our method is generally applicable to off aff-policy RL methods which have Q-networks, i.e., critics, throughout this section, we focus on off- off-policy actor-critic RL algorithms which consist of the policy (i.e., actor) (als) and critic networks Qa (s, a) with an experience replay buffer B, where and 0 are their parameters, respectively. The most commonly used loss for Qa a) (Mnih et al., 2015; Haarnoja et al., 201 8a; Fujimoto et al., 2018) is:
)( 15 [[ms]]
(1)
where || lIsse is the mean-squared error and 6, is the Temporal Difference error (TD-error)
defined as follows: for a given transition (st, Qz, Tt, St+1),
8 Te = iQT (st, az Tt, 56+1) = rt. + "lBaa (Qo (81+1, Qo (st, or).
(2)
We can interpret this value as a measure of how surprising 'unexpected" the transition is.
2.2 MODEL-AUGMENTED PRIORITIZED EXPERIENCE REPLAY (MAPER)
Model-augmented Critic Network (MaCN), An environment is depicted as reward and transition distributions, R(s, a) and T(s, a). We observe that these two distributions and Qo a) have the same input domain, i.e., S x A, the product of state and action spaces. Motivated by it, we slightly modify Qe(s, a) into Ca(s, a) to additionally predict the environment by a reward model Ra (s. a) and transition model To(s, a) in parallel via parameter sharing. We refer to Co(s, a) as a Model- augmented Critic Network (MaCN) such that Co Qo. Ro, To). Here, 2o measures Q-values with the reward estimation Ra as (Hessel et al., 2021) to stimulate the exploration (see Table C. in the supplementary material), In other words, we formulate the loss for 2p using the estimated rewards, that is similar to Eq, (I):
Lg, B(ee, [10 27 (St, Oe, 81+1)
(3)
where
825 de, 81+1) = Qo(se, ax) - (Ro az) + Tla ((eee)))) [Qo(s++1,
(4)
Here, Ro is detached in the backward propagation of the loss L Then the loss for Co is
Lc, = LIL + E2LRg + ELLT.
(5)
where LR, and Lyo are the losses for Rg and Te, respectively, with adaptively changing positive coefficients ε1, εz, and &s by employing a dynamic method in (Liang & Zhang, 2020; Liu et al., 2019b). Assuming deterministic environments, we employ the following losses for Ra and To:
LR. (o > [1|,: [oboo)) LTe B( [1|6 [ss]
3

---

Published as a conference paper at ICLR 2022
Algorithm 1 Model-augmented Prioritized Experience Replay based on Actor-Critic Methods
Initialize the moodll-augment critic network's parameters the actor's parameters 6, a replay buffer B t 0. priority set Ps +- 0, and the batch size m
2: for each timestep do
3: Choose ar from the actor and collect transition (st, Gz S14 from the environment
4: Update replay buffer B +- B U {(St, ot, Tt; 8t41)) and priority set Ps +- Ps U {maxg Pis)
5: for each gradient step do
6:
Sample an index by the given set Ps and Eq. (8) with | = m
7:
Calculate weights {wy hie by Eq. (9)
8: Learn by Eq. (5) and θ by {B, her C B with corresponding weights heeeer
9: Update priority set {ar hez by Eq. (10)
10: end for
end for
where
R,Rg = a.) rv. 6,To = (o(s at) 8c-1-
Formulation of MaPER. We first explain the concept PER (Schaul et al., 2016) since we will augment it with the model-based components. Let [n] be defined as the set {1, _. for a positive
integer n. Without loss of generality, we can suppose that the replay buffer stores the following information as its i-th transition:
B; = (()( (w(s): (w(()) Sw(1)+1)=
(6)
with function (i) from the index of B to corresponding timestep. PER calculates each B,'s priority o as the recently computed TD-error for itself and collects set of priority scores:
Pg (01,... , }.
where each priority o, is updated whenever the corresponding transition sampled for training the actor and critic networks. The TD-error (Eq. (2)) is the most frequently used quantity make the priority set (Eq. (7)) (Schaul et al., 2016; Brittain et al., 2019; Hessel et al., 2018; van Hasselt et al., 2019). Then the sampling strategy of PER is to determine an index set in [[B]] from the probability of i-th transition defined by the priority set:
P:
Σ B %
with hyper- parammer >> 0. Finally, we calculate the importance-sampling weights follows:
B
where 3 > O is also hyper-parametee to compensate the bias of probabilities.
We design new priority score equation with MaCN. Since our method uses Co a) instead of the original critic network, we modify the rule in obtaining the priority set in Eq. (7) for experiences in buffer B accordingly. To this end, we compute the priority of each transition as the sum of the TD-errors and model errors, which is used to compute probabilities and weights for corresponding transitions by Eqs. (8)-1), as follows:
a; E (6 2. + E2l Rolles + Eall Toll ISFF
(10)
where a is the map introduced in Eq. (6). By using the priority scoring Eq. (10), the buffer can sample various experiences that are useful to MaCN for both long-term and short-term viewpoints. The detailed descriptions of our framework MaPER are provided in Algorithm
Finally, we remark that the computational overhead by MaPER is negligible, compared to the vanilla PER. Suppose that a critic network has a linear layer with N hidden units as the final layer. Then one can compute the difference in the number of parameters between the MaCN and the original critic network: N(1+ + [S]), where S is a space of (embedded) states in critic networks. Notice that the number of additional parameters depends only the final hidden units in the critic network. Therefore, computing costs for the additional parameters minor compared to computing costs for both the total parameters in all networks and prioritized experience sampling.
4

---

Published as a conference paper at ICLR 2022
3 EXPERIMENT
In this section, we conduct experiments to answer the following questions:
Can the proposed method enhance the performances of diverse off-policy RI. algorithms having Q-networks in various environments?
What attributes to the success of our method are the most?
To answer them, we first describe experimental setup and show the main results against relevant baselines to show that our method is generally applicable to RI. algorithms including critic networks and experience replays, and largely improves their performance. Then we perform an ablation study of our method to analyze the most crucial components of it. In all experiments, the solid lines and shaded regions in all experimental results represent the mean and standard deviations across five runs with random seeds.
3.1
EXPERIMENTAL. SETUP
RL algorithms. We validate the effectiveness of MaPER with the following algorithms: Soft Actor-Critic (SAC) (Haarnoja et al., 2018 Twin Delayed Deep Deterministic policy gradient (TD3), Rainbow (Hessel et al., 2018), and Model-based Policy Optimization (MBPO) (Janner et al., 2019). Here, we apply data-efficient Rainbow (van Hasselt et al., 2019) since its sample efficiency dramatically higher than the original. When applying our method to given algorithms, we do not alter the original hyper-parameters to show that our method effectively improves the base algorithm's s performance without hyperparameter tuning.
Environments. For continuous control tasks, we consider not only MuJoCo environments (Todorov et al., 2012), which have been frequently used to validate Many RL. algorithms, but also PyBullet Gymperium! which are free implementations of the original MuloCo environments. Other free environments in the OpenAl (Brockman et al., 2016) are also considered. For discrete control tasks, we validate our method on Atari games. Finally, consider sparse reward variants of Pendulum-vO and HumanoidPy to show that our method also effective on them. We provide the details of the environments we used for the experiments in the supplementary material.
Sampling methods. We also compare the performance of our method against the following expe- rience replays to the original algorithms. The first is Experience Replay with Uniform Sampling at Random (RANDOM). The second is the vanilla PER (Schaul et al., 2016), which is rule- based prioritized sampling of the transitions based TD-errors. The third is experience replay with Likelihood-free Importance Weights (LfIW (Sinha et al., 2020), a learning-based prioritizing method, which predicts the importance of each experience.
Reward shaping methods. We also consider reward shaping methods to show that MaPER can seamlessly be combined with them. Since there are various proposed reward shaping methods (Pathak et al., 2017; Stadie et al., 2015; Devlin & Kudenko, 2012; Zou et al., 2019; Hu et al., 2020; Wiewiora al., 2003; Ng et al., 1999), we combine MaPER to some of them. The first is Curiosity-driven Exploration (CE) (Pathak et al., 2017). This method uses additional intrinsic rewards to promote exploration: ri プ 2 ($1+1) & (s(+1) Il, where o(s:) is feature vector and Ō(s;) is predicted feature vector for st, respectively. The second is Incentivizing Exploration (IE) (Stadie et al., 2015). This method uses modified reward ri + V(se, at) with V(st, CT maxe epr where C is
decay rate constant, and e, (|((#++)) M(((s)), lvMEE Here, M is a learnable network by e, for predicting embedding ((s,) of &t- In a similar manner the original results, we implement additional networks for reward shaping independently of the other networks.
The supplementary material provides more detailed descriptions of our experimental settings, including the details about the peram-paraeeters and implementations.
3.2 MAIN EXPERIMENTAL RESULTS
MIRL. Figure 2 shows the learning curves of SAC and TD3 on three PyBullet environments and Bipdaa/WalkerHaaddonn-v.. Our MaPER significantly and consistently outperforms baselines in
5

---

Published as a conference paper at ICLR 2022
2500
HumanoidPy (SAC)
3000
HallChreelahPyBulleee
3000
HopperPyBull Bulllel (SAC)
250
(SAC)
2000
SAC
sdd PIB
2500
2500
200
150
Retumn
1.50
540 UIF!
Reluum
Rellur
1,000
1,500
1,500
Retu
1000
1000
-50
500
500
-100
1.5
-150
Enviranment Steps (x10"))
Environment Steps (x10%)
Environment Steps (x10f)
Environment Staps (x1of)
HumannidPyBullett (TD3)
HaltCheetahPyBullet (TD3)
HopperPy (TD3)
BipedaiWalkerHarrccoom (TD3)
703 NaFER
2000
TD3
123 PSR.
Retum
1500
Relumn
2000
Relumm
2000
Retur
-100-
1000
1550
1,500
1000
1000
-200
Environment Steps (X10%)
Environment Steps (x10f)
Environment Steps (x10f)
Enwironment Steps (x110%
Figure 2:
Learning curves of the baseline offlpollcy model-free algorithms with various sampling methods. Here, the algorithms with our method outperform the original ones. The solid lines and shaded regions represent the mean and standard deviations by ten evaluation across ten runs with random seeds.
Table 1: Average of cumulative rewards under Rainhow on Atari Environments after 2M training steps by 10 evaluations across five runs with random seeds. One can observe that Rainbow with MaPER outperforms Rainbow without MaPER al., 2019; Lee et al., 2020a) except for total timesteps (0.1M - 2M) to observe the much longer time behavior (see Figure C. in the supplementary material for learning curves).
tasks. Here, used the same hyperparameters of Rainhow (van Hasselt
Game
Return MaPER
Return w/o MaPER
Alien
1635.6 (#:390.0)
1351.4 (4:296.3)
Amidar
482.6 (=103.4
427.5 (+:: 163.6)
Assault
866.4 (=144.7
508.1 1 (=+0..1
Asterix
1281.6 (#:215.1)
1122.6 (+200.6)
BankHeist
1035.0 (1:134.3)
542.6 (11330.2)
Boxing
(=2.5)
0.9 (114.6)
Breakout
7.1 (=2.3)
6.5 (12.3)
ChopperCommand
1565.8 (+-95599)
1358.3 (+:525.9)
DemonAttack
1870.6 (1:94.0)
1281.3 (++658.1)
Freeway
32.9 (=0.6)
32.4 (#0.3)
Frostbite
4712.0 (#712.9)
3147.4 (#666.3)
Gopher
714.8 (1:30.0)
578.6 (-1192.9)
Hero
14954.4 (1:2328.5)
14088.6 (1 3495.3)
Jamesbond
831.2 (1.39.8)
515.3 (1 120.3)
Kangaroo
8385.9 (=2149.3)
6120.2 (+ 2944.9)
Krull
4846.8 (+7769.1)
3981.1 (+ 557.8)
KungFuMaster
11483.6 (=1804.1)
10808.2 (# 1668.1)
MsPacman
1823.3 (#226.55)
1791.9 9 (# 445.8)
Pong
19.8 (+11.2)
19.8 (+: 2.3)
PrivateEye
98.3 (110.0
120.0 (1: 40.0)
Qbert
9364.3 (=197..4)
8845.9 (# 2826.9)
RoadRunner
17911.1 (#7441.5)
15088.6 6 (# 2853.6)
Seaquest
742.8 (+: 126.0)
630.4 (# 81.6)
UpNDown
8465.6 6867.0)
7383.5 (# 2320.9)
all tested cases. Next, since we focus on improving Q-learning, we validate our method with data-efficient Rainbow (van Hasselt et al., 2019), which is one of the state-of-the-an Q-learning frameworks. Table I shows the performance ratio of the Rainbow with MaPER to the original Rainbow. One can observe that Rainbow with MaPER achieves overwhelmingly higher performance the base Rainbow most Atari games.

---

Published as a conference paper at ICLR 2022
10000
Humanoid v2 (MBPO)
22 (MBPO)
Walker2d-12 (MSPO)
12000
HaliCheetah (MBPO)
9900
5000
3.00
Retum
Return
6000
4100
Retum
8200
3000
3000
0000
2000
2000
1000
1000
2000
0.53
0.09
0.05
9.3
0.15 0.2
Environmment Steps (x1)
Environment Steps (K10
Environment Steps (X101)
Environment Steps (x107)
The average cumulative rewards obtained on continuous control tasks using MBPO and MBPO + MaPER. Our method significantly improves the performance of MBPO. The solid lines and shaded regions denote the mean and standard deviations by one evaluation across five runs with random seeds.
Figure 3:
MbRL. Figure shows the learning curves of the MBPO with and without MaPER on the MuJoCo environments. We observe that MBPO with MaPER consistently outperforms the vanilla MBPO. In particular, similarly to MfRL on HumanoidPy performance gain on Humanoid-v2, which has the highest-dimensional state and action spaces among tasks in MuJoCo environments, which may require larger number of samples for accurate Q-value estimation. These impressive results with MBPO further show MaPER's versatility and effectiveness.
(Figure 2), MaPER obtains the largest
Sparse reward. We experiment MaPER on the sparse reward tasks by combining with existing reward shaping methods (see Table 2), to show its effectiveness them. Although MaPER does not use specifically designed rewards to solve for sparse reward tasks, it obtains reasonable performance gain over the original SAC, due to stimulated exploration while MaCN's additional learning to predict the environment behavior with MaPER. We further observe that combining MaPER and reward shaping methods results in considerable performance improvements over the reward shaping alone.
We provide additional experimental results in the supplementary material.
Table 2:
Average of cumulative rewards under SAC on sparse reward environments by 10 evaluations after 2M training steps across five runs with random seeds. Combining MaPER (our method) and reward shaping method results in the best performances.
Method Original
Sparse Penduuu-
Sparse-HtumnnidPyBuullee
00.00 (+-00.00)
00.00 (+-00.00)
IE
32.89 (++40.27)
00.08 (+:00.08)
CE
49.19 4#7)
00.14 (=00.06)
MaPER (our method)
16.30 (1:32.61)
00.17 (+:00.13)
IE+MaPER
82.69 (+:01.35)
00.53 (+00.54)
CE+MaPER
81.12 (=22.39)
00.83 (+00.45)
3.3 ABLATION STUDY
We analyze what components in our method are crucial to its improvement of RL's performances.
Effectiveness of MaPER. Figure 4. compares the learning curves of SAC with MaCN, but different sampling methods to verify the effectiveness of MaPER. There a considerably large gap between MaPER and MaCN alone. Moreover, one can observe that MaCN with LITW and PER are inferior to RANDOM. The main reason is that these two sampling methods focus on sampling transitions that are beneficial in updating the Q-network only (i.e.. TD-errors and importance weights for the Q-loss), but estimation of the Q-value is only a single component of MaCN C.. So, is suboptimal in improving Co. Yet, MaPER seeks to improve the estimation of the reward, state, and Q-values, so that it samples diverse and effective experiences. As a result, it outperforms other sampling methods under SAC with MaCN.
Effectiveness of learning the environment behavior. Next, we analyze the effectiveness of the environment estimation in MaPER. To verify it, we examine the performance of MaPER without the reward or transition estimators. Figure 4.(b) shows the learning curves of each case. It shows that leaming both of the environment estimators useful to improve the performance.
To show that computing priority scores by additionally considering model-errors is effective in itself, considered networks separate from the original
Effectiveness of parameter sharing.
7

---

Published as a conference paper at ICLR 2022
3000
HaltCheetaPyyBullee (SAC)
5030
HoirChestahPyBulit (SAC)
3020
HaliCheetahhPyBunke (SAC)
HumancidFy BaaanndddyBBlleeee (SAC)
2500
Maric mettroi
2320
Cittc
Retum
1000
Retum
Return
Heturr
1:000
Large Aden-Cios
1000
1300
500
1856-
-1000
MaCH-
MaCH- PER
Murst (oar maiter
Maack
S/C
-2000
0
0.5
1.5
-1000
0.5
15
0.5
-500
0.3
5.6
0.0
Enviranment Stops (x10")
Enviranment Stops (x10')
Environment Steps (x10")
Environment Stops (X100)
(a) MaPER effect
(b) Model effect
(c) Sharing Effect
(d) Size effect
Figure 4: Ablation study. (a): Different prioritizing methods under MaCN. We observe that MAPER outperforms other prioritizing methods under MaCN. (b): Different estimators in MaPER. One can observe that estimating both reward and transition is the most effective. (c): Learning curves of MaPER with and without sharing networks. Here, networks, which predict rewards and next states, used the same number of hidden units with the same architecture as the original critic networks under SAC. (d): Network size effect under SAC. Large networks have hidden units (2500, 2500). respectively. Simply increasing the number of parameters and gradient steps (GS) leads marginal worse performance. The solid lines and shaded regions represent the mean and standard deviations by ten evaluation across ten runs with random seeds.
critic networks under SAC in Figure 4.(c), i..., networks without parameter sharing. The separated networks have the same number of hidden units with the same architecture as the original critic networks under SAC. One can observe that the final performance of MaPER without parameter sharing is par with MaPER with parameter sharing, i.e., with MaCN.
Effectiveness of increased parameters. One may suspect that if MaPER achieves improved performance due to the increased size of the Q-network, with additional estimators. To show that this is not the case, we use the original SAC with its hidden layers dramatically increased, from (400, 300) to (2500, 2500) for both the Q-network and the policy network. Figure 4.(d) shows the effect of changes by learning curves. We observe that increasing the hidden layer size alone does not improve the performance and may even lead to the performance degeneration. Further, one can observe that increasing gradient steps cannot change the tendency. Therefore, the improvements of our method is not simply coming from the increased size of hidden layers.
4 RELATED WORK
Model-based RL. Although our approach is fundamentally different from Model-based RL (MbRL.) approaches which generate virtual experiences to train agents by planning, we briefly introduce some of them because we want to verify our method in MbRL. Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy across them is to first learn the environment models and use them to generate fictitious experiences for learning a policy. Due to its ability to generate transitions, MbRL's sample efficiency is remarkable certain tasks although much larger computing costs are generally needed.
Off-policy model-free RL. Among Q-learning algorithms, Rainbow (Hessel et al., 2018), which combines various techniques for deep Q-learning (Mnih et al., 2015), is one of the state-of-the-ar methods. Specially, data-efficient Rainbow (van Hasselt et al., 2019) outperforms the original one and other existing methods. In the case of the policy-based methods, Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018), Soft Actor-Critie (SAC) (Haarnoja et al., 20) and their variants (Wang& & Ross, 2019; Yang et al., 2021) are the state-of-the art methods that are frequently used as baselines. Here, TD3 employs double Qentworks, target policy smoothing, and different frequencies to update policy and Q-networks. SAC also adopts double Q-learning and utilizes the entropy measure of an agent policy to the reward to encourage the exploration of the agent. They are also utilized as a component of MbRL (Janner et al., 2019).
Experience replay. Prioritized Experience Replay (PER) and its variants are one of the most frequently used methods to sample relatively important transitions from the replay buffer for Q- learning (Hessel et al., 2018; Schaul et al., 2016; Wang & Ross, 2019; Brittain et al., 2019). For instance, Based on PER, Brittain et al. (2019) proposed a method to increases the priorities of previous
8

---

Published as a conference paper at ICLR 2022
transitions resulting in the important transitions. The effect of increasing capacity and downweighting old transition in the buffer has been also studied (Fedus et al., 2020). Wang et al. (2015); Zhang & Sutton (2015) also stuided the importance of utilizing the newest experiences for RL's learning. Of course, instead of TD-error, a different metric can be also used to PER, e.g., the expected return (Isele & Cosgun, 2018 8; Jaderherg al., 2016) which is also obtainable from critic networks like TD-error. Apart from PER and its variants, Novati & Koumoutsakos (2019) proposed a replay method in which policy updates are penalized if samples are far-policy. Sun et al. (2020) proposed attentive experience replay, which compute similarity between on-policy and past experiences to sample most similar experiences. By the way, some results (Andrychowicz et al., 2017: Fang et al., 201 9b;a; Liu et al., 2019a) designed experience replay methods which relabel experiences to stimulate exploration for sparse reward environments. Experience replay also important to initialize states for planning. Pan et al. (2019; 2020) applied hill climbing to generate initial states to generate virtual experiences (the search control) on the value function estimate. Recently, Icarning-base based experience replay methods, which utilizes neural networks to generate priority scores, have been proposed (Zha et al., 2019; Sinha et al., 2020). Specifically Zha et al. (2019) designed a network, whose input is the concatenation of reward, timestep, and TD-error, which computes the Bernoulli distribution's 'ss probability on each experience to select suitable experiences in DDPG (Lillicrap et al., 2016) algorithm on continuous action tasks. Sinha al. (2020) proposed a learnable likelihood-free density ratio estimator between on-policy and off- off-policy experiences, whose input is tuple of action and state, to compute the importance weights. Finally, Oh et al. (2021) designed a neural sampler, whose imput consists of Q- value, TD-error, transition's 's clement, and timestep, which computes the relative importance between experiences to sample diverse and useful experiences. However, to the best of our knowledge, all learnable features are only obtained from critic networks even in learning- based methods.
Shared weights. Shared weights are frequently used multi-agent RL frameworks (Rashid et al., 2018; Papoudakis et al., 2019), mostly for learning the policy networks. Further, sharing weights between the actor and critic networks is a standard approach (Mnih et al., 2016; Silver et al., 2017: Schulman et al., 2017), since it allows to learn representations that can contribute to both of them. To resolve practical issues with shared weights (c.g., balancing objectives), some methods propose to decouple the actor and the critic networks (Raileanu & Fergus, 2021; Cobbe et al., 2020). Moreover, shared weights between model and Q-value estimators have been used to enhance representation learning for pixel-image observations (Lin et al., 2019; Agarwal et al., 2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner et al., 2020b; Srinivas et al., 2020; Guo et al., 2020; Lee et al., 2019) reward for stimulating exploration (Pathak al., 2017; Shelhamer et al., 2016). However, to the best of our knowledge. none of the existing works attempted to share weights across the critics and environment models to enhance experience replay.
5 DISCUSSION AND CONCLUSION
We propose Model-augmmnned Prioritized Experience Replay (MaPER) which computes priority scores of past experiences based on the model estimation errors, as well as the TD-errors. To implement this, we employ a Model-augmente Q-value but also the environment behavior via weight sharing, where environment estimations are useful to predict Q-values. By providing experiences with high model errors the early stage of training, MaPER brings a curriculum learning effect for predicting Q-values well at later stages.
Critic Network (MaCN) that estimates not only the
The advantages of our proposed method are as follows. First, MAPER dramatically increases sample efficiency of state-of-the-ar algorithms: SAC, TD3, Rainbow, and MBPO since it largely alleviates the underestimation or overestimation of the value with the conventional Q-learning. Second, is simple to implement and is applicable to any model-free and -based algorithms that utilize Q-networks. Third, the computational cost of our algorithms is almost the same as the original algorithms. Finally, it is effective in various environments, including ones with the sparse rewards.
Despite the advantages above, unfortunately, there is no theoretical analysis of under what conditions and domains our method is effective. As a result, in certain complicated environments with high- dimensional state and action spaces for which the model training is extremely difficult, our method may achieve lower performance than that of the model-free baselines. In such environments, MaPER may require an excessive amount of interactions with the environments to train the model estimators. However, in such extreme cases, all-model based methods will be similarly ineffective.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Agents in reinforcement learning have potential threats of damaging human property and life. Indeed, RL agents trained under a wrongly designed reward may choose actions that sacrifice humans to increase their returns. Besides, even if agents select suitable actions in a simulation, it is not easy to guarantee they will behave well in the real world. In the future, RL agents may also cause many workers to lose their jobs due to the overwhelming performance of RL agents in various areas. Accordingly, we claim that RL should developed with consideration of how to tackle the issues. For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed parallel of methods for improving the sample efficiency of RL.
REPRODUCIBILITY STATEMENT
We describe the implementation details for experiments in Appendix B. We also provide our code.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Marlos Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization reinforcement learning, arXiv preprint arXiv:2101.05200 2021.
Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Nickum, and Ufuk Topeu. Safe reinforcement learning via shielding. In Siirt--eecond// AAAI Conference on Artificial Intelligence, 2018.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbecl, and Wojeiech Zaremba. Hindsight experience replay. In NeurlPS, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41 48, 2009.
Mare Brittain, Josh Bertram, Xuxi Yang, and Peng Wei. Prioritized sequence experience replay. arXiv preprint arXiv: 12726, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 54500, 2016.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3387-3395, 2019.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. arXiv preprint arXiv: 1805008805 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Picter Abheel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pp. 617-629, 2018.
Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmented actor-critic: arXiv preprint arXiv:2005.08000 08068, 2020.
Backpropagating through paths.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint arXiv:209.004466. 2020.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings the 1 11th International Conference on Autonomous Agents and Multiagent Syxtems, pp. 433-440. IFAAMAS, 2012.
Benjamin Ellenherger. Pybullet gymperium. 2019.
1eygggg, -gym, 2018-
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience replay for dynamic goals. In ICLR, 2019a.
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 12623- 12634, 2019b.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pp.
PMLR, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastier Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent predictvee representations for multitask reinforeement learning. In International Conference on Machine Learning, pp.
PMLR, 2020.
Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICMI,, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbool, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 2018b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020a.
11

---

Published as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020b.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Matteo Hessel, Ivo Danihelka. Fabio Viola, Arthur Guez. Simon Schmitt, Laurent Sifre. Theophane Weber, David Silver, and Hado van Hasselt. Muesli: Combining improvements policy optimization. arXiv preprint arXiv:2104.08 001999990.66...... 2021.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang. Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach reward shaping. Advances Neural Information Processing Systems, 33, 2020.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI. 2018.
Max Jaderberg. Volodymyr Mnih, Wojeiech Marian Czameekki Tom Schaul, Joel 7. I. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICIR, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519-12530, 2019.
Lukasz Kaiser, Mohammad Babacizadch, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Modcl-based reinforcement learning for atari. arXiv preprint 1rXiv:1903: 00374, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv 2014.
arXivyreprint preprint 1412.6412:::98))
Vijay R Konda and John N Tsitsiklis, Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008 1014, 2000.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference Learning Representutions, 2018
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv:1907.0095: 2019.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04933 2020a.
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl arXiv preprint arXiv:2007.12401, 2020b.
Sicong Liang and Yu Zhang, simple general approach balance task difficulty in multi-task learning. arXiv preprint arXiv:2002.04777 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. ICLR, 2016.
Long-Ji Lin. Seff-improving reactive agents based on reinforeement learning, planning and teaching. Machine learning, 8(3-4):293-322 321, 1992.
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprint arXiv: 005288 2019a
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1871-1880, 2019b.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforvement learning with theoretical guarantees. In International Conference on Learning Representations, 2018
Volodymyr Mnih Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement leamning. Nature, 518(7550))55)) 2015.
12

---

Published as a conference paper at ICLR 2022
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, PMLR, 2016.
Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278-2877 1999.
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. In SMC, 2019.
Youngmin Oh, Kimin Lee, Jinwoo Shin, Funho Yang, and Sung Ju Hwang. Learning sample with local and global contexts in experience replay buffer. International Conference of Representation Learnings, 2021.
Yangchen Pan, Hengshuai Yao, Amir Massoud Farahmand, and Martha White. Hill climbing on value estimates for search-control dyna. Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3209-3215. AAAI Press, 2019.
Yangehen Pan, Jincheng Mei, and preprint arXiv:2002.05822 2020.
Farahmand. Frequency- bassdd search-control dyna. arXiv
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Dealing with non- stationarity in multi-agent deep reinforcement learning. arXiv preprint arXiv:1906.0477 047337 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16 17, 2017.
Lerrel Pinto. James Davidson, Rahul Sukthankar, and Abhinay Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Antonin Raffin and Freek Stulp. Generalized state-dependent exploration for deep reinforcement learning in robotics. arXiv preprint arXiv:2005.057/99 2020.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. arXiv preprint 112:32102.11330. 2021.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. arXiv, pp. arXiv-2004, 2020.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farguhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning pp. 4295 4304. PMLR, 2018.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. ICLR, 2016.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(78899)6644-699) 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1777::17777 2017.
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint 16X2::66/2.07300 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Samarth Sinba, Jiaming Song, Animesh Garg. and Stefano Ermon. Experience replay with likelihood freee importance weights. arXiv preprint arXiv:2006.1 13169, 2020.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.0413 04136. 2020.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1rXiv:1507.008/ 2015.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael I Laskin. Decoupling representation learning from reinforcement learning. In International Conference Machine Learning, pp. 9870-98799 PMIR, 2021.
13

---

Published as a conference paper at ICLR 2022
Peiquan Sun, Wengang Zhou, and Houqiang Li. Attentive experience replay. In Proceedings the AAAI
Conference on Artificial Intelligence, volume pp. 5900-5907, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference Machine Learning, pp. 6215-6224, PMLR, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, pp. 14322-14333, 2019.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning, pp. 9797-9806. PMLR, 2020.
Che Wang and Keith Ross. Boosting soft actor-critic: Emphasizing recent experience without forgetting the past. arXiv preprint arXiv:1906.0e 140000990004009, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 1..051, 2015.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforcement learning agents. In Proceedings of the 20th International Conference Machine Learning (ICML-03), pp. 792-799, 2003.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft critic for safety-constraiied reinforcement learning. In Proceedings of the Thirty- AAAI Conference on Artificial Intelligence. AAAI Press, online, 2021.
Dauchen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. IJCAI, 2019.
Shangtong Zhang and Richard Sutton. A deeper look experience replay. In ICMR, 2015.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta-learning. arXiv preprint /00.::190000033000000 2019.
14

---

Published as a conference paper at ICLR 2022
Supplementary Material:
Model-augmented Prioritized Experience Replay
ORGANIZATION
This supplementary material provides descriptions of the materials that the main paper does not cover and additional experimental results to help readers understand our method. The document is organized as follows.
Appendix A
We describe environments, which belong to MuJoCo, PyBullet, OpenAl Gym, and Atari environments, considered in both the main paper and the supplementary material.
Appendix B
We provide the parameters for model-free RL, model-based RL, and reward shaping experiments conducted in this paper.
Appendix C- We provide additional experimental results about Atari and MuJoCo environments and ablation study to validate MaPER (our method) this appendix.
A ENVIRONMENT DESCRIPTION
A.1 MUJoCo ENVIRONMENTS
Multi-Joint dynamics with Contact (MuJoCo)² (Todorov et al., 2012) a physics engine for robot simulations supported by OpenAl Gym?. The MuloCo environments most widely used toolkits for developing and comparing RL algorithms. In MuloCo environments, Reinforcement Learning (RL) agents should learn policy control the joints (action) for achieving a goal, i.c., cumulative rewards. The observation of each environment includes information about the angular velocity and position of the robot's joints. This paper has considered the following MuJoCo environments.
currently some of the
Humanoid-v2 is an environment where RL agents control a diree-dimensional bipedal robot on the ground. Agents should learn how to walk forward without falling over.
HalfCheetah-v2 an environment where RL agents control two-dimensional cheetah robot for learning sprint.
Hopper-v2 an environment where RL agents control one-legged robot. The robot receives high return if it hops forward soon as possible without falling over.
Walker2d-v2 is an environment where RL agents control robot with tvo-dimensional bipedal legs to walk. Learning to quick walking without failure guarantees high return.
(a) Humanoid v2
(b) HallCheetah-v2
(c) Hopper-v2
(d) Walker2d-v2
Figure A.1: MuJoCo environments.
hhtp://wwww.mjccooor//
hhups://gym.opemai.com//
15

---

Published as a conference paper at ICLR 2022
A.2 PYBULLET ENVIRONMENTS
We also use PyBullet environments which are the open-source implementations of the MuJoCo environments, for our experiments for anyone's 's reproducibility. Besides, compared to MuJoCo environments, these environments are more difficult due the more realistic setting, e.g., adding energy cost.
HummanoidyyBllle(CEnne is environment in which RL agents control a three-dimensional bipedal robot to walk quickly without falling over.
is environment in which RI agents control two-dimensional cheetah robot for learning sprint.
WaifChhetahyyBulleCCCass
HopperPy Bulle((Envvvv)) is environment in which RL agents control two-dimensional one- legged robot hop-forward quickly without falling over.
(a) HumanoidPy Bullet
(b) HallCheetahhyyBullet
(c) HopperPy Bullet
Figure A.2: PyBullet environments.
A.3 OTHER ENVIRONMENTS
3 is a variant of Bipedal in OpenAl Gym. The robot's objective is to move forward as far as possible while escaping many obstacles which do not appear in the original.
Cparse-Pendulum is an enviromment which objective is to balance rod in the upright position as long as possible. It is variant of Pendulum-vO that OpenAl Gym supports. To make it sparser, we impose the following condition: The pendulum begins to receive +1 reward maintaining the rod in the upright position more than 100 steps continuously.
is a variant of the HumanoidPy BuumanoddyBullee(En---eo enviromment in which the robot receives +1 every 200 timesteps the robot not fallen over. The other things are entirely equivalent to the original.
(a) BipedalWalkerHardee
(b) Pharse-Pendulum
(c)
Figure A.3: Other Gym enviromments.
16

---

Published as a conference paper at ICLR 2022
Table A.1: Dimensions of observation and action spaces for continuous control environments.
Environment
Observation space
Action space
Horizon
Humanoid(-v2)
R
[-1, 1, 1 17
1000
Hopper(-vO)
R
[ 1
1000
Walker2d(-v2)
R
[-1, 1]
1000
HalfCheetah(-v22
[-1, 1
1000
Humaaniddyybulle(((En--)))))
R
[-1,1]
1000
HulfCheetaaPybulleerEn--00)))
R
[_
1000
HopperPybullee((nn--0))
15
1]
1000
Bipeda//alkerHarddooeeee
1
2000
Sparse-Pendulum
[ 200
R
1,
1000
Table A. shows the observation and action spaces and the maximum environment steps for each episode (horizon) in MuloCo, Pybullet, and other OpenAl Gym environments in this paper. Here, IR and |-1, 1] denote sets of real numbers and those between 0 and 1, respectively.
17

---

Published as a conference paper at ICLR 2022
A.4 DISCRETE CONTROL ENVIRONMENTS
We now describe the Atari environments that we consider in the main article and supplementary material. The objective of RL agents here is to learn a policy of discrete actions (buttons) by observing the screen (RGB) to get high cumulative rewards defined for each game.
Alien is an environment where a player should destroy all alien eggs in the RGB screen while escaping aliens.
Amidar is an environment similar MsPacman. In this environment, agents control a monkey in fixed rectilinear lattice to eat pellets much as possible while escaping chasing masters.
Assault an environment where a player controls spaceship. whose objective is eliminate the enemies.
Asterix is an environment where player controls tormado. Its objective hamburgers on the screen with avoiding dynamites.
O DOO0
(a) Alien
(b) Amidar
(c) Assault
(d) Asterix
Figure A.4: Atari Games: Alien, Amidar, Assault, and Asterix.
BankHeist is an environment where a player controls robber whose objective to rob banks as many as possible while avoiding the police in maze-like cities.
Boxing is an environment about the sport of boxing. There are two boxers with a top-down view, and RL agents should control one of them. They get one point if their punches hit from long distance and two points if their punches hit from close range. match over after two minutes, or 100 punch hits to the opponent.
Breakout an environment where player should destroy all bricks by controlling a paddle that makes a ball rebound.
ChopperCommane an environment in which a player operates a helicopter a desert. The helicopter should destroy all enemy aircraft and helicopters while protecting convoy of trucks.
3
5
(a) BankHeist
(b) Boxing
(c) Breakout
(d) ChopperCommand
Figure A.5: Atari Games: BankHeist, Boxing, Breakout, and ChopperCommand.
18

---

Published as a conference paper at ICLR 2022
DemonAttack is environment in which player controls guardian who should kill demons that attacks from above.
Freeway is environment where player controls chickens to across a ten-lane highway with traffic. They are only allowed to move up or down. The objective is to get across as possible as they until two minutes.
Frostbite is an environment where a player controls a man who should collect ice blocks to make his igloo, whose objective is to collect 15 ice blocks while avoiding some opponents, e.g., crabs and birds.
Gopher is environment where player controls farmer who should protect three carrots from gopher.
45"
RIWSSON
(a) DemonAttack
(b) Freeway
(c) Frostbite
(d) Gopher
Figure A.6: Atari Games: DemonAttack, Freeway, Frostbite, and Gopher.
Hero is an environment in which player should control a man who has backpack mounted helicopter unit for rescuing miner trapped at the bottom.
Jamesbond an environment where player controls vehicle whose objective is to move forward while avoiding and attacking enemies.
Kangaroo is an environment where a player controls a mother kangaroo, whose objective is to rescue her son while climbing.
Krull is an environment that controls player should complete stages, which are central parts of the film which name is the same.
(a) Hero
(b) Jamesbond
(c) Kangaroo
(d) Krull
Figure A.7: Atari Games: Hero, Jamesbond, Kangaroo, and Krull.
KungFuMasttr is environment where player should control fighter to save his girlfriend. He can use two types of attacks (punch and kick) and move/crunch/jump actions.
MsPacman is environment where player controls Paeman in given mazes for eating pellets as much as possible while avoiding chasing masters. The Pacman loses one life it contacts with monsters.
19

---

Published as a conference paper at ICLR 2022
Pong is environment about table tennis. RL agents control in in-game paddle to hit a ball back and forth. The objective to gain 11 points before the opponent. The agents earn each point when the opponent fails return the ball.
PrivateEye is an environment where player controls a private eye. It requires action, adventure, and memorization. For solving five cases, the private eye should find and return items to suitable places.
2000
1000
3:000
Polcc
KungFuMaster
(b) MsPacman
(c) Pong
(d) PrivateEye
Figure A.8: Atari Games: KungFuMaster, MsPaeman, Pong, and PrivateEye.
Qbert is an environment where a player controls a character under pyramid made of 28 cubes. The character should change the color of all cubes while avoiding obstacles and enemies.
Roaner is an environment where player controls a roadrunner (chaparral bird). The roadrunner runs to the left on the road. RL agents should pick up bird seeds while avoiding chasing coyotes and obstacles such as cars.
Seaquest is an environment where a player controls a submarine, whose objective rescue divers while attacking enemies with missiles.
UpNDown an environment where player should control purple dune buggy for collecting flags while avoiding other cars.
ESNSEN
:
(a) Qbert
(b) RoadRunner
(c) Seaguest
(d) UpNDown
Figure A.9: Atari Games: Jamesbond, Kangaroo, Krull, and Seaquest.
20

---

Published as a conference paper at ICLR 2022
B TRAINING DETAILS
Table B.1: Parameters for Model-free Reinforcement Learning experiments.
Parameter
Value
Shared
Batch size
256 (SAC), 100 (TD3), 32 (Rainbow) 10
Buffer size
Target smoothing coefficient (r) for soft update
0.02
Initial PER exponents (ax, (
(0.7, 0.4) (SAC/TD3), (0.5, 0.4) (Rainbow) 0.98 (SAC/TD3), 0.99 (Rainbow)
Discount factor for the agent reward (7)
Number of initial random actions
10, 000 (SAC/TD3) 000 (Rainbow)
Optimizer
Adam (Kingma & Ba, 2014) ReLU 64 (SAC/TD3), (Rainbow)
Nonlincarity
Replay period
Gradient step
64 (SAC/TD3), (Rainhow)
MaCN
Temperature for dynamic coefficients (Liu al., 2019b)
50
Likelihood-fiee Importance Weights
Temperature for weights
Hidden units networks
256, 256 10*
Fast replay buffer size
TD3
Hidden units of MLP networks
400, 300 10
Learning rate
Policy update frequency
Gaussian action and target noises
0.1, 0.2 0.5 Soft update with interval
Target noise clip
Target network update
SAC
Hidden units ML.P networks
400, 300 7.3 x 10
Learning rate
Target entropy
dim A (AA is action space)
Soft update with interval
Rainbow
Observation lown-sampling for Atari RGB
84 x with grey-scaling
CNN channels for Atari environments
32, 64
CNN filter size for Atari environments
5 5, 5 x 5,
CNN stride for Atari environments
Action repetitions and Frame stack
Reward clipping
True ([-1, 1, 1))
Terminal on loss of life
True
Max frames per episode
1.08 x 10
Target network update
Hard update (every 2,000 updates) 51
Support of Q- Q-distribution for Adam optimizer Learning Rate Max gradient norm
1.5 x 10
10
10
Noisy nets parameter
0.1
Multi-step return length
20
Q-network's hidden units per layer
Table B.1 includes the parameters that are used for the experiments under Model-free Reinforcement Learning (MfRL) in this paper. Since there are few benchmark results of PyBullet environments compared to MuloCo environments, we followed parameters (Raffin Stulp, 2020) for SAC and
$/3 increases 1.0 the rule 3 0.4n + 1.0(1 1)), where 77 the current step/the maximum steps.
21

---

Published as a conference paper at ICLR 2022
Table B.2: Parameters for MBPO (Janner et al., 2019) experiments.
Parameter
Value
Epochs
400 (HalfCCeetahh), 300 (Walker2d, Humanoid),
or 125 (Hopper)
Environments steps per epoch
1,000
Model rollouts per environment step Ensemble size Hidden units of ML.P networks
400
7
hidden layers of size 400 (Humanoid),
or hidden layers of size 200 (otherwise)
Policy updates environment steps Model horizon
40 (HalfCheetah) or 20 (otherwise)
-> 15 over epochs 20 + 100 (Hopper).
1 25 epochs 20 300 (Humanoid), or (otherwise)
Initial PER exponents (a, 3) Temperature for dynamic coefficients
(1.0, 0.4) (Humanoid), (0.5, 1.0) (otherwise) 50
TD3. In the case of MBPO, to use MaPER MBPO, we modified the critic networks in SAC, which the of MBPO, to MaCN for adding the model-learning components. Notice that MBPO utilizes two buffers to collect true and virtual experiences, respectively. Then MaPER is used to compute both true and virtual experiences" priority scores, which are updated whenever updating SAC networks. In the case Rainbow, we followed parameters used in (van Hasselt et al., 2019; Lee et al., 2020a), but we increased the total environment steps to observe tendencies over longer time steps. Notice that MaCN only requires the temperature value of dynamic coefficients (Liu et al., 2019b) for multi-task learning with Eq. ((5). We implemented MaPER Rainbow such that that MaCN's transition estimator learns to predict deep representations from the CNN layers for the current and next RGB arrays input and output, instead of taking the RGB array as input and output. Moreover, since Rainbow (van Hasselt et al., 2019) uses the multi-step return with length 20 to compute the TD-error instead of the reward, we also use it instead of the estimated reward. Table B.2 shows parameters for applying MBPO (Janner et al., 2019), which is one of the state-of-ther siie-offthe-anne art model-based RLs. We basically employed parameters introduced in (Janner et al., 2019) for MBPO experiments. Here, only PER's exponents additionally considered. Finally, Table B.3
(a, 3) and temperature for dynamic coefficients provides parameters for the reward shaping. Here, [O| represents the dimension of observations. We provide source code for the main experiments.
Table B.3: Parameters for the reward shaping (Pathak et al., 2017; Stadie et al., 2015) experiments.
Parameter
Value
Curiosity-driven Exploration
Architecture of feature network & for observation O Inverse model architecture
MLP with hidden layers: ([10)/21, [01/4]) MLP with 2 hidden layers: (256, 256)
Importance weight against the forward model loss
inverse model loss
0.2
Importance weight of the policy gradient loss signal against the learning the intrinsic reward Incentivizing Exploration
0.1
Architecture enender a
MLP with hidden layers: ([(0/22) [0]/4]
Architecture of predictive Model M
MI.P with 2 hidden layers: (1001/2)[[00))
Decay constant C
1.0
The current environment step
T
22

---

Published as a conference paper at ICLR 2022
000
amidar
1020
assault
asterri
1750
500
520
1300
1500
020
1900
730
Return
400
1:00
300
620
750
200
536
420
250
326
Steps
0
25 Environment Steps (x101)
0.5
Environmant Staps (x109)
0.3
1.5
Envfronmant Steps Ix109)
85
25
1200
bank helst
boxing
10
breakout
chopper command
1300
1750
$40
200
Returrn
Return
Rusum mutum
1252
100
752
200
500
252
-250
Environment Steps (x109)
0.3
15
Environment Steps (x10*)
2.5
Environment Steps (x100))
0.5
Envfronment Stops (x10")
2100
demon attack
5000
frostbite
gopher
20003
hero
2300
11300
4000
Return
12:00
2000
7503
300
2500
Environment Steps (x10*)
6.5
Environment Steps (x10*)
0.5
25
Environment Steps (x100)
Environmment Steps Ox 10.1
1200
Jamesbond
D00
seaquest
30000
kangaroo
100
600
0000
5000
540
Ruturn
Rutur
4002
300
2000
2002
200
200
100
1.5 Environment Steps (x10*)
6.5
Environment Steps (K10F)
Enviorment Staps 1
15
Enxfronmant (x10")
pong
freeway
2500
ms pacman
14:00
kung Tu master
10.
2030
12/000
10000
Retum
10
Return
1530
Betum
M
1030
G000
-20
530
c.00
2000
Environment Steps (x10")
0.5
15
%
Environment Steps (x10*)
Environmant Stops (x110))
6.5
Environment Steps (X11f))
12000
obert
25008
road runner
14000
down
1200
privato eye
20000
12000
Baturr
8030
Sature
15000
Butum
10e0e
Retum
902
2000
202
53000
4000
5820
2000
-202
Environment Steps (X10))
L5
Enviroement
255
Steps (x()
15
Environment Stopo 1K10%)
63
13
Figure C.I: Learning curves of Rainbow on Atari games. The solid line and shaded regions represent the mean and standard deviation by 10 evaluations, respectively, across five runs with random seeds.
C ADDITIONAL EXPERIMENTAL RESULTS
Learning curves Atari games. Figure C.1 shows Rainbow's 's learning curves of the average of performances for 2M environment steps with and without MaPER on Atari games. Rainbow with MaPER consistently outperforms the original Rainbow in most experimental results.
23

---

Published as a conference paper at ICLR 2022
14000
HalfCheetah-v2 (SAC)
5000
Walker2d-v2 (SAC)
12000
10000
Return
8000
Retur
3000
6000
4000
2000
SAC w/ MaFER
2000
SAC
w/
1000
SAC w/ LTW
-200
0.2
0.4
0.6
0.8
0.2
Environment Steps (x109)
Environment Steps (x 109)
Figure C.2: Learning curves of SAC on MuloCo environments. The solid line and shaded regions represent the and standard deviation by one evaluation, respectively, across five with random seeds.
Sparse-Pendulum (SAC)
HhannidHumanniiPyBBul (SAC)
1.0
MaPER+CE MaPER+1E
IE
MaPER
Original
100
Return
Return
0.5
50
0
0.0
0.5
1
1.5
0.5
1.5
Environment Steps (x10%)
Environment Steps (x10%)
Figure C.3: Learning curves of SAC on sparse reward environments. The solid line and shaded regions represent the mean and standard deviation by one evaluation, respectively. across five runs with random seeds.
HumanoidPyBullet (SAC)
HumanoidPyBullet (SAC)
4000
2000
MaPER (our method)
MaCN RANDOM
1500
MaCN PER
MaCN LnW
1000
Return
2000
SAC
Return
1000
MaPER (our method)
-500
MaPER wis reward model
0
MaPER transition model
-1000
0.5
1.5
1.5
Environment Steps (x10%)
Environment Steps (x10%)
(a) MaPER effect
(b) Model effect
Figure C.4: Ablation Study. (a): Different prioritizing methods with MaCN. We observe that MaPER outperforms other prioritizing methods under MaCN. (b): Different estimators in MaPER. One observe that estimating both reward and transition the most effective. The solid line and shaded regions represent the mean and standard deviation by ten evaluation, respectively, across ten runs with random seeds.
Table C.I: Average of cumulative rewards under SAC on sparse reward by 10 evaluations after 2M training steps across five runs with random seeds. The values in parentheses denote standard deviations. MaPER Real Reward means that the TD-error equation (sce Eq. (4)) uses the real rewards instead of model rewards.
Sparse- HunneeomdniidPBlllllee (SAC)
Steps
MaPER
MaPER Real Reward
Original
1.0M
00.17
(8)
0.03 (+00.05)
00.00 (=:00.00)
1.5M
00.20
(=0).20)
0.03 (+00.06)
00.00 (+:00.00)
2.0M
00.17 (3:00.13)
0.04 (+-00.06)
00.00 (4:00.00)
24

---

Published as a conference paper at ICLR 2022
The learning curves of SAC on MuJoCo environments shown in Figure C.2. We observe that SAC with MaPER outperforms SAC with other sampling methods. Figure C.3 shows the learning curves of SAC (Haarnoja al., 201 8a) on the sparse reward environments in the main article (see Table B.3), which reveals that the original algorithm could not learn a policy for returns on these environments. We observe that combining MaPER with reward shaping methods guarantees the best results compared to other methods, including reward shaping methods solely.
Under SAC, we computed the training wall clock times between MaPER and the vanilla PER in Table C.2. One can observe that the increased wall clock-times are marginal as we discussed in the final paragraph in Section
Table C.2: The training wall-clock time between MaPER and PER under SAC. We computed the elapsed time for train networks from sampling across five runs with random seeds.
Task HaliChectahPyyullltt HopperPyBullet
MaPER (hours)
PER (hours)
Ratio (MaPER/PER)
5.89
5.50
1.07
5,95
5,84
1.02
HumanoidPyBullles
6.34
5.93
1.07
Figure C,4 shows the effectiveness of both MaPER and learning the enviromment behavior, on HumanoidPyBullet. Comparing Figure 4 which considers HalfCheetahPyBullet one can observe the same tendency, i.e., MaPER is necessary to improve the performance. Table C.1 shows the (SAC) time steps 1.0M, 1.5,
performance of SAC with MaPER on Hatnrss-uummiidPyBuu and 2.0M, respectively. The table shows that using model rewards to compute TD-errors in MACN stimulates agents' exploration.
HumanoidPyBullet (SAC)
HumanoidPyBullet (SAC)
NoityttmancidPyylllleec (SAC)
2500
2500
1500
c=0.7,8-04
MaPER
2000
e- 1-0,8- 0.4
2000
E 0.1, m0.45
1250
SAC w/ LAW
0-0.5 0.4
0.33. 6
1000
SAC w/ PER
1500
p=0..p11..0
1500
61 0.9. -0.05
Return
0.7.8
Return
Return
1000
1000
500
250
500
-500
1.5 Environment Steps (x10%)
0.5
2
-500
1.5 Environment Steps (x10%)
0.5 1
-500
0.5
1.5
Environment Steps (x10*)
(a) PER parameters effect
(b) Loss weights effect
(c) Noise effect
Figure C.5: Ablation Study. (a): Different hyper- parameters a and 3 PER. (b): Comparison between
dynamic weights (our method) and fixed weights of the loss (5) in MaPER under SAC. Some typer--arameters degrade the performance of MaPER. (c): Learning curves on HumanoidPyBullee with Gaussian noise (0, 0.01) to rewards and next states. MaPER considerably outperforms other methods. The solid line and shaded regions represent the mean and standard deviation by one evaluation, respeetively, across five runs with random seeds.
One can observe the sensitivity of MaPER with respect PER's parameters 8) and coefficients in the loss (5) in Figure C.5.(a))(b). In most cases, final results are similar, but some degrade the performance of MaPER. To confirm the robustness of our method on an noise environment, with Gaussian noise (00, 0.01) to rewards and next states in Figure C.5.(c). MaPER considerably outperforms other methods since MaPER provides samples for improving the representation of inputs of critic networks.
we considered HumanoidPy
Transitions of Dearse-Pendulum while training under SAC are displayed in Figure C.6. In both cases (MaPER and PER), the agents did not receive any reward. A position of the pendulum, which is trained by samples based only on TD-errors, in downward. However, the pendulum trained by MaPER explores various states to receive reward. Finally, Table C.3 shows holistic performances for different domains, which are derived in our paper. We normalized each method's 's performance by that of MaPER for each domain. We believe that the considerably large gap between PER and MaPER in continuous control tasks under MiRL results from the actor-critic architecture (SAC and TD3). Compared to Q-learning, the advantage of the actor-critic architecture is to derive faster convergence (Konda& & Tsitsiklis, 2000). Conversely speaking, if the performance of critic networks is consistently poor, then the policy networks can quickly converge to a poorly behaving policy that learning
25

---

Published as a conference paper at ICLR 2022
will be stuck. We think that the prediction of Q-values with MaPER quickly converges the true returns (see Figure I.(c) the manuseript), the policy networks can be well-trained. Therefore, the large gap shows that how to improve the critic networks is crucial to not only Q-learning but also modern algorithms having the actor-critic architecture.
time step:
time step: 80
time step: 120
time step: 160
time step: 200
(a) SAC with MaPER in Sparse-
Sparse-Pendulune Pendulum
time step: 40
time step: 80
time step: 120
time step: 160
time step: 200
(b) SAC with PER in Sparse-Pendulumm
Figure C.6: Transitions on Sparse-Pendulum while training under SAC with MaPER and PER, respectively, from 50,000 environment steps. Although both MaPER and not find any reward, SAC with MaPER have tried various states of transitions, i.c., the rod in downward, upright, and horizontal positions.
Table C.3: Holistic performances for each domain. Here, cach domain, we normalized each method's performance based on MaPER's s. One can observe that MaPER considerably outperforms other methods.
Domain
MaPHR
RANDOM
PER
LNW
MIRL (continuous control tasks)
1.00
0.68
0.27
0.68
MfRL (discrete control tasks)
1.00
N/A
0.75
N/A
MbRL (continuous control tasks)
1.00
0.80
N/A
N/A
26


Reviews:
[{"review_id": "oFSk54tR3Ig", "reviewer": "Reviewer_mk5C", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Writing: In general, the writing is very clear. With that being said, I do have a few questions. \n1. In Algorithm 1, The index set $\\mathcal{I}$ is never used after initialization.\n2. When are the priorities in $\\mathcal{P_B}$​ updated? \n3. In Eq (10), why are all losses weighted equally? \n4. How is MaPER combined with MBPO? \n\nExperiments: The experiments are quite extensive. I do appreciate the experiments of applying MaPER on different algorithms. \n\nI would like to have another ablation to check the effectiveness of MaPER: Can we use MaPER without MaCN? For example, we have two separated networks, one of which predicts Q and another predicts the next state and reward. This method has a larger computational overhead than vanilla PER, but can better decompose MaCN and MaPER. \n\nDoes PER in the experiments (Fig 2, Fig 4) use (4) to stimulate exploration? If not, the comparison is not very fair. \n\nNovelty: The method sounds novel. The idea of MaCN is pretty natural so it might exist in some prior work but I'm not sure. Using model/reward prediction error for prioritized experience replay is indeed novel. ", "summary_of_the_paper": "This paper studies the problem of experience replay. It proposes Model-augmented Prioritized Experience Replay (MaPER), a novel experience replay method, based on the intuition that the model is easier to estimate than Q-value. It also proposes a modification to critic network, Model-augmented Critic Network (MaCN), by predicting the reward and dynamics model additionally. Experiments on MaPER show that MaPER can be applied to both discrete action space (DQN) and continuous action space (SAC), both model-free RL algorithms (SAC) and model-based RL algorithms (MBPO), as well as sparse reward tasks, and improve the baseline algorithms a lot. ", "summary_of_the_review": "Given the novelty of the proposed method and extensive experiments, I like this paper and would recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5VKyu9MQUPN", "reviewer": "Reviewer_rZAM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n=========\n+ The paper is well-written overall and is easy to digest.\n+ Through and strong experimental results\n+ Simple technique that is generalizable\n\nRoom for Improvement\n=========\n- Dive deep into the Private Eye domain. Why didn't the proposed approach work? I love to see that inside the paper.\n- While the additional complexity of the algorithm was discussed (N(1+|S)), it would be really great if they have compared the performance of their algorithm against the others, where the x-axis is wall close-time.\n- Provide a more holistic number across all domains for better sending the message out. For example, normalizing the return between 0-1 and then provide the average across all domains.\n\nDetails\n=========\n- Perhaps remove the acronyms in the abstract.\n- Add line number to your Algorithm.\n- I did not follow the priority set on line 5 of Algorithm. Can you elaborate ?\n- Figure 2, Table 2, Figure 4 captions: \"across five runs with randoms\" => \"across five runs with random seeds\".\n- Please avoid reintroducing the acronyms. Example MaPER in sec 3.1\n- \"Since LfIW was only validated it on\" => remove it\n- Page 8: \"Besides, the performance between\" => \"Besides, the performance difference between\"\n- What was unique about private eye? Any insights there?\n", "summary_of_the_paper": "Authors introduced Model-augmented Prioritized Experience Replay (MaPER) as a variation to the canonical prioritized sweeping. The main difference is that they extended the critic network on its last layer to also predict the reward and transition model. Then they changed the prioritization of sample for replay to also include the error in the reward and transition model, in addition to the TD-error. Experimental results in various MuJoC domains showed the performance boost introduced by MaPER on top of existing state-of-the-art techniques. Furthermore Authors carried an ablation study, investigation the impact of 1) incorporating the MaPER while all methods used the same network 2) incorporating various error values for prioritization, and 3) increasing the network size.", "summary_of_the_review": "I liked this paper. The proposed approach is simple and generic (more sparse reward domains should be investigated for their final claim to be substantiated) yet practical. The experimental results are comprehensive and I think it would be a great value to RL practitioners.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3mCg0zcZi-E", "reviewer": "Reviewer_noEm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1. The paper focuses on an important topic: what should be the sampling distribution for ER? And the proposed method seems new. \n\n2. The presentation is reasonably clear. \n\n3. The empirical results show that the proposed method is promising. \n\nWeaknesses:\n\n1. Although the proposed method is supported by some intuitions, it does not have a solid motivation. \n\nIn page 1, it is said “learning to predict … is difficult, so sampling based on TD-error is far from optimal.” What do you mean by “short term”? And why sampling from TD error is far from optimal? Any reference? \n\nThe effectiveness of PER is questionable …. The authors never introduce what these limitations of PER are. I think the limitations of PER should be reviewed in more detail in this work and should be explained how the proposed method can address these limitations.\n\nAn additional note, reducing TD errors faster does not tell us anything about the sample efficiency of learning performance. I personally have some experience with it. Reducing TD errors fast can still have a bad policy. For example, when your buffer only covers a small subset of the state space.  Hence, I do not understand the effect of showing figure 1(b). \n\nAccording to your intuition, during early learning, samples with large model errors would get sampled more frequently. The authors should provide some insight (either empirically or theoretically) to explain why such sampling distribution is preferred. I am doubting it has some exploration effect. There is also work saying that TD error magnitude can be used for encouraging exploration (Smart Exploration in Reinforcement Learning using Absolute Temporal Difference Errors by Clement et al), so it seems it is not a problem to use TD errors during the early learning stage. You might compare the sample space coverages during the early learning stage between your algorithm and regular PER. \n\nThe intuition is not persuasive. During early learning, how do you know the model error plays a dominant role? It seems domain-dependent. And, it is also doubtful whether a representation can describe the dynamics imply it can predict Q values. They can be quite different. In some environments, learning a model can be much more challenging (e.g., high dimensional), it is possible that the model error term always dominates the priority. \n\n2.  The contribution is incremental. Basically, it adds a model error term to the TD error. \n\n3. What if the transition probability p(s’|s,a) is not Gaussian, how do you measure the error? \n\nThe three losses can have very different numerical scales, how to handle this? \n\n4. The experiments are not sufficient. \n\na. What is the parameter sweep for each algorithm? PER is sensitive to the alpha, beta parameter in (8), (9)\n\nb. Do you ever try to use a separate NN for model learning (but still use the same priority as you defined)? This can help identify if the auxiliary tasks are useful or not. \n\nc. It would be interesting to see how the method works when increasing uncertainty in 1) reward function 2) transition function. This can help identify if the benefit comes from uncertainty measurement when calculating the model errors.  \n\nMissing citations: \n1. The credit of ER should go to Self-Improving Reactive Agents Based On\nReinforcement Learning, Planning and Teaching by Lin, 1992. \n\n2. Page 2, “we remark … ” There are indeed works improving RL’s experience replay by employing model learning. Please see the SGLD-based sampling in MBRL (e.g., Frequency-based search control by Pan et al.) I believe there is a category of work along this direction.", "summary_of_the_paper": "The paper proposes a new method for prioritizing experiences used in the prioritized experience replay (PER) method. The proposed approach is simple: the critic network also learns the reward function and transition function. The two errors (absolute value) are added to the absolute TD error to calculate the priorities in the PER method. The authors provide some intuitions to their method. Experiments on both Mujoco and Atari environments are presented to show the method’s effectiveness. ", "summary_of_the_review": "Please see main review. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0PqbbDNR90Y", "reviewer": "Reviewer_K56x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Main Review\n\nPros:\n1. The experiment section is very complete.\nThere’s a model-free part that considers some of representative algorithms such as SAC, TD3,\nand there's a model-based part that compares against MBPO.\nAnd experiments are implemented on both the continuous robotics tasks and atari games.\n\n2. The details of the paper are adequate.\nThe main paper and the appendix section provides a lot of the details of the algorithm,\nmaking it quite reproducible.\n\n3. Sufficient related work.\n\n4. Compared to the baselines, the authors show that the proposed algorithm has better performance.\n\n5. The code is released.\n\nCons:\n1. The algorithm is relatively simple.\nAnd there’s no theoretical evidence that supports the proposed algorithms,\nand the inspiration from this paper is not clear.\nAdding the dynamics terms in the priority seems pretty random.\n\n2. The sensitivity. \nIt is not clear how sensitive the parameters in the algorithm are.\nFor example $\\xi_1$, $\\xi_2$, $\\xi_2$ in equation (5) are three variables i think that could be quite unstable and sensitive. It would be interesting to see if the algorithm only works on 1 set of hyper-parameters, or it is in general a very robust algorithm\n\n", "summary_of_the_paper": "\n*Summary Of The Paper\nModel-augmented Prioritized Experience Replay\nIn this paper, the authors consider using the dynamics prediction error in the priority calculation.\nThe priority is then used to decide the probability for each sample during training.\nThe method is applicable to both model-based and model-free setting\n", "summary_of_the_review": "\n*Summary Of The Review\n\nI am quite worried about the fact that the algorithm is looking quite simple.\nBut the results are looking good.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new strategy for prioritized experience replay. It is based on the argument that the TD error itself may not be a good indicator for priority, so we should rely on other factors that are easier and more reliable to learn. The new method is based on two modifications: (1) modifying the critic's objective so that it learns a good model of the environment (reward and transition dynamics) as well, and (2) use the combination of the TD loss and the model loss in order to define the priorities in the ER queue.

The majority of reviewers are positive about this work. They believe the method is novel and the experiments are extensive. The authors improved the paper during the discussion phase, so many of the questions have already been answered.

There are some concerns though, some of them shared by reviewers and some after my own reading of the paper:

One concern is about the justifications for the method, which are based on the heuristic and intuitive arguments, rather than principled approach. Currently it is not clear, at least to me, why adding a model error to the objective is a good idea.

Another concern, which is not shared by the reviewers, is that there is much overlap in the confidence band of figures and confidence intervals of tables. For example, in many of the subfigures of Figure 2 or 4, there is a significant overlap in the shaded areas. Or many of the numbers in Table 1 (with and without MaPER) are within each others' confidence intervals. Are the results statistically significant?

Another comment, again not shared by reviewers, is regarding how the loss functions are defined. Consider the loss in Eq. (1):
- Is there a squared missing? I assume that it is missing. Although it does not matter at this stage, when you add other terms to the loss, it would, i.e, the minimizer of $f(x) + g(x)$ is not necessarily the same as the minimizer of $f^2(x) + g^2(x)$.

- Is the target value based on a fixed parameter (not optimized), or do you actually consider the expected of the TD error, which would be equal to the empirical Bellman error. If the latter, it would be a biased estimate of the Bellman error. And it is not what DQN or the TD method optimizes (that's why Sutton and Barto's textbook calls them pseudo-gradient).

These requires some clarifications.

Another question related to the model: Is it assumed that the model $T_\theta$ is deterministic and predicts a next-state, as opposed to predicting a distribution over them? (cf. equations after (5) )?

All strengths and concerns considered together, I believe this is a good paper overall, and can be accepted at ICLR. Hopefully we get a better understanding of what this method is actually doing in the future research.

I have the following suggestions to the authors:
- Perform statistical significant tests on your results. In some cases, it might be helpful to increase the number of runs from five to a larger number. It may also be more visually clear to provide standard error instead of standard deviation.
- Clarify issues about the definition of the loss function.
- Please consider improving and clarifying your argument of why you method works.
- Please consider the remaining comments by reviewers in order to improve your paper.
Decision: Accept (Poster)