Year: 2021

Abstract:
Experience replay is an essential component in off-policy model-free reinforcement learning (MfRL). Due to its effectiveness, various methods for calculating priority scores on experiences have been proposed for sampling. Since critic networks are crucial to policy learning, TD-error, directly correlated to Q-values, is one of the most frequently used features to compute the scores. However, critic networks often underor overestimate Q-values, so it is often ineffective to learn to predict Q-values by sampled experiences based heavily on TD-error. Accordingly, it is valuable to find auxiliary features, which positively support TD-error in calculating the scores for efficient sampling. Motivated by this, we propose a novel experience replay method, which we call model-augmented priority experience replay (MaPER), that employs new learnable features driven from components in model-based RL (MbRL) to calculate the scores on experiences. The proposed MaPER brings the effect of curriculum learning for predicting Q-values better by the critic network with negligible memory and computational overhead compared to the vanilla PER. Indeed, our experimental results on various tasks demonstrate that MaPER can significantly improve the performance of the state-of-the-art offpolicy MfRL and MbRL which includes off-policy MfRL algorithms in its policy optimization procedure.

heading: 1 INTRODUCTION
text: Experience replay (Lin, 1992; Mnih et al., 2015), which provides experiences that different policies may collect, is an essential component of policy training in reinforcement learning (RL). By utilizing many past experiences in a large buffer, experience replay can stabilize and improve policy training. As a result, experience replay with the buffer has significantly contributed to the success of RL on various tasks (Fujimoto et al., 2018; Haarnoja et al., 2018a;b; Mnih et al., 2015).
Due to its importance, various methods for calculating priority scores of experiences in the buffer have been proposed to sample experiences efficiently (Schaul et al., 2016; Zha et al., 2019; Sinha et al., 2020). Since well-trained critic networks lead to effective policy learning, one of the most popular methods is to utilize predefined metrics for prioritizing experiences based on the temporal difference error (TD-error), directly related to the loss of critic networks (Schaul et al., 2016). The prioritized experience replay based on TD-error (PER) has indeed proved its effectiveness in Q-learning (Hessel et al., 2018; Schaul et al., 2016).
However, measuring Q-values requires to predict the expectation of returns, which can be obtained after multi-steps, so learning to predict Q-values generally needs a lot of interactions with an environment. Due to the difficulty of the multi-step estimation, sampling based only on high TD-errors may be often ineffective or even degrade the sample-efficiency of the RL framework on some tasks, compared to the uniform sampling, e.g., see Figure 6 in (Hessel et al., 2018). Moreover, Zha et al. (2019); Wang & Ross (2019); Sinha et al. (2020) show that the effectiveness of PER is questionable under some settings other than Q-learning, e.g., policy-based methods. In particular, Zha et al. (2019) reported that transitions with low TD-errors could be suitable for training a policy for some tasks. To overcome this issue, several works attempt to design learning-based prioritizing strategies (Zha et al., 2019; Sinha et al., 2020; Oh et al., 2021) utilizing various features, including transition’s elements and quantities driven by critic networks. Here, our motivating observation is that all learnable features used in the existing schemes are from critic networks. For instance, although
Zha et al. (2019) utilizes multiple features, e.g., reward, TD-error, and timestep, TD-error is only learnable and is obtained from critic networks.
In this paper, we aim for developing a new prioritizing strategy by employing learnable auxiliary features outside critic networks, but can support TD-errors for efficient critics training. To this end, we consider components in Model-based RL (MbRL). To be specific, we first modify the critic network by additionally predicting the reward and the transition with shared weights, which we call Model-augmented Critic Network (MaCN). Then, we propose a modified experience replay scheme, coined Model-augmented Prioritized Experience Replay (MaPER), which prioritizes the past experiences having both high model estimation errors and the TD-errors (see Figure 1.(a)). Namely, MaPER encourages MaCN to predict both the Q-value and the model (i.e., reward and transition) well. Here, our intuition is two-fold:
• We expect that prioritizing samples additionally with high model estimation errors helps decrease the model errors at the early stage of training, even if MaCN’s estimation of the Q-value is poor.
• Once MaCN has small model errors, MaPER prefers samples of high TD-error. Here, the learned representation to predict the model well is also useful to predict Q-value, as Q-value is decided by the model (Sutton & Barto, 2018).
One may suggest to train MaCN without using MaPER, but this is not effective (see Figure 4). This is because MaPER brings a type of curriculum learning effect (Bengio et al., 2009) to learn Q-value well by MaCN; it gradually increases the complexity of samples for training, from high model estimation errors to high TD-errors. Indeed, Figure 1.(b) shows that TD-errors start to abruptly reduce (around 0.05 × 106 steps) after model errors sufficiently reduce. Figure 1.(b)-(c) also shows that MaPER improves both the decay rate of TD-errors and the convergence rate of Q-values to returns.
To summarize, our contributions are as follows:
• We propose Model-augmented Prioritized Experience Replay (MaPER), which computes the priority scores of the past experiences additionally considering the model estimation error. MaPER helps reduce TD-errors much faster than learning the Q-values alone, as shown in Figure 1.(b)-(c). This results in large improvement in the return with the comparable number of parameters of the original critic networks.
• MaPER can be seamlessly integrated into any modern off-policy RL frameworks with critic networks, including both MfRL (e.g. SAC (Haarnoja et al., 2018a), TD3 (Fujimoto et al., 2018) and Rainbow (van Hasselt et al., 2019)) and MbRL (e.g. MBPO (Janner et al., 2019)) methods with negligible memory or computational overhead.
We remark that to the best of our knowledge, none of the existing methods improve RL’s experience replay by employing model learning although there exist some relevant works for different purposes, e.g., to augment rewards for improving the exploration (Pathak et al., 2017; Shelhamer et al., 2016) or to enhance representation learning for pixel-based observations (Lin et al., 2019; Agarwal et al., 2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner et al., 2020b; Srinivas et al., 2020; Guo et al., 2020; Lee et al., 2019).

heading: 2 METHOD

heading: 2.1 PRELIMINARY
text: In an RL framework, an agent interacts with an environment which consists of reward and transition distributions, R : S × A → ∆(R) and T : S × A → ∆(S), for state and action spaces S and A, where ∆(·) denotes a set of distributions. The agent selects an action at ∼ π(·|st) by its policy π on the current state st ∈ S to receive the next state st+1 ∼ T (st, at) and a reward rt ∼ R(st, at) over discrete timesteps t. The agent’s objective is to learn a policy π that maximizes Es0∼ρ,π [∑∞ k=0 γ krk ] , where ρ is a distribution of initial state s0 at each episode and ∑∞ k=0 γ
krt+k is the discounted cumulative rewards with a discount factor γ ∈ [0, 1). Although our method is generally applicable to off-policy RL methods which have Q-networks, i.e., critics, throughout this section, we focus on off-policy actor-critic RL algorithms which consist of the policy (i.e., actor) πΘ(a|s) and critic networks Qθ(s, a) with an experience replay buffer B, where Θ and θ are their parameters, respectively. The most commonly used loss for Qθ(s, a) (Mnih et al., 2015; Haarnoja et al., 2018a; Fujimoto et al., 2018) is:
E(st,at,rt,st+1)∼B [∥δtQ πΘ θ ∥MSE] , (1)
where ∥·∥MSE is the mean-square error and δtQπΘθ is the Temporal Difference error (TD-error) defined as follows: for a given transition (st, at, rt, st+1),
δtQ πΘ θ = δQ πΘ θ (st, at, rt, st+1) = rt + γEa′∼πΘ(·|st+1) [Qθ (st+1, a ′)]−Qθ (st, at) . (2) We can interpret this value as a measure of how surprising or ‘unexpected’ the transition is.

heading: 2.2 MODEL-AUGMENTED PRIORITIZED EXPERIENCE REPLAY (MAPER)
text: Model-augmented Critic Network (MaCN). An environment is depicted as reward and transition distributions, R(s, a) and T (s, a). We observe that these two distributions and Qθ(s, a) have the same input domain, i.e., S ×A, the product of state and action spaces. Motivated by it, we slightly modify Qθ(s, a) into Cθ(s, a) to additionally predict the environment by a reward model Rθ(s, a) and a transition model Tθ(s, a) in parallel via parameter sharing. We refer to Cθ(s, a) as a Modelaugmented Critic Network (MaCN) such that Cθ = (Qθ,Rθ, Tθ). Here, Qθ measures Q-values with the reward estimationRθ as (Hessel et al., 2021) to stimulate the exploration (see Table C.1 in the supplementary material). In other words, we formulate the loss for Qθ using the estimated rewards, that is similar to Eq. (1):
LQθ = E(st,at,st+1)∼B [∥δtQ πΘ θ ∥MSE] , (3)
where
δtQπΘθ = δQ πΘ θ (st, at, rt, st+1) = Qθ(st, at)− ( Rθ(st, at) + γEa′∼πΘ(·,st+1) [Qθ(st+1, a ′)] ) .
(4)
Here,Rθ is detached in the backward propagation of the loss LQθ . Then the loss for Cθ is LCθ = ξ1LQθ + ξ2LRθ + ξ3LTθ , (5)
where LRθ and LT θ are the losses for Rθ and Tθ, respectively, with adaptively changing positive coefficients ξ1, ξ2, and ξ3 by employing a dynamic method in (Liang & Zhang, 2020; Liu et al., 2019b). In this paper, we use the following losses forRθ and Tθ:
LRθ = E(st,at,rt,st+1)∼B [∥δtRθ∥MSE] , LT θ = E(st,at,rt,st+1)∼B [∥δtTθ∥MSE] ,
Algorithm 1 Model-augmented Prioritized Experience Replay based on Actor-Critic Methods 1: Initialize the model-augmented critic network’s parameters θ, the actor’s parameters Θ, a replay
buffer B ← ∅, priority set PB ← ∅, and the batch size m. 2: for each timestep t do 3: Choose at from the actor and collect a transition (st, at, rt, st+1) from the environment. 4: Update replay buffer B ← B ∪ {(st, at, rt, st+1)} and priority set PB ← PB ∪ {maxB PB}. 5: for each gradient step do 6: Sample an index I by the given set PB and Eq. (8) with |I| = m. 7: Calculate weights {wi}i∈I by Eq. (9) 8: Learn θ by Eq. (5) and Θ by {Bi}i∈I ⊂ B with corresponding weights {wi}i∈I (Eq. (9)). 9: Update a priority set {σi}i∈I by Eq. (10)
10: end for 11: end for
where δtRθ = Rθ(st, at)− Er∼R(st,at) [r] , δtTθ = Tθ(st, at)− Es∼T (st,at) [s] .
Formulation of MaPER. We first explain the concept of PER (Schaul et al., 2016) since we will augment it with the model-based components. Let [n] be defined as the set {1, · · · , n} for a positive integer n. Without loss of generality, we can suppose that the replay buffer B stores the following information as its i-th transition:
Bi = ( sκ(i), aκ(i), rκ(i), sκ(i)+1 ) , (6)
with a function κ (i) from the index of B to a corresponding timestep. PER calculates each Bi’s priority σi as the recently computed TD-error for itself and collects a set of priority scores: PB = {σ1, · · · , σ|B|}, (7) where each priority σi is updated whenever the corresponding transition is sampled for training the actor and critic networks. The TD-error (Eq. (2)) is the most frequently used quantity to make the priority set (Eq. (7)) (Schaul et al., 2016; Brittain et al., 2019; Hessel et al., 2018; van Hasselt et al., 2019). Then the sampling strategy of PER is to determine an index set I in [|B|] from the probability pi of i-th transition defined by the priority set:
pi = σαi∑
k∈[|B|] σ α k
, (8)
with a hyper-parameter α > 0. Finally, we calculate the importance-sampling weights as follows:
wi =
( 1
|B|pi
)β , (9)
where β > 0 is also a hyper-parameter to compensate the bias of probabilities.
We design a new priority score equation with MaCN. Since our method uses Cθ(s, a) instead of the original critic network, we modify the rule in obtaining the priority set in Eq. (7) for experiences in the buffer B accordingly. To this end, we compute the priority of each transition as the sum of the TD-errors and model errors, which is used to compute probabilities and weights for corresponding transitions by Eqs. (8)-(9), as follows:
σi = ξ1∥δκ(i)QπΘθ ∥MSE + ξ2∥δκ(i)Rθ∥MSE + ξ3∥δκ(i)Tθ∥MSE, (10) where κ is the map introduced in Eq. (6). By using the priority scoring in Eq. (10), the buffer can sample various experiences that are useful to MaCN for both long-term and short-term viewpoints.
The detailed descriptions of our framework MaPER are provided in Algorithm 1.
Finally, we remark that the computational overhead by MaPER is negligible, compared to the vanilla PER. Suppose that a critic network has a linear layer with N hidden units as the final layer. Then one can compute the difference in the number of parameters between the MaCN and the original critic network: N(1 + |S|), where S is a space of (embedded) states in critic networks. Notice that the number of additional parameters depends only on the final hidden units in the critic network. Therefore, computing costs for the additional parameters is minor compared to computing costs for both the total parameters in all networks and prioritized experience sampling.

heading: 3 EXPERIMENT
text: In this section, we conduct experiments to answer the following questions:
• Can the proposed method enhance the performances of diverse off-policy RL algorithms having Q-networks in various environments?
• What attributes to the success of our method are the most?
To answer them, we first describe our experimental setup and show the main results against relevant baselines to show that our method is generally applicable to RL algorithms including critic networks and experience replays, and largely improves their performance. Then we perform an ablation study of our method to analyze the most crucial components of it. In all experiments, the solid lines and shaded regions in all experimental results represent the mean and standard deviations across five runs with random seeds.

heading: 3.1 EXPERIMENTAL SETUP
text: RL algorithms. We validate the effectiveness of MaPER with the following algorithms: Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), Twin Delayed Deep Deterministic policy gradient (TD3), Rainbow (Hessel et al., 2018), and Model-based Policy Optimization (MBPO) (Janner et al., 2019). Here, we apply data-efficient Rainbow (van Hasselt et al., 2019) since its sample efficiency is dramatically higher than the original. When applying our method to given algorithms, we do not alter the original hyper-parameters to show that our method effectively improves the base algorithm’s performance without hyperparameter tuning.
Environments. Many RL algorithms frequently have used MuJoCo environments (Todorov et al., 2012) for continuous control tasks, but they are not freely available to everyone since they belong to a commercial physics engine. Accordingly, we consider additionally free implementations of the original MuJoCo environments, called PyBullet Gymperium1 and other free environments in the OpenAI Gym (Brockman et al., 2016) for anyone’s reproducibility. For discrete control tasks, we validate our method on Atari games. Finally, we consider sparse reward variants of Pendulum-v0 and HumanoidPyBulletEnv to show that our method is also effective on them. We provide the details of the environments we used for the experiments in the supplementary material.
Sampling methods. We also compare the performance of our method against the following experience replays to the original algorithms. The first is Experience Replay with Uniform Sampling
1https://github.com/benelot/pybullet-gym
at Random (RANDOM). The second is the vanilla PER (Schaul et al., 2016), which is rule-based prioritized sampling of the transitions based on TD-errors. The third is experience replay with Likelihood-free Importance Weights (LfIW) (Sinha et al., 2020), a learning-based prioritizing method, which predicts the importance of each experience.
Reward shaping methods. We also consider reward shaping methods to show that MaPER can seamlessly be combined with them. Since there are various proposed reward shaping methods (Pathak et al., 2017; Stadie et al., 2015; Devlin & Kudenko, 2012; Zou et al., 2019; Hu et al., 2020; Wiewiora et al., 2003; Ng et al., 1999), we combine MaPER to some of them. The first is Curiosity-driven Exploration (CE) (Pathak et al., 2017). This method uses additional intrinsic rewards to promote exploration: rit = η 2∥ϕ̂ (st+1)− ϕ (st+1) ∥, where ϕ(st) is a feature vector and ϕ̂(st) is a predicted feature vector for st, respectively. The second is Incentivizing Exploration (IE) (Stadie et al., 2015). This method uses a modified reward rt + V(st, at) with V(st, at) = etCT maxt′≤t et′ , where C is a decay rate constant, and et = ∥ζ(st+1) −M(ζ(st), at)∥MSE. Here,M is a learnable network by et for predicting embedding ζ(st) of st. In a similar manner to the original results, we implement additional networks for reward shaping independently of the other networks.
The supplementary material provides more detailed descriptions of our experimental settings, including the details about the hyper-parameters and implementations.

heading: 3.2 MAIN EXPERIMENTAL RESULTS
text: MfRL. Figure 2 shows the learning curves of SAC and TD3 on three PyBullet environments and BipdalWalkerHardcore-v3. Our MaPER significantly and consistently outperforms baselines in all tested cases. Next, since we focus on improving Q-learning, we validate our method with
data-efficient Rainbow (van Hasselt et al., 2019), which is one of the state-of-the-art Q-learning frameworks. Table 1 shows the performance ratio of the Rainbow with MaPER to the original Rainbow. One can observe that Rainbow with MaPER achieves overwhelmingly higher performance over the base Rainbow on most Atari games.
MbRL. Figure 3 shows the learning curves of the MBPO with and without MaPER on the MuJoCo environments. We observe that MBPO with MaPER consistently outperforms the vanilla MBPO. In particular, similarly to MfRL on HumanoidPyBullet (Figure 2), MaPER obtains the largest performance gain on Humanoid-v2, which has the highest-dimensional state and action spaces among tasks in MuJoCo environments, which may require larger number of samples for accurate Q-value estimation. These impressive results with MBPO further show MaPER’s versatility and effectiveness.
Sparse reward. Although we do not directly target for sparse reward tasks, we experiment MaPER on the sparse reward tasks by combining it with existing reward shaping methods (see Table 2), to show its effectiveness on them. Although MaPER does not use specifically designed rewards to solve for sparse reward tasks, it obtains reasonable performance gain over the original SAC, due to stimulated exploration while MaCN’s additional learning to predict the environment behavior with MaPER. We further observe that combining MaPER and reward shaping methods results in considerable performance improvements over the reward shaping alone.
We provide additional experimental results in the supplementary material.

heading: 3.3 ABLATION STUDY
text: We analyze what components in our method are crucial to its improvement of RL’s performances.
Effectiveness of MaPER. Figure 4.(a) compares the learning curves of SAC with MaCN, but different sampling methods to verify the effectiveness of MaPER. One can observe that MaCN with LfIW and PER are inferior to RANDOM. The main reason is that these two sampling methods focus on sampling transitions that are beneficial in updating the Q-network only (i.e., TD-errors and importance weights for the Q-loss), but estimation of the Q-value is only a single component of MaCN Cθ. So, it is suboptimal in improving Cθ. Yet, MaPER seeks to improve the estimation of
the reward, state, and Q-values, so that it samples diverse and effective experiences. As a result, it outperforms other sampling methods under SAC with MaCN. Besides, the performance difference between SAC with and without MaCN is marginal, i.e., MaCN alone cannot improve the sample efficiency.
Effectiveness of learning the environment behavior. Next, we analyze the effectiveness of the environment estimation in MaPER. To verify it, we examine the performance of MaPER without the reward or transition estimators. Figure 4.(b) shows the learning curves of each case. It shows that learning both of the environment estimators are useful to improve the performance.
Effectiveness of parameter sharing. To show that computing priority scores by additionally considering model-errors is effective in itself, we considered networks separate from the original critic networks under SAC in Figure 4.(c), i.e., networks without parameter sharing. The separated networks have the same number of hidden units with the same architecture as the original critic networks under SAC. One can observe that the final performance of MaPER without parameter sharing is on a par with MaPER with parameter sharing, i.e., with MaCN.
Effectiveness of increased parameters. One may suspect that if MaPER achieves improved performance due to the increased size of the Q-network, with additional estimators. To show that this is not the case, we use the original SAC with its hidden layers dramatically increased, from (400, 300) to (2500, 2500) for both the Q-network and the policy network. Figure 4.(d) shows the effect of changes by learning curves. We observe that increasing the hidden layer size alone does not improve the performance and may even lead to the performance degeneration. Further, one can observe that increasing gradient steps cannot change the tendency. Therefore, the improvements of our method is not simply coming from the increased size of hidden layers.
Wall-clock time. Under SAC, we computed the training wall clock times between MaPER and the vanilla PER in Table 3. One can observe that the increased wall clock-times are marginal as we discussed in the final paragraph in Section 2.

heading: 4 RELATED WORK
text: Model-based RL. Although our approach is fundamentally different from Model-based RL (MbRL) approaches which generate virtual experiences to train agents by planning, we briefly introduce some of them because we want to verify our method in MbRL. Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy across them is to first learn the environment models and use them to generate fictitious experiences for learning a policy. Due to its ability to generate transitions, MbRL’s sample efficiency is remarkable on certain tasks although much larger computing costs are generally needed.
Off-policy model-free RL. Among Q-learning algorithms, Rainbow (Hessel et al., 2018), which combines various techniques for deep Q-learning (Mnih et al., 2015), is one of the state-of-the-art methods. Specially, data-efficient Rainbow (van Hasselt et al., 2019) outperforms the original one and other existing methods. In the case of the policy-based methods, Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018), Soft Actor-Critic (SAC) (Haarnoja et al., 2018a) and their variants (Wang & Ross, 2019; Yang et al., 2021) are the state-of-the art methods that are frequently used as baselines. Here, TD3 employs double Q-networks, target policy smoothing, and different frequencies to update a policy and Q-networks. SAC also adopts double Q-learning and utilizes the entropy measure of an agent policy to the reward to encourage the exploration of the agent. They are also utilized as a component of MbRL (Janner et al., 2019).
Experience replay. Prioritized Experience Replay (PER) and its variants are one of the most frequently used methods to sample relatively important transitions from the replay buffer for Qlearning (Hessel et al., 2018; Schaul et al., 2016; Wang & Ross, 2019; Brittain et al., 2019). For instance, Based on PER, Brittain et al. (2019) proposed a method to increases the priorities of previous transitions resulting in the important transitions. The effect of increasing capacity and downweighting old transition in the buffer has been also studied (Fedus et al., 2020). Wang et al. (2015); Zhang & Sutton (2015) also stuided the importance of utilizing the newest experiences for RL’s learning. Of course, instead of TD-error, a different metric can be also used to PER, e.g., the expected return (Isele & Cosgun, 2018; Jaderberg et al., 2016) which is also obtainable from critic networks like TD-error. Apart from PER and its variants, Novati & Koumoutsakos (2019) proposed a replay method in which policy updates are penalized if samples are far-policy. Sun et al. (2020) proposed attentive experience replay, which compute similarity between on-policy and past experiences to sample most similar experiences. By the way, some results (Andrychowicz et al., 2017; Fang et al., 2019b;a; Liu et al., 2019a) designed experience replay methods which relabel experiences to stimulate exploration for sparse reward environments. Experience replay is also important to initialize states for planning. Pan et al. (2019; 2020) applied hill climbing to generate initial states to generate virtual experiences (the search control) on the value function estimate.
Recently, learning-based experience replay methods, which utilizes neural networks to generate priority scores, have been proposed (Zha et al., 2019; Sinha et al., 2020). Specifically Zha et al. (2019) designed a network, whose input is the concatenation of reward, timestep, and TD-error, which computes the Bernoulli distribution’s probability on each experience to select suitable experiences in DDPG (Lillicrap et al., 2016) algorithm on continuous action tasks. Sinha et al. (2020) proposed a learnable likelihood-free density ratio estimator between on-policy and off-policy experiences, whose input is a tuple of action and state, to compute the importance weights. Finally, Oh et al. (2021) designed a neural sampler, whose input consists of Q-value, TD-error, transition’s element, and timestep, which computes the relative importance between experiences to sample diverse and useful experiences. However, to the best of our knowledge, all learnable features are only obtained from critic networks even in learning-based methods.
Shared weights. Shared weights are frequently used in multi-agent RL frameworks (Rashid et al., 2018; Papoudakis et al., 2019), mostly for learning the policy networks. Further, sharing weights between the actor and critic networks is a standard approach (Mnih et al., 2016; Silver et al., 2017; Schulman et al., 2017), since it allows to learn representations that can contribute to both of them. To resolve practical issues with shared weights (e.g., balancing objectives), some methods propose to decouple the actor and the critic networks (Raileanu & Fergus, 2021; Cobbe et al., 2020). Moreover, shared weights between model and Q-value estimators have been used to enhance representation
learning for pixel-image observations (Lin et al., 2019; Agarwal et al., 2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner et al., 2020b; Srinivas et al., 2020; Guo et al., 2020; Lee et al., 2019) or reward for stimulating exploration (Pathak et al., 2017; Shelhamer et al., 2016). However, to the best of our knowledge, none of the existing works attempted to share weights across the critics and environment models to enhance experience replay.

heading: 5 DISCUSSION AND CONCLUSION
text: We propose Model-augmented Prioritized Experience Replay (MaPER) which computes priority scores of past experiences based on the model estimation errors, as well as the TD-errors. To implement this, we employ a Model-augmented Critic Network (MaCN) that estimates not only the Q-value but also the environment behavior via weight sharing, where environment estimations are useful to predict Q-values. By providing experiences with high model errors at the early stage of training, MaPER brings a curriculum learning effect for predicting Q-values well at later stages.
The advantages of our proposed method are as follows. First, MaPER dramatically increases sample efficiency of state-of-the-art algorithms: SAC, TD3, Rainbow, and MBPO since it largely alleviates the underestimation or overestimation of the value with the conventional Q-learning. Second, it is simple to implement and is applicable to any model-free and -based algorithms that utilize Q-networks. Third, the computational cost of our algorithms is almost the same as the original algorithms. Finally, it is effective in various environments, including ones with the sparse rewards.
Despite the advantages above, unfortunately, there is no theoretical analysis of under what conditions and domains our method is effective. As a result, in certain complicated environments with highdimensional state and action spaces for which the model training is extremely difficult, our method may achieve lower performance than that of the model-free baselines. In such environments, MaPER may require an excessive amount of interactions with the environments to train the model estimators. However, in such extreme cases, all-model based methods will be similarly ineffective.

heading: ETHICS STATEMENT
text: Agents in reinforcement learning have potential threats of damaging human property and life. Indeed, RL agents trained under a wrongly designed reward may choose actions that sacrifice humans to increase their returns. Besides, even if agents select suitable actions in a simulation, it is not easy to guarantee they will behave well in the real world. In the future, RL agents may also cause many workers to lose their jobs due to the overwhelming performance of RL agents in various areas. Accordingly, we claim that RL should be developed with consideration of how to tackle the issues. For instance, various methods for safety (Tessler et al., 2019; Pinto et al., 2017; Chow et al., 2018; Wachi & Sui, 2020; Alshiekh et al., 2018; Cheng et al., 2019) should be proposed in parallel of methods for improving the sample efficiency of RL.

heading: REPRODUCIBILITY STATEMENT
text: We describe the implementation details for experiments in Appendix B. We also provide our code.

heading: Model-augmented Prioritized Experience Replay

heading: ORGANIZATION
text: This supplementary material provides descriptions of the materials that the main paper does not cover and additional experimental results to help readers understand our method. The document is organized as follows.
Appendix A - We describe environments, which belong to MuJoCo, PyBullet, OpenAI Gym, and Atari environments, considered in both the main paper and the supplementary material.
Appendix B - We provide the parameters for model-free RL, model-based RL, and reward shaping experiments conducted in this paper.
Appendix C - We provide additional experimental results about Atari and MuJoCo environments and ablation study to validate MaPER (our method) in this appendix.

heading: A ENVIRONMENT DESCRIPTION

heading: A.1 MUJOCO ENVIRONMENTS
text: Multi-Joint dynamics with Contact (MuJoCo)2 (Todorov et al., 2012) is a physics engine for robot simulations supported by OpenAI Gym3. In MuJoCo environments, Reinforcement Learning (RL) agents should learn a policy to control the joints (action) for achieving a goal, i.e., cumulative rewards. The observation of each environment includes information about the angular velocity and position of the robot’s joints. This paper has considered the following MuJoCo environments.
Humanoid-v2 is an environment where RL agents control a three-dimensional bipedal robot on the ground. Agents should learn how to walk forward without falling over.
HalfCheetah-v2 is an environment where RL agents control a two-dimensional cheetah robot for learning sprint.
Hopper-v2 is an environment where RL agents control a one-legged robot. The robot receives a high return if it hops forward as soon as possible without falling over.
Walker2d-v2 is an environment where RL agents control a robot with two-dimensional bipedal legs to walk. Learning to quick walking without failure guarantees a high return.

heading: A.2 PYBULLET ENVIRONMENTS
text: The MuJoCo environments are currently some of the most widely used toolkits for developing and comparing RL algorithms. However, since they are commercial and hinder open research, we also use
2http://www.mujoco.org/ 3https://gym.openai.com/
PyBullet environments4, which are the open-source implementations of the MuJoCo environments, for our experiments for anyone’s reproducibility. Besides, compared to MuJoCo environments, these environments are more difficult due to the more realistic setting, e.g., adding energy cost.
HumanoidPyBullet(Env-v0) is an environment in which RL agents control a three-dimensional bipedal robot to walk quickly without falling over.
HalfCheetahPyBullet(Env-v0) is an environment in which RL agents control a two-dimensional cheetah robot for learning sprint.
HopperPyBullet(Env-v0) is an environment in which RL agents control a two-dimensional onelegged robot hop-forward quickly without falling over.

heading: A.3 OTHER ENVIRONMENTS
text: BipedalWalkerHardcore(-v3) is a variant of BipedalWalker-v3 in OpenAI Gym. The robot’s objective is to move forward as far as possible while escaping many obstacles which do not appear in the original.
Sparse-Pendulum is an environment which objective is to balance a rod in the upright position as long as possible. It is a variant of Pendulum-v0 that OpenAI Gym supports. To make it sparser, we impose the following condition: The pendulum begins to receive +1 reward if maintaining the rod in the upright position more than 100 steps continuously.
Sparse-HumanoidPyBullet is a variant of the HumanoidPyBullet(Env-v0) environment in which the robot receives +1 at every 200 timesteps if the robot has not fallen over. The other things are entirely equivalent to the original.
Table A.1 shows the observation and action spaces and the maximum environment steps for each episode (horizon) in MuJoCo, Pybullet, and other OpenAI Gym environments in this paper. Here, R and [−1, 1] denote sets of real numbers and those between 0 and 1, respectively.
4https://github.com/benelot/pybullet-gym/

heading: A.4 DISCRETE CONTROL ENVIRONMENTS
text: We now describe the Atari environments that we consider in the main article and supplementary material. The objective of RL agents here is to learn a policy of discrete actions (buttons) by observing the screen (RGB) to get high cumulative rewards defined for each game.
Alien is an environment where a player should destroy all alien eggs in the RGB screen while escaping aliens.
Amidar is an environment similar to MsPacman. In this environment, agents control a monkey in a fixed rectilinear lattice to eat pellets as much as possible while escaping chasing masters.
Assault is an environment where a player controls a spaceship, whose objective is to eliminate the enemies.
Asterix is an environment where a player controls a tornado. Its objective is to eat hamburgers on the screen with avoiding dynamites.
BankHeist is an environment where a player controls a robber whose objective is to rob banks as many as possible while avoiding the police in maze-like cities.
Boxing is an environment about the sport of boxing. There are two boxers with a top-down view, and RL agents should control one of them. They get one point if their punches hit from a long distance and two points if their punches hit from a close range. A match is over after two minutes, or 100 punch hits to the opponent.
Breakout is an environment where a player should destroy all bricks by controlling a paddle that makes a ball rebound.
ChopperCommandis an environment in which a player operates a helicopter in a desert. The helicopter should destroy all enemy aircraft and helicopters while protecting a convoy of trucks.
DemonAttack is an environment in which a player controls a guardian who should kill demons that attacks from above.
Freeway is an environment where a player controls chickens to run across a ten-lane highway with traffic. They are only allowed to move up or down. The objective is to get across as possible as they can until two minutes.
Frostbite is an environment where a player controls a man who should collect ice blocks to make his igloo, whose objective is to collect 15 ice blocks while avoiding some opponents, e.g., crabs and birds.
Gopher is an environment where a player controls a farmer who should protect three carrots from a gopher.
Hero is an environment in which a player should control a man who has a backpack-mounted helicopter unit for rescuing a miner trapped at the bottom.
Jamesbond is an environment where a player controls a vehicle whose objective is to move forward while avoiding and attacking enemies.
Kangaroo is an environment where a player controls a mother kangaroo, whose objective is to rescue her son while climbing.
Krull is an environment that controls a player should complete stages, which are central parts of the film which name is the same.
KungFuMaster is an environment where a player should control a fighter to save his girlfriend. He can use two types of attacks (punch and kick) and move/crunch/jump actions.
MsPacman is an environment where a player controls a Pacman in given mazes for eating pellets as much as possible while avoiding chasing masters. The Pacman loses one life if it contacts with monsters.
Pong is an environment about table tennis. RL agents control an in-game paddle to hit a ball back and forth. The objective is to gain 11 points before the opponent. The agents earn each point when the opponent fails to return the ball.
PrivateEye is an environment where a player controls a private eye. It requires action, adventure, and memorization. For solving five cases, the private eye should find and return items to suitable places.
Qbert is an environment where a player controls a character under a pyramid made of 28 cubes. The character should change the color of all cubes while avoiding obstacles and enemies.
RoadRunner is an environment where a player controls a roadrunner (chaparral bird). The roadrunner runs to the left on the road. RL agents should pick up bird seeds while avoiding chasing coyotes and obstacles such as cars.
Seaquest is an environment where a player controls a submarine, whose objective is to rescue divers while attacking enemies with missiles.
UpNDown is an environment where a player should control a purple dune buggy for collecting flags while avoiding other cars.

heading: B TRAINING DETAILS
text: Table B.1 includes the parameters that are used for the experiments under Model-free Reinforcement Learning (MfRL) in this paper. Since there are few benchmark results of PyBullet environments compared to MuJoCo environments, we followed parameters in (Raffin & Stulp, 2020) for SAC and
5β increases to 1.0 by the rule β = 0.4η + 1.0(1− η), where η = the current step/the maximum steps.

heading: C ADDITIONAL EXPERIMENTAL RESULTS
text: Figure C.1 shows Rainbow’s learning curves of the average of performances for 2M environment steps with and without MaPER on Atari games. Rainbow with MaPER consistently outperforms the original Rainbow in most experimental results.
The learning curves of SAC on MuJoCo environments are shown in Figure C.2. We observe that SAC with MaPER outperforms SAC with other sampling methods. Figure C.3 shows the learning curves of SAC (Haarnoja et al., 2018a) on the sparse reward environments in the main article (see Table B.3), which reveals that the original algorithm could not learn a policy for returns on these environments. We observe that combining MaPER with reward shaping methods guarantees the best results compared to other methods, including reward shaping methods solely.
Figure C.4 shows the effectiveness of both MaPER and learning the environment behavior, on HalfCheetahPyBullet. Comparing Figure 4 which considers HopperPyBullet, one can observe the same tendency, i.e., MaPER is necessary to improve the performance. Table C.1 shows the performance of SAC with MaPER on Sparse-HumanoidPyBullet (SAC) at time steps 1.0M, 1.5, and 2.0M, respectively. The table shows that using model rewards to compute TD-errors in MACN stimulates agents’ exploration.
One can observe the sensitivity of MaPER with respect to PER’s parameters (α, β) and coefficients in the loss (5) in Figure C.5.(a)-(b). In most cases, final results are similar, but some hyper-parameters degrade the performance of MaPER. To confirm the robustness of our method on an noise environment, we considered HumanoidPyBullet with Gaussian noise N (0, 0.01) to rewards and next states in Figure C.5.(c). MaPER considerably outperforms other methods since MaPER provides samples for improving the representation of inputs of critic networks.
Transitions of Sparse-Pendulum while training under SAC are displayed in Figure C.6. In both cases (MaPER and PER), the agents did not receive any reward. A position of the pendulum, which is trained by samples based only on TD-errors, is in downward. However, the pendulum trained by MaPER explores various states to receive a reward. Finally, Table C.2 shows holistic performances for different domains, which are derived in our paper. We normalized each method’s performance by that of MaPER for each domain. We believe that the considerably large gap between PER and MaPER in continuous control tasks under MfRL results from the actor-critic architecture (SAC and TD3). Compared to Q-learning, the advantage of the actor-critic architecture is to derive faster convergence (Konda & Tsitsiklis, 2000). Conversely speaking, if the performance of critic networks is consistently poor, then the policy networks can quickly converge to a poorly behaving policy so that learning will be stuck. We think that as the prediction of Q-values with MaPER quickly converges to the true returns (see Figure 1.(c) in the manuscript), the policy networks can be well-trained. Therefore, the large gap shows that how to improve the critic networks is crucial to not only Q-learning but also modern algorithms having the actor-critic architecture.

Review summaries:
This paper studies the problem of experience replay. It proposes Model-augmented Prioritized Experience Replay (MaPER), a novel experience replay method, based on the intuition that the model is easier to estimate than Q-value. It also proposes a modification to critic network, Model-augmented Critic Network (MaCN), by predicting the reward and dynamics model additionally. Experiments on MaPER show that MaPER can be applied to both discrete action space (DQN) and continuous action space (SAC), both model-free RL algorithms (SAC) and model-based RL algorithms (MBPO), as well as sparse reward tasks, and improve the baseline algorithms a lot. 

The paper proposes a new method for prioritizing experiences used in the prioritized experience replay (PER) method. The proposed approach is simple: the critic network also learns the reward function and transition function. The two errors (absolute value) are added to the absolute TD error to calculate the priorities in the PER method. The authors provide some intuitions to their method. Experiments on both Mujoco and Atari environments are presented to show the method’s effectiveness. 

Authors introduced Model-augmented Prioritized Experience Replay (MaPER) as a variation to the canonical prioritized sweeping. The main difference is that they extended the critic network on its last layer to also predict the reward and transition model. Then they changed the prioritization of sample for replay to also include the error in the reward and transition model, in addition to the TD-error. Experimental results in various MuJoC domains showed the performance boost introduced by MaPER on top of existing state-of-the-art techniques. Furthermore Authors carried an ablation study, investigation the impact of 1) incorporating the MaPER while all methods used the same network 2) incorporating various error values for prioritization, and 3) increasing the network size.

 *Summary Of The Paper Model-augmented Prioritized Experience Replay In this paper, the authors consider using the dynamics prediction error in the priority calculation. The priority is then used to decide the probability for each sample during training. The method is applicable to both model-based and model-free setting