Year: 2020

Abstract:
Policy entropy regularization is commonly used for better exploration in deep reinforcement learning (RL). However, policy entropy regularization is sampleinefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account. In order to take advantage of the previous sample distribution from the replay buffer for sample-efficient exploration, we propose sample-aware entropy regularization which maximizes the entropy of weighted sum of the policy action distribution and the sample action distribution from the replay buffer. We formulate the problem of sample-aware entropy regularized policy iteration, prove its convergence, and provide a practical algorithm named diversity actor-critic (DAC) which is a generalization of soft actor-critic (SAC). Numerical results show that DAC significantly outperforms SAC baselines and other state-of-the-art RL algorithms.

text: Policy entropy regularization is commonly used for better exploration in deep reinforcement learning (RL). However, policy entropy regularization is sampleinefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account. In order to take advantage of the previous sample distribution from the replay buffer for sample-efficient exploration, we propose sample-aware entropy regularization which maximizes the entropy of weighted sum of the policy action distribution and the sample action distribution from the replay buffer. We formulate the problem of sample-aware entropy regularized policy iteration, prove its convergence, and provide a practical algorithm named diversity actor-critic (DAC) which is a generalization of soft actor-critic (SAC). Numerical results show that DAC significantly outperforms SAC baselines and other state-of-the-art RL algorithms.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) aims to maximize the expectation of the discounted reward sum under Markov decision process (MDP) environments (Sutton & Barto, 1998). When the given task is complex, i.e. the environment has high action-dimensions or sparse rewards, it is important to well explore state-action pairs for high performance (Agre & Rosenschein, 1996). For better exploration, recent RL considers various methods: maximizing the policy entropy to take actions more uniformly (Ziebart et al., 2008; Fox et al., 2015; Haarnoja et al., 2017), maximizing diversity gain that yields intrinsic rewards to explore rare states by counting the number of visiting states (Strehl & Littman, 2008; Lopes et al., 2012), maximizing information gain (Houthooft et al., 2016; Hong et al., 2018), maximizing model prediction error (Achiam & Sastry, 2017; Pathak et al., 2017), and so on. In particular, based on policy iteration for soft Q-learning, (Haarnoja et al., 2018a) considered an offpolicy actor-critic framework for maximum entropy RL and proposed the soft actor-critic (SAC) algorithm, which has competitive performance for challenging continuous control tasks.
In this paper, we reconsider the problem of policy entropy regularization in off-policy learning and propose a generalized approach to policy entropy regularization. In off-policy learning, we store and reuse old samples to update the current policy (Mnih et al., 2015), and it is preferable that the old sample distribution in the replay buffer is uniformly distributed for better performance. However, the simple policy entropy regularization tries to maximize the entropy of the current policy irrespective of the distribution of previous samples. Since the uniform distribution has maximum entropy, the current policy will choose previously less-sampled actions and more-sampled actions with the same probability and hence the simple policy entropy regularization is sample-unaware and sample-inefficient. In order to overcome this drawback, we propose sample-aware entropy regularization, which tries to maximize the weighted sum of the current policy action distribution and the sample action distribution from the replay buffer. We will show that the proposed sampleaware entropy regularization reduces to maximizing the sum of the policy entropy and the α-skewed Jensen-Shannon divergence (Nielsen, 2019) between the policy distribution and the buffer sample action distribution, and hence it generalizes SAC. We will also show that properly exploiting the sample action distribution in addition to the policy entropy over learning phases will yield far better performance.

heading: 2 RELATED WORKS
text: Entropy regularization: Entropy regularization maximizes the sum of the expected return and the policy action entropy. It encourages the agent to visit the action space uniformly for each given state, and the regularized policy is robust to modeling error (Ziebart, 2010). Entropy regularization is considered in various domains for better optimization: inverse reinforcement learning (Ziebart et al., 2008), stochastic optimal control problems (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2013), and off-policy reinforcement learning (Fox et al., 2015; Haarnoja et al., 2017). (Lee et al., 2019) shows that Tsallis entropy regularization that generalizes usual Shannon-entropy regularization is helpful. (Nachum et al., 2017a) shows that there exists a connection between value-based and policybased RL under entropy regularization. (O’Donoghue et al., 2016) proposed an algorithm combining them, and it is proven that they are equivalent (Schulman et al., 2017a). The entropy of state mixture distribution is better for pure exploration than a simple random policy (Hazan et al., 2019).
Diversity gain: Diversity gain is used to provide a guidance for exploration to the agent. To achieve diversity gain, many intrinsically-motivated approaches and intrinsic reward design methods have been considered, e.g., intrinsic reward based on curiosity (Chentanez et al., 2005; Baldassarre & Mirolli, 2013), model prediction error (Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018), divergence/information gain (Houthooft et al., 2016; Hong et al., 2018), counting (Strehl & Littman, 2008; Lopes et al., 2012; Tang et al., 2017; Martin et al., 2017), and unification of them (Bellemare et al., 2016). For self-imitation learning, (Gangwani et al., 2018) considered the Steinvariational gradient decent with the Jensen-Shannon kennel.
Off-policy learning: Off-policy learning can reuse any samples generated from behaviour policies for the policy update (Sutton & Barto, 1998; Degris et al., 2012), so it is sample-efficient as compared to on-policy learning. In order to reuse old samples, a replay buffer that stores trajectories generated by previous policies is used for Q-learning (Mnih et al., 2015; Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018a). To enhance both stability and sample-efficiency, several methods are considered, e.g., combining on-policy and off-policy (Wang et al., 2016; Gu et al., 2016; 2017), and generalization from on-policy to off-policy (Nachum et al., 2017b; Han & Sung, 2019).
In order to guarantee the convergence of Q-learning, there is a key assumption: Each state-action pair must be visited infinitely often (Watkins & Dayan, 1992). If the policy does not visit diverse state-action pairs many times, it converges to local optima. Therefore, exploration for visiting different state-action pairs is important for RL, and the original policy entropy regularization encourages exploration (Ahmed et al., 2019). However, we found that the simple policy entropy regularization can be sample-inefficient in off-policy RL, so we aim to propose a new entropy regularization method that significantly enhances the sample-efficiency for exploration by considering the previous sample distribution in the buffer.

heading: 3 BACKGROUND
text: In this section, we briefly introduce the basic setup and the soft actor-critic (SAC) algorithm.

heading: 3.1 SETUP
text: We assume a basic RL setup composed of an environment and an agent. The environment follows an infinite horizon Markov decision process (S,A, P, γ, r), where S is the state space, A is the action space, P is the transition probability, γ is the discount factor, and r : S × A → R is the reward function. In this paper, we consider a continuous state-action space. The agent has a policy distribution π : S × A → [0,∞) which selects an action at for a given state st at each time step t, and the agent interacts with the environment and receives reward rt := r(st, at) from the environment. Standard RL aims to maximize the discounted return Es0∼p0,τ0∼π[ ∑∞ t=0 γ
trt], where τt = (st, at, st+1, at+1 · · · ) is an episode trajectory.

heading: 3.2 SOFT ACTOR-CRITIC
text: Soft actor-critic (SAC) (Haarnoja et al., 2018a) includes a policy entropy regularization term in the objective function for better exploration by visiting the action space uniformly for each given state.
The entropy-augmented policy objective function of SAC is given by
JSAC(π) = Eτ0∼π [ ∞∑ t=0 γt(rt + βH(π(·|st))) ] , (1)
where H is the entropy function and β ∈ (0,∞) is the entropy coefficient. SAC is a practical offpolicy actor-critic based on soft policy iteration (SPI) that alternates soft policy evaluation to estimate the true soft Q-function and soft policy improvement to find the optimal policy that maximizes (1). In addition, SPI theoretically guarantees convergence to the optimal policy that maximizes (1).

heading: 4 THE DIVERSITY ACTOR-CRITIC ALGORITHM

heading: 4.1 MOTIVATION OF THE SAMPLE-AWARE ENTROPY
text: As explained in Section 2, the policy should visit diverse samples to learn the policy without converging to the local optima. In off-policy learning, we can reuse previous samples stored in the replay buffer to learn the policy, so it is efficient to draw diverse samples while avoiding frequently selected samples before. The policy entropy maximization enhances exploration to yield better performance, but it is sample-inefficient for off-policy RL because it does not take advantage of the previous sample action distribution obtainable from the replay buffer: If we assume bounded action space, the simple policy entropy maximization will choose all actions with the equal probability without considering the previous action samples because maxπH(π) = minπ DKL(π||U) is achieved when π = U , where U is a uniform distribution and DKL is the Kullback-Leibler (KL) divergence. In order to overcome the limitation of the simple policy entropy maximization, we consider maximizing a sample-aware entropy defined as the entropy of a mixture distribution of the policy distribution π and the current sample action distribution q in the replay buffer. Here, q is defined as
q(·|s) := ∑ a∈DN(s, a)δa(·)∑ a′∈DN(s, a ′) , (2)
where D is the replay buffer that stores previous samples (st, at, rt, st+1) at each time t, δa(·) is the Dirac measure at a ∈ A, and N(s, a) is the number of state-action pair (s, a) in D. Then, we define a target distribution qπ,αtarget as the mixture distribution of π and q, which is expressed as qπ,αtarget := απ + (1 − α)q, where α ∈ [0, 1] is the weighting factor. Note that we draw samples from policy π and store them in the replay buffer, so the target distribution can be viewed as the updated sample action distribution in the future replay buffer. Then, maximizing the sample-aware entropy H(qπ,αtarget) can encourage sample-efficient exploration because π will choose actions rare in the buffer with high probability and actions stored many times in the buffer with low probability in order to make the target distribution uniform. We provide a simple example below:
Let us consider a simple 1-step MDP in which s0 is the unique initial state, there exist Na actions (A = {A0, · · · , ANa−1}), s1 is the terminal state, and r is a deterministic reward function. Then, there exist Na state-action pairs in total and let us assume that we already have Na − 1 state-action samples in the replay buffer as R = {(s0, A0, r(s0, A0)), · · · , (s0, ANa−2, r(s0, ANa−2))}. In order to estimate the Q-function for all state-action pairs, the policy should sample the last action ANa−1 (After then, we can reuse all samples infinitely to estimate Q). Here, we will compare two exploration methods.
1) First, if we consider the simple entropy maximization, the policy that maximizes its entropy will choose all actions with equal probability 1/Na (uniformly). Then, Na samples should be taken on average by the policy to visit the action ANa−1.
2) Consider the sample-aware entropy maximization. Here, the sample action distribution q in the buffer becomes q(a0|s0) = 1/(Na − 1) for a0 ∈ {A0, · · · , ANa−2} and q(ANa−1|s0) = 0, the target distribution becomes qπ,αtarget = απ + (1 − α)q, and we set α = 1/Na. Then, the policy that maximizes the sample-aware entropy becomes π(ANa−1|s0) = 1 to make q π,α target uniform because maxπH(qπ,αtarget) = minπ DKL(q π,α target||U). In this case, we only needs one sample to visit the
action ANa−1. In this way, the simple entropy maximization is sample-inefficient for off-policy RL, and the proposed sample-aware entropy maximization can enhance the sample-efficiency for exploration by using the previous sample distribution and choosing a proper α. With this motivation, we propose the sample-aware entropy regularization for off-policy RL and the corresponding αadaptation method.

heading: 4.2 SAMPLE-AWARE ENTROPY REGULARIZATION
text: Our approach is to maximize the return while maximizing the sample-aware entropy. Under this approach, previously many times sampled actions will be given low probabilities and previously less taken actions will be given high probabilities by the current policy π for sample-efficient exploration as shown in Section 4.1. Hence, we set the objective function for the proposed sample-aware entropy regularization as
J(π) = Eτ0∼π [ ∞∑ t=0 γt(rt + βH(qπ,αtarget(·|st))) ] . (3)
Here, the sample-aware entropyH(qπ,αtarget) for given st can be decomposed as
H(qπ,αtarget) = − ∫ a∈A (απ + (1− α)q) log(απ + (1− α)q) = αH(π) +DαJS(π||q) + (1− α)H(q), (4)
whereDαJS(π||q) := αDKL(π||q π,α target)+(1−α)DKL(q||q π,α target) is the α skew-symmetric JensenShannon (JS) divergence (Nielsen, 2019). Note that DαJS reduces to the standard JS divergence for α = 12 and to zero for α = 0 or 1. Hence, for α = 1, (4) reduces to the simple entropy, but for α 6= 1, it is a generalization incorporating the distribution q. Thus, our objective function aims to maximize the return while simultaneously maximizing the discounted sum of policy entropy, sample entropy, and the divergence between π and q. In this way, the policy will choose more diverse actions that are far from the samples stored in the replay buffer while maintaining its entropy for better exploration.

heading: 4.3 DIVERSE POLICY ITERATION WITH THE PROPOSED OBJECTIVE
text: In this section, we derive the diverse policy evaluation and diverse policy improvement to maximize the objective function with the sample-aware entropy regularization (3). Note that the sample action distribution q is updated as the iteration goes on. However, it changes very slowly since the buffer size is much larger than the time steps of one iteration. Hence, for the purpose of proof we regard the action distribution q as a fixed distribution in this section.
First, we define the true diverse Q-function Qπ as Qπ(st, at) := 1β rt + Eτt+1∼π [∑∞ l=t+1 γ l−t−1 ( 1 β rl + αH(π(·|sl)) +D α JS(π(·|sl)||q(·|sl)) + (1− α)H(q(·|sl)) )] .
We defined the sample distribution q in equation (2), but we do not want to compute actual q, which requires a method such as discretization and counting for continuous samples. Even if q is obtained by counting, a generalization of q for arbitrary state-action pairs is needed again to estimate Qπ . We circumvented this difficulty by defining the ratio Rπ,α of απ to qπ,αtarget as
Rπ,α(st, at) = απ(at|st)
απ(at|st) + (1− α)q(at|st) , (5)
and we will show later that all objective (or loss) functions for practical implementation can be represented by using the ratio only, without using the explicit q in Appendix B.
Then, we can decompose DαJS(π(·|sl)||q(·|sl)) as
DαJS(π||q) = αEal∼π(·|sl)[logR π,α(sl, al)] + (1− α)Eal∼q(·|sl)[log(1−R π,α(sl, al))] +H(α), (6)
where H(α) = −α logα− (1− α) log(1− α) is the binary entropy function. The modified Bellman backup operator for Qπ estimation is given by
T πQ(st, at) := 1
β rt + γEst+1∼P [V (st+1)], (7)
where V (st) = Eat∼π[Q(st, at) + α logRπ,α(st, at)− α logαπ(at|st)] + (1− α)Eat∼q[log(1− Rπ,α(st, at))− log(1− α)q(at|st)] is an estimated diverse state value function, Q : S ×A → R is an estimated diverse state-action value function.
Proof of the convergence of diverse policy evaluation that estimates Qπ by repeating the Bellman operator (7) is provided in Appendix A. Then, the policy is updated from πold to πnew as πnew = arg maxπ Jπold(π), where Jπold(π) is the objective of π estimated under Q πold defined as1
Jπold(π(·|st)) := β{Eat∼π [Qπold(st, at) + α logRπ,α(st, at)− α logαπ(at|st)] + (1− α)Eat∼q [log(1−Rπ,α(st, at))− log(1− α)q(at|st)]}. (8)
The monotone improvement of this step is proved in Appendix A. Now, we can find the optimal policy that maximizes J(π)(= Jπ(π)) by the following theorem:
Theorem 1 (Diverse Policy Iteration) By repeating iteration of the diverse policy evaluation and the diverse policy improvement, any initial policy converges to the optimal policy π∗ s.t. Qπ ∗ (st, at) ≥ Qπ ′ (st, at), ∀ π′ ∈ Π, ∀ (st, at) ∈ S × A. Also, such π∗ achieves maximum J , i.e., Jπ∗(π∗) ≥ Jπ(π) for any π ∈ Π.
Proof. See Appendix A.1.
Note that Jπold(π) for diverse policy iteration above requires the ratio function R π,α of the current policy π, but we can only estimate Rπold,α for the previous policy πold in practice. Thus, we circumvent this difficulty by defining a practical objective function Jπold(π) given by
J̃πold(π(·|st)) := βEat∼π[Qπold(st, at) + α logRπold,α(st, at)− α log π(at|st)], (9)
Regarding the practically computable objective function J̃πold(π), we have the following result:
Theorem 2 Suppose that the policy is parameterized with parameter θ. For parameterized policy πθ, two objective functions Jπθold (πθ(·|st)) and J̃πθold (πθ(·|st)) have the same gradient direction for θ at θ = θold for all st ∈ S.
Proof. See Appendix A.2.
By Theorem 2, we can replace the objective function Jπold(π) of policy improvement with the practically computable objective function J̃πold(π) for parameterized policy without loss of optimality.

heading: 4.4 DIVERSITY ACTOR CRITIC IMPLEMENTATION
text: We first define Rα as an estimate for the ratio function Rπold,α. For implementation, we parameterize π, Rα, Q, and V by neural network parameters θ, η, φ, and ψ, respectively. Then, we setup the practical objective (or loss) functions Ĵπ(θ), ĴRα(η), L̂Q(φ), and L̂V (ψ) for the parameter update. Detailed DAC implementation based on Section 4 is provided in Appendix B. The proposed DAC algorithm is summarized in Appendix C. Note that DAC becomes SAC when α = 1, and becomes standard off-policy RL without entropy regularization when α = 0.
5 α-ADAPTATION
In the proposed sample-aware entropy regularization, the weighting factor α plays an important role in controlling the ratio between the policy distribution π and the sample action distribution q. However, it is difficult to estimate optimal α directly. Hence, we further propose an adaptation method for α based on max-min principle widely considered in game theory, robust learning, and decision making problems (Chinchuluun et al., 2008). Since we do not know optimal α, an alternative formulation is that we maximize the return while maximizing the worst-case sample-aware entropy, i.e., minαH(qπ,αtarget). Then, the max-min approach can be formulated as follows:
max π Eτ0∼π [∑ t γt(rt + βmin α [H(qπ,αtarget)− αc]) ] (10)
1Note that if we replace πold with π and view every state st as an initial state, then (8) reduces to J(π).
where c is a control hyperparameter for α adaptation. We learn α to minimize H(qπ,αtarget) − αc, so the role of c is to maintain the target entropy at a certain level to explore the state-action well. Detailed implementation for α-adaptation is given in Appendix B.1.

heading: 6 EXPERIMENTS
text: In this section, we evaluate the proposed DAC algorithm on various continuous-action control tasks and provide ablation study. In order to see the superiority of the sample-aware entropy regularization, we here focus on comparison with two SAC baselines: SAC and SAC-Div. SAC-Div is SAC combined with the method in (Hong et al., 2018) that diversifies policies from buffer distribution by simply maximizing J(π) + αdD(π||q) for J(π) in (1) and some divergence D. Note that the key difference between SAC-Div and DAC is that SAC-Div simply adds the single divergence term to the policy objective function J(π), whereas DAC considers the discounted sum of target entropy terms as seen in (3). For SAC-Div, we consider KL divergence (MSE if the policy is Gaussian) and adaptive scale αd with δd = 0.2 for the divergence term as suggested in (Hong et al., 2018). In order to rule out the influence of factors other than exploration, we use the common simulation setup for DAC and SAC baselines except for the parts about entropy or divergence.
In addition, we provide comparison of DAC to random network distillation (RND) (Burda et al., 2018) and MaxEnt (Hazan et al., 2019), which are the recent exploration methods based on finding rare states in Appendix F.2, and to other recent RL algorithms in Appendix F.3. The result shows that DAC yields the best performance for all considered tasks as compared to recent RL algorithms. We also provide the source code of DAC implementation that requires Python Tensorflow. Detailed simulation setup for experiments is summarized in Appendix E.

heading: 6.1 PURE EXPLORATION COMPARISON
text: In order to see the exploration performance of DAC (α = 0.5) as compared to the SAC baselines, we compare state visitation on a 100 × 100 continuous 4-room maze task. The maze environment is made by modifying a continuous grid map available at https://github.com/huyaoyu/ GridMap, and it is shown in Fig. 1(a). State is (x, y) position in the maze, action is (dx, dy) bounded by [−1, 1], and the agent location after the action becomes (x + dx, y + dy). The agent starts from the left lower corner (0.5, 0.5) and explores the maze without any reward, and Fig. 1(b) shows the mean number of new state visitations over 30 seeds, where the number of state visitation is obtained for each integer interval. As seen in Fig. 1(b), DAC visited much more states than SAC/SAC-Div, which means that the exploration performance of DAC is superior to that of the SAC baselines. In addition, Fig. 1(c) shows the corresponding state visit histogram of all seeds. Here, as the color of the state becomes brighter, the state is visited more times. Note that SAC/SAC-Div rarely visit the right upper room even at 500k time steps for all seeds, but DAC starts visiting the
right upper room at 5k time steps and frequently visit the right upper room at 500k time steps. Thus, Fig. 1(c) clearly shows that DAC has better sample-efficiency for exploration than SAC/SAC-Div.

heading: 6.2 PERFORMANCE COMPARISON WITH THE SAC BASELINES
text: The final goal of RL is to achieve high scores for given tasks. For this, exploration techniques are needed to ensure that the policy does not converge to local optima, as explained in Section 2. We first showed the improvement of the exploration performance in a pure exploration task (continuous 4-room maze), and experiments in this section will show that DAC has better return performance than SAC baselines on several sparse-rewarded tasks. Note that having high scores on the sparsereward tasks means that the policy can get rewards well without falling into local optima, which implies that the agent successfully explores more state-action pairs that have positive (or diverse) rewards. Therefore, the performance comparison on sparse tasks fits well to the motivation and also note that sparse-rewarded tasks has been widely used as a verification method of exploration in many previous exploration studies (Hong et al., 2018; Mazoure et al., 2019; Burda et al., 2018).
Fixed α case: In order to see the advantage of the sample-aware entropy regularization for rewarded tasks, we compare the performance of DAC with α = 0.5 and the SAC baselines on simple MDP tasks: SparseMujoco tasks. SparseMujoco is a sparse version of Mujoco and the reward is 1 if the agent exceeds the x-axis threshold, otherwise 0 (Hong et al., 2018; Mazoure et al., 2019).
The performance results averaged over 10 random seeds are shown in Fig. 2. As seen in Fig. 2, DAC has significant performance gain for most tasks as compared to SAC. On the other hand, SAC-Div also enhances the convergence speed compared to SAC for some tasks, but it fails to enhance the final performance. Fig. 3 shows the α-skewed JS divergence curve (α = 0.5) of DAC and SAC/SAC-Div for sparse Mujoco tasks and we provide Fig. F.1 in Appendix F.1 that shows the corresponding mean number of discretized state visitation curve on sparse Mujoco tasks. For SAC/SAC-Div, the ratio function R is estimated separately from (B.2) in Appendix B and the divergence is computed from R. The performance table for all tasks is given by Table F.1 in Appendix F.1. As seen in Fig. 3, the divergence of DAC is much higher than that of SAC/SAC-Div throughout the learning time. It means that the policy of DAC choose more diverse actions from the distribution far away from the sample action distribution q, then DAC visits more diverse states than the SAC baselines as seen in Fig. F.1. Thus, DAC encourages better exploration and it yields better performance. Thus, we can conclude that the proposed sample-aware entropy regularization is superior to the simple policy entropy regularization of SAC and single divergence regularization of SAC-Div in terms of exploration and the convergence.
Adaptive α case: Now, we compare the performance of DAC with α = 0.5, 0.8, α-adaptation, and the SAC baselines to see the need of α-adaptation. To maintain controllability and prevent saturation
of Rαη , we used regularization for α learning and restricted the range of α as 0.5 ≤ α ≤ 0.99 for α adaptation so that a certain level of entropy regularization is enforced. Here, we consider more complicated tasks: HumanoidStandup and delayed Mujoco tasks (DelayedHalfCheetah, DelayedHopper, DelayedWalker2d, and DelayedAnt). HumanoidStandup is one of high-action dimensional Mujoco tasks. Delayed Mujoco tasks suggested by (Zheng et al., 2018; Guo et al., 2018) have the same state-action spaces with original Mujoco tasks but reward is sparsified. That is, rewards for D time steps are accumulated and the accumulated sum is delivered to the agent once every D time steps, so the agent receives no reward during the accumulation time. The performance results averaged over 5 random seeds are shown in Fig. 4. The result of the max average return of these Mujoco tasks for DAC and SAC/SAC-Div is provided in Table F.2 in Appendix F.1. As seen in Fig. 4, all versions of DAC outperform SAC. Here, SAC-Div also outperforms SAC for several tasks, but the performance gain by DAC is much higher. In addition, it is seen that the best α depends on the tasks in the fixed α case. For example, α = 0.8 is the best for DelayedHalfCheetah, but α = 0.5 is the best for DelayedAnt. Thus, we need to adapt α for each task. Finally, DAC with α-adaptation has the top-level performance for most tasks and the best performance for HumanoidStandup and DelayedHopper tasks. Further consideration for α is provided in Section 6.3.

heading: 6.3 ABLATION STUDY
text: In this section, we provide ablation study for important parameters in the sample-aware entropy regularization on the DelayedHalfCheetah task. Ablation studies on the other DelayedMucoco tasks are provided in Appendix G.
Weighting factor α: As seen in Section 6.2, α-adaptation is necessary because one particular value of α is not best for all environments. Although the proposed α-adaptation in Section 5 is suboptimal, it shows good performance across all the considered tasks. Thus, we study more on the proposed α-adaptation and the sensible behavior of sample-awareness in entropy regularization.
Fig. 5(a) shows the averaged learning curve of α, α-skewed JS divergenceDJS(π||q) and the policy entropy H(π) for DAC with the proposed α-adaptation method on DelayedHalfCheetah. Here, we fix the control coefficient c as −2.0dim(A). As seen in (3), the return, the policy entropy and the JS divergence are intertwined in the cost function, so their learning curves are also intertwined over time steps. Here, the learned policy entropy term decreases and the learned α increases to one as time step goes on. Then, the initially nonzero JS divergence term DJS(π||q) diminishes to zero, which means that the sample action distribution is exploited for roughly initial 2.5M time steps, and then DAC operates like SAC. This adaptive exploitation of the sample-aware entropy leads to better overall performance across time steps as seen in Fig. 4, so DAC with α-adaptation seems to properly exploit both the policy entropy and the sample action distribution depending on the learning stage.
Control coefficient c: In the proposed α-adaptation (10), the control coefficient c affects the learning behavior of α. Since H(π) and DαJS are proportional to the action dimension, we tried a few values such as 0, −0.5d, −1.0d and −2.0d where d = dim(A). Fig. 5(b) shows the corresponding performance of DAC with α-adaptation on DelayedHalfCheetah. As seen in Fig. 5(b), the performance depends on the change of c as expected, and c = −2.0 ·dim(A) performs well. We observed that −2.0d performed well for all considered tasks, thus we set c = −2.0d in (B.8). Entropy coefficient β: As mentioned in (Haarnoja et al., 2018a), the performance of SAC depends on β. It is expected that the performance of DAC depends on β too. Fig 5(c) shows the performance of DAC with fixed α = 0.5 for three different values of β: β = 0.1, 0.2 and 0.4 on DelayedHalfCheetah. It is seen that the performance of DAC indeed depends on β. Although there exists performance difference for DAC depending on β, the performance of DAC is much better than SAC for a wide range of β. One thing to note is that the coefficient of pure policy entropy regularization term for DAC is αβ, as seen in (3). Thus, DAC with α = 0.5 and β = 0.4 has the same amount of pure policy entropy regularization as SAC with β = 0.2. However, DAC with α = 0.5 and β = 0.4 has much higher performance than SAC with β = 0.2, as seen in Fig. 5(c). So, we can see that the performance improvement of DAC comes from joint use of policy entropyH(π) and the sample action distribution from the replay buffer via DαJS(π||q). The effect of JS divergence: In order to see the effect of the JS divergence on the performance, we also provide an additional ablation study that we consider a single JS divergence for SAC-Div by using the ratio function in Section 4.3. 5(d) shows the performance comparison of SAC, SACDiv(KL), SAC-Div(JS), and DAC. For SAC-Div(JS), we used δd = 0.5 for adaptive scaling in (Hong et al., 2018). As a result, there was no significant difference in performance between SAC-Div with JS divergence and SAC-Div with KL divergence. On the other hand, the DAC still shows a greater performance increase than both SAC-Div(KL) and SAC-Div(JS), and this means that the DAC has more advantages than simply using JS divergence.

heading: 7 CONCLUSION AND FUTURE WORKS
text: In this paper, we have proposed a sample-aware entropy framework for off-policy RL to overcome the limitation of simple policy entropy for sample-efficient exploration. With the sample-aware entropy regularization, we can achieve diversity gain by exploiting sample history in the replay buffer in addition to policy entropy. For practical implementation of sample-aware entropy regularized policy optimization, we have proposed the DAC algorithm with convergence proof. We have also provided an adaptation method for DAC to control the ratio of the sample action distribution to the policy action entropy. DAC is an actor-critic algorithm for sample-aware regularized policy optimization and generalizes SAC. Numerical results show that DAC significantly outperforms SAC baselines in Maze exploration and various Mujoco tasks.
For further study, we consider a generalization of our method in order to deal with the entropy of the state-action distribution. Currently, many recent papers only consider one of the entropy of state distribution dπ(s) or that of action distribution π(a|s) only since they have much different properties (e.g. the state-based entropy is non-convex on π and the action-based entropy is convex on π). However, both entropies can be handled simultaneously as one fused entropy that deals with the entropy of the state-action distribution, factorized as log dπ(s, a) = log dπ(s) + log π(a|s). Then, the generalization of our method for the fused entropy may be able to further enhance the exploration performance by considering the exploration on the entire state-action space.

heading: A PROOFS
text: A.1 PROOF OF THEOREM 1
For a fixed policy π, Qπ can be estimated by repeating the Bellman backup operator by Lemma 1. Lemma 1 is based on usual policy evaluation but has a new ingredient of the ratio condition in the sample-aware case.
Lemma 1 (Diverse Policy Evaluation) Define a sequence of diverse Q-functions as Qk+1 = T πQk, k ≥ 0, where π is a fixed policy and Q0 is a real-valued initial Q. Assume that the action space is bounded, and Rπ,α(st, at) ∈ (0, 1) for all (st, at) ∈ S ×A. Then, the sequence {Qk} converges to the true diverse state-action value Qπ .
Proof. Let rπ,t := 1β rt+γEst+1∼P [Eat+1∼π[α logR π,α(st+1, at+1)−α logαπ(at+1|st+1)]+(1− α)Eat+1∼q[log(1 − Rπ,α(st+1, at+1)) − log(1 − α)q(at+1|st+1)]]. Then, we can formulate the standard Bellman equation form for the true Qπ as
T πQ(st, at) = rπ,t + γEs+1∼P, at+1∼π [Q(st+1, at+1)] (A.1)
Under the assumption of a bounded action space and Rπ,α ∈ (0, 1), the reward rπ,t is bounded and the convergence is guaranteed as the usual policy evaluation (Sutton & Barto, 1998; Haarnoja et al., 2018a).
Now, we prove diverse policy improvement in Lemma 2 and diverse policy iteration in Theorem 1 by using Jπold(π) in a similar way to usual RL or SAC.
Lemma 2 (Diverse Policy Improvement) Let πnew be the updated policy obtained by solving πnew = arg max
π∈Π Jπold(π). Then, Q πnew(st, at) ≥ Qπold(st, at), ∀ (st, at) ∈ S ×A.
Proof. We update the policy to maximize Jπold(π), so Jπold(πnew) ≥ Jπold(πold). Hence,
Eat∼πnew [Qπold(st, at) + α logRπnew,α(st, at)− α logαπnew(at|st)] + (1− α)Eat∼q[log(1−Rπnew,α(st, at))− log(1− α)q(at|st)]
≥Eat∼πold [Qπold(st, at) + α logRπold,α(st, at)− α logαπold(at|st)] + (1− α)Eat∼q[log(1−Rπold,α(st, at))− log(1− α)q(at|st)] =V πold(st) (A.2)
By repeating the Bellman equation (7) and (A.2) at Qπold ,
Qπold(st, at) = 1
β rt + γEst+1∼P [V πold(st+1)]
≤ 1 β rt + γEst+1∼P [Eat+1∼πnew [Qπold(st+1, at+1) + α logRπnew,α(st+1, at+1)
− α logαπnew(at+1|st+1)] + (1− α)Eat+1∼q[log(1−Rπnew,α(st+1, at+1)) − log(1− α)q(at+1|st+1)]]
... ≤ Qπnew(st, at), (A.3)
for each (st, at) ∈ S ×A.
Theorem 1 (Diverse Policy Iteration) By repeating iteration of the diverse policy evaluation and the diverse policy improvement, any initial policy converges to the optimal policy π∗ s.t. Qπ ∗ (st, at) ≥ Qπ ′ (st, at), ∀ π′ ∈ Π, ∀ (st, at) ∈ S × A. Also, such π∗ achieves maximum J , i.e., Jπ∗(π∗) ≥ Jπ(π) for any π ∈ Π.
Proof. Let {πi : i ≥ 0, πi ∈ Π} be a sequence of policies s.t. πi+1 = arg maxπ∈Π Jπi(π). For arbitrary state action pairs (s, a) ∈ S×A, {Qπi(s, a)}monotonically increases by Lemma 2 and each Qπi(s, a) is bounded. Also, πi+1 is obtained by the policy improvement that maximizes Jπi(π(·|s)), so Jπi(πi+1(·|s)) ≥ Jπi(πi(·|s)) as stated in the proof of Lemma 2. From the definition of Jπold(π) in (8), all terms are the same for Jπi+1(πi+1(·|s)) and Jπi(πi+1(·|s)) except βEa∼πi+1 [Qπi+1(s, a)] in Jπi+1(πi+1(·|s)) and βEa∼πi+1 [Qπi(s, a)] in Jπi(πi+1(·|s)). Since {Qπi(s, a)} monotonically increases, Jπi+1(πi+1(·|s)) ≥ Jπi(πi+1(·|s)). Finally, Jπi+1(πi+1(·|s)) ≥ Jπi(πi+1(·|s)) ≥ Jπi(πi(·|s)) for any state s ∈ S, so the sequence {Jπi(πi(·|s)} also monotonically increases, and each Jπi(πi(·|s)) is bounded because Q-function and the target entropy are bounded. By the monotone convergence theorem, {Qπi} and {Jπi(πi)} pointwisely converge to their optimal functions Q∗ : S ×A → R and J∗ : S → R, respectively. Here, note that J∗(s) ≥ Jπi(πi(·|s)) for any i because the sequence {Jπi(πi)} is monotonically increasing. From the definition of convergent sequence, for arbitrary > 0, there is a large N ≥ 0 s.t. Jπi(πi(·|s)) ≥ J∗(s)− (1−γ) γ satisfies for all i ≥ N and any s ∈ S.
Now, we can easily show that Jπk(πk(·|s)) ≥ Jπk(π(·|s))− (1−γ) γ for any k > N , any policy π ∈ Π, and any s ∈ S. (If not, Jπk(πk+1) = maxπ′ Jπk(π′) ≥ Jπk(π), and then Jπk+1(πk+1(·|s′)) ≥ Jπk(πk+1(·|s′)) ≥ Jπk(π(·|s′)) > Jπk(πk(·|s′)) + (1−γ) γ ≥ J
∗(s′) for some s′ ∈ S . Clearly, it contradicts the monotone increase of the sequence {Jπi(πi)}.) Then, by the similar way with (A.3),
Qπk(st, at) = 1
β rt + γEst+1∼P [V πk(st+1)] =
1 β rt + γEst+1∼P [Jπk(πk(·|st+1))]
≥ 1 β rt + γEst+1∼P
[ Jπk(π(·|st+1))−
(1− γ) γ ] = 1
β rt + γEst+1∼P [Eat+1∼π[Qπk(st+1, at+1) + α logRπ,α(st+1, at+1)− α logαπ(at+1|st+1)]
+ (1− α)Eat+1∼q[log(1−Rπ,α(st+1, at+1))− log(1− α)q(at+1|st+1)]]− (1− γ) ... ≥ Qπ(st, at)− . (A.4)
Note that the state action pair (s, a), the policy π, and > 0 were arbitrary, so we can conclude that Qπ∞(s, a) ≥ Qπ(s, a) for any π ∈ Π and (s, a) ∈ S×A. In addition, we show that Jπk(πk(·|s)) ≥ Jπk(π(·|s)) − (1−γ) γ , so Jπ∞(π∞(·|s)) ≥ Jπ(π(·|s)) for any π ∈ Π and any s ∈ S. Thus, π∞ is the optimal policy π∗, and we can conclude that {πi} converges to the optimal policy π∗.
A.2 PROOF OF THEOREM 2
Theorem 2 Suppose that the policy is parameterized with parameter θ. Then, for parameterized policy πθ, two objective functions Jπθold (πθ(·|st)) and J̃πθold (πθ(·|st)) have the same gradient direction for θ at θ = θold for all st ∈ S.
Proof. Under the parameterization of πθ, the two objective functions become
Jπθold (πθ(·|st)) = β(Eat∼πθ [Q πθold (st, at) + α logR πθ,α(st, at)− α log πθ(at|st)] + (1− α)Eat∼q[log(1−Rπθ,α(st, at))− log q(at|st)]) +H(α)
J̃πθold (πθ(·|st)) = βEat∼πθ [Q πold(st, at) + α logR πold,α(st, at)− α log πθ(at|st)].
We can ignore the common Q-function and log πθ terms, and the constant terms w.r.t. θ that leads zero gradient in both objective functions. Thus, we only need to show
∇θ[αEat∼πθ [logRπθ,α] + (1− α)Eat∼q[log(1−Rπθ,α)]] = ∇θEat∼πθ [α logRπθold ,α] (A.5)
at θ = θold. Now, the gradient of the left term in (A.5) at θ = θold can be expressed as
∇θ[αEat∼πθ [logRπθ,α] + (1− α)Eat∼q[log(1−Rπθ,α)]] = αEat∼πθ [logRπθ,α · ∇θ log πθ]
+ αEat∼πθ [∇θ logRπθ,α] + (1− α)Eat∼q[∇θ log(1−Rπθ,α)] = ∇θαEat∼πθ [α logRπθold ,α]
+ αEat∼πθ [∇θ logRπθ,α] + (1− α)Eat∼q[∇θ log(1−Rπθ,α)]. (A.6)
Here, the gradient of the last two terms in (A.6) becomes zero as shown below:
αEat∼πθ [∇θ logRπθ,α] + (1− α)Eat∼q[∇θ log(1−Rπθ,α)] = αEat∼πθ [∇θRπθ,α/Rπθ,α] + (1− α)Eat∼q[∇θ(1−Rπθ,α)/(1−Rπθ,α)] = αEat∼πθ [∇θRπθ,α/Rπθ,α]− (1− α)Eat∼q[∇θRπθ,α/(1−Rπθ,α)]
= αEat∼πθ [∇θRπθ,α/Rπθ,α]− (1− α)Eat∼q [ απθ + (1− α)q
(1− α)q · ∇θRπθ,α ] (1) = αEat∼πθ [∇θRπθ,α/Rπθ,α]− αEat∼πθ [ απθ + (1− α)q
απθ · ∇θRπθ,α ] = αEat∼πθ [∇θRπθ,α/Rπθ,α]− αEat∼πθ [∇θRπθ,α/Rπθ,α] = 0, (A.7)
where we used an importance sampling technique at Eat∼q[f(st, at)] = Eat∼πθ [ q(at|st) πθ(at|st)f(st, at) ] for Step (1). By (A.6) and (A.7), Jπθold (πθ(·|st)) and Jπθold (πθ(·|st)) have the same gradient at θ = θold.

heading: B DETAILED DAC IMPLEMENTATION
text: To compute the final objective function (9), we need to estimate Qπold and Rπold,α. Qπold can be estimated by diverse policy evaluation. For estimation of Rπold,α, we use function Rα. If we set the objective function of the ratio function as J(Rα(st, ·)) = αEat∼π[logRα(st, at)] + (1 − α)Eat∼q[log(1−Rα(st, at))]. In the α = 0.5 case, Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has shown that the ratio function for α = 0.5 can be estimated by maximizing J(R0.5). By a similar way, we can easily show that maximizing J(Rα) can estimate our ratio function as below:
For given s, J(Rα(s, ·)) = ∫ a απ(a|s) logRα(s, a) + (1 − α)q(a|s) log(1 − Rα(s, a))da. The integrand is in the form of y → a log y + b log(1 − y) with a = απ and b = (1 − α)q. For any (a, b) ∈ R2\(0, 0), the function y → a log y+ b log(1− y) has its maximum at a/(a+ b). Thus, the optimal R∗,α maximizing J(Rα(s, ·)) is R∗,α(s, a) = απ/(απ + (1− α)q) = Rπ,α(st, at). Here, note that J(Rα) becomes just an α-skewed Jensen-Shannon (JS) divergence except some constant terms if Rα = Rπ,α.
For implementation we use deep neural networks to approximate the policy π, the diverse value functions Q, V , and the ratio function Rα, and their network parameters are given by θ, φ, ψ, and η, respectively. Based on Section 4.3 and we provide the practical objective (or loss) functions for parameter update as Ĵπ(θ), ĴRα(η), L̂Q(φ), and L̂V (ψ). The objective functions for the policy π and the ratio function Rα are respectively given by
Ĵπ(θ) = Est∼D, at∼πθ [Qφ(st, at) + α logRαη (st, at)− α log πθ(at|st)], (B.1)
ĴRα(η) = Est∼D[αEat∼πθ [logRαη (st, at)] + (1− α)Eat∼D[log(1−Rαη (st, at))]]. (B.2) Furthermore, based on the Bellman operator, the loss functions for the value functions Q and V are given respectively given by
L̂Q(φ) = E(st, at)∼D [ 1
2 (Qφ(st, at)− Q̂(st, at))2
] , (B.3)
L̂V (ψ) = Est∼D [ 1
2 (Vψ(st)− V̂ (st))2
] , (B.4)
where the target values are defined as
Q̂(st, at) = 1
β rt + γEst+1∼P [Vψ̄(st+1)] (B.5)
V̂ (st) = Eat∼πθ [Qφ(st, at) + α logRαη (st, at)− α logαπθ(at|st)] + (1− α)Eat∼D[log(1−Rαη (st, at))− log(1− α)q(at|st)]. (B.6)
By using the property of ratio function that satisfies log(1−Rπ,α)− log(1−α)q = − log(απ+(1− α)q) = logRπ,α− logαπ, we can replace the last term in V̂ (st) as (1−α)Eat∼D[logRαη (st, at)− logαπ(at|st)]. However, the probability of π for actions sampled from D can have high variance, so we clip the term in the expectation over at ∼ D by action dimension for stable learning, then the final target value becomes
V̂ (st) = Eat∼πθ [Qφ(st, at) + α logRαη (st, at)− α logαπθ(at|st)] + (1− α)Eat∼D[clip(logRαη (st, at)− logαπ(at|st),−d, d)], (B.7)
where d = dim(A) is the action dimension. We will use (B.7) for implementation. Then, note that all objective (or loss) functions does not require the explicit q, and they can be represented by using the ratio function Rα only as explained in Section 4.3.
In addition, Rα ∈ (0, 1) should be guaranteed in the proof of Theorem 1, and Rα ∈ (0, 1) satisfies when π and q are non-zero for all state-action pairs. For practical implementation, we clipped the ratio function as ( , 1− ) for small > 0 since some q values can be close to zero before the replay buffer stores a sufficient amount of samples. π is always non-zero since we consider Gaussian policy.
Here, ψ̄ is the network parameter of the target value Vψ̄ updated by exponential moving average (EMA) of ψ for stable learning (Mnih et al., 2015). Combining all up to now, we propose the
diversity actor-critic (DAC) algorithm summarized as Algorithm 1 in Appendix C. Note that DAC becomes SAC when α = 1, and becomes standard off-policy RL without entropy regularization when α = 0.
To compute the gradient of Ĵπ(θ), we use the reparameterization trick proposed by (Kingma & Welling, 2013; Haarnoja et al., 2018a). Note that the policy action at ∼ πθ is the output of the policy neural network with parameter θ. So, it can be viewed as at = fθ( t; st), where f is a function parameterized by θ and t is a noise vector sampled from spherical normal distribution N . Then, the gradient of Ĵπ(θ) is represented as∇θĴπ(θ) = Est∼D, t∼N [∇a(Qφ(st, a) +α logRαη (st, a)− α log πθ(a|st))|a=fθ( t;st)∇θfθ( t; st)− α(∇θ log πθ)(fθ( t; st)|st)]. For implementation, we use two Q-functions Qφi , i = 1, 2 to reduce overestimation bias as proposed in (Fujimoto et al., 2018), and each Q-function is updated to minimize their loss function L̂Q(φi). For the policy and the value function update, the minimum of two Q-functions is used (Haarnoja et al., 2018a).
Note that one version of SAC (Haarnoja et al., 2018b) considers adaptation of the entropy control factor β by using the Lagrangian method with constraint H(π) ≥ c. In our case, this approach can also be generalized, but it is beyond the scope of the current paper and we only consider fixed β in this paper.
B.1 DETAILED IMPLEMENTATION OF THE α-ADAPTATION
In order to learn α, we parameterize α as a function of st using parameter ξ, i.e., α = αξ(st), and implement αξ(st) with a neural network. Then, ξ is updated to minimize the following loss function deduced from (10):
L̂α(ξ) = Est∼D[αξH(πθ) +D αξ JS(πθ||q) + (1− αξ)H(q)− αξc] (B.8)
Here, all the updates for diverse policy iteration is the same except that α is replaced with αξ(st). Then, the gradient of L̂α(ξ) with respect to ξ can be estimated as below:
The loss function of α is defined as L̂α(ξ) = Est∼D[αξH(πθ)+D αξ JS(πθ||q)+(1−αξ)H(q)−αξc]. The gradient of L̂α(ξ) can be computed as
∇ξL̂α(ξ) = ∇ξEst∼D[αξH(πθ) +D αξ JS(πθ||q) + (1− αξ)H(q)− αξc]
=∇ξEst∼D[αξEat∼πθ [− log(αξπθ + (1− αξ)q)− c] + (1− αξ)Eat∼q[− log(αξπθ + (1− αξ)q)]] =Est∼D[(∇ξαξ)(Eat∼πθ [− log(αξπθ + (1− αξ)q)− c]− Eat∼q[− log(αξπθ + (1− αξ)q)])]
+ Est∼D[αξEat∼πθ [−∇ξ log(αξπθ + (1− αξ)q)] + (1− αξ)Eat∼q[−∇ξ log(αξπθ + (1− αξ)q)]] =Est∼D[(∇ξαξ)(Eat∼πθ [− logαξπθ + logRπθ,αξ − c]− Eat∼q[log(1−Rπθ,αξ)− log(1− αξ)q])]
+ Est∼D [∫
at∈A (αξπθ + (1− αξ)q)[−∇ξ log(αξπθ + (1− αξ)q)︸ ︷︷ ︸
=0
]
]
=Est∼D[(∇ξαξ)(Eat∼πθ [− logαξπθ + logRπθ,αξ − c]− Eat∼q[logRπθ,αξ − logαξπθ])] (B.9)
Note that Rπθ,αξ can be estimated by the ratio function Rαξη . Here, we use the same clipping technique as used in (B.7) for the last term of (B.9).

heading: C ALGORITHM
text: Algorithm 1 Diversity Actor Critic Initialize parameter θ, η, ψ, ψ̄, ξ, φi, i = 1, 2 for each iteration do
Sample a trajectory τ of length N by using πθ Store the trajectory τ in the buffer D for each gradient step do
Sample random minibatch of size M from D Compute Ĵπ(θ), ĴRα(η), L̂Q(φi), L̂V (ψ) from the minibatch θ ← θ + δ∇θĴπ(θ) η ← η + δ∇ηĴRα(η) φi ← φi − δ∇φiL̂Q(φi), i = 1, 2 ψ ← ψ − δ∇ψL̂V (ψ) Update ψ̄ by EMA from ψ if α-Adpatation then
Compute L̂α(ξ) from the minibatch ξ ← ξ − δ∇ξL̂α(ξ)
end if end for
end for

heading: D HYPERPARAMETER SETUP AND ENVIRONMENT DESCRIPTION
text: In Table D.1, we provide the detailed hyperparameter setup for DAC and the SAC baselines: SAC, and SAC-Div. Table D.2 shows the environment description, the corresponding entropy control coefficient β, threshold for sparse Mujoco tasks, and reward delay D for delayed Mujoco tasks.

heading: E SIMULATION SETUP
text: We compared our DAC algorithm with the SAC baselines and other RL algorithms on various types of Mujoco tasks with continuous action spaces (Todorov et al., 2012) in OpenAI GYM (Brockman et al., 2016). For fairness, both SAC/SAC-Div and DAC used a common hyperparameter setup that basically follows the setup in (Haarnoja et al., 2018a). Detailed hyperparameter setup and environment description are provided in Appendix D, and the entropy coefficient β is selected based on the ablation study in Section 6.3. For the policy space Π we considered Gaussian policy set widely considered in usual continuous RL. For the performance plots in this section, we used deterministic evaluation which generated an episode by deterministic policy for each iteration, and the shaded region in the figure represents standard deviation (1σ) from the mean.

heading: F PERFORMANCE COMPARISONS
text: In this section, we provide more performance plots and tables. In Section F.1, Fig. F.1 shows the mean number of discretized state visitation curve of DAC and SAC/SAC-Div. For discretization, we simply consider 2 components of observations of Mujoco tasks, which indicate the position of the agent: x, z axis position for SparseHalfCheetah, SparseHopper, and SparseWalker, and x, y axis position for SparseAnt. We discretize the position by setting the grid spacing per axis to 0.01 in range (−10, 10). Table. F.1 shows the performance on sparse Mujoco tasks. Table F.2 shows max average return for HumanoidStandup and delayed Mujoco tasks. In Section F.3, Fig. F.3 and Table. F.3 shows the performance comparison to other RL algorithms on HumanoidStandup and delayed Mujoco tasks.
F.1 PERFORMANCE COMPARISON WITH THE SAC BASELINES
0.0 0.2 0.4 0.6 0.8 1.0 Time Steps (1e6)
6000
8000
10000
12000
14000
16000
M ea
n Nu
m be
r o f V
isi te
d Bl
oc ks
SAC SAC-Div DAC
(a) SparseHalfCheetahv1
0.0 0.2 0.4 0.6 0.8 1.0 Time Steps (1e6)
2000
2250
2500
2750
3000
3250
3500
3750
4000
M ea
n Nu
m be
r o f V
isi te
d Bl
oc ks
SAC SAC-Div DAC
(b) SparseHopper-v1
0.0 0.2 0.4 0.6 0.8 1.0 Time Steps (1e6)
4000
6000
8000
10000
12000
M ea
n Nu
m be
r o f V
isi te
d Bl
oc ks
SAC SAC-Div DAC
(c) SparseWalker2d-v1
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
8000
9000
10000
11000
12000
13000
M ea
n Nu
m be
r o f V
isi te
d Bl
oc ks
SAC SAC-Div DAC
(d) SparseAnt-v1
Figure F.1: The number of discretized state visitation on sparse Mujoco tasks
DAC (α = 0.5) SAC SAC-Div SparseHalfCheetah 915.90±50.71 386.90±404.70 394.70±405.53 SparseHopper 896.90±10.57 900.60±5.22 901.40±4.25 SparseWalker2d 573.10±404.96 301.30±408.15 373.10±433.13 SparseAnt 935.80±37.08 870.70±121.14 963.80±42.51
Table F.1: Max average return of DAC algorithm and SAC baselines for fixed α setup
DAC (α = 0.5) DAC (α = 0.8) DAC (α-adapt.) SAC SAC-Div
HumanoidS 202491.81±25222.77 170832.05 ±12344.71
197302.37 ±43055.31
167394.36 ±7291.99
165548.76 ±2005.85
Del. HalfCheetah 6071.93±1045.64 6552.06±1140.18 7594.70±1259.23 3742.33±3064.55 4080.67±3418.07 Del. Hopper 3283.77±112.04 2836.81±679.05 3428.18±69.08 2175.31±1358.39 2090.64±1383.83 Del. Walker2d 4360.43±507.58 3973.37±273.63 4067.11±257.81 3220.92±1107.91 4048.11±290.48 Del. Ant 4088.12±578.99 3535.72±1164.76 4243.19±795.49 3248.43±1454.48 3978.34±1370.23
Table F.2: Max average return of DAC algorithms and SAC baselines for adaptive α setup
F.2 COMPARISON TO RND AND MAXENT
We first compared the pure exploration performance of DAC to random network distillation (RND) (Burda et al., 2018) and MaxEnt (Hazan et al., 2019), which are state-of-the-art exploration methods, on the continuous 4-room maze task described in Section 6.1. RND adds an intrinsic reward rint,t to MDP extrinsic reward rt as rRND,t = rt+ cintrint,t based on the model prediction error rint,t = ||f̂(st+1) − f(st+1)||2 of prediction network f̂ and random target network f for given state st+1. The parameter of the target network is initially given randomly and the prediction network learns to minimize the MSE of the two models. Then, the agent goes to rare states since rare states have higher prediction errors. On the other hand, MaxEnt considers maximizing the entropy of state mixture distribution dπ mix
by setting the reward functional in (Hazan et al., 2019) as − log dπmix(s) + cM , where dπ is a state distribution of the trajectory generated from π and cM is a smoothing constant. Here, MaxEnt mainly considers large or continous state space, so the reward functional is computed based on several projection/discretization methods. Then, MaxEnt explores the state space better than a simple random policy on various tasks with continuous state space.
For RND, for both the prediction network and the target network, we use MLP with 2 ReLu hidden layers of size 256, where the input dimension is equal to the state dimension and the output dimension is 20, and we use cint = 1. For MaxEnt, we compute the reward functional at each iteration by using Kernel density estimation with a bandwidth 0.1 as stated in (Hazan et al., 2019) on previous 10000 states stored in the buffer, and we use cM = 0.01. For RND and MaxEnt, we change the entropy term of SAC/DAC to the intrinsic reward and the reward functional term respectively, and we use the Gaussian policy with fixed standard deviation σ = 0.1. Fig. F.2(a) shows the mean number of state visitation curve over 30 seeds of the 4-room maze task and Fig. F.2(b) shows the corresponding state visit histogram of all seeds. As seen in Fig. F.2, DAC explores more number of states than RND and MaxEnt on continuous 4-room maze task, so it is seen that the exploration of DAC is more sample-efficient than that of RND/MaxEnt on the maze task.
0 1 2 3 4 5 Time Steps (1e5)
0
500
1000
1500
2000
2500
3000
3500
M ea
n Nu
m be
r o f V
isi te
d Bl
oc ks
RND MaxEnt DAC
(a) Number of state visitation
0 50 100
0
50
100
RN D
0
50
100
M ax
En t
0
50
100
DA C
0
5
>10
Exploration Flow
(b) State visit histogram at 5k (left) 50k (middle) 500k (right) steps
Figure F.2: Pure exploration comparison with RND/MaxEnt
F.3 COMPARISON TO OTHER RL ALGORITHMS
We also compare the performance of DAC with α-adaptation to other state-of-the-art RL algorithms. Here, we consider various on-policy RL algorithms: Proximal Policy Optimization (Schulman et al., 2017b) (PPO, a stable and popular on-policy algorithm), Actor Critic using Kronecker-factored Trust Region (Wu et al., 2017) (ACKTR, actor-critic that approximates natural gradient by using Kronecker-factored curvature), and off-policy RL algorithms: Twin Delayed Deep Deterministic Policy Gradient (Fujimoto et al., 2018) (TD3, using clipped double-Q learning for reducing overestimation); and Soft Q-Learning (Haarnoja et al., 2017) (SQL, energy based policy optimization using Stein variational gradient descent). We used implementations in OpenAI baselines (Dhariwal et al., 2017) for PPO and ACKTR, and implementations in author’s Github for other algorithms. We provide the performance results as Fig. F.3 and Table F.3.
0 2 4 6 8 10 Time Steps (1e6)
50000
100000
150000
200000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
ACKTR SQL TD3 PPO SAC SAC-Div DAC( -adapt.)
(a) HumanoidStandup-v1
0 1 2 3 4 5 Time Steps (1e6)
0
2000
4000
6000
8000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
ACKTR SQL TD3 PPO SAC SAC-Div DAC( -adapt.)
(b) DelayedHalfCheetah-v1
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
0
500
1000
1500
2000
2500
3000
3500
Av er
ag e
Ep iso
de R
ew ar
d Su
m
ACKTR SQL TD3 PPO SAC SAC-Div DAC( -adapt.)
(c) DelayedHopper-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
ACKTR SQL TD3 PPO SAC SAC-Div DAC( -adapt.)
(d) DelayedWalker2d-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
ACKTR SQL TD3 PPO SAC SAC-Div DAC( -adapt.)
(e) DelayedAnt-v1
Figure F.3: Performance comparison to other RL algorithms
DAC PPO ACKTR SQL TD3 SAC
HumanoidS 197302.37±43055.31 160211.90 ±3268.37
109655.30 ±49166.15
138996.84 ±33903.03
58693.87 ±12269.93
167394.36 ±7291.99
Del. HalfCheetah 7594.70±1259.23 2247.92 ±640.69
3295.30 ±824.05
5673.34 ±1241.30
4639.85 ±1393.95
3742.33 ±3064.55
Del. Hopper 3428.18±69.08 2740.15 ±719.63
2864.81 ±1072.64
2720.32 ±127.71
2276.58 ±1471.66
2175.31 ±1358.39
Del. Walker2d 4067.11±257.81 2859.27 ±1938.50
1927.32 ±1647.49
3323.63 ±503.18
3736.72 ±1806.37
3220.92 ±1107.91
Del. Ant 4243.19±795.49 1224.33 ±521.62
2956.51 ±234.89
6.59 ±16.42
904.99 ±1811.78
3248.43 ±1454.48
Table F.3: Max average return of DAC and other RL algorithms

heading: G ABLATION STUDIES
text: Here, we provide more ablation studies for remaining delayed Mujoco tasks. Fig. G.2 shows the averaged learning curves of α, DαJS , and H(π) of DAC considering α-adaptation, where the control coefficient c is −2.0d and d = dim(A). Fig. G.2 shows the performance of DAC considering α-adaptation with control coefficient c = 0,−0.5d,−1.0d, and −2.0d. Fig. G.3 shows the performance of DAC with α = 0.5 and β = 0.1, 0.2, 0.4. Fig. G.4 shows the performance of SAC, SAC-Div with KL-divergence (SAC-Div(KL)), SAC-Div with JS-divergence (SAC-Div(JS)), and DAC to see the effect of JS divergence on the performance as explained in Section 6.3. Other hyperparameters follow the default setup given in Table D.1.
Weighting factor α
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Va lu
e
DJS ( )/dim( )
(a) DelayedHopper-v1
0 1 2 3 4 5 Time Steps (1e6)
0.4
0.2
0.0
0.2
0.4
0.6 0.8 Va lu e
DJS ( )/dim( )
(b) DelayedWalker2d-v1
0 1 2 3 4 5 Time Steps (1e6)
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Va lu
e
DJS ( )/dim( )
(c) DelayedAnt-v1
Figure G.1: Ablation study for α
Control coefficient c
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
0
500
1000
1500
2000
2500
3000
3500
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div DAC( -adapt., c = 0.0d) DAC( -adapt., c = 0.5d) DAC( -adapt., c = 1.0d) DAC( -adapt., c = 2.0d)
(a) DelayedHopper-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div DAC( -adapt., c = 0.0d) DAC( -adapt., c = 0.5d) DAC( -adapt., c = 1.0d) DAC( -adapt., c = 2.0d)
(b) DelayedWalker2d-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div DAC( -adapt., c = 0.0d) DAC( -adapt., c = 0.5d) DAC( -adapt., c = 1.0d) DAC( -adapt., c = 2.0d)
(c) DelayedAnt-v1
Figure G.2: Ablation study for c
Entropy coefficient β
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
0
500
1000
1500
2000
2500
3000 3500 Av er ag e Ep iso de R ew ar d Su m
SAC( = 0.2) DAC( = 0.5, = 0.1) DAC( = 0.5, = 0.2) DAC( = 0.5, = 0.4)
(a) DelayedHopper-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC( = 0.2) DAC( = 0.5, = 0.1) DAC( = 0.5, = 0.2) DAC( = 0.5, = 0.4)
(b) DelayedWalker2d-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC( = 0.2) DAC( = 0.5, = 0.1) DAC( = 0.5, = 0.2) DAC( = 0.5, = 0.4)
(c) DelayedAnt-v1
Figure G.3: Ablation study for β
The Effect of JS divergence
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Time Steps (1e6)
0
500
1000
1500
2000
2500
3000
3500
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div(KL) SAC-Div(JS) DAC( -adapt.)
(a) DelayedHopper-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div(KL) SAC-Div(JS) DAC( -adapt.)
(b) DelayedWalker2d-v1
0 1 2 3 4 5 Time Steps (1e6)
0
1000
2000
3000
4000
5000
Av er
ag e
Ep iso
de R
ew ar
d Su
m
SAC SAC-Div(KL) SAC-Div(JS) DAC( -adapt.)
(c) DelayedAnt-v1
Figure G.4: Ablation study for JS divergence

Review summaries:
The paper proposes DAC, an actor-critic method exploiting the replay buffer to do policy entropy regularisation. The main idea of DAC is to use the data from the replay buffer to induce a distribution  $q(\cdot, s_t)$ and replace the entropy part of the Soft Actor-Critic objective with a convex combination of $q$ and $\pi$. This results positively on exploration properties and leads to sample-efficiency gains on some of the considered MuJoCo benchmarks.

This paper considers the exploration efficiency issues in off-policy deep reinforcement learning (DRL). The authors identify a sample efficiency limitation in the classical entropy regularization, which does not take into account the existing samples in the replay buffer. To avoid repeated sampling of previously seen scenarios/actions, the authors propose to replace the current policy in the entropy term with a mixture of the empirical policy estimation from the replay buffer and the current policy, and term this approach as sample-aware entropy regularization. The authors then propose a theoretical algorithm called sample-aware entropy regularized policy iteration, which is a generalization of the soft policy iteration (SPI) algorithm, and show that it converges assuming that the empirical policy estimation is fixed. A practical algorithm based on the sample-aware entropy regularized policy iteration, called Diversity Actor-Critic (DAC), is then proposed. This algorithm is a generalization of the well-known soft actor-critic (SAC) algorithm. Finally, numerical experiments show that DAC outperforms SAC and other SOTA RL algorithms, and some ablation studies are also provided to demonstrate the effect of hyper-parameter choices in DAC.