Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
Authors: Seungyul Han
Youngchul Sung
Venue:  (2021)
Keywords: Reinforcement Learning
Entropy Regularization
Exploration

Abstract: Policy entropy regularization is commonly used for better exploration in deep reinforcement learning (RL). However, policy entropy regularization is sample-inefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account. In order to take advantage of the previous sample distribution from the replay buffer for sample-efficient exploration, we propose sample-aware entropy regularization which maximizes the entropy of weighted sum of the policy action distribution and the sample action distribution from the replay buffer. We formulate the problem of sample-aware entropy regularized policy iteration,  prove its convergence, and provide a practical algorithm named diversity actor-critic (DAC) which is a generalization of soft actor-critic (SAC). Numerical results show that DAC significantly outperforms SAC baselines and other state-of-the-art RL algorithms.

Under review as a conference paper at ICLR 2021
DIVERSITY ACTOR-CRITIC: SAMPLE-AWARE ENTROPY REGULARIZATION FOR SAMPLE-EFFICIENT EXPLORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Policy entropy regularization is commonly used for better exploration in deep reinforcement learning (RI.). However, policy entropy regularization sample- inefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account. the previous sample distribution from the replay buffer for sample-efficient ex- ploration, we propose sample-aware entropy regularization which maximizes the entropy of weighted sum of the policy action distribution and the sample action distribution from the replay buffer. We formulate the problem of sample-aware entropy regularized policy iteration, prove its convergence, and provide practi- cal algorithm named diversity actor-critic (DAC) which is a generalization of soft actor-critic (SAC). Numerical results show that DAC significantly outperforms SAC baselines and other state-of-the-an RL algorithms.
order to take advantage
INTRODUCTION
Reinforcement learning (RL) aims to maximize the expectation of the discounted reward sum under Markov decision process (MDP) environments (Sutton & Barto 1998). When the given task is complex, i.e. the environment has high action-dimensions or sparse rewards, it is important to well explore state-action pairs for high performance (Agre & Rosenschein 1996). For better exploration, recent RL considers various methods: maximizing the policy entropy to take actions more uniformly (Ziebart et al., 2008 Fox et al. 2013: Haarnoja et al.) 2017). maximizing diversity gain that yields intrinsic rewards to explore rare states by counting the number of visiting states (Strehl & Littman
2008 Lopes al., 2012), maximizing information gain (Houthooft et al.) [006]] Hong et al.) (0018), maximizing model prediction error (Achiam & Sastry 2017: Pathak et al.) (017), and so on. In particular, based on policy iteration for soft Q-learning, (Haarnoja et al.) 2018a) considered an off- policy actor-eritic framework for maximum entropy RL and proposed the soft actor-critic (SAC)
algorithm, which has competitive performance for challenging continuous control tasks.
In this paper, reconsider the problem of policy entropy regularization in off-policy learning and propose generalized approach to policy entropy regularization. In off-policy learning, we store and reuse old samples update the current policy (Mnih al.) (015),, and it is preferable that the old sample distribution in the replay buffer is uniformly distributed for better performance. However, the simple policy entropy regularization tries maximize the entropy of the current policy irrespective of the distribution of previous samples. Since the uniform distribution has maximum entropy, the current policy will choose previously less-sampled actions and more-samplee sampled actions with the same probability and hence the simple policy entropy regularization is sample-unaware and sample-ineffccintt In order to overcome this drawback, we propose sample-aware entropy regularization, which tries to maximize the weighted sum of the current policy action distribution and the sample action distribution from the replay buffer. We will show that the proposed sample- aware entropy regularization reduces to maximizing the sum of the policy entropy and the a-skewed -haneenn-hanonnn divergence (Nielsen 2019) between the policy distribution and the buffer sample action distribution, and hence generalizes SAC. We will also show that properly exploiting the sample action distribution in addition to the policy entropy over learning phases will yield far better performance.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
Entropy regularization: Entropy regularization maximizes the sum of the expected return and the policy action entropy. It encourages the agent to visit the action space uniformly for each given state, and the regularized policy is robust to modeling error (Ziebart (2100). Entropy regularization is
considered in various domains for better optimization: inverse reinforcement learning (Ziebart et al.)
2008), stochastic optimal control problems (Todorov] 2008 TTossaint] 2009) Rawlik et al. 2013), and off-po reinforcement learning (Fox et al. |015] Haarnoja et al. (017). (Lee et al. 2019)
shows that Tsallis entropy regularization that generalizes usual Shannon- entropy regularization is helpful. (Nachum et al. 2017a) shows that there exists a connection between value-based and policy- based RL under entropy regularization. (O Donoghue et al. (016) proposed algorithm combining them, and it is proven that they are equivalent 1Schulman et al. 2017a). The entropy of state mixture distribution is better for pure exploration than simple random policy (Hazan al. 2019).
Diversity gain: Diversity gain is used to provide a guidance for exploration to the agent. To achieve diversity gain, many mirrisically-motivateed approaches and intrinsic reward design methods have been considered, e.g., intrinsic reward based on curiosity (Chentanez et al.] [2055] Baldassarre & Mirolli 2013). model prediction error (Achiam & Sastry, 2017: Pathak al., 2017 Burda et al.
2018), divergence/informationn gain (Houthooft et al. Hong et al. 2018). counting (Strehl & Martin et al. 7). and unification of them
Littman 2008 Lopes et al., 2012] Tang et al. [017]]
(Bellemare et al. 2016). For self-imitation learning, (Gangwani et al. 2018) considered the Stein-
variational gradient decent with the Jensen-Shane Sensen-Shannon kennel.
Off-policy learning: Off-policy learning can reuse any samples generated from behaviour policies for the policy update (Sutton& & Barto 1998 Degris et al. (012)) so it is sample-efficien as compared to on-policy learning. In order to reuse old samples, replay buffer that stores trajectories generated by previous policies is used for Q-learning (Mnih et al.] 2013 Lillicrap ct al. 2015) Fujimoto et al. 2018 Haarnoja al., (0) To enhance both stability and sample-efficiency. several methods are considered, e.g., combining on-policy and off-policy (Wang et al. 2016 Gu et al. 2016 (017). and generalization from on-policy to off-policy (Nachum al. 2017b Han & Sung 2019).
In order to guarantee the convergence of Q-learning, there is key assumption: Each state-action pair must be visited infinitely often (Watkins & Dayan 1992.. If the policy does not visit diverse state-action pairs many times, it converges to local optima. Therefore, exploration for visiting differ- ent state-action pairs is important for RL, and the original policy entropy regularization encourages exploration (Ahmed et al. 2019). However, we found that the simple policy entropy regulariza- in off- officc RL, so we aim to propose a new entropy regularization method that significantly enhances the sample-efficiency for exploration by considering the previous sample distribution in the buffer.
tion can be cumple-inetficien
3 BACKGROUND
In this section, we briefly introduce the basic setup and the soft actor-critic (SAC) algorithm.
3.1 SETUP
We assume a basic RL setup composed of an environment and an agent. The environment follows an infinite horizon Markov decision process A,,A, P.y,r), where is the state space, A is the action space, P is the transition probability, 7 is the discount factor, and r : S x A - R is the reward function. In this paper, we consider a continuous state-action space. The agent has a policy distribution S x A j0, oo) which selects an action de for given state St at each time step I, and the agent interacts with the environment and receives reward r, = r(s,, ax) from the environment. Standard RL aims to maximize the discounted return Ego Ei= rro,,, where T = (St, 04, S1+1, a1+1 ..) is an episode trajectory.
3.2 SOFT ACTOR-CRITEE
Soft actor-critic (SAC) (Haarnoj et al. 20) includes a policy entropy regularization term in the objective function for better exploration by visiting the action space uniformly for each given state.
2

---

Under review as a conference paper at ICLR 2021
The
policy objective function of SAC is given by
(s) Bronx
Σ +r (H(T((1(1))) +++++M(((())))
where H is the entropy function and 3 E o0) is the entropy coefficient. SAC is a practical off- policy actor-critic based soft policy iteration (SPI) that alternates soft policy evaluation to esti- mate the true soft Q-function and soft policy improvement to find the optimal policy that maximizes TT.
In addition, SPI theoretically guarantees convergence to the optimal policy that maximizes (
THE DIVERSITY ACTOR-CRITIC ALGORITHM
4.1 MOTIVATION OF THE SAMPLE-AWARE ENTROPY
As explained in Section 2 the policy should visit diverse samples to learn the policy without con- verging to the local optima. In off-policy learning, we can reuse previous samples stored in the replay buffer to learn the policy, it is efficient draw diverse samples while avoiding frequently selected samples before. The policy entropy maximization enhances exploration to yield better performance, but it is sample-inefficiett for off-policy RL because it does not take advantage of the previous sam- ple action distribution obtainable from the replay buffer: If we assume bounded action space, the simple policy entropy maximization will choose all actions with the equal probability without con- sidering the previous action samples because max, H(TT) = min., D (T] (T||U') ) is achieved when π U, where U is uniform distribution and DR is the Kullback- Leibler (KL) divergence.
In order to overcome the limitation of the simple policy entropy maximization, we consider maxi- mizing a sample-aware entropy defined as the entropy of mixture distribution of the policy distri- bution π and the current sample action distribution q in the replay buffer. Here, q is defined as
(-115) :- EOED N(s N(,,a)
where the replay buffer that stores previous samples (8t, at, Te, 8t++1 at each time 4, 6.).) the Dirac measure at a E A. and (ss, a) is the number of state-action pair (s, a) in D.
Then, we define a target distribution Starget as the mixture distribution of and q, which is expressed a, as Vaarget := am + (1 ar)g, where a E [0, 1] is the weighting factor. Note that we draw samples from policy and store them in the replay buffer, so the target distribution can be viewed as the updated sample action distribution in the future replay buffer. Then, maximizing the sumple-aware entropy (((target) can encourage sample-efficient exploration because will choose actions rare in the buffer with high probability and actions stored many times the buffer with low probability in order make the target distribution uniform. We provide simple example below:
Let us consider a simple 1 1-step MDP in which so is the unique initial state, there exist N₂ actions (A (Ao,.... ))) 81 is the terminal state, and is a deterministic reward function. Then, there exist N, state-action pairs in total and let us assume that we already have N. - 1 state-action samples in the replay buffer as
(su, AN. r(so. 2N)- In order to estimate the Q-function for all state-action pairs, the policy should sample the last action A AN.-1 (After then, we can reuse all samples infinitely to estimate Q). Here, we will compare two exploration methods.
{(so. Ao. r(so. Ag)).
1) First, if we consider the simple entropy maximization, the policy that maximizes its entropy will choose all actions with equal probability 1/N, (uniformly). Then, Na samples should be taken on average by the policy to visit the action AN
2) Consider the sample-aware entropy maximization. Here, the sample action distribution q in the buffer becomes q(a))=)) 1/(N, 1) for dg E {Ag.... AN.-2) and q( (ANN_1188)) = 0, the target distribution becomes gergget + (1 a)g. and set 1/No. Then, the policy that maximizes the sample-aware entropy becomes 5 x(AN.- (s)) to make guanget uniform because max, H (Ltarget) min, DxI (g (9lrgeellUU) In this case, we only needs sample visit the
3

---

Under review as a conference paper at ICLR 2021
action AN.-1- In this way, the simple entropy maximization is sample-ineffcient for off-policy RL, and the proposed sample-aware entropy maximization can enhance the sample-effiiinncy for exploration by using the previous sample distribution and choosing proper a. With this motivation, we propose the sample-aware entropy regularization for off-policy RI. and the corresponding a- adaptation method.
4.2 SAMPLE- AWARE ENTROPY REGULARIZATION
Our approach is to maximize the return while maximizing the sample-aware entropy. Under this approach, previously many times sampled actions will be given low probabilities and previously less taken actions will be given high probabilities by the current policy æ for sample-efficient exploration as shown in Section 4.1 Hence, we set the objective function for the proposed sample-aware entropy regularization
J(x) = Ero
tr, + + (1ss)))
(3)
-1=0
Here, the sample-aware entropy H. H((aarget) for given St can be decomposed as
H((targett - == aed
(us (1 a)g) log(er + (1 a)4) = aH(()) Dis + (1-))H(()) (4)
where (T)(() aDk (T | Ptarger + (1 (1-ol a D k (gl (harget)) is the skew-symmetric Jensen- Shannon (JS) divergence (Nielsen 2019). 1... Note that Dÿs reduces to the standard JS divergence for a = 2 and to zero for a = 0 or 1. Hence, for a = 1, (4 reduces to the simple entropy. but for a * 1, it is generalization incorporating the distribution q. Thus, our objective function aims maximize the return while simultaneously maximizing the discounted sum of policy entropy, sample entropy, and the divergence between and q. In this way, the policy will choose more diverse actions that far from the samples stored in the replay buffer while maintaining its entropy for better exploration.
4.3 DIVERSE POLICY ITERATION WITH THE PROPOSED OBJECTIVE
In this section, we derive the diverse policy evaluation and diverse policy improvement to maximize the objective function with the sample-aware entropy regularization (3). Note that the sample action distribution q is updated as the iteration goes on. However, it changes very slowly since the buffer size is much larger than the time steps of one iteration. Hence, for the purpose of proof regard the action distribution q as a fixed distribution this section.
First,
define the true diverse Q-function QT as QT ar) " 1-1-1 ( m + (H((T()):)) + ()s(a((1s))la + (1 - a))(((( (1)())
are
E
We defined the sample distribution q in equation 2 but do not want to compute actual g. which requires method such as discretization and counting for continuous samples. Even if q is obtained by counting, a generalization of for arbitrary state-action pairs is needed again estimate QT. We 3,
circumvented this difficulty by defining the ratio a, of az to Starget as
an ( (st)
R (St, az)
an 8t) (1 ar)g(a:)s:)
and we will show later that all objective (or loss) functions for practical implementation can be represented by using the ratio only, without using the explicit q in Appendix
Then, we can decompose D3s( ( s)) ())]lq(-(ss)) as ()s(()lq)) a log R
, (st, a:) + - (1-a)En,-o)) 10) [log(1
R RT." a:)) + H(a),
where H(a)= a loga (1 a) log(1 a) is the binary entropy function.
The modified Bellman backup operator for Q estimation is given by
T=Q(s, az) 3" + BB (B
4

---

Under review as a conference paper at ICLR 2021
where V(st) (Q
log (st, az) - a log (t) + + - a)) a ~g
~log( log((
( R (St, Qz)) - log(1 a { (s)] is an estimated diverse state value function, Q S x A an estimated diverse state-action value function.
is
Proof of the convergence of diverse policy evaluation that estimates Q by repeating the Bellman operator (7) is provided in Appendix A Then, the policy is updated from Vold Nnew Nnww
arg max; Jaoia (T), where Janea (T) is the objective и estimated under Qa defined as
Jmnes (T( (x)) = [[ [Q (st, as) + a log Itt (st, ax) a log OCT (ar
+ (1 a>)o [log(1 R (st, at)) - log (s)((1)))))
(8)
The monotone improvement of this step is proved in Appendix A Now, we can find the optimal policy that maximizes J (= (T)) by the following theorem:
Theorem (Diverse Policy Iteration) By repeating iteration of the diverse policy evaluation and the diverse policy improvement, any initial policy converges to the optimal policy xt s.I. QF (st, at) 2 Q as), V x' E II, v (st,a) E A Also, such Ht" achieves maximum J, L.., Jx (x(") (T) for any E 11.
Proof. See Appendix A.1
Note that Juai (I) for diverse policy iteration above requires the ratio function R of the current policy n, but we can only estimate Rotas for the previous policy Hold in practice. Thus, we circumvent this difficulty by defining practical objective function (T) given by
Jn Jnna (-1se))
(Q 0e) + log Ro 04) log (aees))))
Regarding the practically computable objective function J (I), we have the following result:
Theorem 2 Suppose that the policy is parameterized with parameter 0. For parameterized policy two objective functions Jremi (TT(-1ss)) and Jao.s (TT(+)se) have the same gradient direction for 0 at = Oota. all E E.
Proof. See Appendix
A.2
By Theorem
we can replace the objective function Jn (T) of policy improvement with the prac- tically computable objective function Jrand (xt for parameterized policy without loss of optimality.
2
4.4 DIVERSITY ACTOR CRITIC IMPL IMPLEMENTATION
We first define R as an estimate for the ratio function R For implementation, we parameter- ize R Q, and V by neural network parameters 0, 77, ó, and vh respectively. Then, we setup the practical objective (or loss) functions (#), J.z (17), LQ (Ø), and Ly (1)) for the parameter update. Detailed DAC implementation based on Section 4 is provided in Appendix B The proposed DAC algorithm is summarized in Appendix C Note that DAC becomes SAC when
and becomes
standard off-policy RL without entropy regularization when a 0.
5 A-ADAPTATION
In the proposed sample-aware entropy regularization, the weighting factor a plays an important role in controlling the ratio between the policy distribution and the sample action distribution q. How- ever, is difficult to estimate optimal a directly. Hence, we further propose an adaptation method for based on max-min principle widely considered in game theory, robust learning, and decision making problems (Chinchuluun et al., 20088. Since we do not know optimal a, an alternative for- mulation is that we maximize the return while maximizing the worst-case sample-aware entropy, i... ming H (target)) Then, the max-min approach can be formulated as follows:
Σn' E (re + min (H(gas - -ac))
max E Enyer
(10)
Note that if replace Totd with and view every state an initial state, then reduces J(T).

---

Under review as a conference paper at ICLR 2021
where is control hyperparameter for adaptation. We leam to minimize H(Atarget) ac, so the role of с to maintain the target entropy at certain level to explore the state-action well. Detailed implementation for a-adaptation is given in Appendix B.1
6 EXPERIMENTS
In this section, we evaluate the proposed DAC algorithm on various continuous-action control tasks and provide ablation study. In order see the superiority of the sample- entropy regulariza- tion, we here focus on comparison with two SAC baselines: SAC and SAC-Div. SAC-Div is SAC combined with the method in (Hong et al. 2018) that diversifies policies from buffer distribution by simply maximizing J(r) + (( x ll for J(x) in ①I and some divergence D. Note that the key difference between SAC-Div and DAC is that SAC-Div simply adds the single divergence term to the policy objective function (Tr), whereas DAC considers the discounted sum of target entropy terms as seen in (3) For SAC-Div, we consider KL divergence (MSE if the policy is Gaussian) and adaptive scale Odd with dd 0.2 for the divergence term suggested in (Hong al. 2018). In order to rule out the influence of factors other than exploration, we use the common simulation setup for DAC and SAC baselines except for the parts about entropy or divergence.
In addition, we provide comparison of DAC to random network distillation (RND) (Burda et al, 2018) and MaxEnt (Hazan et al. 2019), which the recent exploration methods based on finding rare states in Appendix F.2 and to other recent RL algorithms in Appendix F.3 The result shows that DAC yields the best performance for all considered tasks compared to recent RL algorithms. We also provide the source code of DAC implementation that requires Python Tensorflow. Detailed simulation setup for experiments summarized in Appendix
50
100
100
SAC
100
SAC SAC-Div Div
(a) Continuous 4-room maze
Mai Ro00
100
DAC
50
Exploration Flow
Time Sieps Io8ns
(b) Number of state visitation
(c) State visit histogram 5k (left) 50k (middle) 500k (right) steps
Figure 1: Pure exploration task: Continuous 4-room maze
6.1 PURE EXPLORATION COMPARISON
In order to the exploration performance of DAC (ar 0.5) compared the SAC baselines. we compare state visitation on a 100 x 100 continuous 4-room maze task. The maze environment made by modifying a continuous grid map available at https:: / /ttpttttbb../// com/ huyaoyu/ GridMap) and it is shown in Fig. I(a) State is (x, y) position in the maze, action is (dr, dy) bounded by [-1, 1), and the agent location after the action becomes (x + dr, y + dy). The agent starts from the left lower corner (0.5, 0.5) and explores the maze without any reward, and Fig. ((b) shows the mean number of new state visitations over 30 seeds, where the number state visitation is obtained for each integer interval. As seen in Fig. 1(b) DAC visited much more states than SAC/SAC-Div, which means that the exploration performance of DAC superior to that of the SAC baselines. In addition, Fig. [(c) shows the corresponding state visit histogram of all seeds. Here, as the color of the state becomes brighter, the state visited more times. Note that SAC/SAC-Div rarely visit the right upper room at 500k time steps for all seeds, but DAC starts visiting the

---

Under review as a conference paper at ICLR 2021
right upper room at 5k time steps and frequently visit the right upper room 500k time steps. Thus, Fig. I[c) clearly shows that DAC has better sample-efficiency for exploration than SACISAC-Div.
6.2 PERFORMANCE COMPARISON WITH THE SAC BASELINES
The final goal of RL is to achieve high scores for given tasks. For this, exploration techniques needed to ensure that the policy does not converge to local optima, as explained in Section 2 We first showed the improvement of the exploration performance in a pure exploration task (continuous 4-room maze), and experiments in this section will show that DAC has better return performance than SAC baselines on several sparse-rewarded tasks. Note that having high scores on the sparse- reward tasks means that the policy can get rewards well without falling into local optima, which implies that the agent successfully explores more state-action pairs that have positive (or diverse) rewards. Therefore, the performance comparison note that sparse-rewarded tasks has been widely used as a verification method of exploration in many
sparse tasks fits well to the motivation and also
previous exploration studies (Hong et al. 2018 Mazoure et al. 2019 Burda al. 2018
Tiue Stps
Taae Gord
Trus
(a) SparselfalCChettah-vI
(b) Sparsel
(c) SparseWalker2d-v
(d) SparseAnt-vl
Figure 2: Performance comparison: Fixed a case
0.50
531
635
(a) SparselfalCCheetahi
(b) Sparsellopper-v1
(c) SparseWalker-y Walker-v1
(d) SparseAnt-vl
Figure 3: a-skewed JS divergence for DAC and SACISAC-Div
Fixed a case: order to see the advantage of the sample-aware entropy regularization for rewarded tasks, compare the performance of DAC with = 0.5 and the SAC baselines simple MDP tasks: SparseMujoco tasks. SparseMujoco is a sparse version of Mujoco and the reward is the agent exceeds the x-axis threshold, otherwise 0 (Hong et al. 2018 Mazoure al. 2019).
The performance results averaged over 10 random seeds are shown in Fig. As seen in Fig. 2 DAC has significant performance gain for most tasks as compared to SAC. On the other hand, SAC-Div also enhances the convergence speed compared to SAC for some tasks, but it fails to enhance the final performance. Fig. shows the a-skewed JS divergence curve
0.5) of DAC and SAC/SAC-Div for sparse Mujoco tasks and we provide Fig. F.I in Appendix F. that shows the corresponding mean number of discretized state visitation curve on sparse Mujoco tasks. For SAC/SAC-Div, the ratio function R is estimated separately from B.2) in Appendix B and the divergence is computed from R. The performance table for all tasks is given by Table F. in Appendix F.T As seen in Fig. 3 the divergence of DAC is much higher than that of SAC/SAC-Div throughout the learning time. means that the policy of DAC choose more diverse actions from the distribution far away from the sample action distribution g. then DAC visits more diverse states than the SAC baselines as seen in Fig. F. Thus, DAC encourages better exploration and it yiclds better performance. Thus, conclude that the proposed sample-aware entropy regularization is superior to the simple policy entropy regularization of SAC and single divergence regularization of SAC-Div in terms of exploration and the convergence.
Adaptive a case: Now, we compare the performance of DAC with a = 0.5, 0.8, a-adaptation, and the SAC baselines the need of o-adaptation. To maintain controllability and prevent saturation
7

---

Under review as a conference paper at ICLR 2021
340000
1420
1010
OP6 3.0
CINCE HOLD
220080
Toe Septe
Tiur Oryn (1-))
(a) HumanoidStandup-vv
(b) DelayedHalfCCoctth--v
(c) DelayedHopper-vl
(d) DelayedWalker2d-v1
(e) DelayedAnt-vl
Figure 4: Performance comparison: Adaptive a case
2008
petlenck p0311
14.00 22
(a) Weighting factor c
(b) Control coefficient e (c) Entropy coefficient B
(d) JS divergence
Figure 5: Averaged learning curve for ablation study
of Reg. we used regularization for learning and restricted the range of 0.5 a 0.99 for adaptation so that a certain level of entropy regularization is enforced. Here, we consider more complicated tasks: HumanoidStandup and delayed Mujoco tasks (DelayedHalffheetthh Delayed- Hopper, Delayed Walkerd,, and DelayedAnt). HumanoidStandup is one of high-action dimensional Mujoco tasks. Delayed Mujoco tasks suggested by (Zheng et al. 2018 Guo al., 2018) have the same state-action spaces with original Mujoco tasks but reward is sparsified. That is, rewards for D time steps are accumulated and the accumulated sum delivered the agent once every D time steps, so the agent receives reward during the accumulation time. The performance results aver- aged over random seeds are shown in Fig. 4 The result the max average return of these Mujoco tasks for DAC and SAC/SAC-Div is provided in Table P2 in Appendix F.I As seen in Fig. all versions of DAC outperform SAC. Here, SAC-Div also outperforms SAC for several tasks, but the the
performance gain by DAC is much higher. In addition, it is seen that the best depends tasks in the fixed a case. For example, a = 0.8 is the best for DelayedHalfCheettah, but a = 0.5 is the best for DelayedAnt. Thus, we need to adapt a for each task. Finally, DAC with a-adaptation has the top-level performance for most tasks and the best performance for HumanoidStandup and DelayedHopper tasks. Further consideration for is provided in Section 6.3
6.3 ABLATION STUDY
In this section, provide ablation study for important parameters in the sample-aware entropy regularization on the DelavedHallChettah task. Ablation studies on the other DelayedMucoco tasks provided in Appendix G
a-adaptation is necessary because one particular value of is not best for all environments. Although the proposed a-adaptation in Section 5 is sub- optimal, it shows good performance across all the considered tasks. Thus, we study more on the proposed a-adaptation and the sensible behavior of sample-awareness in entropy regularization.
Weighting factor a: As seen in Section 6.2
8

---

Under review as a conference paper at ICLR 2021
Fig. ((a) shows the averaged learning curve of a, a-skewed JS divergence (T)(() and the policy entropy (T)T) for DAC with the proposed a-adaptation method on DelayeHHaliCheetah. Here, we fix the control coefficient e as -2.0dim(A). As seen in (3), the return, the policy entropy and the IS divergence are intertwined in the cost function, so their learning curves are also intertwined over time steps. Here, the learned policy entropy term decreases and the learned increases to as time step goes on. Then, the initially nonzero JS divergence term s (T lg) diminishes to zero, which means that the sample action distribution exploited for roughly initial 2.5M time steps, and then DAC operates like SAC. This adaptive exploitation of the sample-aware aware entropy leads to better overall performance across time steps as seen in Fig. 4 so DAC with a-adaptation seems to properly exploit both the policy entropy and the sample action distribution depending on the learning stage.
Control coefficient c: In the proposed o-adaptation (10), the control coefficient e affects the learn- ing behavior of a. Since H(rr)) and D9s are proportional to the action dimension, we tried a few values such as 0, -0.5d. -1.0d and -2.0d where dim((A). Fig. 5(b) shows the corresponding performance of DAC with a-adaptation on DelayedHallCheetahh As seen in Fig, [(b) the perfor- mance depends on the change of c as expected, and c = -2.0 dim(AA performs well. We observed that 220d performed well for all considered tasks, thus we set e -2.0d in (B.8).
Entropy coefficient 3: As mentioned in (Haarnoja et al., 201 8a). the performance of SAC depends on B. is expected that the performance of DAC depends on 3 too. Fig (howss the performance of DAC with fixed a = 0.5 for three different values of 8: , = 0.1, 0.2 and 0.4 on Delayed- HalfCheetah. It is seen that the performance of DAC indeed depends on B. Although there exists performance difference for DAC depending on B, the performance of DAC is much better than SAC for a wide range of B. One thing to note is that the coefficient of pure policy entropy regularization term for DAC is aB, as seen in (3) Thus, DAC with a = 0.5 and B = 0.4 has the same amount of pure policy entropy regularization as SAC with 3 0.2. However, DAC with a 0.5 and B = 0.4 has much higher performance than SAC with 3 = 0.2, as seen in Fig. 5(c) So, we can see that the performance improvement DAC comes from joint use of policy entropy H((T) and the sample action distribution from the replay buffer via (xll))
The effect of JS divergence: In order to see the effect of the JS divergence on the performance, we also provide additional ablation study that we consider single JS divergence for SAC-Div by using the ratio function in Section 4.3 5(d)shows the performance comparison of SAC, SAC- Div(KL), SAC-Div(JS) and DAC. For SAC-Div(JS),
used 6 0.5 for adaptive scaling in (Hong ct al. 2018). As a result, there was no significant difference in performance between SAC-Div with IS divergence and SAC-Div with KI. divergence. On the other hand, the DAC still shows a greater performance increase than both SAC-Div(KL) and SAC-Div(SS) and this means that the DAC has more advantages than simply using JS divergence.
CONCLUSION AND FUTURE WORKS
In this paper, we have proposed sample-aware entropy framework for off-policy RL to overcome the limitation of simple policy entropy for sample-efficient exploration. With the sample-aware en- tropy regularization, we can achieve diversity gain by exploiting sample history in the replay buffer in addition to policy entropy. For practical implementation of sample-aware entropy regularized policy optimization, we have proposed the DAC algorithm with convergence proof. We have also provided an adaptation method for DAC to control the ratio of the sample action distribution to the policy action entropy. DAC is an actor-critic algorithm for sample-aware regularized policy opti- mization and generalizes SAC. Numerical results show that DAC significantly outperforms SAC basclines in Maze exploration and various Mujoco tasks.
For further study, we consider generalization of our method in order to deal with the entropy of the state-action distribution. Currently, many recent papers only consider one of the entropy of state distribution d' (s) that of action distribution IT (a]s) only since they have much different properties (e.g. the state-based entropy is non-convex on π and the action-based entropy is convex on r). However, both entropies can be handled simultaneously the entropy of the state-action distribution, factorized as log d" (s,a) = log d" (s) + log π (a]s). Then, the generalization of our method for the fused entropy may be able further enhance the exploration performance by considering the exploration on the entire state-action space.
one fused entropy that deals with

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning, arXiv preprint arXiv:1703.01332, 2017.
Philip Agre and Stanley J Rosenschein. Computational theories of interaction and agency. Mit Press, 1996.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, Pp- 151-160. PMLR, 2019.
Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning in natural and artificial systems. Springer, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprini arXiv:1606.0155: 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, pp. 1281-1288, 2005.
Altannar Chinchuluun, Panos M Pardalos, Athanasios Migdalas, and Leonidas Pitsoulis. Pareto optimality, game theory and equilibria. Springer, 2008
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint 12Xiv:/205.48399 2012.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//github.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv: 151 1512.08562, 2015.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 18xiv:1802.094777 2018.
Tanmay Gangwani. Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv preprint 180iv:1805. 10309 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Shixiang Gu. Timothy Lillicrap. Zoubin Ghahramani. Richard Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.0222 2016.
Shixiang Shane Gu, Timothy Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on on-policy and off-policy gradient esti- mation for deep reinforcement learning. In Advances Neural Information Processing Systems, pp. 3846-3855, 2017.
Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning. arXiv preprint 18 2018.
Tuomas Haarojj,, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv: /tt::/70,.000 08165, 2017.
10

---

Under review as a conference paper at ICLR 2021
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05900 2018b.
Seungyul Han and Youngchul Sung. Dimension-win wise importance sampling weight clipping for sample-efficient reinforcement learning. International Conference on Machine Learning, 2019.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. International Conference on Machine Learning, pp. 2681-2691. PMLR, 2019.
Zhang-Way Wei Hong, Tzu- Yun Shann, Shin-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10489-10500, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck. and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Diederik P Kingma and Max Welling, orXiv:1312.66. 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforce- ment learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint 1rXiv:1902.00137 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrXivv:509.02977 2015.
Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre-Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems, pp. 206-214, 2012.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based explo- ration in feature space for reinforcement learning. arXiv preprint arXiv: / 7706008090 2017.
Bogdan Mazoure, Thang Doan, Audrey Durand, R Devon Hjelm, and Joelle Pineau. Leveraging exploration in off off-policy algorithms via normalizing flows. arXiv preprint arXiv:1905.0689; 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-55 2015.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2775-27778 2785, 2017a.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. arXiv preprint arXiv:1707.01891 2017b.
Frank Nielsen. On the sensennshannon symmetrization of distances relying on abstract means. Entropy, 21(5):485, 2019.
Brendan O' Donnoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning, arXiv preprint arXiv:1611.01623 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.

---

Under review as a conference paper at ICLR 2021
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. In Twenty- Third International Joint Conference on Artificial Intelligence, 2013.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint
2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1xXiv:1707.06.44 2017b.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331 2008.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. The MIT Press, Cambridge, MA, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
Emanuel Todorov. General duality between optimal control and estimation. 2008 47th IEEE Conference Decision and Control, pp. 4286 4286-4292. IEEE, 2008
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
Mare Toussaint. Robot trajectory optimization using approximate inference, In Proceedings of the 26th annual international conference on machine learming, pp. 1049-1056. ACM, 2009.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224. 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning. 8(3):279-299 292,, 1992.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factoree approximation. In Advances in neural information processing systems, pp. 5279-5288, 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp.
2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
12

---

Under review as a conference paper at ICLR 2021
A PROOFS
A. PROOF OF THEOREM 1
For fixed policy n, Q can be estimated by repeating the Bellman backup operator by Lemma 1. Lemma 1 is based on usual policy evaluation but has a new ingredient of the ratio condition in the sample-awar case.
Lemma (Diverse Policy Evaluation) Define a sequence of diverse Q-functions as Qk+1 = T k 2 O, where is fixed policy and Qo is real-valued initial Q. Assume that the ac- tion space is bounded, and R (st, at) (0, 1) for all (St, (e, E x A. Then, the sequence {Qx} converges to the true diverse state-action value Q".
Proof. Let Fx, a (1m9 [log(1 (81+1 1, (x+1)) )) - log(1 1()) Then, we can formulate the standard Bellman equation form for the true Q?
Barss (a log R (8t+1, 0++1) log ar (4(+111/4144)) + (1
T (*Q(ss, 0z) = n,, + 7Bs+1 a.. (Q(5+11,441))
(A.1)
Under the assumption of a bounded action space and R E (0, 1), the reward Tx. is bounded and the convergence is guaranteed as the usual policy evaluation (Sutton & Barto 1998 Haarnoja al.,
2018a).
Now, prove diverse policy improvement in Lemma 2 and diverse policy iteration in Theorem by using Jzua (T) in a similar way to usual RL or SAC.
Lemma (Diverse Policy Improvement) Let Tnew be the updated policy obtained by solving Tnew arg max (T). Then, Q Thee (st, az) (Sel 0t), V (st,at) E SSXx A
well
Proof. We update the policy to maximize Jania (T), so Janea (Tnew) 2 Junee (Tota). Hence,
Ear [Qaate (st, at) + a log R R (8 (8t, az) - a log new (as [s)]
+ (1 Ba [log(1 1 n (st, (s),a))) - log(1 - a)g((())))
Mani (Q (st, az) + a log R (st, az) a log (
+ +11 Bary llo a (st, az)) - log(1 - ()(())s)))))))))))))))))))))))))
=VRald(S))
(A.2)
By repeating the Bellman equation
and A.2 at QTaid
Tolld az) = r re + 7B
1
3 r't + 7B,,, 1 ER,, new Fold (8c+1, a(+1) log Re
(8t-1, (8t-1 ac41)
a log a ae (0 (1 a Be log(( - Re ($1+1, 1+11)
log(1 a))(a++1
< InTe (st, at); for each (st, .:) E S x A.
(A.3)
13

---

Under review as a conference paper at ICLR 2021
Theorem (Diverse Policy Iteration) By repeating iteration of the diverse policy evaluation and the diverse policy improvement. any initial policy converges to the optimal policy x s.t. Q (st, at) 2 Q (8), (t), A E II, v (st,a) E ESS A. Also, such x² achieves maximum J, Le., Jx (x² (T) for any π E II.
Proof. Let {T i2 O, mi E II] be a sequence of policies s.t. TT+11 arg max Jn.(T) (T). For ar-
bitrary state action pairs (s, a) E S x A. (Q* a}} monotonically increases by Lemma and each QT a) is bounded. Also, TH+1 obtained by the policy improvement that maximizes Jn, (-1ss)), so ( (Ta+1 (-1s)) 2 Jn (+s)) as stated in the proof of Lemma 2. From the definition of J (T) in (). all terms are the same for Junn (+1+)(+|s) and Jx. ( (-|s) except BBaxx.. I [Q (s, a)]
in J m44 (+14a(+)s) and (B 10th (s. in Jr (Ti+1 (-1s)). Since (Q"((8 monotonically
increases, JT (T++1(-18))
Jn, (as+1(-())) Finally, Ju (Te41 (+(4) 2 Jr, (+1s) Z
Js. (-[)) for any state s E S, so the sequence {Ja, (m((- also monotonically increases, and
each Jz, (IT (-Is) is bounded because Q-function and the target entropy are bounded.
By the monotone convergence theorem, { and {Jm, (m:)) pointwisely converge to their optimal functions Q : S x A - R and J S - IR, respectively. Here, note that (s 2 Jn, (mT() for any because the sequence {J (T,)} is monotonically increasing. From the definition of convergent
sequence, for arbitrary e > 0, there is a large N 2 s.t. Jr, (T((-)s)) 2 (s) <(1->) satisfies for all i 2 N and any s E S.
Now, we can easily show that J.n (TR(-1))) 2 Joc (T(116) e(l-a) for any h > N. any policy a E 11, and any E S. (If not, Jay (T = max Joc (x') 2 Jmn (T), and then Jmas (T=+1(-|s')) 2
Jr. (T (+16)) 2 Jn, (T(()))) > Jr. (-16)) + ((177) 2 (s') for some s E S. Clearly, contradicts the monotone increase of the sequence (Ja, (T:)
Then, by the similar way with A..
Q (st, 0z) = g + 7 ME (sc+1)] 8 p (TA((
Jan (T((1111))1)) c(1-7)
2 3 rt + BELHINP
1
3"r t + +11E, -.Ees1iap( (I 1 10th a(41) log R ( 1411 - - -0loga((44411))
+ (1 - (aaa -a) log(1 - R (1 log(1 - - -7)
2 (s) c.
(A.4)
Note that the state action pair (s, a). the policy and e >> 0 were arbitrary, so we can conclude that and (s,a) E 8 A. In addition, we show that J (T& > Jr. (-(+)) <(1-1) so J= (+1s)) 2 Ja (m([s)) for any π E II and any 8 E S. Thus, Toa is the optimal policy n², and we can conclude that {m:) converges to the optimal policy x².
QTx (s, a) 2 a) for any
A.2 PROOF OF THBBREM? 2
Theorem Suppose that the policy is parameterized with parameter 0. Then, for parameterized policy ne, two objective functions Jasgu (TT(-Is))) and J (To(-(s:)) have the same gradient direction 0 at 0 = for all st S.
Proof. Under the parameterization of ny. the two objective functions become
Janus (T)(- (Euu aad (st, az) + a log Re (st, a:) - a log No (a((s)) + (1 ax) (log(1 Re (st, a.)) logq(ar(s())) + H(a) Jaoan (TO(-Is:))- = BEa,~x To (st, az) + a log am (s,, az) a lologmp(alss))
14

---

Under review as a conference paper at ICLR 2021
We ignore the common Q-function and log Te terms, and the constant terms w.r.t 0 that leads zero gradient in both objective functions. Thus, we only need to show
Vo VolaBa [log R + (1 - (1-a))aa,na log(1 R(aaa)]] []] VolBa (ar log R (A.5)
Oatd- Now, the gradient of the left term in AA.5) 0 Ootd can be expressed
V Lal [log RTv:0] + (1 a:) a)) Bar [og(1 - RT)aa)]]
a.Ba [log R V log mo)
+ aaBa Va log 1= (1 (x) Bay Vo log(1 [m
VeaBo, (a log utar")
+ +aBa V log 18 (1 ax) a)) Ve log(1 RTo.a)].
(A.6)
Here, the gradient of the last two terms in A.6) becomes zero as shown below:
a.BawNTO (V log R
(1 a E Vo log - R
V R R + (1 a E Ea Vo R= RT
V R
R (1 - a E ~ VoR R /(1 -R
a + (1 - (1-a)g
VoR / R
(1 - a Earng
VoR
(1 a)-
aTT + +(1
aBar
V No [R aBaara
V VoR
aTT
a.Bayoyy V R /
al RT
Ro. RR = 0,
(A.7)
where we used an importance sampling technique at laaio of (f az)] = Barwt [e(ae|).) f(r,, ar) for Step (1). By (A.6) and (.7). Juru (To(-1se)) and J Janee (To())8))) have the same gradient at f Bold-
15

---

Under review as a conference paper at ICLR 2021
B DETAILED DAC IMPLEMENTATION
To compute the final objective function 9 we need to estimate Quola and Raoldia QRald can
be estimated by diverse policy evaluation. For estimation of R use function R If we set the objective function of the ratio function as (st: ))) = alLoa llog (8r, az]] + (1
a a) log(1 R (st, d.).]. In the a = 0.5 case, Generative Adversarial Network (GAN) (Good-
Fellow et al. 2014) has shown that the ratio function for a = 0.5 can be estimated by maximizing
)(.-)). By a similar way, we can easily show that maximizing J(R)) can estimate our ratio
function as below:
J. auT log (s, a) + (1- a)g(als) log(1 H a)) The integrand is in the form of y + a alogy- blog(1 y) with a ar and (1 a)q. For any (a, E ( 0), the function + a blog(+ y) has its maximum at a/ ( b). Thus, the optimal R maximizing J [ (s,-) R* () = at/ (a/ + (1 a)g)= R= (st, at). Here, (JS) divergence except some constant
For given s, (s,
note that ((R") becomes just an a-skewed terms if R R=.0.
For implementation we use deep neural networks to approximate the policy the diverse value functions Q. V. and the ratio function R and their network parameters are given by 9, Φ. 4/ and 7, respectively. Based on Section 4.3 and we provide the practical objective (or loss) functions for parameter update as Jx (0), JR= (1)). L (Φ). and fy (vb) The objective functions for the policy and the ratio function Rd are respectively given by
(0) = Es,-D, ar~ I (0, az) + a log (8), 0z) - a (Tt(aa)))))))
(B.1)
JR- (1) = Ee,~D JTT [log Ry (sc, ax)l + (1 a)Ear-D(logl [log(1 Ry (s )]]]-
(B.2)
Furthermore, based on the Bellman operator, the loss functions for the value functions Q and V are given respectively given by
I (0olse (e) Q((,,
Lo(Ø)= = Biee, anD
2
(B.3)
Lv (v) = EEND (Vo(s)) - V(si))2
(B.4)
where the target values are defined as
1 3 Te + P P1
Q(5,, a1)
(B.5)
(se) BT Qo(st, az) + a log (s), (st, az) log anoo (nrlss))
+
(s,, at) log(1 a)g(ar(se))
(B.6)
By using the property of ratio function that satisfies log(1 - RT.aa= - log(1 -a)y log(am (1- a)g) log R log ax, we can replace the last term in ((x) (1 a Ba, -D (log R3 (st, at) log (a, However, the probability of π for actions sampled from D can have high variance, so we clip the term in the expectation over a, ~ D by action dimension for stable learning, then the final target value becomes
((e) Ea,~TO Qo(st, a:) + a log (st,,, (s) log ano (o + (1 (1- (1, (clip(log R. (st, ax) loog azt (ag - d)),
(B.7) where d = dii is the action dimension. We will use B.7 for implementation. Then, note that all objective (or loss) functions does not require the explicit g, and they can be represented by using the ratio function R only explained in Section
In addition, Rº E (0, 1) should be guaranteed in the proof of Theorem 1, and Rº E (0, 1) satisfies when and q non-zero for all state-action pairs. For practical implementation, we clipped the ratio function as (c, 1-)) <) for small > 0 since some values can be close to zero before the replay buffer stores sufficient amount of samples. π is always non-zero since consider Gaussian policy.
Here, T is the network parameter of the target value Vú updated by exponential moving average (EMA) of ae for stable learning (Mnih et al. 2015). Combining all up to now, we propose the
16

---

Under review as a conference paper at ICLR 2021
diversity actor-critic (DAC) algorithm summarized Algorithm 1 in Appendix C Note that DAC becomes SAC when a = 1, and becomes standard off-policy RL without entropy regularization when 0.
To compute the gradient (0). we use the reparameterization trick proposed by (Kingma & Welling, 2013 Haarojj et al. 2018a). Note that the policy action Oş No is the output of the policy neural network with parameter 0. So, it can be viewed as = fo (c; 8t), where f is a function parameterized by 0 and e, noise vector sampled from spherical normal distribution N. Then, the gradient of J, (0) is represented as Ve Be,DD e (V.ooVVaVsssr.. a) + log Ry (st, a) log Te (T(aa==))) (==)s))))== Vele st) VVV, log (E) st)]s.)]
For implementation. we use two Q-functions Q i - 1, 2 to reduce overestimation bias as pro- posed in (Fujimoto et al., 2018), and each Q-function is updated to minimize their loss function Lo(e)). For the policy and the value function update, the minimum of two Q-functions is used (Haamoja et al. 2018a).
Note that one version of SAC (Haarnoja et al., 2018b) considers adaptation of the entropy control factor 3 by using the Lagrangian method with constraint H(T) 2 In our case, this approach can also be generalized, but beyond the scope of current paper and we only consider fixed /3 in this paper.
B. DETAILED IMPLEMENTATION OF THE A-AAPTTTIO
In order to learn a, we parameterize as function of 8; using parameter &, i.e. a = ogg (8)), and implement og (&t) with neural network. Then, & is updated minimize the following loss function deduced from (I0):
(G) - (., + DJs + (1 (xg))()) agc]
(B.8)
Here, all the updates for diverse policy iteration is the same except that a is replaced with (s(s)). Then, the gradient of (g) with respect to be estimated as below:
The loss function of is defined as (E) E,, (a<H((m)) The gradient of i.g (E) can be computed as
-acd-
VeLu(E) VeB.,
Djg no (q) + (1 ag)-()))(() açe)
-VBBnDDee L.BnnDlagE.u. (ag log(aç (1 ag)4) c| + (1 a1l-a)1a -o) log(a (1 ag)4)]
=EADD (V (Veae) (B8 -la-
log + (1 kg)g) c] - -Eangl ( + (1 (01-ag)4))) og)g)))
EANDD (ag Eo, - Vo
az) (1 - Bo log( (ac To + (1 oz a))
EE (Vea)) [E,, (Bu, log log R cf Dog R log(1 ae)4))
EUND
(aETT (1 a3) log(ae + aa)4)]
a.EA
=0
=E E (VVaa)(((V))Veenne
log actl + log R - cj Bag vg [log Ro log loga)))))] (B.9)
Note that R can be estimated by the ratio function R4 Here, we use the same clipping technique as used in (B.7) for the last term of B..
17

---

Under review as a conference paper at ICLR 2021
C ALGORITHM
Algorithm Diversity Actor Critic
Initialize parameter 0, 77, 1h, v. E, Qs, i 1,2 for each iteration do
Sample trajectory of length N by using Te
Store the trajectory I in the buffer D
for each gradient step do
Sample random minibatch of size M from D Compute (0), JR (n), Lo())), Lv (v) from the minibatch 0 +- 0 (
7 17 + oV, R=,
Φ - Φ: - oV L. (oi).. i 1,2
1 < oV (1b))
Update v by EMA from +b
if ed-ppaatation then Compute (C) from the minibatch { dV vV. (E(
end if end for end for
D HYPERPARAMETER SETUP AND ENVIRONMENT DESCRIPTION
In Table D.I we provide the detailed hyperparameter setup for DAC and the SAC baselines: SAC, and SAC-Div. Table D.2 shows the environment description, the corresponding entropy control coefficient 3, threshold for sparse Mujoco tasks, and reward delay D for delayed Mujoco tasks.
SAC SAC-Div
DAC
Learning rate Discount factor
3 10
0.99 (0.999 for pure exploration)
Horizon N
1000
Mini-batch size M
256
Replay buffer length
106
Smoothing coefficient of EMA for Vi
0.005
Optimizer
Adam
Num. hidden layers (all networks)
2
Size of hidden layers (all networks)
256
Policy distribution Activation layer
Independent Gaussian distribution
ReLu Linear
Output layer for Qs. V V
Output layer for org, R2
Sigmoid
Regularize coefficient for dgg Control coefficient. for a-adaptation
10-3
- -dim(A)
Table D.I: Hyperparamter setup
18

---

Under review as a conference paper at ICLR 2021
State dim.
Action dim.
B
Threshold
SparseealiCheetah--il
17
0.02
5.0
SparseHopper-il
11
0.04
1.0
Sparse WalnseWall22dvv1
17
0.02
1.0
SparseAnt-vl
111
0.02
1.0
State dim.
Action dim.
Delay D
HumanoidStandup-vi
376
17
DelayedHaliCheetaa--r
17
0,2
20
DelayedHopper-vl
11
0.2
20
WelayedWallkr22-v1
17
0.2
20
DelayedAnt-vl
111
0.2
20
Table D.2: State and action dimensions of Mujoco tasks and the corresponding B
E SIMULATION SETUP
We compared our DAC algorithm with the SAC baselines and other RL algorithms on various types of Mujoco tasks with continuous action spaces (Todorov et al. 2012) in OpenAl GYM (Brockman ctt al. 2016). For fairness, both SAC/SAC-Div and DAC used a common hyperparameter setup that basically follows the setup in (Haarnoja et al., (018a). Detailed hyperparameter setup and environ- ment description are provided in Appendix [ and the entropy coefficient ,3 is selected based on the ablation study in Section 6.33 For the policy space II considered Gaussian policy set widely considered in usual continuous RL. For the performance plots in this section, we used deterministic evaluation which generated an episode by deterministic policy for each iteration, and the shaded region in the figure represents standard deviation (Io) from the mean.
F PERFORMANCE COMPARISONS
In this section, we provide more performance plots and tables. In Section F. Fig. F.I shows the mean number of discretized state visitation curve of DAC and SAC/SAC-Div. For discretization, we simply consider 2 components of observations of Mujoco tasks, which indicate the position of the agent: x, 2 axis position for SparseHalfCheetah, SparseHopper, and SparseWalker, Walker, and xr axis position for SparseAnt. We discretize the position by setting the grid spacing per axis to 0.01 in range (-10, 10). Table. F.I shows performance on sparse Mujoco tasks. Table F.2 shows max average return for HumanoidStandup and delayed Mujoco tasks. In Section F:3
Fig. 3 and Table.
F3 shows the performance comparison to other RL algorithms on HumanoidStandup and delayed Mujoco tasks.
F.1 PERFORMANCE COMPARISON WITH THE SAC BASELINES
$5C
12
-2.
(a)
SparseHalfCheetah-
(b) SparseHopper-vl
(c) Sparse
(d) SparseAnt-vl
Figure F.1: The number of discretized state visitation sparse Mujoco tasks
19

---

Under review as a conference paper at ICLR 2021
DAC (a 0.5)
SAC
SAC-Div
SparseHalfCheetah SparseHopper SpurseWalker
915.90+50.77
386.90-404.70
394.70++405.5
19.590=110.57
900.60:15.22
901.40:14.25 373.10:443.11 963.80=:42.51
573.10:104496 935.80437.08
301.30=448.15
SparseAnt
870.70=121.14
Table F.1: Max average return of DAC algorithm and SAC baselines for fixed setup
DAC
DAC 0.8)
DAC (a-adapt.)
SAC
SAC-Div
2012491881
170832.05
197302.37
167394.36
165548.76
HumanoidS
1:25222.77
=11244.71
144305,331
1.721.99
+:2005,88 8,005
Del. HalfChectah
1045.69
6552.06=1140.18
7594,70=:1259.23
3742.33++3064.5
4080.67:::418.07
Del. Hopper
3283.77::112.00
2836.811679.05
3428.18::69.06
2175.31+
2090. 138338
Del. Walker2d
5360..33--507.58
3973:37-4273.63
4067.
11-7171--25.88
3220,92+++1177,99
4048.11 1120990448
Del. Ant
4088. 12::57999
1536.72:11164.6
4243.19:175549
3248. 13:8.433114444..0
3978.3411170.22
Table F2: Max average return of DAC algorithms and SAC baselines for adaptive setup
F.2 COMPARISON TO RND AND MAXENT
We first compared the pure exploration performance of DAC to random network distillation (RND) (Burda et al. 2018) and MaxEnt (Hazan et al. (019). which are state-of-the-art exploration methods, on the continuous 4-room maze task described in Section 6. RND adds an intrinsic reward Tint, to MDP extrinsic reward Tn TR т₁ CantPint,t based on the model prediction error Tint, (1(+1) f(++1)) 2 of prediction network and random target network f for given state $1+1- The parameter of the target network is initially given randomly and the prediction network learns to minimize the MSE the two models. Then, the agent goes rare states since rare states have higher prediction errors. On the other hand, MaxFnt considers maximizing the entropy of state mixture distribution mii by setting the reward functional in (Hazan et al. 2019) as log dT (s) + CM. where dm is a state distribution of the trajectory generated from ग and Car is a smoothing constant. Here, MaxEnt mainly considers large or continous state space, the reward functional is computed based on several than simple random policy on various tasks with continuous state space.
methods. Then, MaxEnt explores the state space better
For RND, for both the prediction network and the target network, we use MLP with 2 ReLu hidden layers of size 256, where the input dimension is equal to the state dimension and the output dimen- sion is 20, and we use Cint = 1. For MaxEnt, we compute the reward functional at each iteration by using Kernel density estimation with a bandwidth 0.1 as stated in (Hazan et al. 2019) on previous 10000 states stored in the buffer, and we use eM 0.01. For RND and MaxEnt, we change the entropy term SAC/DAC to the intrinsic reward and the reward functional term respectively, and the Gaussian policy with fixed standard deviation o 0.1. Fig. F[(() shows the mean number of state visitation curve over 30 seeds of the 4-room maze task and Fig. F[(() shows the corresponding state visit histogram of all seeds. As seen in Fig. [22] DAC explores more number of states than RND and MaxEnt on continuous 4-room maze task, so it is seen that the exploration of DAC is more sample-efficient than that of RND/MaxEnt the maze task.
20

---

Under review as a conference paper at ICLR 2021
50
100
100
1500
3000
AND
50
Visted
100
2000
MaxEnt
50
1500
1000
1000
RND
MaxEnt
DAC
0-
Exploration Flow
Time Steps (1e5)
(a) Number of state visitation
(b) State visit histogram at 5k (left) 50k (middle)
500k (right) steps
Figure F.2: Pure exploration comparison with RND/MaxEnt
F.3 COMPARISON OTHER RL ALGORITHMS
We also compare the performance of DAC with o-adaptation to other state-of-the-art RL. algorithms. Here, we consider various on-policy RI. algoritmms: Proximal Policy Optimization (Schulman et al.) 2017b) (PPO, stable and popular on-policy algorithm), Actor Critic using Kronecker-factored Trust Region (Wu et al. (017) (ACKTR, actor-critic that approximates natural gradient by using Fronecker-factored curvature). and off-policy RL algorithms: Twin Delayed Deep Deterministic Policy Gradient (Fujimoto et al.) 2018) (TD3, using clipped double-Q learning for reducing over- estimation); and Soft LeLemrnig (Haarnoja et al.) (017) (SQL, energy based policy optimization using Stein variational gradient descent). We used implementations in OpenAl baselines (Dhariwal jet al. 2017) for PPO and ACKTR, and implementations in author's Github for other algorithms. We
provide the performance results Fig. F3 and Table F3
A000
600
ADTTR
ACKTR
ACCR
4000
Sai
TD3
SAC
1AC.0
SAC-O
04
DACID
(a) HumanoidStandupp-l
(b) DelayedHallCCheetah-vv
(c) DelayedHopper-vl
5000
1000
4000
4000
3000
2000
105
1P0
Eno
1009
0
Time Swer (UnO)
The Sep 1100
(d) DelayedWaller2d-v1
(c) DelayedAnt-vl
Figure F3: Performance comparison to other RL algorithms
21

---

Under review as a conference paper at ICLR 2021
DAC
PPO
ACKTR
SQL
TD3
SAC
197302.37
160211.90)
109655.30
138996.84
58693.87
167394.36
HumanoidS
1:43055.31
+3268.37
149166.15
+33903.03
112269.93
17211.9
7594,70
2247.92
3295.30
5673.34
4639.85
3742.33
Del. HalfCheetah
11259.23
1440.69
1824.05
+1241.30
11393.95
+:304.55
3428.18
2740.15
2864.81
2720.32
2276.58
2175.31
Del. Hopper
+69.08
+719.63
-1072.64
+127.71
+1471,66
+ +1358.39
4067.11
2859.27
1927.32
3323,63
3736.72
32201.92
Del. Walker2d
+257.81
+1938.50
-1647.49
+503.18
+1806.37
+1107.91
4243.19
1224.33
2956.51
6.59
904.99
3248.43
Del. Ant
1795.49
#521.62
+234.89
116.4
1811178
#1454.48
Table F.3: Max average return of DAC and other RL algorithms
22

---

Under review as a conference paper at ICLR 2021
G ABLATION STUDIES
Here, we provide more ablation studies for remaining delayed Mujoco tasks. Fig. G.2 shows the averaged learning curves of a, g. and ((x) of DAC considering a-adaptation, where the con-
trol coefficient с is -2.0d and d = dim[[A (A). Fig. G.2 shows the performance of DAC considering a-adaptation with control coefficient 0.5 and 8 = 0.1, 0.2, 0.4. Fig. G.4 shows the performance of
0, -0.5d, 1.0d, and -2.0d. Fig. G.3 shows the per-
formance of DAC with SAC, SAC-Div with KL-divergence (SAC-Div(KL)), SAC-Div with JithJS-din JS-divergence (SAC-Div(IS)), and DAC to see the effect of IS divergence on the performance as explained in Section
6.3 Other
hyperparameters follow the default setup given in Table D.I
Weighting factor
10-
0.6
0.
0.4
03
-12
-02
02
-04
-04
-08
Time Sters (246)
Time Stats (100)
Time Stars (lel)
(a) DelayedHoopper-vl
(b) DelayedWalker2d-vl
(c) DelayedAnt-vl
Figure G.I: Ablation study for
Control coefficient
4020
2310
SAC
SAC
2000
SAC Biy
S42
Sac Bay
Bactevthet... 7.5c1
D4CIovddgg L 2.09
BACtevthett.. 2,501
DAClevillL L 235
500
Biclesmpe. -200
DeCivedest.coo
DAGleamgtt.c -2.6m
BAClesdapt -200
63
05
155
20
25
30
Tere
Time Stepo
(a) Delayedllopper-vl
(b) DelayedWaller2d-v1
(c) DelayedAnt-vl
Figure G.2: Ablation study for
23

---

Under review as a conference paper at ICLR 2021
Entropy coefficient
5200
4203
6200
9300
2340
2003
2.21
SACES
0.0 0000p001,11222 2=221
2320
34 6-1.3
1009
040c=05,1=34 J-241
DaClomab, 5m04
1.5 Time Sotpa nut)
Thme Stape Dedi
Tims Slapo Geei
(a) DelayedHopper-vl
(b) DelayedWalker2d-vl
(c) DelayedAnt-vl
Figure G.3: Ablation study for
The Effect of JS divergence
430
200E
200
3000
SAC-O viiLI
SAcovidi
54C.0 +1.0
540.0 vibi BACIa-Start
DAG 2daggp
(ACCa-38911)
00
LG
(a) DelayedHopper-vll
(b) DelayedWalker2d-vl
(c) DelayedAnt-vl
Figure G.4: Ablation study for JS divergence
24


Reviews:
[{"review_id": "o2VsOTAJEOC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes diversity actor-critic (DAC) for exploration in reinforcement learning. The main idea of the proposed algorithm is to take advantage of the previous sample distribution from the replay buffer for sample-efficient exploration. The authors provide convergence analysis of DAC and conduct empirical investigations on several benchmarks. \n\nPros \n\nThe idea of using previous sample distribution from the replay buffer for better exploration seems interesting. The proposed exploration bonus $\\mathcal{H}(q^{\\pi, \\alpha}_{\\text{target}})$ can be decomposed into three terms as shown in (4). Since the last term does not depend on $\\pi$, intuitively this exploration bonus encourages the exploration of $\\pi$ (first term), and tries to make $\\pi$ different with previous policies approximated by the replay buffer (second term). The authors provide a reasonable method to optimized the proposed objective, which can be naturally combined with state-of-the-art algorithms like SAC. \n\nCons\n\n1. Theorem 1 seems misleading. The diverse policy iteration can only guarantee the converge to the optimal policy with respect to the regularized value function, not the optimal policy of the original problem. The authors should make the definition of $\\pi^*$ clear. \n\n2. It’s hard to see the motivation of using a mixture of $q$ and $\\pi$. Could you explain more about this choice?  \n\n3. It’s worth to provide the results of SAC-div with JS divergence as it’s more similar to the proposed objective (4).\n\n4. The experiment results are not convincing enough as some important baselines are missing. For example, [1] also uses a mixture of previous polices to encourage exploration with strong theoretical guarantees. I believe this is closely related to the proposed algorithms. \nAlso, the experiment results are not very promising compared with the baseline algorithms based on SAC. \n\n[1] Hazan, E., Kakade, S., Singh, K. and Van Soest, A., 2019, May. Provably efficient maximum entropy exploration. In International Conference on Machine Learning (pp. 2681-2691). \n\nOther suggestions\n\nThe main idea of the proposed method is to make the current policy different with previous policies. The paper uses a nonparametric method  (2) to approximate the previous policies. I think it’s also worth to try parametric $q$. For example, $q$ could be learned by fitting the replay buffer, or use a moving average of previous policies. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "k4QaQQz9Cdn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper proposes DAC, an actor-critic method exploiting the replay buffer to do policy entropy regularisation. The main idea of DAC is to use the data from the replay buffer to induce a distribution  $q(\\cdot, s_t)$ and replace the entropy part of the Soft Actor-Critic objective with a convex combination of $q$ and $\\pi$. This results positively on exploration properties and leads to sample-efficiency gains on some of the considered MuJoCo benchmarks.\n\n### Pros\n- Formulating the diversity using the entropy of the replay buffer frequences is an interesting idea.\n- Using the convex combination of $q$ and $\\pi$ for entropy regularisation is a nice way of generalising SAC for the considered purpose.\n- The paper shows the convergence of their method to an optimal policy and derives a surrogate objective whose gradient direction coincides with the original one, but which can be practically used. (However, I have not checked the proofs which are in the appendix).\n\n### Cons\n- It is not clear, what is the problem the paper tackles. Is it exploration? Is it a generic RL setup? What kind of problems is DAC good for?\n- If DAC is for improving exploration, then it should be compared with other exploration methods, not with vanilla SAC. Comparison with RND should not be in the appendix and there should be more details on this. Related work in this case should have a paragraph on exploration methods in RL.\n- The paper is based on assumptions not challenged/tested by the authors, e.g. policy entropy regularisation is inefficient, because it does not take the distribution of the samples into account.\n- The paper focuses more on the technical details of the solution rather than justifying the assumptions and making the research question clear.\n\n### Reasoning behind the score\n\nI believe, the paper has a great potential. However, at the moment I vote for rejection. The paper has to have a clear research question and its motivation. This should define the experimental part of the work. Lack of a clear positioning makes it unclear if the baselines of the experimental sections are the right ones and whether the claims have been properly supported by the results.\n\n### Questions to the authors\n- Can you formulate the exact problem you are solving?\n- How can you justify the claim that 'entropy regularization is sample inefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account.\n- \"it is preferable that the old sample distribution in the replay buffer is uniformly distributed\". Why is it true? Doesn't prioritized experience replay refute this claim?\n- You define $\\beta$ in Equation 1 in $(0, \\infty)$, can it really be infinite?\n- \"The rationale behind this is that it is preferable to have as diverse actions stored in the replay buffer as possible for better Q estimation in off-policy learning.\" What are the assumptions for this? Do you care more about better Q estimates or finding an better policy faster? How can you support your rationale?\n- In section 4.1. you define the target distribution as a convex combination of $\\pi$ and $q$. You assume that the buffer is generated by $q$. Does such a policy always exist? What are the assumptions for this?\n- You prove the convergence of your algorithm (I did not check the proof in the appendix), what are the assumptions for which the convergence is guaranteed?\n- Why do you use sparse/delayed MuJoCo benchmarks, but not the original ones?\n- The variance across different seeds seems to be huge for your method (as well as for the others). What do you think is the reason behind this? This also happens for the pure exploration task in 6.1, why do you think it happens?\n- For the adaptive $\\alpha$ case, you restrict the range of possible values, what is the reasoning behind the left boundary?\n- I think your paper can find an important application in Imitation Learning or off-line RL. Have you considered this? Are you aware of works which do something similar in those subfields?\n\n### Additional feedback not affecting the score\n- \"Reinforcement learning aims to maximize the discounted sum of rewards...'. Should be 'expected discounted sum'.\n- There should be a distribution over initial states under the expectation sign in 3.1.\n- 'A is the continuous action space'. This is not true for the general MDP definition, specify that this is specific for your paper.\n- Section 3.1, a policy is a mapping from states to distribution over actions, not to actions.\n- In off-policy, we can learn from any other samples, not only from 'previous samples' from our policy.\n- typo \"propoed\" at the bottom of page 4.\n- Equation 9 does not have a left hand side.\n- DAC acronym has been used in RL. I would choose a different one to avoid confusion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hEe7fmcWT3y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a novel exploration method in off-policy learning. Compared to previous methods which do not take care into account the distribution of the samples in the replay buffer, the proposed method maximizes the entropy of the mixture of the policy distribution and the distribution of the samples in the replay buffer, hereby making exploration efficient.\n\nReasons for score\n\nI vote for accepting the paper. The paper proposes an intuitive and efficient exploration method that generalizes existing methods, including them as special cases. The authors provide a theoretical guarantee (Theorem 1) that the policy obtained from the iteration of evaluation and improvement under this new regime converges to the optimal policy.  The presentation is clear and concrete, and the experiments are convincing.\n\nPros\n\nThe experiment results are not limited to just showing that the proposed method achieves higher reward than state of the art methods, but they also address important questions such as \n(i) the pure exploration when rewards are assumed to be 0\n(i) the necessity of the adaptation of alpha, the parameter that controls the ratio of the current policy to the sample distribution in the target distribution.\n(ii) the effect of controlling alpha, the entropy weighting factor beta, and the control coefficient c (required for adapting alpha), and also, the robustness of the proposed method to these parameters.\n\nThe authors have stated the experiment details clearly and the results are convincing.\n\nCons\n\nThe methodology part in Section 3 and 4 could be improved. Some notations are confusing.\n(a) In Section 3, the policy \\pi is defined as a function from S to A. It looks like it is a  fixed function over time.\n(b) An explanation on the definition of J_{pi 1}(pi 2) would be helpful,e.g.,  J_{pi 1}(pi 2) is value of J(pi_2) computed under pi_1.\n\nMinor Comments\n\nIt would be good to add the line of SAC and SAC-Div in Figure 5 (c ) to show that the performance of DAC with adaptive alpha is robust to control coefficient c. For now, one has to go back to Figure 4 (b) to check that most of the case (when c is not 0), DAC with adaptive alpha performs better than SAC and SAC-Div.  \n\nIn Section 6 in the 5th line, J(\\pi) should be specified as “J(\\pi) in (1)”. It is done in the next sentence, but I prefer that it is done when it first appears. It was confusing\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IXFJcbejoak", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the exploration efficiency issues in off-policy deep reinforcement learning (DRL). The authors identify a sample efficiency limitation in the classical entropy regularization, which does not take into account the existing samples in the replay buffer. To avoid repeated sampling of previously seen scenarios/actions, the authors propose to replace the current policy in the entropy term with a mixture of the empirical policy estimation from the replay buffer and the current policy, and term this approach as sample-aware entropy regularization. The authors then propose a theoretical algorithm called sample-aware entropy regularized policy iteration, which is a generalization of the soft policy iteration (SPI) algorithm, and show that it converges assuming that the empirical policy estimation is fixed. A practical algorithm based on the sample-aware entropy regularized policy iteration, called Diversity Actor-Critic (DAC), is then proposed. This algorithm is a generalization of the well-known soft actor-critic (SAC) algorithm. Finally, numerical experiments show that DAC outperforms SAC and other SOTA RL algorithms, and some ablation studies are also provided to demonstrate the effect of hyper-parameter choices in DAC.\n\nIn general, the approach is novel to my knowledge and the high level idea of using mixed policies in the entropy regularization to avoid repeated sampling and encourage unseen scenarios/actions is also interesting and reasonable. However, there are some clarity and technical issues that should be addressed and improved, as listed below:\n1. The authors study finite horizon MDPs, for which the optimal policy should be non-stationary in general. However, the authors only consider stationary policies. Instead, the authors should either change the underlying setting to infinite horizon MDPs or consider non-stationary policies.  \n2. In (2), $s_t$ should be replaced by an arbitrary $s$ in the state space. Otherwise there may be contradicting definitions of the policy $q$ if $s_t$ and $s_{t’}$ are equal for some two different timestamps $t$ and $t’$. And in (3), it is better to write the $q_{\\rm target}^{\\pi,\\alpha}$ in the entropy term as $q_{\\rm target}^{\\pi,\\alpha}(\\cdot|s_t)$, to be consistent with (1). \n3. It’s not very clear why the authors propose to estimate $R^{\\pi,\\alpha}$ with some (neural network) parametrized $R^{\\alpha}$. The authors mention that one can only estimate $R^{\\pi_{\\rm old},\\alpha}$ for the previous policy $\\pi_{\\rm old}$ in practice. However, since in $R^{\\pi,\\alpha}$, all the quantities including $\\pi$, $q$ and $\\alpha$ are known, I’m confused why one cannot evaluate it directly. On a related point, it’s not very clear why the estimation procedure for $\\eta$ (the parameter of $R^{\\alpha}$) using hat $J_{R^{\\alpha}}(\\eta)$ makes sense. The form of hat $J_{R^{\\alpha}}(\\eta)$ looks like an entropy term extracted from the $J_{\\pi_{\\rm old}}$ function, but it’s unclear why maximizing it gives a good estimation of $R^{\\pi,\\alpha}$. Some more explanations are needed. \n4. There seem to be several errors (at least inaccuracies) in the proof of Theorem 1 (in the Appendix). Firstly, in the proof of Lemma 1, the term “correctly estimates” is not very accurate, and should be simply stated as something like “equals”. Also, it’s not very clear when the assumption $R^{\\alpha}\\in(0,1)$ can be guaranteed (e.g., using Gaussian/soft-max policies?). Secondly, in the main proof of Theorem 1, convergence of $Q^{\\pi_i}$ to some $Q^{\\star}$ is correct, but this does not immediately imply convergence of $J_{\\pi_i}$, let alone the convergence of $\\pi_i$ to some policy $\\pi^\\star$. On a related point, the proof for the optimality of $\\pi^\\star$ in terms of $J$ is not clear. In particular, it is not clear why (7) and Lemma 2 implies the chained inequality $J_{\\pi_{\\rm new}}(\\pi_{\\rm new})\\geq J_{\\pi_{\\rm old}}(\\pi_{\\rm new})\\geq J_{\\pi_{\\rm old}}(\\pi_{\\rm old})$. I understand that the authors may feel that the proofs are similar to that of SPI, but indeed there are several significant differences (e.g., the definitions of $\\pi_{\\rm new}$ and $J_{\\pi}$). More rigorous proofs are needed for these claims. \n5. In Section 5, it is unclear why the authors need to include the parameter $c$, how to choose it and what it serves for. Some additional explanations are needed. \n6. On a high level, the eventual goal of the paper is not clearly stated. From the experiments, it seems that the average episode reward is the actual goal of concern. However, the problem setting and the theoretical results (Theorem 1) seem to indicate that the problem of concern is the discounted entropy regularized reward. Some discussion about this is needed. \n\nFinally, here are some more minor comments and suggestions:\n1. In the analysis of the sample-aware entropy regularized policy iteration, the authors assume that $q$ is fixed. However, in practice, especially in the long run (as concerned in the analysis), such an assumption will not hold (even in just an approximate sense). Can you still obtain some sort of convergence when taking into account the $q$ changes?\n2. Why do you need to divide the reward and entropy regularization term in $Q^{\\pi}$ by $\\beta$? \n3. It’s better to write out the “binary entropy function $H$\" explicitly for clarity. \n4. At the beginning of Section 4.3, “propoed” should be “proposed”, and In Section 5, “a function $s_t$” should be “a function of $s_t$”. \n5. Some high level explanations on why the $(1-\\alpha)$ term can also be dropped in (8) will be helpful.\n6. The theoretical results only show that the algorithm converges, which is already guaranteed by SPI. Is there any possibility to show that there is also some theoretical improvement?\n\nSo in short, the paper proposes an interesting modification of the max-entropy regularization framework, but contains several technical and clarity issues. Hence I think it is not yet ready for publication in its current form. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: First, I'd like to thank both the authors and the reviewers for extensive and constructive discussion. The paper proposes a generalization of SAC, which considers the entropy of both the current policy and the action samples in the replay pool. The method is motivated by better sample complexity, as it avoids retaking actions that already appear in the pool. The paper formulates a theoretical algorithm and proves its convergence, as well as a practical algorithm that is compared to SAC and SAC-Div in continuous sparse-reward tasks.

Generally, the reviewers found the method interesting. After rounds of discussion and revisions, the reviewers identified two remaining issues. Theoretical analysis still requires improvement and the positioning of the paper is not clear. Particularly, the method is motivated as an exploration method, and it should be evaluated as such, for example, by comparing to a more representative set of baseline methods. Therefore, I'm recommending rejection, but encourage the authors to improve the work bases on the reviews, and submit to a future conference.
Decision: Reject