A Max-Min Entropy Framework for Reinforcement Learning
Authors: Seungyul Han
Youngchul Sung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
entropy regularization
max-min optimization framework
exploration

Abstract: In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.

A Max-Min Entropy Framework for Reinforcement Learning
Seungyul Han
Youngchul Sung'
Graduate School of Artificial Intelligence UNIST Ulsan, South Korea 44919 syhan@uniste kr
School of Electrical Engineering
KAIST
Daejeon, South Korea 34141 ycsung@kaiste wtt.kaiist..... kr
Abstract
In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-crite (SAC) algorithm implement- ing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglemen of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement the current ofte-oo-thhearle art RL algorithms.
1 Introduction
The maximum entropy framework has been considered in various RL domains [22, 23, 30, 45, 51, 53, 58). Maximum entropy RL adds the expected policy entropy to the return objective of standard RL in order to maximize both the return and the entropy of policy distribution. Maximum entropy RL encourages the policy to choose multiple actions probabilistically and yields a significant improvement in exploration and robustness and good final performance in various control tasks [15, 20, 21, 25, 26, 29, 50). particular, soft actor-critic (SAC) implements maximum entropy RL in an efficient iterative manner based on soft policy iteration and guarantees convergence to the optimal policy for finite MDPs, yielding significant performance improvement over various on-policy and off-policy recent RL algorithms in many continuous control tasks. However, we observe that such an iterative implementation of the maximum entropy strategy of optimizing for policies that aim to reach states with high entropy in the future does not necessarily result in the desired exploration behavior but may yield positive feedback hindering exploration model-fre fadel-free sample-bar function approximation. In order to overcome such limitations associated with implementation of the maximum entropy RL, we propose max-min entropy framework for RL, which aims to learn policies reaching states with low entropy and maximizing the entropy of these low-entropy states, whereas the conventional maximum entropy RI. optimizes for policies that aim to visit states with high entropy and maximize the entropy of those high-entropy states for high entropy of the entire trajectory. We implemented the proposed max-min entropy framework into a practical iterative actor-critic algorithm based on policy iteration with disentangled exploration and exploitation. It is demonstrated that the proposed algorithm significantly enhances exploration capability due to the fairness across states induced by the max-min framework and yields drastic performance improvement over existing RL algorithms including maximum-entrope SAC on difficult control tasks.
learning with
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2 Related Works
Maximum Entropy RL: The maximum entropy framework has been considered in various RL domains: inverse reinforcement learning [58], stochastic optimal control [45, 51, 53], guided policy search [30], and off-policy learning [22, 231. There is a connection between value-based and policy-based RL under the policy entropy regularization [38], [42] combines them, and finally [46] proves that they are equivalent. Maximum entropy RL also related to probabilistic inference [40, 45). Recently, maximizing the entropy of state distribution instead of the policy distribution [26] and maximizing the entropy considering the previous sample action distribution [25] have been investigated for better exploration.
Max-Min Optimization: Max-min optimization aims maximize the minimum of the objective function [11]. Under the convex-concave assumption, there exist many algorithms to find the solution to max-min problem by using optimistic mirror descent [44], Frank-Wolfs algorithm [17], and Primal-Dual method [24]. However, non-convex max-min problems are more challenging [37] and there are several recent studies to find (approximate) solutions to non-convex max max-min optimization problems [6, 41, 43]. This framework has been used in various optimization/contro) domains: fair resource allocation [31], inference [4, 56). generative adversarial network (GAN) [2, 18], robust training [33], and reinforcement learning [54].
Exploration in RL: Exploration is one of the most important issues in model-free RI., there is the key assumption that all statc-action pairs must be visited infinitely often to guarantee the convergence of Q-funcction [55]. In order explore diverse state-action pairs in the joint state-action space, various methods have been considered in prior works: patrinsiaall-motiiaaee
reward based on curiosity [5. 10], model prediction error [1, 9], information gain [25, 27, 28], and counting states [32, 34]. These exploration techniques improve exploration and performance in challenging sparse-reward environments [3, 9, 12].
3 Background
3.1 Basic RL Setup
We consider an infiite-hhrrzon MDP (S, P,7,,). where S is the state space, A is the action space, P is the transition probability, 7 is the discount factor, and is the bounded reward function. We assume that each action dimension is bounded. The RL agent has a policy π S x A + R+ E II, which chooses an action o, for given state 8, according to de T--1se) at each time step where II is the policy space. For action a, the environment yields the reward r == r(t: a+) and the next state St+1 P(sc+1lss,, az). Standard RL learns policy maximize the discounted return Esaxpo, Torna Dr 7're], where Tt = (St, De St+1, ...) is an episode trajectory.
3.2 Maximum Entropy RL and Soft Actor-Critic
Maximum entropy RL maximizes both the expected return and the expected policy entropy simulta- neously to achieve an improvement exploration and robustness. The entropy- -ugmeettd objective function of maximum entropy RL is given by
(T) = Esporoo TorK
Σ or + (
(I)
where H(T(--s))- Eu-r(1aal- log (a]s)) the entropy function and a > is the entropy coeffi- cient. A key point here is that the policy entropy is included in the reward not used as external regularizer at each time step. Thus, this maximum entropy RL framework optimizes for policies that aim to reach states on which policies have high entropy in the future [21].
Soft actor-critic (SAC) is an efficient off-policy actor-critic algorithm to solve the maximum entropy RL problem [22]. SAC maximizes (1) based on soft policy iteration, which consists of soft policy evaluation and soft policy improvement. For this, the soft Q-value of given (st, ae) is defined
(T rt E E not
Σ J=t+1
(r + aM((((1s))))
(2)

---

which does not include the policy entropy of the current time step but includes the sum of all future policy entropy and the sum of all current and future rewards. For given 9, soft policy evaluation guarantees the convergence of soft Q-function estimation, which estimates Q" by iteratively applying a modified Bellman operator T to a real-valued estimate function Q : A - IR, given by
T (st,, ae) = Ti + 7Bs,, 1- (V(++)))
where
(3)
v(s)) = (.. [Q(s]](([[])))]) a.) - a log ( [sr]]
and V(st) is the soft state value function. Then, at each iteration, SAC updates the policy
(xp(Q at)/a)
Tnew = arg min DEL (T(161))]
(4)
Zhold
xEll
arg max E (a (st, 0z) - a log π (ap(8))
(5)
mell
where (s) is the log partition function which is function of s only. Soft policy improvement guarantees QTHe= (st, at) 2 QTald (s),a) at) for all (s;, az) E S x A. Finally, soft policy evaluation and soft policy improvement are repeated. Then, any initial policy policy x², i.e., Q (st, az) 2 Q" (st, at) for all x' II and all (st, az) S x A, and x² maximizes Jaaz [22]. Proof of soft policy iteration assumes finite MDPs. SAC approximates the soft policy iteration by sample-based learning with function approximation in continuous-space cases.
πI converges the optimal
4 Motivation: Limitation of Maximum Entropy SAC in Pure Exploration
In this section, we will consider only the maximum entropy SAC in a pure exploration setup without the reward function (the reward function r = 0 in MDPs). As seen in Sec. 3, SAC efficiently solves the maximum entropy RL problem to maximize (1) in an iterative manner based on judiciously-dfingg state and action value functions and the step-wise optimization cost (5). The well-defined value functions and the local cost function as such enable proof of soft policy improvement for finite MDPs in similar way the proof of the classical policy improvement theorem. Note that at each time step, SAC updates the policy to maximize the cost (5), composed of two terms: -a (QTaia (st, ax)] and a.a,--r(Iss log π (a, [s)]] = aH((a(a (s)) As aforementioned, the soft Q-function contains the sum of current and future rewards and the sum of only future policy entropy. Since we consider only the entropy terms without rewards here, the first term Eg [ 3 [QF (st, )] is the current estimate of the sum of future entropy when action a is taken from policy π at state st, whereas the second term a (c) is the entropy of the policy π itself. Hence, at each time step, SAC tries to update the policy N to yield the maximum sum of the estimated future entropy and the policy entropy itself. Here, the term (e) [Quate (se, ax)) plays the role of guiding the policy towards the direction of high future entropy.
Saturation: In sample- update with function approximation, however, the SAC iteration does not yield the desired result, contrary to the intention behind maximum entropy. To see this, let us consider a pure exploration task in which there is no reward. The considered task is a 100 x 100 continuous 4-room maze proposed in [25], modified from the continuous grid map available at https: gittb..t..uugggg/// com/ huy hoyyaoyu/au /oroyyrrriiia /etrrriiaapp Fig. 1(a) shows the maze environment, where state is the (Ir, -p-poiition of the agent in the map, action is (dz, dy) bounded by [-1, x [1,, and the next state of the agent is (x + dx, y + dy). Starting from the left-lower corner (0.5, 0.5), the agent explores the maze without any external reward. First, note that for this pure exploration task, the optimal policy maximizing JMaxEnel (T) is given by the uniform policy that selects all actions A = [] x 1, uniformly regardless of the value of st. This is because the uniform distribution has maximum entropy for a bounded space [13]. Then, we compare the exploration behaviour of SAC and the uniform policy in the maze task. Fig. 1(b) shows the mean accumulated number of different visited states averaged over 30 random seeds as time goes, where the shaded region in the curve represents standard deviation (lo) from the mean and different state is meant as a nonoverlapping quantized 1 x 1 square. As seen in Fig. 1(b), SAC explores more states than the uniform policy at the early stage of learning. As learning progresses, however, SAC fails to visit new states after 300k time steps, whereas the uniform policy continues visiting new states. As a result, SAC eventually visits fewer states than the uniform policy on average. The result shows that SAC fails to converge to the optimal uniform policy and its performance become saturated.
3

---

50
100
100
SAC
(do
(a) Continuous 4-room maze
t =011)
of
2802
SAC
50
1301
100
Uniferm
SAC
TIme Stape 12a5)
(do
(h) Number of state visits
SACCav
Unitomm
3.00
50
Exploration Flow
Time Soapa (2551
(c) Gradient Q-function
(d) State histogram over time steps starting from 300k, 350k, 400k, and 450k time steps (from the left in order)
Figure I: Comparison of SAC and the uniform policy the continuous 4-room maze
To examine the saturation behavior of SAC in the above pure explo- ration task, we investigate the policy update SAC in (5). Since the current Q-function estimate (implemented by neural network) replaces Futt in (5) in implementation with function approxima- tion, the policy update is rewritten as
Narrow Exploration Radius:
arg max (- (9,))) 4z)] + oHH(T(-ls:))).
*Ell
As mentioned already, the first term Ee, (.. (8,,, (+)] )] is the current estimate of the sum of future entropy this pure exploration case when action a is taken from policy π at state 8t, whereas the second term a HH(m ( (a)]s:)) is the entropy of the policy itself. The first term (e [Q(ss, at)] intends to direct the policy towards the direction of high future entropy. Note that maximizing the second term already yields the uniform policy, but the Q-function term affects the policy update. In order to see how the Q-function term actually affects the policy update, we differentiate the entropy coefficient in the policy update part (5) (6) as the policy entropy coefficient a and that the soft value function part (2) and (3) as the value entropy coefficient aq. We fix an as is original SAC). With this
(
a and change oig 0, 0.1, and 1 (note that the case of aq change of aq, we conducted the same pure exploration maze task. Fig. 1(c) shows the average norm of the gradient of Q-function with respect action, i.e., E, [Ill[lVV [ ((s,(((s) a) a=a, II] over time with a ~ (-(18)) and 8 from a mini-batch drawn from the replay buffer ' of SAC update, where the Q neural network weights were initialized randomly. Fig. 1(d) shows the histogram of states that the policy visits over 50k time steps starting from 300k, 350k, 400k, and 450k time steps. When aq 0 with no reward, the Q-function update by the Bellman operator T in (3) is trivial as (ss a) + EersP(-)s.a) (f [Q(ss/ ; .a)], i.e., replacement. When the initial ((s.a) a) is (nearly) flat over S
by initial random weight assignment for the Q-neural network, the flat Q is maintained by this trivial update. Indeed, it is seen in Fig. 1(c) that E,,~2
a) a=)==] with a, 7 (-(ss) nearly zero across all time for aq = With flat function Q -) e over the action space A, the first term (- [Q(s,, at)] in (6) does not affect the policy update, only the second term (((((-(ss) works, and thus the policy update yields to converge to the uniform policy for every state maximizing the total entropy. Hence, the exploration radius in the case of aq is almost the same as that of the uniform policy, as seen in Fig. 1(d). When ag > O, on the other hand, the Q-function starts be updated nontrivially by the Bellman operator T in (3) due the log ग ([4++11$44) term in ((S++1) in (3), with x given by the current policy. It is now seen in Fig. I(c) that Ex,=- [I] VI V.Q((s (st, [ama; is not zero anymore and the gradient norm becomes larger as ag increases from 0.1 to 1.0. Non-zero Eg,-D| [|| V. Q (8t, a)la=aill means that Q(s,, as function of action o for given 8, is not flat anymore and the first term in (6) affects the policy update so that
the policy is updated for the direction of high Q-value (with intention for high future entropy) well

---

22
Secsupe
TILLEO maa TEETITATT CHEDRy lanforni
115-221
125.311
2000
(a) Number of state visits
(b) Q-value difference
(c) Empirical entropy
(5.5)
(10,10)
(20,20)
(30,30)
4y 4y-(42-11 10.00
G.D
Q-value
log
10000
100000
200000
300000
400000
500000
Timestep
(d) Cross-section of Q- functioo and log ı along the action line at square center
Figure 2: Analysis on sample trajectories of SAC in the continuous maze task
as high policy entropy H(x)). As seen in Fig. I(d), however, the exploration radius reduces as aq increases from 0 to 1. The iteration process does not evolves for wider exploration intended.
5 Methodology
5.1 A Deeper Look at Pure Exploration
In order to propose our new approach overcoming the limitation of SAC implementation of the maximum entropy framework,
first take deeper look at how the Q-function term in (6) hinders exploration, as SAC (with 0g = aq = learns the maze task. For this, we consider four 2 x 2 squares centered at (5, 5), (10, 10). (20, 20), and (30, 30) in the 100 100 maze, where every episode starts from (0.5,0.5). Fig. 2(a) shows the number of accumulated visits to each square as time elapses. Figs. 2(b) and 2(c) show the estimated Q value and the average empirical entropy of each square, respectively, as time goes. For Fig. 2(b), every 1000 time steps. we sampled 1000 states uniformly from each square and an action from the current policy for each sampled state, and computed the Q-value average over the 1000 state-action samples for each square. Then, we computed the mean value of the four average values of the four squares. Fig. 2(b) shows the average Q value of each square relative to the four-square mean value as time goes. For Fig 2(c), every 1000 time steps, we sampled 1000 states uniformly from each square and computed the average empirical entropy E., [Ba,s<((ss))-=- log ( [s]]] of the current policy
time & averaged over the 1000 sampled states {s.} from each square. The upper row of Fig. 2(d) shows the cross-section of the estimate Q-function Q(x,, a) along the diagonal action line from (-1,- 11 to (1, at the center state s of each square, as time goes, where each curve is shifted in y-axis so that the mean value averaged over samples along the action line is matched to zero in y-axis. The lower row of Fig. 2(d) shows the value of log π (a|s) of the current policy
time step along the diagonal action line from (-1, - to (1, 1) at the center state s for each square as time goes, where the curve shifted in y-axis to match the mean value to zero in y-axis.
First, note from Fig. 2(a) that the farther state from the starting point (0.5, 0.5), the less the agent visits the state, and the visitation difference is large. At the early stage of learning starting with random Q-network weight initialization and random policy-network weight initialization, there little Q-value difference with respect to either state or action, as seen in Figs. 2(b) and 2(d), so the entropy term is dominant the policy update (6) and the policy entropy increases with the policy distribution approaching the uniform distribution,
in Fig. 2(c). As time goes, learning of
5

---

the Q-function with the Bellman backup (3) progresses. Basically, the Bellman backup (3) with no reward adds AQ;- = =>(Ba,1><n(e ( a++1)] + a' (+((((+++())))))))) Q(s,, a,) to Q (s,, az) for every (st, at) E S A. However, this is approximated in practical RL. In sample-based off-policy learning with function approximation, RL typically stores visited state-action pairs in the replay buffer D and the above Bellman backup approximated as updating the Q-function by minimizing the loss (()
(Q+++((++
( ((Q(s), ax) Qtarget (st, Qz) ]]] based on a mini-bate batch uniformly drawn from the buffer. Under this off off-policy learning with experience replay, when the initial Q-function is roughly flat and small, AQr soon becomes positíve (the policy update increases the entropy of the visited states and AQ, soon becomes positive for the visited states), and hence the Q-values of frequently- -issttd states are updated more and thus have higher Q-values as seen in 2ib. This is because these states are stored more into D and sampled more from D at mini-batch generation. Then, the initial Q-value difference biases the policy to visit the states with high Q-values more frequently than the states with low Q-values because the policy is updated to choose actions that maximizes the expectation of Q-value E,,~~((-ss) (-,.) (st,, [x)] in the policy update (6). This is evident in the Fig. 2(d), which shows the estimate of Q- Q-function and the value of log π along the diagonal line. At the early stage of learning (10k time step in the figure), Q-function is roughly flat and the policy is almost close to the uniform distribution for the action line. As the time steps go on, the Q-values of actions close to (-1,-1) 1) becomes higher than the Q-value of actions near (1, 1) due to the off-policy learning with experience replay, as explained above. Then, the policy is updated to choose actions with high Q-values more frequently to maximize the Q-value expectation,
the probability of choosing action (1, towards the left-lower corner becomes higher than that of action (1,1) for the opposite direction. As the policy distribution leans toward a certain action and becomes asymmetric away from uniformity, the policy entropy decreases further. As seen in Fig. 2(c), the speed of the policy entropy decrease varies depending on the Q-value difference along the action line in Fig. 2(d), and the policy entropy difference deepens the Q-value difference between states in Fig. 2(b) because the Q-value estimates the policy entropy sum of future states. This positive feedback continues until saturation, as seen in Fig. 2(b), and it results in the narrow exploration radius in Fig. 1(d) because the policy will be forced to visit states with high Q-values only. Note that this positive feedback reduces the policy entropy due to the Q-value difference, contrary to the intention behind maximum entropy.
5.2 Max-Min Entropy RL
In order to break the unwanted positive feedback loop occurring when implementing the maximum entropy framework (i.e., max-max entropy framework) in the previous subsection, we must reduce the policy entropy difference between states to reduce the Q-value difference between states in the feedback loop. For this, we aim to learn the Q-function so that the policy visits states with low entropy, and the policy update increases the policy entropy of these low-entropy states. Under this principle, we propose a new max-min entropy (MME) framework that aims to learn the Q-function to estimate the negative sum of policy entropy, while maintaining the policy entropy maximization term H(r (-Iss)) in the policy update to increase the policy entropy of the visited states. Thus, we define the reversed soft Q-function Qh az) for MME as
2h (s:, at) == To + E ETLINT
(' aQ ((H(((())
1+++1
whereas the original soft Q-function of SAC in (2) is given by
Et' + a, H(m(()))) )
("((s, a.) == ri + E B
li+++
Note that the original soft Q-function Q adds the policy entropy to the reward and drives the policy visit states with high entropy. On the other hand, our reversed soft Q-function subtracts the policy entropy from the reward and drives the policy to visit states with low entropy. In this sense, we call QR as the "reversed" soft Q-function because it desires the reverse behavior of soft -ufuctiion.
Then, Q is estimated by real-valued function Qn : S x A + R based on a Bellman operator T T=QR(s, a.) = rr + 7B,,, I-P(( [V 1)l,
(8)
where Vn(ss) = Eg (1).)) IQ (((( St, ar) + ag log (aa is the reversed soft state value function. At each iteration, the policy of MME is updated as
Nnew = arg max (e QQ (st, at) - or log ((aess)))
rEll

---

where QR is substituted by the estimate function QR at the iteration. Then, in pure exploration with no reward r; O, Vt. the policy of MME will visit the states with low entropy due the first term ( (ex) 17 (st. at)], and the policy entropy of those states will increase by the second term Be.=
log [a)]s;)] = (-[(-=s))) ). as we intended. Note that the behaviour of the proposed method follows the max-min principle [11], so we expect that our MME fairly increase the policy entropy of all states based on the faimess perspective of max-min min optimization, whereas SAC increases the policy entropy of states with high entropy only. The MME is expected to reduce the entropy difference and the Q-value difference between states to reduce the unwanted feedback loop and solve the saturation problem. Furthermore, SAC considers the same entropy coefficient or, for its policy update and the soft Q-function Q", but our MME distinguishes the policy entropy coefficient 0x in the policy update (9) and the value-entropy coefficient aq in the reversed soft Q-function Q% experimented in Section 4. Changing aq and at allows for us to control the amount of the reversed Q-function in the policy update, and it will determine the ratio between the exploration due to the policy entropy and the exploration due to the reversed soft Q-function.
in (7),
In actual implementation, the negative entropy in (7) is plus-offsetted to make the Q-update increase the Q-value. The detailed implementation and algorithm of MME are provided in Appendix A.
5.3 Disentangled Exploration and Exploitation for Rewarded Setup
In the previous subsection, we considered the problem from pure exploration perspective. However, the ultimate goal of RL is to maximize the sum of rewards rewarded environments, and the goal of exploration is to receive higher rewards without falling into local optima. With non-zero reward in (7)- (9), the policy will not only visit states with low entropy but also states with higher return. In this case, the reward and the entropy are intertwined in the Q-function and then it is difficult to expect the intended MME exploration behavior through the intertwined Q-function. Therefore, we disentangle exploration from exploitation for rewarded setup, considered in several previous works [7, 49), and propose disentangled MME DE-MM)) for rewarded setup. For this, we consider two policies: pure exploration policy gæ that samples actions for pure exploration described in Sec. 5.2, and target policy wr that actually interacts with the environment. We decompose the reversed soft Q-function Q% in (7) into terms Q% Q7, R. + Q% QR. where Qh. is the expected current and future reward sum considered in standard RL and QhE is the expected sum of future entropy:
(n.n(st, ax) + BT
Σ U=41
(T, ae) = =-a))=aa@appppa a.
(+H(((()))))
Then, we update the policy Fg for pure exploration as
Te arg max Ea (a)e)))))))
[o (st, az) Ox log (ael)aa))
(10)
x'Ell Note that increasing the expectation of Q makes the policy visit states with low entropy of nE. as we intended in the pure exploration case in Sec. 5.2. Finally, we update the target policy or by using Q R.. as
Tee arg max (elee [QT. @ (s), + The (st, 0z) ax log nं' (aclss))
(11)
Ell
For implementation, QR. and ala estimated by real-valued functions QR.R and QR,E based on their own Bellman operators (see Appendix A). Note that the policy update (9) in Sec. 5.2 can be expressed as maximizing ( (st, az) - ag log n' r'(ar(s)) over the target Fr. Thus, we can view that the policy update in (11) replaces policy, where Q7 TT. = Q or. + *
Q in the previous policy update (9) with T to disentangle exploration from exploitation. In this way, the policy update (11) will simultaneously increase the expectation of QR to maximize the reward sum, the expectation of Q to visit states with low entropy, and the policy entropy for diverse action. The detailed implementation and algorithm for DE-MME are provided in Appendix A.
6 Experiments
We provide numerical results to show the performance of the proposed MME and DE-MME in pure exploration and various control tasks. We provide source code for the proposed method at
7

---

3000
Mean Namber Veileee (ibrco
4000
2000
MMElao
sic B 1
MMEleo
1000
MHelaa u.S:
SAC
NMEle,
unitorm
NMEloe -
Time Steps (1e51
Exploration Row
(b) State histogram of every 50k steps after 300k steps Figure 3: Comparison of MME (Proposed), SAC, and the uniform policy in the 4-room maze
(a) Number of visited states
http:/ / //ggttutbbb..gggg com/ seungyulhar. han.yuullaa//mmee. /me/ that requires Python Tensorflow. For all plots. the solid line represents the mean over random seeds and the shaded region represents standard deviation from the mean.
6.1 Pure Exploration
To see how the proposed method be- haves in pure exploration, con- sidered the maze task described in Sec.4 again. We compared the ploration performance
Average Avergge fteracce
MME in Sec.5.2, SAC, and the uniform pol- icy. For MME, we considered several
3.0
MINEI 51
MINEIRG 201
ag E (0.1, 0.2, 0.5, 2, 10) with
ma manmmum encrogy (urdism))
Ox = 1. Fig. 3(a) in the next page shows the mean number of accumu- lated quantized visited states averaged over 30 random seeds corresponding to Fig. 1(b), and Fig. 3(b) shows the histogram of visited states, of which setup is the same as Fig.1 As seen in Fig. 3(a), the proposed MME visits much more states than SAC or the uniform policy. In addition, we observe that MME continues discovering new states throughout the learning. while SAC rarely visits new states as learning progresses. As seen in Fig. 3(b), MME explores far and rare states compared to SAC or uniform policy, and this leads to a large enhancement exploration performance, as intended in Sec. 5.2. Note that the larger oi in update (7) (9) with Q-function term to visit states with low entropy and the weaker is the effect the policy entropy term to explore widely in the action space, between the two terms and aq = 0.5 seems best the maze task when a₂ 1.0, as seen in Fig. 3(a). Thus, the result clearly shows why distinguish the policy entropy coefficient a₂ and the value entropy coefficient og for MME, whereas SAC uses the common entropy coefficient = an = aQ. We also plotted the Q-value difference and the empirical entropy of the four squares centered at (5,5), (10, (20,20) and (30,30) for MME, as done in Figs. 2(b) and 2(c). The result is shown in Fig. 4. It is seen that the Q-value difference and the entropy difference among the states are clearly reduced as compared to Figs. 2(b) and 2(c). means that MME breaks the unwanted positive feedback loop and improves the policy entropy of diverse states uniformly compared to SAC in terms of fairness under our max-min framework. This leads to better exploration, as seen in Fig 3.
Time Staps (345)
Time Staps (145)
(a) Q-value difference
(b) Empirical entropy
Figure 4: Performance of MME
O, the stronger is the effect of the
expected in Section 5.2. Hence, there is trade-off
6.2 Performance in Rewarded Environments
As mentioned Sec.5.3, the ultimate goal of RI. to maximize the reward sum in rewarded environments and exploration is one of the means to achieve this goal. Based on the enhanced

---

600
19.14
DOAVE
200
02
06 Shipe Dal))
2.3
0.0
04
055
0.3
0.
04
3.0
00
8.5
3,0
2.
25
Time Sopa
Time Stepe Dest
Stgpa
(180)
(a) SpprseHopper-vl
(b) SpanseHalrCheetah-vv
(c) Sparse Walkerrall
(d) SparseAnt-vl
Figure 5: Performance comparison on Sparse Mujoco tasks
20000
5000
5000
2506
2000
2200
300
1000
1000
30
25
30
35
25
3.
(a) Del. Hopper-v1
(b) Del.HalfCheetah-vl
(c) Del.Walker2d-y
(d) Del.Ant-vl
Figure 6: Performance comparison Delayed Mujoco tasks
exploration performance of MME, we expect MME/DE-MME to show good performance in rewarded environments too. In order verify this, considered three types of difficult control tasks for which current state-of-the-art RL algorithms do not show satisfactory performance: Two types of sparse-reward tasks (SparseMujoco tasks and DelayedMujoco tasks) and high dimensional Humanoid tasks. SparseMujoco [27, 35] is a sparse version of Mujoco [52] in OpenAl Gym [8], and the reward is 1 if the agent crosses the z-axis threshold T, otherwise O. DelayedMujoco [19, 57] is delayed version of Mujoco in which the reward is accumulated for D time steps and the agent receives the accumulated reward sum once every D time steps. During the accumulation time, the agent receives no reward. These sparse-reward environments have widely been considered as challenging environments for validating the performance of exploration in many previous works [9, 25, 27].
First, we compared the performance of MME algorithms to that of maximum entropy SAC in the spwrre-reward tasks. For MME, considered two versions: vanilla MME proposed in Sec.5.2, and disentangled MME (DE-MME) proposed in Sec.5.3. For MME/DE-MME, we fixed a, of MME and DE-MME be equal of SAC, and chose proper oq for each task. Detailed experimental setup is provided in Appendix B. Figs. 5 and 6 show the performance averaged over 10 random seeds on SparseMMuooco tasks and random seeds on DelayedMujoco tasks, respectively. It is seen that the proposed MME shows much higher performance than SAC in the considered environments with rewards. is also seen that MME itself performs well enough in most environments but DE-MME indeed yields performance gain over vanilla MME and the gain is large in SparseWalker Thus, disentanglement of exploration from exploitation is beneficial to MME for better reward performance in rewarded environments, as discussed in Sec.5.3. We provided the corresponding max average return tables in Appendix C and ablation study for further analysis in Appendix D. There, one of ablation study empirically shows that the performance enhancement by MME is caused by improved exploration of MME as we intended.
Finally, we compared the performance of MME/DE-MME to that of popular general RI. algorithms and recent exploration methods on the considered sparse-reward environments (SparseMujoco and DelayedMujoco tasks) and dense-reward figh--imensional Mujoco tasks (Humanoid, Humanoid- Standup). We considered several action-based exploration methods: SAC combined with divergence [27] (SAC-Div) and diversity actor-critic critic (DAC) [25], and state- based exploration methods with random network distillation (RND) [9] and MaxEnt (State) [26]. For general RL algorithms, we con- sidered several on-policy RL algorithms: proximal policy optimization (PPO) [48] and trust-region policy optimization (TRPO) [47], and entropy-based off-policy RL algorithms: soft Q-learning (SQL) [21] and SAC [22]. We provided detailed explanation and implementation for each algorithm in Appendix C. Table summarizes the max average return result. It seen that MME/DE-W MMEE have superior performance to other methods.

---

MINE
DE-MME
DAC
SAC-DW
RND
MaxEnt
Sps
4.2.00:4.36
893.30 30
3.0
217.402253.54
697
879.50=
Sps. Hallhhhhhh
903.50:134.97
915.90-150.7
924-70-24.44
Sps. Walker2d
818.00-208.60
886.60- 28,77
665. 15-350--5500
278. 278.50+-900 498-999.00 29.23
750.90+1720 179.99
705. 30+-27488
Sps.Ant
953,70:228.05 39
973,601 12,55
935,80 37 08
870.70=1214
920.60 107 50
940.70::43.86
Hopper
3421.32-88829
3435,28 39.55
3428.18
2090 64+-1
3254,
Del HallChoctah
7299.28=1162.119
1451.20
7594,70=12550 23
4080.67::3118.00
7429 138333
7907: 9307.9
Del. Walker2d
5148,58:1 193.78
S274.89 186.35
4067. 1067711::220 11:2257.81
0148.11:290..4
4098, 63:3666336666366
4420.61=344..2
40640044836551
4851.64 -116648880.0
4243, 2245.194795.49
3978, 344-1570.28
1361.
1156.61- 112.44
MKT
DE-MME
SAC
SQL
PEO
TRPO
Humanoid-
267734.03
250935.55 53
167394,30 36
389996 84
160211 90
153919.84
Standup
=74302.99
47291.99
4:3268.37
Humanood
9080.54-1768.52
$607.75
6760.81:226.788
5010.72:-248.55
6153.54.1246.5
5730.74::455.90
Table 1: Max average return of MME/DE-MME and other recent RL algorithms
7 Conclusion
In this paper, we have proposed a MME framework for RL to resolve the unwanted exploration behavior of maximum entropy RL in off-policy learning with function approximation. In pure exploration, to implement MME, we train the Q-function to visit states with low entropy contrary to the maximum entropy strategy. while maintaining the policy entropy maximization term in the policy update. Then, we extended MME to rewarded environments. In rewarded environments we disentangle exploration from exploitation for MME to explore diverse states as in pure exploration as well as to achieve high return. Numerical results show that the proposed MME explores farther and wider in the state space than maximum entropy realization, alleviates possible positive feedback of off-policy maximum entropy learning, and yields a significant enhancement in exploration and final performance over existing RL methods in various difficult tasks. As for potential impacts, RL can be applied to sensitive areas that require control, such drone control. However, it is only risk that RL itself has, and it not very relevant to the work that we are trying to address in this paper.
Acknowledgement
This work is supported by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by Defense Acquisition Program Administration (DAPA) and Agency for Defense Development (ADD) (UD19003 IRD),. Dr. Seungyul Han is currently with AI Graduate School of UNIST and his work is partly supported by Artificial Intelligence Graduate School support (UNIST), Institute of Information Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (N0.2020-0)))33).
10

---

References
[1] Joshua Achiam and Shankar Sastry. Surprise-basee intrinsic motivation for deep reinforcement learning. arXiv preprint 1rXiv:1703.01732, 2017.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[3] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06036 2020.
[4] Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn. Ryeeyi fair inference. arXiv preprint arXiv: 12005, 2019.
[5] Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning in natural and artificial systems. Springer, 2013.
[6] Babak Barazandeh and Meisam Razaviyayn. Solving non-convex non-differentiable min-max games using proximal gradient method. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3162-3166. IEEE, 2020.
[7] Lucas Beyer, Damien Vincent, Olivier Teboul, Sylvain Gelly, Matthieu Geist, and Olivier Pietquin. Mulex: Disentangling exploitation from exploration in deep rl. arXiv preprint 1oXiv:1907.00868, 2019.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0111 1606.01566, 2016.
[9] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1810v:1810.12880 2018.
[10] Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated rein-
forcement learning. In Advances in neural information processing systems, pages 1281-1288, 2005.
[11] Altannar Chinchuluun, Panos M Pardalos, Athanasios Migdalas, and Leonidas Pitsoulis. Pareto optimality, game theory and equilibria. Springer, 2008.
[12] Jongwook Choi, Yijie Guo, Marcin Moczulski. Junhyuk Oh, Neal Wu. Mohammad Norouzi, and Honglak Lee. Contingency-awee exploration reinforcement learning. arXiv preprint 181 2018.
[13] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
[14] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: / htt/tttttu/bb////o com www.oeenniiiaaeeeeeeooee /bwee/aalleeeeeee ines, 2017.
[15] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. arXiv preprint arXiv:2103.06257, 2021.
[16] Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09477 2018.
[17] Gauthier Gidel, Tony Jebara, and Simon Lacoste-Julien Frank-wolfe algorithms for saddle point problems. In Artificial Intelligence and Statistics, pages 362-371. PMLR, 2017.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[19] Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning, arXiv preprint
2018.

---

[20] Tuomas Haarnoja, Schoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint arXiv:1812.11103. 2018.
[21] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine Reinforcement learning with deep energy-based policies. arXiv preprini arXiv: wrXiv:1702.08166 2017.
[22] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
[23] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: /812.05905 2018.
[24] Yazdandoost Hamedani, A Jalilzadeh, NS Aybat, and UV Shanbhag. Iteration complexity of randomized primal-dual methods for convex-concave saddle point problems. arXiv preprint 1806. 04118, 2018.
[25] Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. arXiv preprint arXiv:2006.0/419 2020.
[26] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
[27] Zhang- Wei Hong, Tzu- Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 10489-10500, 2018
[28] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109-1111 117,, 2016.
[29] Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Svqn: Sequential variational soft q-learning networks. In International Conference on Learning Representations, 2019.
[30] Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pages 1-9, 2013.
[31] Ya-Feng Liu, Yu-Hong Dai, and Zhi-Quan Luo. Max-min fairness linear transceiver design for a multi-user mimo interference channel. IEEE Transactions on Signal Processing, 61(9):2413- 2423, 2013.
[32] Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems, pages 206-214, 2012.
[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXivv1706.060s: 2017.
[34] Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based exploration in feature space for reinforcement learning, arXiv preprint
2017.
[35] Bogdan Mazoure, Thang Doan, Audrey Durand, R Devon Hjelm, and Joelle Pineau. Leveraging exploration in off-policy algorithms via normalizing flows. arXiv preprint 1005::990500680 2019.
[36] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
[37] Katta Murty and Santosh Kabadi. Some np-complete problems in quadratic and nonlinear programming. Technical report, 1985.
12

---

[38] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pages 2775-2785, 2017.
[39] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. arXiv preprini arXiv:1707.0189), 2017.
[40] Gerhard Neumann et al. Variational inference for policy search in changing situations. In Proceedings of the 28th International Conference on Machine Learning, ICML 20 pages 817-824, 2011.
[41] Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving class of non-convex min-max games using iterative first order methods. arXiv preprint urXiv:/902.08220 2019.
[42] Brendan O' D'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning. arXiv preprint arXiv: 2016.
[43] Hassan Rafique, Mingrui Liu, Qihang Lin, and Tianbao Yang. Non-convex min-max op- timization: Provable algorithms and applications in machine learning. arXiv preprini arXiv:1810.022660, 2018.
[44] Alexander Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. arXiv preprint arXiv: 1869, 2013.
[45] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and re- inforcement learning by approximate inference. In Twenty-Third International Joint Conference on Artificial Intelligence, 2013.
[46] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft 2017.
q-learning arXiv preprint
[47] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1889-1899 1897, 2015.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[49] Riley Simmons-Edler, Ben Eisner, Daniel Yang. Anthony Bisulco, Eric Mitchell, Sebastian Seung, and Daniel Lee. Qxplore: Q-learning exploration by maximizing temporal difference error. 2019.
[50] Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint 1904v:1904.0788 07855, 2019.
[51] Emanuel Todorov. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control, pages 4296-4292. IEEE, 2008
[52] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-50000000 5033. IEEE, 2012.
[53] Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pages 1049-1056. ACM, 2009.
[54] Hoi-To Wai, Mingyi Hong, Zhuoran Yang, Zhaoran Wang, and Kexin Tang. Variance reduced policy evaluation with smooth function approximation. Advances in Neural Information Processing Systems, 32:5784-5795 2019.
[55] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):799-292, 1992.
13

---

[56] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on Al, Ethics, and Society, pages 335-340, 2018.
[57] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pages 4644-46544 2018.
[58] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. 2008
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The abstract and introduction reflect all contributions in the paper well.
(b) Did you describe the limitations of your work? [Yes] See Appendix E.I.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] The paper conforms to the ethics review.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] The paper does not deal with theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] The paper does not deal with theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Appendix B.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] We plot the standard deviation with respect to the multiple random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 6 and Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 6.
(b) Did you mention the license of the assets? [Yes] See Section B.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
The paper does not contain new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section B.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section B.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "n8qMmG4gIBE", "reviewer": "Reviewer_rp8A", "summary": "This paper proposes a conjecture that the agent should visit the state with low entropy and then do the exploration of this low entropy state with the maximum-entropy principle. To prove this conjecture, the author does a simple experiment on the 4-room maze and illustrates the empirical entropy of the action at different states. There is an interesting observation that the rarely visited states have low entropy maybe due to the function approximation error or off-policy learning explained by the author. Given this observation, the author proposes an algorithm which penalizes the reward with negative entropy （eq7) . Based on this modified Q function, the agent does the policy improvement as the original soft-actor-critic algorithm. The empirical result shows that the proposed algorithm outperforms the baselines such as SAC and many others.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n1. The authors provide a simple experiment to demonstrate their observations which leads to the easy understanding of the core idea.\n2. The empirical performance of the proposed algorithm is strong.\n\ncons:\n\n1. The contribution of this paper is somewhat incremental, since the negative entropy can be thought a special case of the intrinsic reward.\n2. In some experiment it seems true that the rarely visited state have low entropy. The author explained that in some heuristic way in page 5 and 6. Does this conclusion hold for most RL problems? I hope the author can provide more rigorous analysis, since it is the core of this paper.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "YCSpX8CG9hj", "reviewer": "Reviewer_NuTp", "summary": "This paper investigates the undesired behavior of soft actor-critic (SAC) algorithm that implementing maximum entropy reinforcement learning (RL). \n\nOne of the main contributions of this paper is the empirical investigation of the undesired behavior of SAC. By running SAC with function approximation on no-reward environment, the authors illustrate that the policy obtained by SAC failed to maximize the entropy, while maximizing the entropy is the optimal situation in no-reward environment and the theoretical study of SAC in finite MDP setups guarantees its optimal convergence. The authors pointed out a possible defect in SAC with function approximation: Q value of already visited states tend to be greater than the other states because the policy update increases the entropy of the visited states. \n\nThe second contribution is the proposal of a solution to the above mentioned defect. The authors propose Max-Min Entropy RL (MME). To improve the exploration, the soft Q-function has been revised. Instead of adding the entropy term to the Q-function, the revised soft Q-function subtracts the entropy, if I borrow the authors statement, to enhance the chance to visit rare states with low entropy. This is, however, counter intuitive to me as I will describe in the main review part.\n\nThe proposed approach has been tested on sparse mujoco tasks and delayed mujoco tasks, where the environments have sparse-reward shapes, as well as on some standard mujoco environments with dense rewards. The algorithm has been compared to different SOTA approaches. Promising performance has been observed.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The first contribution mentioned above is a very nice finding of a possible defect of SAC. It may open a new direction of future works. One small comment: This behavior of over-estimation of Q-value for already visited states may be related to the over-estimation bias of Q-value reported in TD3. The authors might want to comment on the similarity between them.\n\nMy concern is in the second contribution, i.e., the revised soft Q-function (7). This definition follows the authors idea of \"requiring the policy to visit rare states with lower entropy more frequently.\" However, the rare states do not necessarily have lower entropy. Based on the authors empirical investigation on the defect of SAC, I understand that SAC tends to over-estimate the entropy of already visited states. Therefore, unvisited states have relatively smaller estimated value of the entropy. However, this is the matter of estimation, and the entropy itself is not necessarily lower for unvisited states. If we run the approach sufficiently long, Q-function is expected to tend to the right value, which is the right entropy value in case of no-reward situation. Therefore, I couldn't understand why this is the right modification to do. I appreciate if the authors give more justification of (7). \n\nAlthough I am uncertain about the validness of the proposed modification, the empirical results presented in this paper are promising. One minor comment: What happens if the proposed approach is applied to swimmer environment? It is known to be deceptive and more exploration tends to lead to better performance, as is often shown in evolutionary RL and related approaches.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation is the lack of theoretical guarantee, which the original SAC has. This is stated in Appendix. However, since the statement of the limitation is a very important part of a research paper that may promote a future investigation, I would like to suggest to put it in the main text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "PQdJjqACC1f", "reviewer": "Reviewer_M4pi", "summary": "The paper addresses limitation in max-entropy RL frameworks. These frameworks usually add a regularization term to the reward function to increase policy entropy to encourage exploration, but the effect can be limited in states where the Q-function dominates the expression.\nThe authors propose a new objective that better encourages RL agents to visit rare states with low entropy, and show empirically that their method is able to outperform the baselines used for comparison.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I very much enjoyed reading this paper. The writing was clear, there was a nice flow of arguments, there was some interesting insights on commonly used MaxEnt RL frameworks, and a concise change to address the limitations found.\n\nI think RL practicioners might find this paper useful, specially when dealing with sparse reward settings.\n\nI only have a few suggestions that I think might help the reader.\n\n- Line 98: why is the first term \"the sum of future entropy\"? It's the sum of Q, so wouldn it be more like the sum of \"entropy-regularized reward\"?\n\nline 120-121: From the plot I do see that SAC behaves very differently than Uniform, however I don't know if you could definitely say that SAC visits fewer states than uniform, as the variance is huge and sometimes it visits a lot more states than uniform.\n\n- Eq. 7 defines the new Q function for MME but the difference to Eq. 2 is very subtle, it took me a bit of scrolling back and forth to find the difference. It would be useful for the reader if as the new objective is derived either the previous one is re-stated on the same page, or the differences being explained as they are introduced.\n\n- why is it called \"reversed\" soft Q-function?\n\n- I'm a bit confused with the notation alpha_{Q}. Does this mean that the value of alpha_{Q} depends on the current Q value?\n\n- Just for clarification, Q_{R,R} and Q_{R,E}, are two separate functions maintained by two separate approximators (neural nets), right?\n\n\n- Typo line 47: (approximiate) -> (approximate)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not much has been said about limitations, the authors could expand a bit on this point.\nNo negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-qN3IhedQVC", "reviewer": "Reviewer_P2Px", "summary": "The authors describe a positive feedback loop, that arises in SAC-based implementation of the maximum entropy reinforcement learning and hinders efficient exploration. Q values of frequently visited states are updated more, because these states are stored more into the replay buffer. The policy update increases the entropy of the visited states, i.e. if a high entropy is rewarded in the policy the frequently visited states will be selected even more. In the paper, Max-Min Entropy Rl is proposed that is supposed to break the unwanted feedback loop. It considerably improved the empirical performance in common Maze tasks and Mujoco tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n* as far as I know the described limitation was not studied before. The proposed method is new, too. \n\nQuality:\n* The paper does not contain a theoretical analysis, but the experiments are analyzed in quite a detail. I like that the authors plotted the magnitude of the Q-values and the entropy. \n\nClarity:\n* In the first sentence of the abstract you write \"the limitation of ...\", but the abstract never says what this limitation is. If you can summarizes the positive feedback loop in one sentence, I would add that. Otherwise I would change \"the limitation\" to \"a limitation\".\n* the described problem is rather a problem with the SAC implementation of the maximum entropy framework than with the maximum entropy framework itself. The problem is that the sampling based approximation in SAC results in a positive feedback loop, that keeps SAC from converging to the optimal solution of the maximum entropy objective $J_{MaxEnt}$. While this is described later on, reading the abstract and the introduction I misunderstood it as a limitation of the maximum entropy framework itself. \n* The paper is well written in general. Just one suggestion for improvement of the reading flow: The caption of Section 4 mentions that Section 4 considers pure exploration only, but the text itself does not mention it. This made me stumble across the sentence starting in line 97, that only holds for pure exploration. \n\nSignificance:\n* The method is widely applicable. However, I am wondering if the situation would be different in MDPs with stochastic transitions $P$. The described example and the experiments are all with deterministic transitions, right? But even if not, I would still consider it significant. \n\nFurther comments:\n* Is it easy to see what $ J_{MaxMin}$ would look like?\n* In Section 5.1, in line 164, it says: \"For Fig. 2(b), every 1000 time steps, we sampled ...\", but the x-axis label in Figure 2(b) is \"Time Steps (1e5)\". \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal Impact (last two sentences of the paper): I perceive the formulations \"only a risk\" and \"not very relevant\" slightly unlucky in the context of ethics, but contentwise I agree with the authors that the paper does not pose a risk beyond the general dual use problematic in RL. ", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting and novel view on MaxEnt RL: It observes that  SAC may suffer from a feedback loop since the Q values of already visited states tend to be greater than that of other states because the policy update increase the entropy. This in turn encourages the policy to visit these states even more. This defect is confirmed in a no-reward environment and a method to address this is presented with encouraging experimental results. While there could have been more rigorous investigation into the claim "rarely visited state have low entropy", the overall contribution of the paper is appreciated by most of the reviewers and the interesting take on MaxEnt RL warrants acceptance.
Decision: Accept (Poster)