A Max-Min Entropy Framework for Reinforcement Learning

In this paper , we propose a max-min entropy framework for reinforcement learning ( RL ) to overcome the limitation of the soft actor-critic ( SAC ) algorithm implementing the maximum entropy RL in model-free sample-based learning . Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future , the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration . For general Markov decision processes ( MDPs ) , an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation . Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms . 1 Introduction . The maximum entropy framework has been considered in various RL domains [ 22 , 23 , 30 , 45 , 51 , 53 , 58 ] . Maximum entropy RL adds the expected policy entropy to the return objective of standard RL in order to maximize both the return and the entropy of policy distribution . Maximum entropy RL encourages the policy to choose multiple actions probabilistically and yields a significant improvement in exploration and robustness and good final performance in various control tasks [ 15 , 20 , 21 , 25 , 26 , 29 , 50 ] . In particular , soft actor-critic ( SAC ) implements maximum entropy RL in an efficient iterative manner based on soft policy iteration and guarantees convergence to the optimal policy for finite MDPs , yielding significant performance improvement over various on-policy and off-policy recent RL algorithms in many continuous control tasks . However , we observe that such an iterative implementation of the maximum entropy strategy of optimizing for policies that aim to reach states with high entropy in the future does not necessarily result in the desired exploration behavior but may yield positive feedback hindering exploration in model-free sample-based learning with function approximation . In order to overcome such limitations associated with implementation of the maximum entropy RL , we propose a max-min entropy framework for RL , which aims to learn policies reaching states with low entropy and maximizing the entropy of these low-entropy states , whereas the conventional maximum entropy RL optimizes for policies that aim to visit states with high entropy and maximize the entropy of those high-entropy states for high entropy of the entire trajectory . We implemented the proposed max-min entropy framework into a practical iterative actor-critic algorithm based on policy iteration with disentangled exploration and exploitation . It is demonstrated that the proposed algorithm significantly enhances exploration capability due to the fairness across states induced by the max-min framework and yields drastic performance improvement over existing RL algorithms including maximum-entropy SAC on difficult control tasks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 2 Related Works . Maximum Entropy RL : The maximum entropy framework has been considered in various RL domains : inverse reinforcement learning [ 58 ] , stochastic optimal control [ 45 , 51 , 53 ] , guided policy search [ 30 ] , and off-policy learning [ 22 , 23 ] . There is a connection between value-based and policy-based RL under the policy entropy regularization [ 38 ] , [ 42 ] combines them , and finally [ 46 ] proves that they are equivalent . Maximum entropy RL is also related to probabilistic inference [ 40 , 45 ] . Recently , maximizing the entropy of state distribution instead of the policy distribution [ 26 ] and maximizing the entropy considering the previous sample action distribution [ 25 ] have been investigated for better exploration . Max-Min Optimization : Max-min optimization aims to maximize the minimum of the objective function [ 11 ] . Under the convex-concave assumption , there exist many algorithms to find the solution to a max-min problem by using optimistic mirror descent [ 44 ] , Frank-Wolfe algorithm [ 17 ] , and Primal-Dual method [ 24 ] . However , non-convex max-min problems are more challenging [ 37 ] and there are several recent studies to find ( approximate ) solutions to non-convex max-min optimization problems [ 6 , 41 , 43 ] . This framework has been used in various optimization/control domains : fair resource allocation [ 31 ] , inference [ 4 , 56 ] , generative adversarial network ( GAN ) [ 2 , 18 ] , robust training [ 33 ] , and reinforcement learning [ 54 ] . Exploration in RL : Exploration is one of the most important issues in model-free RL , as there is the key assumption that all state-action pairs must be visited infinitely often to guarantee the convergence ofQ-function [ 55 ] . In order to explore diverse state-action pairs in the joint state-action space , various methods have been considered in prior works : intrinsically-motivated reward based on curiosity [ 5 , 10 ] , model prediction error [ 1 , 9 ] , information gain [ 25 , 27 , 28 ] , and counting states [ 32 , 34 ] . These exploration techniques improve exploration and performance in challenging sparse-reward environments [ 3 , 9 , 12 ] . 3 Background . 3.1 Basic RL Setup . We consider an infinite-horizon MDP ( S , A , P , γ , r ) , where S is the state space , A is the action space , P is the transition probability , γ is the discount factor , and r is the bounded reward function . We assume that each action dimension is bounded . The RL agent has a policy π : S ×A → R+ ∈ Π , which chooses an action at for given state st according to at ∼ π ( ·|st ) at each time step t , where Π is the policy space . For action at , the environment yields the reward rt : = r ( st , at ) and the next state st+1 ∼ P ( st+1|st , at ) . Standard RL learns policy π to maximize the discounted return Es0∼p0 , τ0∼π [ ∑∞ t=0 γ trt ] , where τt = ( st , at , st+1 , at+1 , · · · ) is an episode trajectory . 3.2 Maximum Entropy RL and Soft Actor-Critic . Maximum entropy RL maximizes both the expected return and the expected policy entropy simultaneously to achieve an improvement in exploration and robustness . The entropy-augmented objective function of maximum entropy RL is given by JMaxEnt ( π ) = Es0∼p0 , τ0∼π [ ∞∑ t=0 γt ( rt + αH ( π ( ·|st ) ) ) ] , ( 1 ) whereH ( π ( ·|s ) ) = Ea∼π ( ·|s ) [ − log π ( a|s ) ] is the entropy function and α > 0 is the entropy coefficient . A key point here is that the policy entropy is included in the reward not used as an external regularizer at each time step . Thus , this maximum entropy RL framework optimizes for policies that aim to reach states on which policies have high entropy in the future [ 21 ] . Soft actor-critic ( SAC ) is an efficient off-policy actor-critic algorithm to solve the maximum entropy RL problem [ 22 ] . SAC maximizes ( 1 ) based on soft policy iteration , which consists of soft policy evaluation and soft policy improvement . For this , the soft Q-value of given ( st , at ) is defined as Qπ ( st , at ) : = rt + Eτt+1∼π [ ∞∑ l=t+1 γl−t ( rl + αH ( π ( ·|sl ) ) ) ] , ( 2 ) which does not include the policy entropy of the current time step but includes the sum of all future policy entropy and the sum of all current and future rewards . For given π , soft policy evaluation guarantees the convergence of soft Q-function estimation , which estimates Qπ by iteratively applying a modified Bellman operator T π to a real-valued estimate function Q : S ×A → R , given by T πQ ( st , at ) = rt + γEst+1∼P ( ·|st , at ) [ V ( st+1 ) ] , where ( 3 ) V ( st ) = Eat∼π ( ·|st ) [ Q ( st , at ) − α log π ( at|st ) ] and V ( st ) is the soft state value function . Then , at each iteration , SAC updates the policy as πnew = arg min π∈Π DKL ( π ( ·|st ) || exp ( Qπold ( st , at ) /α ) Zπold ( st ) ) ( 4 ) = arg max π∈Π Eat∼π ( ·|st ) [ Q πold ( st , at ) − α log π ( at|st ) ] ( 5 ) where Zπold ( st ) is the log partition function which is a function of st only . Soft policy improvement guarantees Qπnew ( st , at ) ≥ Qπold ( st , at ) for all ( st , at ) ∈ S × A . Finally , soft policy evaluation and soft policy improvement are repeated . Then , any initial policy π ∈ Π converges to the optimal policy π∗ , i.e. , Qπ ∗ ( st , at ) ≥ Qπ ′ ( st , at ) for all π′ ∈ Π and all ( st , at ) ∈ S ×A , and π∗ maximizes JMaxEnt [ 22 ] . Proof of soft policy iteration assumes finite MDPs . SAC approximates the soft policy iteration by sample-based learning with function approximation in continuous-space cases . 4 Motivation : Limitation of Maximum Entropy SAC in Pure Exploration . In this section , we will consider only the maximum entropy SAC in a pure exploration setup without the reward function ( the reward function r = 0 in MDPs ) . As seen in Sec . 3 , SAC efficiently solves the maximum entropy RL problem to maximize ( 1 ) in an iterative manner based on judiciously-defined state and action value functions and the step-wise optimization cost ( 5 ) . The well-defined value functions and the local cost function as such enable proof of soft policy improvement for finite MDPs in a similar way to the proof of the classical policy improvement theorem . Note that at each time step , SAC updates the policy to maximize the cost ( 5 ) , composed of two terms : Eat∼π ( ·|st ) [ Qπold ( st , at ) ] and αEat∼π ( ·|st ) [ − log π ( at|st ) ] = αH ( π ( at|st ) ) . As aforementioned , the soft Q-function contains the sum of current and future rewards and the sum of only future policy entropy . Since we consider only the entropy terms without rewards here , the first term Eat∼π ( ·|st ) [ Qπold ( st , at ) ] is the current estimate of the sum of future entropy when action at is taken from policy π at state st , whereas the second term αH ( π ( at|st ) ) is the entropy of the policy π itself . Hence , at each time step , SAC tries to update the policy π to yield the maximum sum of the estimated future entropy and the policy entropy itself . Here , the term Eat∼π ( ·|st ) [ Qπold ( st , at ) ] plays the role of guiding the policy towards the direction of high future entropy . Saturation : In sample-based update with function approximation , however , the SAC iteration does not yield the desired result , contrary to the intention behind maximum entropy . To see this , let us consider a pure exploration task in which there is no reward . The considered task is a 100 × 100 continuous 4-room maze proposed in [ 25 ] , modified from the continuous grid map available at https : //github.com/huyaoyu/GridMap . Fig . 1 ( a ) shows the maze environment , where state is the ( x , y ) -position of the agent in the map , action is ( dx , dy ) bounded by [ −1 , 1 ] × [ −1 , 1 ] , and the next state of the agent is ( x + dx , y + dy ) . Starting from the left-lower corner ( 0.5 , 0.5 ) , the agent explores the maze without any external reward . First , note that for this pure exploration task , the optimal policy maximizing JMaxEnt ( π ) is given by the uniform policy that selects all actions in A = [ −1 , 1 ] × [ −1 , 1 ] uniformly regardless of the value of st . This is because the uniform distribution has maximum entropy for a bounded space [ 13 ] . Then , we compare the exploration behaviour of SAC and the uniform policy in the maze task . Fig . 1 ( b ) shows the mean accumulated number of different visited states averaged over 30 random seeds as time goes , where the shaded region in the curve represents standard deviation ( 1σ ) from the mean and a different state is meant as a nonoverlapping quantized 1× 1 square . As seen in Fig . 1 ( b ) , SAC explores more states than the uniform policy at the early stage of learning . As learning progresses , however , SAC fails to visit new states after 300k time steps , whereas the uniform policy continues visiting new states . As a result , SAC eventually visits fewer states than the uniform policy on average . The result shows that SAC fails to converge to the optimal uniform policy and its performance become saturated . Narrow Exploration Radius : To examine the saturation behavior of SAC in the above pure exploration task , we investigate the policy update of SAC in ( 5 ) . Since the current Q-function estimate ( implemented by a neural network ) replaces Qπold in ( 5 ) in implementation with function approximation , the policy update is rewritten as arg max π∈Π { Eat∼π ( ·|st ) [ Q ( st , at ) ] + αH ( π ( ·|st ) ) } . ( 6 ) As mentioned already , the first term Eat∼π ( ·|st ) [ Q ( st , at ) ] is the current estimate of the sum of future entropy in this pure exploration case when action at is taken from policy π at state st , whereas the second term αH ( π ( at|st ) ) is the entropy of the policy π itself . The first term Eat∼π ( ·|st ) [ Q ( st , at ) ] intends to direct the policy towards the direction of high future entropy . Note that maximizing the second term already yields the uniform policy , but the Q-function term affects the policy update . In order to see how the Q-function term actually affects the policy update , we differentiate the entropy coefficient α in the policy update part ( 5 ) or ( 6 ) as the policy entropy coefficient απ and that in the soft value function part ( 2 ) and ( 3 ) as the value entropy coefficient αQ . We fix απ as απ = 1 and change αQ as 0 , 0.1 , and 1 ( note that the case of αQ = 1 is original SAC ) . With this change of αQ , we conducted the same pure exploration maze task . Fig . 1 ( c ) shows the average norm of the gradient of Q-function with respect to action , i.e. , Est∼D [ ||∇aQ ( st , a ) |a=at || ] over time with at ∼ π ( ·|st ) and st from a mini-batch drawn from the replay buffer D of SAC update , where the Q neural network weights were initialized randomly . Fig . 1 ( d ) shows the histogram of states that the policy visits over 50k time steps starting from 300k , 350k , 400k , and 450k time steps . When αQ = 0 with no reward , the Q-function update by the Bellman operator T π in ( 3 ) is trivial as Q ( s , a ) ← Es′∼P ( ·|s , a ) , a′∼π ( ·|s′ ) [ Q ( s′ , a′ ) ] , i.e. , replacement . When the initial Q ( s , a ) is ( nearly ) flat over S × A by initial random weight assignment for the Q-neural network , the flat Q is maintained by this trivial update . Indeed , it is seen in Fig . 1 ( c ) that Est∼D [ ||∇aQ ( st , a ) |a=at || ] with at ∼ π ( ·|st ) is nearly zero across all time for αQ = 0 . With a flat function Q ( st , · ) ≈ c over the action space A , the first term Eat∼π ( ·|st ) [ Q ( st , at ) ] in ( 6 ) does not affect the policy update , only the second termH ( π ( ·|st ) ) works , and thus the policy update yields π to converge to the uniform policy for every state maximizing the total entropy . Hence , the exploration radius in the case of αQ = 0 is almost the same as that of the uniform policy , as seen in Fig . 1 ( d ) . When αQ > 0 , on the other hand , the Q-function starts to be updated nontrivially by the Bellman operator T π in ( 3 ) due to the − log π ( at+1|st+1 ) term in V ( st+1 ) in ( 3 ) , with π given by the current policy . It is now seen in Fig . 1 ( c ) that Est∼D [ ||∇aQ ( st , a ) |a=at || ] is not zero anymore and the gradient norm becomes larger as αQ increases from 0.1 to 1.0 . Non-zero Est∼D [ ||∇aQ ( st , a ) |a=at || ] means that Q ( st , · ) as a function of action at for given st is not flat anymore and the first term in ( 6 ) affects the policy update so that the policy is updated for the direction of high Q-value ( with intention for high future entropy ) as well as high policy entropy H ( π ) . As seen in Fig . 1 ( d ) , however , the exploration radius reduces as αQ increases from 0 to 1 . The iteration process does not evolves for wider exploration as intended .

Summary: This paper investigates the undesired behavior of soft actor-critic (SAC) algorithm that implementing maximum entropy reinforcement learning (RL).   One of the main contributions of this paper is the empirical investigation of the undesired behavior of SAC. By running SAC with function approximation on no-reward environment, the authors illustrate that the policy obtained by SAC failed to maximize the entropy, while maximizing the entropy is the optimal situation in no-reward environment and the theoretical study of SAC in finite MDP setups guarantees its optimal convergence. The authors pointed out a possible defect in SAC with function approximation: Q value of already visited states tend to be greater than the other states because the policy update increases the entropy of the visited states.   The second contribution is the proposal of a solution to the above mentioned defect. The authors propose Max-Min Entropy RL (MME). To improve the exploration, the soft Q-function has been revised. Instead of adding the entropy term to the Q-function, the revised soft Q-function subtracts the entropy, if I borrow the authors statement, to enhance the chance to visit rare states with low entropy. This is, however, counter intuitive to me as I will describe in the main review part.  The proposed approach has been tested on sparse mujoco tasks and delayed mujoco tasks, where the environments have sparse-reward shapes, as well as on some standard mujoco environments with dense rewards. The algorithm has been compared to different SOTA approaches. Promising performance has been observed.