Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration

Policy entropy regularization is commonly used for better exploration in deep reinforcement learning ( RL ) . However , policy entropy regularization is sampleinefficient in off-policy learning since it does not take the distribution of previous samples stored in the replay buffer into account . In order to take advantage of the previous sample distribution from the replay buffer for sample-efficient exploration , we propose sample-aware entropy regularization which maximizes the entropy of weighted sum of the policy action distribution and the sample action distribution from the replay buffer . We formulate the problem of sample-aware entropy regularized policy iteration , prove its convergence , and provide a practical algorithm named diversity actor-critic ( DAC ) which is a generalization of soft actor-critic ( SAC ) . Numerical results show that DAC significantly outperforms SAC baselines and other state-of-the-art RL algorithms . 1 INTRODUCTION . Reinforcement learning ( RL ) aims to maximize the expectation of the discounted reward sum under Markov decision process ( MDP ) environments ( Sutton & Barto , 1998 ) . When the given task is complex , i.e . the environment has high action-dimensions or sparse rewards , it is important to well explore state-action pairs for high performance ( Agre & Rosenschein , 1996 ) . For better exploration , recent RL considers various methods : maximizing the policy entropy to take actions more uniformly ( Ziebart et al. , 2008 ; Fox et al. , 2015 ; Haarnoja et al. , 2017 ) , maximizing diversity gain that yields intrinsic rewards to explore rare states by counting the number of visiting states ( Strehl & Littman , 2008 ; Lopes et al. , 2012 ) , maximizing information gain ( Houthooft et al. , 2016 ; Hong et al. , 2018 ) , maximizing model prediction error ( Achiam & Sastry , 2017 ; Pathak et al. , 2017 ) , and so on . In particular , based on policy iteration for soft Q-learning , ( Haarnoja et al. , 2018a ) considered an offpolicy actor-critic framework for maximum entropy RL and proposed the soft actor-critic ( SAC ) algorithm , which has competitive performance for challenging continuous control tasks . In this paper , we reconsider the problem of policy entropy regularization in off-policy learning and propose a generalized approach to policy entropy regularization . In off-policy learning , we store and reuse old samples to update the current policy ( Mnih et al. , 2015 ) , and it is preferable that the old sample distribution in the replay buffer is uniformly distributed for better performance . However , the simple policy entropy regularization tries to maximize the entropy of the current policy irrespective of the distribution of previous samples . Since the uniform distribution has maximum entropy , the current policy will choose previously less-sampled actions and more-sampled actions with the same probability and hence the simple policy entropy regularization is sample-unaware and sample-inefficient . In order to overcome this drawback , we propose sample-aware entropy regularization , which tries to maximize the weighted sum of the current policy action distribution and the sample action distribution from the replay buffer . We will show that the proposed sampleaware entropy regularization reduces to maximizing the sum of the policy entropy and the α-skewed Jensen-Shannon divergence ( Nielsen , 2019 ) between the policy distribution and the buffer sample action distribution , and hence it generalizes SAC . We will also show that properly exploiting the sample action distribution in addition to the policy entropy over learning phases will yield far better performance . 2 RELATED WORKS . Entropy regularization : Entropy regularization maximizes the sum of the expected return and the policy action entropy . It encourages the agent to visit the action space uniformly for each given state , and the regularized policy is robust to modeling error ( Ziebart , 2010 ) . Entropy regularization is considered in various domains for better optimization : inverse reinforcement learning ( Ziebart et al. , 2008 ) , stochastic optimal control problems ( Todorov , 2008 ; Toussaint , 2009 ; Rawlik et al. , 2013 ) , and off-policy reinforcement learning ( Fox et al. , 2015 ; Haarnoja et al. , 2017 ) . ( Lee et al. , 2019 ) shows that Tsallis entropy regularization that generalizes usual Shannon-entropy regularization is helpful . ( Nachum et al. , 2017a ) shows that there exists a connection between value-based and policybased RL under entropy regularization . ( O ’ Donoghue et al. , 2016 ) proposed an algorithm combining them , and it is proven that they are equivalent ( Schulman et al. , 2017a ) . The entropy of state mixture distribution is better for pure exploration than a simple random policy ( Hazan et al. , 2019 ) . Diversity gain : Diversity gain is used to provide a guidance for exploration to the agent . To achieve diversity gain , many intrinsically-motivated approaches and intrinsic reward design methods have been considered , e.g. , intrinsic reward based on curiosity ( Chentanez et al. , 2005 ; Baldassarre & Mirolli , 2013 ) , model prediction error ( Achiam & Sastry , 2017 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) , divergence/information gain ( Houthooft et al. , 2016 ; Hong et al. , 2018 ) , counting ( Strehl & Littman , 2008 ; Lopes et al. , 2012 ; Tang et al. , 2017 ; Martin et al. , 2017 ) , and unification of them ( Bellemare et al. , 2016 ) . For self-imitation learning , ( Gangwani et al. , 2018 ) considered the Steinvariational gradient decent with the Jensen-Shannon kennel . Off-policy learning : Off-policy learning can reuse any samples generated from behaviour policies for the policy update ( Sutton & Barto , 1998 ; Degris et al. , 2012 ) , so it is sample-efficient as compared to on-policy learning . In order to reuse old samples , a replay buffer that stores trajectories generated by previous policies is used for Q-learning ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ) . To enhance both stability and sample-efficiency , several methods are considered , e.g. , combining on-policy and off-policy ( Wang et al. , 2016 ; Gu et al. , 2016 ; 2017 ) , and generalization from on-policy to off-policy ( Nachum et al. , 2017b ; Han & Sung , 2019 ) . In order to guarantee the convergence of Q-learning , there is a key assumption : Each state-action pair must be visited infinitely often ( Watkins & Dayan , 1992 ) . If the policy does not visit diverse state-action pairs many times , it converges to local optima . Therefore , exploration for visiting different state-action pairs is important for RL , and the original policy entropy regularization encourages exploration ( Ahmed et al. , 2019 ) . However , we found that the simple policy entropy regularization can be sample-inefficient in off-policy RL , so we aim to propose a new entropy regularization method that significantly enhances the sample-efficiency for exploration by considering the previous sample distribution in the buffer . 3 BACKGROUND . In this section , we briefly introduce the basic setup and the soft actor-critic ( SAC ) algorithm . 3.1 SETUP . We assume a basic RL setup composed of an environment and an agent . The environment follows an infinite horizon Markov decision process ( S , A , P , γ , r ) , where S is the state space , A is the action space , P is the transition probability , γ is the discount factor , and r : S × A → R is the reward function . In this paper , we consider a continuous state-action space . The agent has a policy distribution π : S × A → [ 0 , ∞ ) which selects an action at for a given state st at each time step t , and the agent interacts with the environment and receives reward rt : = r ( st , at ) from the environment . Standard RL aims to maximize the discounted return Es0∼p0 , τ0∼π [ ∑∞ t=0 γ trt ] , where τt = ( st , at , st+1 , at+1 · · · ) is an episode trajectory . 3.2 SOFT ACTOR-CRITIC . Soft actor-critic ( SAC ) ( Haarnoja et al. , 2018a ) includes a policy entropy regularization term in the objective function for better exploration by visiting the action space uniformly for each given state . The entropy-augmented policy objective function of SAC is given by JSAC ( π ) = Eτ0∼π [ ∞∑ t=0 γt ( rt + βH ( π ( ·|st ) ) ) ] , ( 1 ) where H is the entropy function and β ∈ ( 0 , ∞ ) is the entropy coefficient . SAC is a practical offpolicy actor-critic based on soft policy iteration ( SPI ) that alternates soft policy evaluation to estimate the true soft Q-function and soft policy improvement to find the optimal policy that maximizes ( 1 ) . In addition , SPI theoretically guarantees convergence to the optimal policy that maximizes ( 1 ) . 4 THE DIVERSITY ACTOR-CRITIC ALGORITHM . 4.1 MOTIVATION OF THE SAMPLE-AWARE ENTROPY . As explained in Section 2 , the policy should visit diverse samples to learn the policy without converging to the local optima . In off-policy learning , we can reuse previous samples stored in the replay buffer to learn the policy , so it is efficient to draw diverse samples while avoiding frequently selected samples before . The policy entropy maximization enhances exploration to yield better performance , but it is sample-inefficient for off-policy RL because it does not take advantage of the previous sample action distribution obtainable from the replay buffer : If we assume bounded action space , the simple policy entropy maximization will choose all actions with the equal probability without considering the previous action samples because maxπH ( π ) = minπ DKL ( π||U ) is achieved when π = U , where U is a uniform distribution and DKL is the Kullback-Leibler ( KL ) divergence . In order to overcome the limitation of the simple policy entropy maximization , we consider maximizing a sample-aware entropy defined as the entropy of a mixture distribution of the policy distribution π and the current sample action distribution q in the replay buffer . Here , q is defined as q ( ·|s ) : = ∑ a∈DN ( s , a ) δa ( · ) ∑ a′∈DN ( s , a ′ ) , ( 2 ) where D is the replay buffer that stores previous samples ( st , at , rt , st+1 ) at each time t , δa ( · ) is the Dirac measure at a ∈ A , and N ( s , a ) is the number of state-action pair ( s , a ) in D. Then , we define a target distribution qπ , αtarget as the mixture distribution of π and q , which is expressed as qπ , αtarget : = απ + ( 1 − α ) q , where α ∈ [ 0 , 1 ] is the weighting factor . Note that we draw samples from policy π and store them in the replay buffer , so the target distribution can be viewed as the updated sample action distribution in the future replay buffer . Then , maximizing the sample-aware entropy H ( qπ , αtarget ) can encourage sample-efficient exploration because π will choose actions rare in the buffer with high probability and actions stored many times in the buffer with low probability in order to make the target distribution uniform . We provide a simple example below : Let us consider a simple 1-step MDP in which s0 is the unique initial state , there exist Na actions ( A = { A0 , · · · , ANa−1 } ) , s1 is the terminal state , and r is a deterministic reward function . Then , there exist Na state-action pairs in total and let us assume that we already have Na − 1 state-action samples in the replay buffer as R = { ( s0 , A0 , r ( s0 , A0 ) ) , · · · , ( s0 , ANa−2 , r ( s0 , ANa−2 ) ) } . In order to estimate the Q-function for all state-action pairs , the policy should sample the last action ANa−1 ( After then , we can reuse all samples infinitely to estimate Q ) . Here , we will compare two exploration methods . 1 ) First , if we consider the simple entropy maximization , the policy that maximizes its entropy will choose all actions with equal probability 1/Na ( uniformly ) . Then , Na samples should be taken on average by the policy to visit the action ANa−1 . 2 ) Consider the sample-aware entropy maximization . Here , the sample action distribution q in the buffer becomes q ( a0|s0 ) = 1/ ( Na − 1 ) for a0 ∈ { A0 , · · · , ANa−2 } and q ( ANa−1|s0 ) = 0 , the target distribution becomes qπ , αtarget = απ + ( 1 − α ) q , and we set α = 1/Na . Then , the policy that maximizes the sample-aware entropy becomes π ( ANa−1|s0 ) = 1 to make q π , α target uniform because maxπH ( qπ , αtarget ) = minπ DKL ( q π , α target||U ) . In this case , we only needs one sample to visit the action ANa−1 . In this way , the simple entropy maximization is sample-inefficient for off-policy RL , and the proposed sample-aware entropy maximization can enhance the sample-efficiency for exploration by using the previous sample distribution and choosing a proper α . With this motivation , we propose the sample-aware entropy regularization for off-policy RL and the corresponding αadaptation method .

Summary: This paper considers the exploration efficiency issues in off-policy deep reinforcement learning (DRL). The authors identify a sample efficiency limitation in the classical entropy regularization, which does not take into account the existing samples in the replay buffer. To avoid repeated sampling of previously seen scenarios/actions, the authors propose to replace the current policy in the entropy term with a mixture of the empirical policy estimation from the replay buffer and the current policy, and term this approach as sample-aware entropy regularization. The authors then propose a theoretical algorithm called sample-aware entropy regularized policy iteration, which is a generalization of the soft policy iteration (SPI) algorithm, and show that it converges assuming that the empirical policy estimation is fixed. A practical algorithm based on the sample-aware entropy regularized policy iteration, called Diversity Actor-Critic (DAC), is then proposed. This algorithm is a generalization of the well-known soft actor-critic (SAC) algorithm. Finally, numerical experiments show that DAC outperforms SAC and other SOTA RL algorithms, and some ablation studies are also provided to demonstrate the effect of hyper-parameter choices in DAC.