MQES: Max-Q Entropy Search for Efficient Exploration in Continuous Reinforcement Learning

1 INTRODUCTION . In Reinforcement Learning ( RL ) , one of the fundamental problems is exploration-exploitation dilemma , i.e. , the agents explore the states with imperfect knowledge to improve future reward or instead maximize the intermediate reward at the perfectly understood states . The main obstacle of designing efficient exploration strategies is how the agents decide whether the unexplored states leading high cumulative reward or not . Popular exploration strategies , like -greedy ( Sutton & Barto , 1998 ) and sampling from stochastic policy ( Haarnoja et al. , 2018 ) , lead to undirected exploration through additional random permutations . Recently , uncertainty of systems are introduced to guide the exploration ( Kirschner & Krause , 2018 ; Mavrin et al. , 2019 ; Clements et al. , 2019 ; Ciosek et al. , 2019 ) . Basically , as Moerland et al . ( 2017 ) points out , two source of uncertainty exists in the RL system , i.e. , epistemic and aleatoric uncertainty . Epistemic uncertainty is also called parametric uncertainty , which is the ambiguity of models arisen from the imperfect knowledge to the environment , and can be reduced with more data . Aleatoric uncertainty is an intrinsic variation associated with the environment , which is caused by the randomness of environment , and is not affected by the model . In the RL system , if the states are seldom visited , the epistemic uncertainty at these states are relatively large . Hence , the exploration methods should encourage exploration when epistemic uncertainty is large . Moreover , heteroscedastic aleatoric uncertainty means that different states may have difference randomness , which renders different aleatoric uncertainty . If we do not distinguish these two uncertainties and formulate them separately , we may explore the states visited frequently but with high randomness , i.e. , low epistemic uncertainty and high aleatoric uncertainty , which is undesirable . By introducing uncertainty , the exploration objectives like Thompson Sampling ( TS ) ( Thompson , 1933 ; Osband et al. , 2016 ) and Upper Confidence Bound ( UCB ) ( Auer , 2002 ; Mavrin et al. , 2019 ; Chen et al. , 2017 ) are utilized to guide the exploration in RL . However , since the aleatoric uncertainty in the RL systems are heteroscedastic , i.e. , the aleatoric uncertainty depends on states and actions and can be different , the above methods are not efficient . Hence , Nikolov et al . ( 2019 ) proposes novel exploration objective called Information-Directed Sampling ( IDS ) accounting for epistemic uncertainty and heteroscedastic aleatoric uncertainty . However , these methods ( Nikolov et al. , 2019 ; Mavrin et al. , 2019 ; Chen et al. , 2017 ; Osband et al. , 2016 ) can only be applied in the environment with discrete action space . In this paper , we propose a generally information-theoretic principle called Max-Q Entropy Search ( MQES ) for off-policy continuous RL algorithms . Further , as an application example of MQES , we combine distributional RL with soft actor-critic method , where the epistemic and aleatoric uncertainty are formulated accordingly . Then , we incorporate MQES to Distributional Soft Actor-Critic ( DSAC ) ( Ma et al. , 2020 ) method , and show how MQES utilizes both uncertainty to explore . Finally , our results on Mujoco environments show that our method can substantially outperform alternative state-of-the-art algorithms . 2 RELATED WORK . Efficient exploration can improve the efficiency and performance of RL algorithms . With the increasing emphasis on exploration efficiency , various exploration methods have been developed . One kind of methods use intrinsic motivation to stimulate agent to explore from different perspectives , such as count-based novelty ( Martin et al. , 2017 ; Ostrovski et al. , 2017 ; Bellemare et al. , 2016 ; Tang et al. , 2017 ; Fox et al. , 2018 ) , prediction error ( Pathak et al. , 2017 ) , reachability ( Savinov et al. , 2019 ) and information gain on environment dynamics ( Houthooft et al. , 2016 ) . Some recently proposed methods in DRL , originating from tracking uncertainty , do efficient exploration under the principle of OFU ( optimism in the face of uncertainty ) , such as Thompson Sampling ( Thompson , 1933 ; Osband et al. , 2016 ) , IDS ( Nikolov et al. , 2019 ; Clements et al. , 2019 ) and other customized methods ( Moerland et al. , 2017 ; Pathak et al. , 2019 ) . Methods for tracking uncertainty . Bootstrapped DQN ( Osband et al. , 2016 ) combines Thompson sampling with value-based algorithms in RL . It is similar to PSRL ( Strens , 2000 ; Osband et al. , 2013 ) , and leverages the uncertainty produced by the value estimations for deep exploration . Bootstrapped DQN has become the common baseline for lots of recent works , and also the well-used approach for capturing epistemic uncertainty ( Kirschner & Krause , 2018 ; Ciosek et al. , 2019 ) . However , this takes only epistemic uncertainty into account . Distributional RL approximates the return distribution directly , such as Categorical DQN ( C51 ) ( Bellemare et al. , 2017 ) , QR-DQN ( Dabney et al. , 2018b ) and IQN ( Dabney et al. , 2018a ) . Return distribution can be used to approximate aleatoric uncertainty , but those methods do not take advantage of the return distribution for exploration . Exploration with two types of uncertainty . Traditional OFU methods either focus only on the epistemic uncertainty , or consider the two kinds of uncertainty as a whole , which can easily lead the naive solution to favor actions with higher variances . To address that , Mavrin et al . ( 2019 ) studies how to take advantage of distributions learned by distributional RL methods for efficient exploration under both kinds of uncertainty , proposing Decaying Left Truncated Variance ( DLTV ) . Nikolov et al . ( 2019 ) and Clements et al . ( 2019 ) propose to use Information Direct Sampling ( Kirschner & Krause , 2018 ) for efficient exploration in RL ( IDS for RL ) , which estimate both kinds of uncertainty and use IDS to make decision for acting with environment . We refer to the practice of uncertainty estimation in ( Clements et al. , 2019 ) as shown in Sec . 4.2.1 . IDS integrates both uncertainty and has made progress on the issue of exploration , but this is limited on discrete action space . We do focus on how best to exploit both uncertainty for efficient exploration in a continuous action space in our paper . Optimistic Actor Critic . More closely related to our work is the paper of OAC ( Ciosek et al. , 2019 ) , which uses epistemic uncertainty to build the upper bound of Q estimation QUB . OAC is based on Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) , additionally proposing exploration bonus to facilitate exploration . Despite the advantages that OAC has achieved over SAC , it does not consider the potential impact of the aleatoric uncertainty , which may cause misleading for exploration . 3 PRELIMINARIES . 3.1 DISTRIBUTIONAL RL . Distributional RL methods study distributions rather than point estimates , which introduce aleatoric uncertainty from distributional perspective . There are different approaches to represent distribution in RL . In our paper , we focus on quantile regression used in QR-DQN ( Dabney et al. , 2018b ) , where the randomness of state-action value is represented by the quantile random variable Z with value z . Z maps the state-action pair to a uniform probability distribution supported on zi , where zi indicates the value of the corresponding quantile estimates . If τi is defined as the quantile fraction , the cumulative probabilities of such quantile distribution is denoted by FZ ( zi ) = Pr ( Z < zi ) = τi = 1/N for i ∈ 1 , ... , N . Similar to the Bellman operator in the traditional Q-Learning ( Watkins & Dayan , 1992 ) , the distributional Bellman operator T πD under policy π is given as : T πDZ ( st , at ) D = R ( st , at ) + γZ ( st+1 , at+1 ) , at+1 ∼ π ( ·|st+1 ) . ( 1 ) Notice that this operates on random variables , D= denotes that distributions on both sides have equal probability laws . Based on the distributional Bellman operator , Dabney et al . ( 2018b ) proposes QR-DQN to train quantile estimations via the quantile regression loss , which is denoted as : LQR ( θ ) = 1 N N∑ i=1 N∑ j=1 [ ρτ̂i ( δi , j ) ] ( 2 ) where δi , j = R ( st , at ) + γzj ( st+1 , at+1 ; θ ) − zi ( st , at ; θ ) , ρτ ( u ) = u ∗ ( τ − 1u < 0 ) , and τ̂i means the quantile midpoints , which is defined as τ̂i = τi+1+τi 2 . 3.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHODS . Following Ma et al . ( 2020 ) , Distributional RL has been successfully integrated with soft ActorCritic ( SAC ) algorithm . Here , considering the maximum entropy RL , the distributional soft Bellman operator T πDS is defined as follows : T πDSZ ( st , at ) D = R ( st , at ) + γ [ Z ( st+1 , at+1 ) − αlogπ ( at+1|st+1 ) ] ( 3 ) where at+1 ∼ π ( ·|st+1 ) , st+1 ∼ P ( ·|st , at ) . The quantile regression loss in DSAC is different from original QR-DQN only on the δi , j by considering the maximum entropy RL framework . DSAC extends the clipped double Q-Learning proposed on TD3 ( Fujimoto et al. , 2018 ) to overcome the overestimation problem . Two Quantile Regression Deep Q Networks have the same structure that are parameterized by θk , k = 1 , 2 . Following the clipped double Q-Learning , the TD-error of DSAC is defined as : yti = min k=1,2 zi ( st+1 , at+1 ; θ̄k ) ( 4 ) δki , j = R ( st , at ) + γ [ y t i − α log π ( at+1|st+1 ; φ̄ ) ] − zj ( st , at ; θk ) ( 5 ) where θ̄ and φ̄ represents their target networks respectively . DSAC has the modified version of critic , while the update of actors is unaffected . It is worth noticing that the state-action value is the minimum value of the expectation on certain distributions , as Q ( st , at ; θ ) = min k=1,2 Q ( st , at ; θk ) = 1 N min k=1,2 N−1∑ i=0 zi ( st , at ; θk ) ( 6 ) Thus , in DSAC , the original problem aims to maximize the following objective function : Jπ ( φ ) = Est∼D , ∼N [ logπ ( f ( st , t ; φ ) |st ) −Q ( st , f ( st , t ; φ ) ; θ ) ] , ( 7 ) where D is the replay buffer , f ( st , t ; φ ) means sampling action with re-parameterized policy . 4 ALGORITHM . This paper proposes a new exploration principle for continuous RL algorithms , i.e. , MQES , which leverages epistemic and aleatoric uncertainties to explore optimistically and avoid over-exploration . To make MQES practically tractable , distributional and ensembleQ function approximations are introduced to formulate the epistemic and aleatoric uncertainty accordingly . Nevertheless , a constraint is introduced in the MQES to stabilize the training , and the approximated exploration policy is derived in the closed form . All these mechanisms are detailed in the following sections accordingly .

Summary: This paper proposes MQES, a Max-Q entropy search for policy optimization in continuous RL. The authors propose to combine advantages of the information-theoretic principle and distributional RL, in which epistemic and aleatoric uncertainty are estimated using similar entropy-search acquisition functions in the Bayesian Optimization (BO). As said, this is a new method to introduce a more efficient exploration strategy. As a result, policy improvement is formulated as a constraint optimization problem where a next exploration policy can be solved in a closed-form. The proposed method is evaluated on Mujoco tasks and compared against other off-policy approaches, SAC, and DSAC. The results show MQES outperforms other methods in domains where exploration is needed.