OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning

1 INTRODUCTION . The exploration and exploitation trade-off is critical in reinforcement learning ( Thompson , 1933 ; Sutton & Barto , 2018 ) . Many exploration strategies have been proposed in the literature ( Osband et al. , 2016 ; Lillicrap et al. , 2016 ; Martin et al. , 2017 ; Pathak et al. , 2017 ; Ciosek et al. , 2019 ) . Among these strategies , those following the Optimism in the Face of Uncertainty ( OFU ) principle ( Auer et al. , 2002 ) provide efficient guidance for exploration ( Chen et al. , 2017 ; Ciosek et al. , 2019 ) . Generally , OFU-based methods regard the uncertainty as the result of insufficient exploration for the state-action pair , and refer to such uncertainty as epistemic uncertainty . Besides , there is another uncertainty called aleatoric uncertainty , which captures environmental stochasticity : • Epistemic uncertainty ( a.k.a . parametric uncertainty ) represents the ambiguity of the model arisen from insufficient knowledge , and is high at those state-action pairs seldom visited ( Dearden et al. , 1998 ; Osband et al. , 2016 ; Moerland et al. , 2017 ) . • Aleatoric uncertainty ( a.k.a . intrinsic uncertainty ) is the variation arisen from environment randomness , caused by the stochasticity of policy , reward and/or transition probability , and is characterized by return distribution . ( Bellemare et al. , 2017 ; Moerland et al. , 2017 ) . In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs , those OFU-based methods may be inefficient without properly tackling aleatoric uncertainty ( Nikolov et al. , 2019 ) . As shown in Figure 1 , if the epistemic uncertainty estimation is disturbed due to the volatility caused by aleatoric uncertainty , the exploration strategy ( i.e. , policy A ) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncertainty . Visiting such areas results in unstable and risky transitions . In the real world , the aleatoric uncertainty can be caused easily , for example , unpredictable wind can shift the trajectory of an robot ’ s action . If such aleatoric uncertainty is not modelled , the RL agent may be trapped because the state transitions in such area can be wrongly considered novel and worth exploring due to the high uncertainty . This issue , to explore overly the state-action pairs visited frequently but with high aleatoric uncertainty , is referred to as the over-exploration issue . Intuitively , such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty ( i.e. , policy B in Figure 1 ) . The similar concern has been raised in the discrete RL , where Nikolov et al . ( 2019 ) proposes to use the Information-Directed Sampling ( IDS ) to avoid the over-exploration issue in the environments with heteroscedastic noise . However , IDS needs to calculate information-regret ratio for each action , thus applying IDS for continuous RL with explosive or even infinite action space could be non-trivial and ineffective . Meanwhile , many advanced continuous RL approaches suffer from the over-exploration issue . Soft Actor-Critic ( SAC ) is a well performed continuous RL algorithm , but does not account for efficient exploration beyond maximizing policy entropy ( Haarnoja et al. , 2018 ) . OAC improves the exploration follow- ing OFU principle , but it ignores to avoid exploration towards higher aleatoric uncertainty ( Ciosek et al. , 2019 ) . Therefore , an effective exploration method to address the over-exploration issue for continuous RL is of urgent need . To address the over-exploration issue , we propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary . The new principle need to properly trade-off between the epistemic and aleatoric uncertainties , so as to explore seldom visited state-action pairs while avoiding trapped in areas with high aleatoric uncertainty . In this paper , we propose a general information-theoretic exploration approach OVD-Explorer , which enriches the OFU exploration principle with a complement : “ avoid areas with high aleatoric uncertainty ” . Furthermore , to guide the exploration following the new principle , OVD-Explorer maximizes the mutual information between policy and corresponding upper bounds . Notable , the upper bounds represent the best return that policy can reach , formulated using Optimistic Value Distributions ( OVD ) , and the value distribution characterises the aleatoric uncertainty . From the theoretical derivation , we show that maximizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate over-exploration issue by avoiding the areas with lower information gain , i.e. , the areas that have higher aleatoric uncertainty but low epistemic uncertainty . To make OVD-Explorer tractable for continuous action space , we derive its closed form , and propose a scheme to incorporate OVD-Explorer with any policy-based RL algorithm . Practically , we demonstrate the exploration benefits based on SAC . Evaluations on many challenging continuous RL tasks , including a toy task GridChaos , tasks in Mujoco as well as their stochastic variants are conducted , and the results verify our analysis . To the best of our knowledge , OVD-Explorer firstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL . 2 PRELIMINARIES . 2.1 DISTRIBUTIONAL RL . To model the randomness in the observed long-term return , which characterises aleatoric uncertainty , distributional RL methods ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ; a ) are used to estimate Q-value distribution rather than expected Q-value ( Mnih et al. , 2015 ) . In our paper , we focus on quantile regression used in QR-DQN ( Dabney et al. , 2018b ) , where the distribution of Q-value is represented by the quantile random variable Z . Z maps the state-action pair to a uniform probability distribution supported on the values at all corresponding quantile fractions . Based on value distribution estimator parameterized as θ , we denote the value at quantile fraction τi as Ẑi ( s , a , θ ) given state-action pair ( s , a ) . Similar to the Bellman operator in the traditional Q-Learning ( Watkins & Dayan , 1992 ) , the distributional Bellman operator ( Bellemare et al. , 2017 ) T πD under policy π is given as : T πDZ ( st , at ) D = R ( st , at ) + γZ ( st+1 , at+1 ) , at+1 ∼ π ( ·|st+1 ) . ( 1 ) Notice that T πD operates on random variables , D = denotes that distributions on both sides have equal probability laws . Based on operator T πD , Dabney et al . ( 2018b ) propose QR-DQN to train quantile estimations via the quantile regression loss ( Koenker & Hallock , 2001 ) , which is denoted as : LQR ( θ ) = 1 N N∑ i=1 N∑ j=1 [ ρτ̂i ( δi , j ) ] , ( 2 ) where TD error δi , j = R ( st , at ) + γẐi ( st+1 , at+1 ; θ̄ ) − Ẑj ( st , at ; θ ) , the quantile Huber loss ρτ ( u ) = u ∗ ( τ − 1u < 0 ) , and τ̂i means the quantile midpoints , which is defined as τ̂i = τi+1+τi2 . 2.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHOD . Distributional Soft Actor-Critic ( DSAC ) ( Ma et al. , 2020 ) seamlessly integrates distributional RL with Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) . Basically , based on the Equation 1 , the distributional soft Bellman operator T πDS is defined considering the maximum entropy RL as follows : T πDSZ ( st , at ) D = R ( st , at ) + γ [ Z ( st+1 , at+1 ) − αlogπ ( at+1|st+1 ) ] , ( 3 ) where at+1 ∼ π ( ·|st+1 ) , st+1 ∼ P ( ·|st , at ) . Then , the quantile regression loss differs from Equation 2 on δi , j , by extending clipped double Q-Learning ( Fujimoto et al. , 2018 ) based on the maximum entropy RL framework , to overcome overestimation of Q value estimation : δki , j = R ( st , at ) + γ [ min k=1,2 Ẑi ( st+1 , at+1 ; θ̄k ) − α log π ( at+1|st+1 ; φ̄ ) ] − Ẑj ( st , at ; θk ) , ( 4 ) where θ̄ and φ̄ represents their target networks respectively . The actor is optimized by minimizing the following actor loss , the same as SAC , Jπ ( φ ) = E st∼D , t∼N [ log π ( f ( st , t ; φ ) |st ) −Q ( st , f ( st , t ; φ ) ; θ ) ] , ( 5 ) where D is the replay buffer , f ( s , ; φ ) means sampling action with re-parameterized policy and is a noise vector sampled from any fixed distribution , like standard spherical Gaussian , and Q value is the minimum value of the expectation on certain distributions , as Q ( st , at ; θ ) = min k=1,2 Q ( st , at ; θk ) = 1 N min k=1,2 N−1∑ i=0 Ẑi ( st , at ; θk ) . ( 6 ) 3 METHODOLOGY . We introduce a general exploration method , named OVD-Explorer , to achieve efficient and robust exploration for continuous RL . Overall , OVD-Explorer estimates the upper bounds of the policy ’ s return , and uses the bounds as criteria to find a behavior policy for online exploration from the information-theoretic perspective . Innovatively , to avoid over-exploration , OVD-Explorer particularly takes the aleatoric uncertainty into account , leveraging the optimistic value distribution ( OVD ) to formulate the upper bounds of policy ’ s return . Then , OVD-Explorer derives the exploration policy via maximizing the mutual information between policy and corresponding upper bounds for an optimistic exploration . Meanwhile , the derived behavior policy avoids being optimistic to aleatoric uncertainty , thus avoiding over-exploration issue . In the following , we first give the intuition together with the theoretical derivation of OVD-Explorer . Then we practically describe how we formulate OVD based on uncertainty estimation . Lastly , analysis are given to further illustrate why OVD-Explorer can explore effectively and robustly . 3.1 OPTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION . Most OFU-based algorithms follow the exploration principle : “ select the action that leads to areas with high uncertainty ” . However , this principle is incomplete as the high uncertainty may also be caused by the aleatoric uncertainty , which misleads the exploration direction . Hence , we complement the principle by introducing a constrain : “ Not only select the action that leads to areas with high uncertainty , but also avoid the ones that only lead to the area with high aleatoric uncertainty ” . Notable , this new principle intuitively describes a good exploration ability , and we will illustrate how to quantitatively measure a policy ’ s exploration ability in the later section . By following this principle , OVD-Explorer derives the behavior policy for exploration . Intuitively , each policy π in the policy space Π has different exploration ability ( at state s ) , denoted by Fπ ( s ) . Given current state s , OVD-Explorer aims at finding the exploration policy πE with the best exploration ability by solving the following optimization problem : πE = arg max π∈Π Fπ ( s ) . ( 7 ) We propose to quantitatively measure the policy ’ s exploration ability from an information-theoretic perspective by measuring the mutual-information of multi-variables as follows : Fπ ( s ) = MI ( Z̄π ( s , a0 ) , · · · , Z̄π ( s , ak−1 ) ; π ( ·|s ) |s ) , ( 8 ) where π ( ·|s ) is the action random variable , and Z̄π ( s , ai ) the random variable describing the upper bounds of return that action ai can reach under policy π at state s. Here , ai ∈ A denotes any legal action , thus k could be infinite in continuous action space . The higher Fπ ( s ) is , the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty , which satisfies the principle we raised . ( see analysis in Section 3.3 . ) Now we state the Theorem 1 to measure the mutual information above in continuous action space . Theorem 1 . The mutual information in Equation 8 at state s can be approximated as : Fπ ( s ) ≈ 1 C E a∼π ( ·|s ) z̄ ( s , a ) ∼Z̄π ( s , a ) [ ΦZπ ( z̄ ( s , a ) ) log ΦZπ ( z̄ ( s , a ) ) C ] . ( 9 ) Φx ( · ) is the cumulative distribution function ( CDF ) of random variable x , z̄ ( s , a ) is the sampled upper bound of return from its distribution Z̄π ( s , a ) following policy π , Zπ describes the current return distribution of the policy π , and C is a constant ( See proof in Appendix B.1 ) . Theorem 1 reveals that Fπ ( s ) is only proportional to ΦZπ ( z̄ ( s , a ) ) , by maximizing which the policy ’ s exploration ability can be improved . Concretely , given any policy πθ ( parameterized by θ ) , the derivative ∇θΦZπθ ( z̄ ( s , a ) ) can be measured , thus gradient ascent can be iteratively performed to derive a better exploration policy . ( The complete closed form solution is given in later Section 4 ) Note that , to perform above optimization procedure , we need to formulate two critical components in ΦZπ ( z̄ ( s , a ) ) : 1 the return of the policy Zπ ( s , a ) and 2 upper bound of the return of the policy Z̄π ( s , a ) . Therefore , we first details the formulations in Section 3.2 , and then analyze why maximizing ΦZπ ( z̄ ( s , a ) ) can derive a better exploration in Section 3.3 .

Summary: This paper proposes a new information-theoretic exploration approach, called OVD-Explorer, for deep reinforcement learning with continuous action space. The OVD-Explorer is capable of exploring epistemic uncertainty while avoiding aleatoric uncertainty. The paper also contains extensive experiments to evaluate the performance of OVD-Explorer.