OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning

1 INTRODUCTION . The exploration and exploitation trade-off is critical in reinforcement learning ( Thompson , 1933 ; Sutton & Barto , 2018 ) . Many exploration strategies have been proposed in the literature ( Osband et al. , 2016 ; Lillicrap et al. , 2016 ; Martin et al. , 2017 ; Pathak et al. , 2017 ; Ciosek et al. , 2019 ) . Among these strategies , those following the Optimism in the Face of Uncertainty ( OFU ) principle ( Auer et al. , 2002 ) provide efficient guidance for exploration ( Chen et al. , 2017 ; Ciosek et al. , 2019 ) . Generally , OFU-based methods regard the uncertainty as the result of insufficient exploration for the state-action pair , and refer to such uncertainty as epistemic uncertainty . Besides , there is another uncertainty called aleatoric uncertainty , which captures environmental stochasticity : • Epistemic uncertainty ( a.k.a . parametric uncertainty ) represents the ambiguity of the model arisen from insufficient knowledge , and is high at those state-action pairs seldom visited ( Dearden et al. , 1998 ; Osband et al. , 2016 ; Moerland et al. , 2017 ) . • Aleatoric uncertainty ( a.k.a . intrinsic uncertainty ) is the variation arisen from environment randomness , caused by the stochasticity of policy , reward and/or transition probability , and is characterized by return distribution . ( Bellemare et al. , 2017 ; Moerland et al. , 2017 ) . In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs , those OFU-based methods may be inefficient without properly tackling aleatoric uncertainty ( Nikolov et al. , 2019 ) . As shown in Figure 1 , if the epistemic uncertainty estimation is disturbed due to the volatility caused by aleatoric uncertainty , the exploration strategy ( i.e. , policy A ) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncertainty . Visiting such areas results in unstable and risky transitions . In the real world , the aleatoric uncertainty can be caused easily , for example , unpredictable wind can shift the trajectory of an robot ’ s action . If such aleatoric uncertainty is not modelled , the RL agent may be trapped because the state transitions in such area can be wrongly considered novel and worth exploring due to the high uncertainty . This issue , to explore overly the state-action pairs visited frequently but with high aleatoric uncertainty , is referred to as the over-exploration issue . Intuitively , such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty ( i.e. , policy B in Figure 1 ) . The similar concern has been raised in the discrete RL , where Nikolov et al . ( 2019 ) proposes to use the Information-Directed Sampling ( IDS ) to avoid the over-exploration issue in the environments with heteroscedastic noise . However , IDS needs to calculate information-regret ratio for each action , thus applying IDS for continuous RL with explosive or even infinite action space could be non-trivial and ineffective . Meanwhile , many advanced continuous RL approaches suffer from the over-exploration issue . Soft Actor-Critic ( SAC ) is a well performed continuous RL algorithm , but does not account for efficient exploration beyond maximizing policy entropy ( Haarnoja et al. , 2018 ) . OAC improves the exploration follow- ing OFU principle , but it ignores to avoid exploration towards higher aleatoric uncertainty ( Ciosek et al. , 2019 ) . Therefore , an effective exploration method to address the over-exploration issue for continuous RL is of urgent need . To address the over-exploration issue , we propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary . The new principle need to properly trade-off between the epistemic and aleatoric uncertainties , so as to explore seldom visited state-action pairs while avoiding trapped in areas with high aleatoric uncertainty . In this paper , we propose a general information-theoretic exploration approach OVD-Explorer , which enriches the OFU exploration principle with a complement : “ avoid areas with high aleatoric uncertainty ” . Furthermore , to guide the exploration following the new principle , OVD-Explorer maximizes the mutual information between policy and corresponding upper bounds . Notable , the upper bounds represent the best return that policy can reach , formulated using Optimistic Value Distributions ( OVD ) , and the value distribution characterises the aleatoric uncertainty . From the theoretical derivation , we show that maximizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate over-exploration issue by avoiding the areas with lower information gain , i.e. , the areas that have higher aleatoric uncertainty but low epistemic uncertainty . To make OVD-Explorer tractable for continuous action space , we derive its closed form , and propose a scheme to incorporate OVD-Explorer with any policy-based RL algorithm . Practically , we demonstrate the exploration benefits based on SAC . Evaluations on many challenging continuous RL tasks , including a toy task GridChaos , tasks in Mujoco as well as their stochastic variants are conducted , and the results verify our analysis . To the best of our knowledge , OVD-Explorer firstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL . 2 PRELIMINARIES . 2.1 DISTRIBUTIONAL RL . To model the randomness in the observed long-term return , which characterises aleatoric uncertainty , distributional RL methods ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ; a ) are used to estimate Q-value distribution rather than expected Q-value ( Mnih et al. , 2015 ) . In our paper , we focus on quantile regression used in QR-DQN ( Dabney et al. , 2018b ) , where the distribution of Q-value is represented by the quantile random variable Z . Z maps the state-action pair to a uniform probability distribution supported on the values at all corresponding quantile fractions . Based on value distribution estimator parameterized as θ , we denote the value at quantile fraction τi as Ẑi ( s , a , θ ) given state-action pair ( s , a ) . Similar to the Bellman operator in the traditional Q-Learning ( Watkins & Dayan , 1992 ) , the distributional Bellman operator ( Bellemare et al. , 2017 ) T πD under policy π is given as : T πDZ ( st , at ) D = R ( st , at ) + γZ ( st+1 , at+1 ) , at+1 ∼ π ( ·|st+1 ) . ( 1 ) Notice that T πD operates on random variables , D = denotes that distributions on both sides have equal probability laws . Based on operator T πD , Dabney et al . ( 2018b ) propose QR-DQN to train quantile estimations via the quantile regression loss ( Koenker & Hallock , 2001 ) , which is denoted as : LQR ( θ ) = 1 N N∑ i=1 N∑ j=1 [ ρτ̂i ( δi , j ) ] , ( 2 ) where TD error δi , j = R ( st , at ) + γẐi ( st+1 , at+1 ; θ̄ ) − Ẑj ( st , at ; θ ) , the quantile Huber loss ρτ ( u ) = u ∗ ( τ − 1u < 0 ) , and τ̂i means the quantile midpoints , which is defined as τ̂i = τi+1+τi2 . 2.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHOD . Distributional Soft Actor-Critic ( DSAC ) ( Ma et al. , 2020 ) seamlessly integrates distributional RL with Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) . Basically , based on the Equation 1 , the distributional soft Bellman operator T πDS is defined considering the maximum entropy RL as follows : T πDSZ ( st , at ) D = R ( st , at ) + γ [ Z ( st+1 , at+1 ) − αlogπ ( at+1|st+1 ) ] , ( 3 ) where at+1 ∼ π ( ·|st+1 ) , st+1 ∼ P ( ·|st , at ) . Then , the quantile regression loss differs from Equation 2 on δi , j , by extending clipped double Q-Learning ( Fujimoto et al. , 2018 ) based on the maximum entropy RL framework , to overcome overestimation of Q value estimation : δki , j = R ( st , at ) + γ [ min k=1,2 Ẑi ( st+1 , at+1 ; θ̄k ) − α log π ( at+1|st+1 ; φ̄ ) ] − Ẑj ( st , at ; θk ) , ( 4 ) where θ̄ and φ̄ represents their target networks respectively . The actor is optimized by minimizing the following actor loss , the same as SAC , Jπ ( φ ) = E st∼D , t∼N [ log π ( f ( st , t ; φ ) |st ) −Q ( st , f ( st , t ; φ ) ; θ ) ] , ( 5 ) where D is the replay buffer , f ( s , ; φ ) means sampling action with re-parameterized policy and is a noise vector sampled from any fixed distribution , like standard spherical Gaussian , and Q value is the minimum value of the expectation on certain distributions , as Q ( st , at ; θ ) = min k=1,2 Q ( st , at ; θk ) = 1 N min k=1,2 N−1∑ i=0 Ẑi ( st , at ; θk ) . ( 6 ) 3 METHODOLOGY . We introduce a general exploration method , named OVD-Explorer , to achieve efficient and robust exploration for continuous RL . Overall , OVD-Explorer estimates the upper bounds of the policy ’ s return , and uses the bounds as criteria to find a behavior policy for online exploration from the information-theoretic perspective . Innovatively , to avoid over-exploration , OVD-Explorer particularly takes the aleatoric uncertainty into account , leveraging the optimistic value distribution ( OVD ) to formulate the upper bounds of policy ’ s return . Then , OVD-Explorer derives the exploration policy via maximizing the mutual information between policy and corresponding upper bounds for an optimistic exploration . Meanwhile , the derived behavior policy avoids being optimistic to aleatoric uncertainty , thus avoiding over-exploration issue . In the following , we first give the intuition together with the theoretical derivation of OVD-Explorer . Then we practically describe how we formulate OVD based on uncertainty estimation . Lastly , analysis are given to further illustrate why OVD-Explorer can explore effectively and robustly . 3.1 OPTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION . Most OFU-based algorithms follow the exploration principle : “ select the action that leads to areas with high uncertainty ” . However , this principle is incomplete as the high uncertainty may also be caused by the aleatoric uncertainty , which misleads the exploration direction . Hence , we complement the principle by introducing a constrain : “ Not only select the action that leads to areas with high uncertainty , but also avoid the ones that only lead to the area with high aleatoric uncertainty ” . Notable , this new principle intuitively describes a good exploration ability , and we will illustrate how to quantitatively measure a policy ’ s exploration ability in the later section . By following this principle , OVD-Explorer derives the behavior policy for exploration . Intuitively , each policy π in the policy space Π has different exploration ability ( at state s ) , denoted by Fπ ( s ) . Given current state s , OVD-Explorer aims at finding the exploration policy πE with the best exploration ability by solving the following optimization problem : πE = arg max π∈Π Fπ ( s ) . ( 7 ) We propose to quantitatively measure the policy ’ s exploration ability from an information-theoretic perspective by measuring the mutual-information of multi-variables as follows : Fπ ( s ) = MI ( Z̄π ( s , a0 ) , · · · , Z̄π ( s , ak−1 ) ; π ( ·|s ) |s ) , ( 8 ) where π ( ·|s ) is the action random variable , and Z̄π ( s , ai ) the random variable describing the upper bounds of return that action ai can reach under policy π at state s. Here , ai ∈ A denotes any legal action , thus k could be infinite in continuous action space . The higher Fπ ( s ) is , the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty , which satisfies the principle we raised . ( see analysis in Section 3.3 . ) Now we state the Theorem 1 to measure the mutual information above in continuous action space . Theorem 1 . The mutual information in Equation 8 at state s can be approximated as : Fπ ( s ) ≈ 1 C E a∼π ( ·|s ) z̄ ( s , a ) ∼Z̄π ( s , a ) [ ΦZπ ( z̄ ( s , a ) ) log ΦZπ ( z̄ ( s , a ) ) C ] . ( 9 ) Φx ( · ) is the cumulative distribution function ( CDF ) of random variable x , z̄ ( s , a ) is the sampled upper bound of return from its distribution Z̄π ( s , a ) following policy π , Zπ describes the current return distribution of the policy π , and C is a constant ( See proof in Appendix B.1 ) . Theorem 1 reveals that Fπ ( s ) is only proportional to ΦZπ ( z̄ ( s , a ) ) , by maximizing which the policy ’ s exploration ability can be improved . Concretely , given any policy πθ ( parameterized by θ ) , the derivative ∇θΦZπθ ( z̄ ( s , a ) ) can be measured , thus gradient ascent can be iteratively performed to derive a better exploration policy . ( The complete closed form solution is given in later Section 4 ) Note that , to perform above optimization procedure , we need to formulate two critical components in ΦZπ ( z̄ ( s , a ) ) : 1 the return of the policy Zπ ( s , a ) and 2 upper bound of the return of the policy Z̄π ( s , a ) . Therefore , we first details the formulations in Section 3.2 , and then analyze why maximizing ΦZπ ( z̄ ( s , a ) ) can derive a better exploration in Section 3.3 .

Summary: The paper proposes the so-called Optimistic Value Distribution Explorer (OVD-Explorer) to deal with over-exploration issue in reinforcement learning (RL). It is claimed that the proposed algorithm is tractable for continuous action space and can avoid exploring  the areas with high aleatoric uncertainty. The paper tests the performances of OVD-Explorer and some of its competitors on some common environments.