Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning

1 INTRODUCTION . Distributional reinforcement learning ( DRL ) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in several benchmark tasks . Its model generates the approximated distribution of returns , where the mean value implies the traditional Qvalue ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ; Choi et al. , 2019 ) . Learning procedure with stochasticity through return distribution is represented by parametric ( epistemic ) uncertainty , which is due to insufficient or inaccurate data , and intrinsic ( aleatoric ) uncertainty , which is inherently possessed randomness in the environment . ( Chow et al. , 2015 ; Dabney et al. , 2018a ) The learned stochasticity gives rise to the notion of risk-sensitivity , and some distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking policy . Another way to employ the uncertainty is to design an efficient exploration method which is essential to find an optimal behavior with a few number of trials . Optimism in the face of uncertainty ( OFU ) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set . Most OFU algorithms select an action with the highest upper-confidence bound ( UCB ) of uncertainty which can be considered as the optimistic decision at the moment ( Chen et al. , 2017 ; Ciosek et al. , 2019 ) . In deep RL , several OFU studies often model the uncertainty explicitly through the Bayesian posterior , which is estimated by using neural networks . However , learning the representation of high-dimensional state-action space and Bellman update simultaneously leads to unstable propagation ( Yang et al. , 2021 ) . On the other hand , DRL can provide more statistical information during control such as mode , median , or variance by addressing full characteristics of the return distribution . Despite the richness of information for return distribution , only a few DRL methods have tried to employ the benefits of distributional perspective for exploration ( Tang & Agrawal , 2018 ; Mavrin et al. , 2019 ; Clements et al. , 2019 ) . To estimate the uncertainty from distributional outputs , descriptive statistics that is composed of a mixture of intrinsic and parametric uncertainty can be utilized . Unfortunately , sepa- rating these two types of uncertainty during learning is not a trivial task . Mavrin et al . ( 2019 ) propose a distribution-based OFU exploration that schedules a decaying bonus rate to suppress the effect of intrinsic uncertainty , which unintentionally induces a risk-seeking policy . Although OFU based approaches try to reduce parametric uncertainty by revisiting the state with high uncertainty , there exists the side effect that the criteria unfortunately forces the agent to chase the intrinsic uncertainty ( risk ) simultaneously during updates . Relying on a specific criteria causes an one-sided tendency on risk and makes an agent consistently select certain actions during exploration as degrading performance . We call this phenomenon as stuckness . In this paper , we introduce perturbed quantile regression ( PQR ) which perturbs the criterion on uncertainty by randomizing the risk criterion in action selection . First , the distributional perturbation on return distribution is to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight . Unlike the typical worst-case approach in risk-sensitive settings or OFU based approaches , we instead randomly sample a risk measure from an ambiguity set , which represents that the risk setting is ambiguous when the characteristics of a given environment are unknown . We empirically demonstrate on the stochastic variant of N-Chain environment ( Osband et al. , 2016 ) that a randomized scheme is more effective than OFU to alleviate the sub-optimality problem of tendency to obtain risk-seeking policies . Second , any risk-measure with some time-varying perturbation constraint allows us to average over all possible risk-sensitive behaviors to achieve the optimal policy . We provide a sufficient condition for the convergence of return distribution in the weaker contraction property . Our method covers the full elements of the reinforcement learning with distributional perspective from the novel exploration method with distributional perturbation to the theoretically guaranteed convergence of return distribution , which has the same fixed-point with the standard Bellman optimality operator . The proposed algorithm is based on QR-DQN , which is a baseline of DRL architecture to learn a return distribution , and we show that PQR outperforms QR-DQN and other DRL baselines in several benchmark environments , LunarLander-v2 and Atari games . 2 BACKGROUNDS & RELATED WORKS . 2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING . We consider a Markov decision process ( MDP ) which is defined as a tuple ( S , A , P , R , γ ) where S is a finite state space , A is a finite action space , P : S×A×S → [ 0 , 1 ] is the transition probability , R is the random variable of rewards in [ −Rmax , Rmax ] , and γ ∈ [ 0 , 1 ) is the discount factor . We define a stochastic policy π ( ·|s ) which is a conditional distribution overA given state s. For a fixed policy π , we denote Zπ ( s , a ) as a random variable of return distribution of state-action pair ( s , a ) following the policy π . We attain Zπ ( s , a ) = ∑∞ t=0 γ tR ( St , At ) , where St+1 ∼ P ( ·|St , At ) , At ∼ π ( ·|St ) and S0 = s , A0 = a . Then , we define an action-value function as Qπ ( s , a ) = E [ Zπ ( s , a ) ] in [ −Vmax , Vmax ] where Vmax = Rmax/ ( 1−γ ) . For regularity , we further notice that the space of actionvalue distributions Z has the first moment bounded by Vmax : Z = { Z : S ×A →P ( R ) ∣∣ E [ |Z ( s , a ) | ] ≤ Vmax , ∀ ( s , a ) } . In distributional RL , the return distribution for the fixed π can be computed via dynamic programming with the distributional Bellman operator defined as ( Bellemare et al. , 2017 ) , τ πZ ( s , a ) D= R ( s , a ) + γZ ( S′ , A′ ) , S′ ∼ P ( ·|s , a ) , A′ ∼ π ( ·|S′ ) where D= denotes that both random variables share the same probability distribution . We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as , τZ ( s , a ) D= R ( s , a ) + γZ ( S′ , a∗ ) , S′ ∼ P ( ·|s , a ) , a∗ = argmax a′ EZ [ Z ( S′ , a′ ) ] . Bellemare et al . ( 2017 ) have shown that τ π is a contraction in a maximal form of the Wasserstein metric but τ is not a contraction in any metric . Combining with the expectation operator , Eτ is a contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value , while the convergence of a return distribution itself is not guaranteed . 2.2 EXPLORATION ON DISTRIBUTIONAL REINFORCEMENT LEARNING . In this section , we will briefly describe the main practical algorithm for DRL with a deep neural network , and explain the derived exploration method . To combine with deep RL , a parametric distribution Zθ is used to learn a return distribution by using τ . Dabney et al . ( 2018b ) have employed a quantile regression to approximate the full distribution by letting Zθ ( s , a ) = 1N ∑N i=1 δθi ( s , a ) where the parameter θ represents the locations of a mixture of N Dirac delta functions . Each θi represents the value where the cumulative probability is τi = iN . Then , by using the quantile representation with the distributional Bellman optimality operator , the problem can be formulated as a minimization problem as , θ = argmin θ′ D ( Zθ′ ( st , at ) , τ Zθ− ( st , at ) ) : = N∑ i=1 N∑ j=1 1 N [ ρκτ̂i ( rt + γθ − j ( st+1 , a ′ ) − θ′i ( st , at ) ) ] where ( st , at , rt , st+1 ) is a given transition pair , τ̂i = τi−1+τi 2 , a ′ : = argmaxa′ EZ [ Zθ ( st+1 , a′ ) ] , ρκτ̂i ( x ) : = |τ̂i−δ { x < 0 } |Lκ ( x ) , andLκ ( x ) : = x 2/2 for |x| ≤ κ andLκ ( x ) : = κ ( |x|− 12κ ) , otherwise . Based on the quantile regression , Dabney et al . ( 2018b ) have proposed a quantile regression deep Q network ( QR-DQN ) that shows better empirical performance than the categorical approach ( Bellemare et al. , 2017 ) since the quantile regression does not restrict the bounds for return . As deep RL typically did , QR-DQN adjusts ϵ-greedy schedule , which selects the greedy action with probability 1− ϵ and otherwise selects random available actions uniformly . The majority of QR-DQN variants ( Dabney et al. , 2018a ; Yang et al. , 2019 ) rely on the same exploration method . However , such approaches do not put aside inferior actions from the selection list and thus suffers from a loss ( Osband et al. , 2019 ) . Hence , selecting a statistically plausible action is crucial for efficient exploration . In recent studies , Mavrin et al . ( 2019 ) modifies the criterion of selecting an action for efficient exploration in the face of uncertainty . Using left truncated variance as a bonus term to estimate optimistic way and decaying ratio ct to suppress the intrinsic uncertainty , DLTV was proposed as an uncertainty-based exploration in DRL without using ϵ-greedy exploration . At timestep t , the action selection of DLTV can be described as : a∗ = argmax a′ ( EP [ Z ( s′ , a′ ) ] + ct √ σ2+ ( s ′ , a′ ) ) , ct = c √ log t t , σ2+ = 1 2N N∑ i=N2 ( θN 2 − θi ) 2 , where θi ’ s are the values of quantile level τi . DLTV shows that a constant schedule degrades the performance significantly compared to a decaying schedule . 2.3 RISK IN DISTRIBUTIONAL REINFORCEMENT LEARNING . Instead of an expected value , risk-sensitive RL tries to maximize a risk measure such as MeanVariance ( Zhang et al. , 2020 ) , Value-at-Risk ( VaR ) ( Chow et al. , 2017 ) , or Conditional Value-at-Risk ( CVaR ) ( Rockafellar et al. , 2000 ; Rigter et al. , 2021 ) , which result in different classes of optimal policy . Especially , Dabney et al . ( 2018a ) interprets risk measures as the expected utility function of the return , i.e. , EZ [ U ( Z ( s , a ) ) ] . Under this interpretation , risk-sensitive RL can be formulated as the maximization problem with various types of utility functions . If the utility function U is linear , the policy obtained under such risk measure is called risk-neutral . If U is concave or convex , the resulting policy is termed as risk-averse or risk-seeking , respectively . In general , a distortion risk measure is a generalized expression of risk measure generated from the distortion function . Definition 1 . Let h : [ 0 , 1 ] → [ 0 , 1 ] be a distortion function such that h ( 0 ) = 0 , h ( 1 ) = 1 and nondecreasing . Given a probability space ( Ω , F , P ) and a random variable Z : Ω → R , a distortion risk measure ρh corresponding to a distortion function h is defined by : ρh ( Z ) : = Eh ( P ) [ Z ] = ∫ ∞ −∞ z ∂ ∂z ( h ◦ FZ ) ( z ) dz , where FZ is the cumulative distribution function of Z . In fact , non-decreasing property of h makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF . Note that the concavity and the convexity of distortion function also imply risk-averse or risk-seeking behavior , respectively . Dhaene et al . ( 2012 ) showed that any distorted expectation can be expressed as weighted averages of quantiles . In other words , generating a distortion risk measure is equivalent to choosing a reweighting distribution . Fortunately , distributional RL has a suitable configuration to apply those uncertainty-based approaches that could naturally expand the class of policies . Chow et al . ( 2015 ) and Stanko & Macek ( 2019 ) considered risk-sensitive RL with a CVaR objective , where risk is related to robust decision making . Dabney et al . ( 2018a ) expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets τ . Unlike the usual risk-sensitive RL , DLTV applied the risk measure only on action selection , while it keeps the standard objective to obtain a risk-neutral optimal policy . Our analysis shows that risk-based exploration can utilize risk measures in two different ways : ( 1 ) selecting action and ( 2 ) evaluating the value function by using distorted ( perturbed ) expectation .

Summary: This paper considers distributional reinforcement learning with the goal of seeking risk-neutral optimal policy. Prior approaches often have a convergence issue due to either being just risk-seeking or risk-averse. Towards resolving the problem, the authors propose a simple, yet effective approach by randomizing the risk criterion during the training process. On the theoretical front, the authors prove that the method converges to an optimal return distribution under certain conditions. Several experimental results also confirm the validity of the method.