Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning

1 INTRODUCTION . Distributional reinforcement learning ( DRL ) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in several benchmark tasks . Its model generates the approximated distribution of returns , where the mean value implies the traditional Qvalue ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ; Choi et al. , 2019 ) . Learning procedure with stochasticity through return distribution is represented by parametric ( epistemic ) uncertainty , which is due to insufficient or inaccurate data , and intrinsic ( aleatoric ) uncertainty , which is inherently possessed randomness in the environment . ( Chow et al. , 2015 ; Dabney et al. , 2018a ) The learned stochasticity gives rise to the notion of risk-sensitivity , and some distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking policy . Another way to employ the uncertainty is to design an efficient exploration method which is essential to find an optimal behavior with a few number of trials . Optimism in the face of uncertainty ( OFU ) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set . Most OFU algorithms select an action with the highest upper-confidence bound ( UCB ) of uncertainty which can be considered as the optimistic decision at the moment ( Chen et al. , 2017 ; Ciosek et al. , 2019 ) . In deep RL , several OFU studies often model the uncertainty explicitly through the Bayesian posterior , which is estimated by using neural networks . However , learning the representation of high-dimensional state-action space and Bellman update simultaneously leads to unstable propagation ( Yang et al. , 2021 ) . On the other hand , DRL can provide more statistical information during control such as mode , median , or variance by addressing full characteristics of the return distribution . Despite the richness of information for return distribution , only a few DRL methods have tried to employ the benefits of distributional perspective for exploration ( Tang & Agrawal , 2018 ; Mavrin et al. , 2019 ; Clements et al. , 2019 ) . To estimate the uncertainty from distributional outputs , descriptive statistics that is composed of a mixture of intrinsic and parametric uncertainty can be utilized . Unfortunately , sepa- rating these two types of uncertainty during learning is not a trivial task . Mavrin et al . ( 2019 ) propose a distribution-based OFU exploration that schedules a decaying bonus rate to suppress the effect of intrinsic uncertainty , which unintentionally induces a risk-seeking policy . Although OFU based approaches try to reduce parametric uncertainty by revisiting the state with high uncertainty , there exists the side effect that the criteria unfortunately forces the agent to chase the intrinsic uncertainty ( risk ) simultaneously during updates . Relying on a specific criteria causes an one-sided tendency on risk and makes an agent consistently select certain actions during exploration as degrading performance . We call this phenomenon as stuckness . In this paper , we introduce perturbed quantile regression ( PQR ) which perturbs the criterion on uncertainty by randomizing the risk criterion in action selection . First , the distributional perturbation on return distribution is to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight . Unlike the typical worst-case approach in risk-sensitive settings or OFU based approaches , we instead randomly sample a risk measure from an ambiguity set , which represents that the risk setting is ambiguous when the characteristics of a given environment are unknown . We empirically demonstrate on the stochastic variant of N-Chain environment ( Osband et al. , 2016 ) that a randomized scheme is more effective than OFU to alleviate the sub-optimality problem of tendency to obtain risk-seeking policies . Second , any risk-measure with some time-varying perturbation constraint allows us to average over all possible risk-sensitive behaviors to achieve the optimal policy . We provide a sufficient condition for the convergence of return distribution in the weaker contraction property . Our method covers the full elements of the reinforcement learning with distributional perspective from the novel exploration method with distributional perturbation to the theoretically guaranteed convergence of return distribution , which has the same fixed-point with the standard Bellman optimality operator . The proposed algorithm is based on QR-DQN , which is a baseline of DRL architecture to learn a return distribution , and we show that PQR outperforms QR-DQN and other DRL baselines in several benchmark environments , LunarLander-v2 and Atari games . 2 BACKGROUNDS & RELATED WORKS . 2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING . We consider a Markov decision process ( MDP ) which is defined as a tuple ( S , A , P , R , γ ) where S is a finite state space , A is a finite action space , P : S×A×S → [ 0 , 1 ] is the transition probability , R is the random variable of rewards in [ −Rmax , Rmax ] , and γ ∈ [ 0 , 1 ) is the discount factor . We define a stochastic policy π ( ·|s ) which is a conditional distribution overA given state s. For a fixed policy π , we denote Zπ ( s , a ) as a random variable of return distribution of state-action pair ( s , a ) following the policy π . We attain Zπ ( s , a ) = ∑∞ t=0 γ tR ( St , At ) , where St+1 ∼ P ( ·|St , At ) , At ∼ π ( ·|St ) and S0 = s , A0 = a . Then , we define an action-value function as Qπ ( s , a ) = E [ Zπ ( s , a ) ] in [ −Vmax , Vmax ] where Vmax = Rmax/ ( 1−γ ) . For regularity , we further notice that the space of actionvalue distributions Z has the first moment bounded by Vmax : Z = { Z : S ×A →P ( R ) ∣∣ E [ |Z ( s , a ) | ] ≤ Vmax , ∀ ( s , a ) } . In distributional RL , the return distribution for the fixed π can be computed via dynamic programming with the distributional Bellman operator defined as ( Bellemare et al. , 2017 ) , τ πZ ( s , a ) D= R ( s , a ) + γZ ( S′ , A′ ) , S′ ∼ P ( ·|s , a ) , A′ ∼ π ( ·|S′ ) where D= denotes that both random variables share the same probability distribution . We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as , τZ ( s , a ) D= R ( s , a ) + γZ ( S′ , a∗ ) , S′ ∼ P ( ·|s , a ) , a∗ = argmax a′ EZ [ Z ( S′ , a′ ) ] . Bellemare et al . ( 2017 ) have shown that τ π is a contraction in a maximal form of the Wasserstein metric but τ is not a contraction in any metric . Combining with the expectation operator , Eτ is a contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value , while the convergence of a return distribution itself is not guaranteed . 2.2 EXPLORATION ON DISTRIBUTIONAL REINFORCEMENT LEARNING . In this section , we will briefly describe the main practical algorithm for DRL with a deep neural network , and explain the derived exploration method . To combine with deep RL , a parametric distribution Zθ is used to learn a return distribution by using τ . Dabney et al . ( 2018b ) have employed a quantile regression to approximate the full distribution by letting Zθ ( s , a ) = 1N ∑N i=1 δθi ( s , a ) where the parameter θ represents the locations of a mixture of N Dirac delta functions . Each θi represents the value where the cumulative probability is τi = iN . Then , by using the quantile representation with the distributional Bellman optimality operator , the problem can be formulated as a minimization problem as , θ = argmin θ′ D ( Zθ′ ( st , at ) , τ Zθ− ( st , at ) ) : = N∑ i=1 N∑ j=1 1 N [ ρκτ̂i ( rt + γθ − j ( st+1 , a ′ ) − θ′i ( st , at ) ) ] where ( st , at , rt , st+1 ) is a given transition pair , τ̂i = τi−1+τi 2 , a ′ : = argmaxa′ EZ [ Zθ ( st+1 , a′ ) ] , ρκτ̂i ( x ) : = |τ̂i−δ { x < 0 } |Lκ ( x ) , andLκ ( x ) : = x 2/2 for |x| ≤ κ andLκ ( x ) : = κ ( |x|− 12κ ) , otherwise . Based on the quantile regression , Dabney et al . ( 2018b ) have proposed a quantile regression deep Q network ( QR-DQN ) that shows better empirical performance than the categorical approach ( Bellemare et al. , 2017 ) since the quantile regression does not restrict the bounds for return . As deep RL typically did , QR-DQN adjusts ϵ-greedy schedule , which selects the greedy action with probability 1− ϵ and otherwise selects random available actions uniformly . The majority of QR-DQN variants ( Dabney et al. , 2018a ; Yang et al. , 2019 ) rely on the same exploration method . However , such approaches do not put aside inferior actions from the selection list and thus suffers from a loss ( Osband et al. , 2019 ) . Hence , selecting a statistically plausible action is crucial for efficient exploration . In recent studies , Mavrin et al . ( 2019 ) modifies the criterion of selecting an action for efficient exploration in the face of uncertainty . Using left truncated variance as a bonus term to estimate optimistic way and decaying ratio ct to suppress the intrinsic uncertainty , DLTV was proposed as an uncertainty-based exploration in DRL without using ϵ-greedy exploration . At timestep t , the action selection of DLTV can be described as : a∗ = argmax a′ ( EP [ Z ( s′ , a′ ) ] + ct √ σ2+ ( s ′ , a′ ) ) , ct = c √ log t t , σ2+ = 1 2N N∑ i=N2 ( θN 2 − θi ) 2 , where θi ’ s are the values of quantile level τi . DLTV shows that a constant schedule degrades the performance significantly compared to a decaying schedule . 2.3 RISK IN DISTRIBUTIONAL REINFORCEMENT LEARNING . Instead of an expected value , risk-sensitive RL tries to maximize a risk measure such as MeanVariance ( Zhang et al. , 2020 ) , Value-at-Risk ( VaR ) ( Chow et al. , 2017 ) , or Conditional Value-at-Risk ( CVaR ) ( Rockafellar et al. , 2000 ; Rigter et al. , 2021 ) , which result in different classes of optimal policy . Especially , Dabney et al . ( 2018a ) interprets risk measures as the expected utility function of the return , i.e. , EZ [ U ( Z ( s , a ) ) ] . Under this interpretation , risk-sensitive RL can be formulated as the maximization problem with various types of utility functions . If the utility function U is linear , the policy obtained under such risk measure is called risk-neutral . If U is concave or convex , the resulting policy is termed as risk-averse or risk-seeking , respectively . In general , a distortion risk measure is a generalized expression of risk measure generated from the distortion function . Definition 1 . Let h : [ 0 , 1 ] → [ 0 , 1 ] be a distortion function such that h ( 0 ) = 0 , h ( 1 ) = 1 and nondecreasing . Given a probability space ( Ω , F , P ) and a random variable Z : Ω → R , a distortion risk measure ρh corresponding to a distortion function h is defined by : ρh ( Z ) : = Eh ( P ) [ Z ] = ∫ ∞ −∞ z ∂ ∂z ( h ◦ FZ ) ( z ) dz , where FZ is the cumulative distribution function of Z . In fact , non-decreasing property of h makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF . Note that the concavity and the convexity of distortion function also imply risk-averse or risk-seeking behavior , respectively . Dhaene et al . ( 2012 ) showed that any distorted expectation can be expressed as weighted averages of quantiles . In other words , generating a distortion risk measure is equivalent to choosing a reweighting distribution . Fortunately , distributional RL has a suitable configuration to apply those uncertainty-based approaches that could naturally expand the class of policies . Chow et al . ( 2015 ) and Stanko & Macek ( 2019 ) considered risk-sensitive RL with a CVaR objective , where risk is related to robust decision making . Dabney et al . ( 2018a ) expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets τ . Unlike the usual risk-sensitive RL , DLTV applied the risk measure only on action selection , while it keeps the standard objective to obtain a risk-neutral optimal policy . Our analysis shows that risk-based exploration can utilize risk measures in two different ways : ( 1 ) selecting action and ( 2 ) evaluating the value function by using distorted ( perturbed ) expectation .

Summary: The paper proposes a novel distributional RL algorithm that is neither risk-seeking (exploratory) or risk-averse (exploitatory) -- but rather risk-neutral. The paper motivates the need for risk neutrality from the family of works applying OFU (optimism in the face of uncertainty) to distributional RL -- it states that prior works in this space induce a one-sided risk tendency (risk-seeking or risk-averse) which is undesirable as it leads to “biased exploration”. An N-chain environment is initially studied where it is shown that prior works (DLTV, QRDQN)  are not able to identify the optimal action as fast as the proposed method (PQR) -- the prior works are misled by the low probability high reward side of the N-chain environment as opposed to the optimal side which provides maximum averaged reward. The paper then proposes a method for perturbing the risk-measure in risk-sensitive RL such that the resulting perturbed distributional bellman operator (PDBOO) converges to the same fixed point as the standard bellman operator (with some weak assumptions) and hence, produces a risk-neutral policy. The paper presented results in the LunarLander-v2 and Atari domain (4 Atari games selected), demonstrating that prior works with one-sided risk tendencies do not perform as well as the proposed risk-neutral method (PQR).