Year: 2021

Abstract:
Distributional reinforcement learning aims to learn distribution of return under stochastic environments. Since the learned distribution of return contains rich information about the stochasticity of the environment, previous studies have relied on descriptive statistics, such as standard deviation, for optimism in the face of uncertainty. However, using the uncertainty from an empirical distribution can hinder convergence and performance when exploring with the certain criterion that has an one-sided tendency on risk in these methods. In this paper, we propose a novel distributional reinforcement learning that explores by randomizing risk criterion to reach a risk-neutral optimal policy. First, we provide a perturbed distributional Bellman optimality operator by distorting the risk measure in action selection. Second, we prove the convergence and optimality of the proposed method by using the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return distribution. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari games.

heading: 1 INTRODUCTION
text: Distributional reinforcement learning (DRL) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in several benchmark tasks. Its model generates the approximated distribution of returns, where the mean value implies the traditional Qvalue (Bellemare et al., 2017; Dabney et al., 2018b; Choi et al., 2019). Learning procedure with stochasticity through return distribution is represented by parametric (epistemic) uncertainty, which is due to insufficient or inaccurate data, and intrinsic (aleatoric) uncertainty, which is inherently possessed randomness in the environment. (Chow et al., 2015; Dabney et al., 2018a) The learned stochasticity gives rise to the notion of risk-sensitivity, and some distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking policy.
Another way to employ the uncertainty is to design an efficient exploration method which is essential to find an optimal behavior with a few number of trials. Optimism in the face of uncertainty (OFU) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set. Most OFU algorithms select an action with the highest upper-confidence bound (UCB) of uncertainty which can be considered as the optimistic decision at the moment (Chen et al., 2017; Ciosek et al., 2019). In deep RL, several OFU studies often model the uncertainty explicitly through the Bayesian posterior, which is estimated by using neural networks. However, learning the representation of high-dimensional state-action space and Bellman update simultaneously leads to unstable propagation(Yang et al., 2021).
On the other hand, DRL can provide more statistical information during control such as mode, median, or variance by addressing full characteristics of the return distribution. Despite the richness of information for return distribution, only a few DRL methods have tried to employ the benefits of distributional perspective for exploration (Tang & Agrawal, 2018; Mavrin et al., 2019; Clements et al., 2019). To estimate the uncertainty from distributional outputs, descriptive statistics that is composed of a mixture of intrinsic and parametric uncertainty can be utilized. Unfortunately, sepa-
rating these two types of uncertainty during learning is not a trivial task. Mavrin et al. (2019) propose a distribution-based OFU exploration that schedules a decaying bonus rate to suppress the effect of intrinsic uncertainty, which unintentionally induces a risk-seeking policy. Although OFU based approaches try to reduce parametric uncertainty by revisiting the state with high uncertainty, there exists the side effect that the criteria unfortunately forces the agent to chase the intrinsic uncertainty (risk) simultaneously during updates. Relying on a specific criteria causes an one-sided tendency on risk and makes an agent consistently select certain actions during exploration as degrading performance. We call this phenomenon as stuckness.
In this paper, we introduce perturbed quantile regression (PQR) which perturbs the criterion on uncertainty by randomizing the risk criterion in action selection. First, the distributional perturbation on return distribution is to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight. Unlike the typical worst-case approach in risk-sensitive settings or OFU based approaches, we instead randomly sample a risk measure from an ambiguity set, which represents that the risk setting is ambiguous when the characteristics of a given environment are unknown. We empirically demonstrate on the stochastic variant of N-Chain environment (Osband et al., 2016) that a randomized scheme is more effective than OFU to alleviate the sub-optimality problem of tendency to obtain risk-seeking policies. Second, any risk-measure with some time-varying perturbation constraint allows us to average over all possible risk-sensitive behaviors to achieve the optimal policy. We provide a sufficient condition for the convergence of return distribution in the weaker contraction property. Our method covers the full elements of the reinforcement learning with distributional perspective from the novel exploration method with distributional perturbation to the theoretically guaranteed convergence of return distribution, which has the same fixed-point with the standard Bellman optimality operator. The proposed algorithm is based on QR-DQN, which is a baseline of DRL architecture to learn a return distribution, and we show that PQR outperforms QR-DQN and other DRL baselines in several benchmark environments, LunarLander-v2 and Atari games.

heading: 2 BACKGROUNDS & RELATED WORKS

heading: 2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING
text: We consider a Markov decision process (MDP) which is defined as a tuple (S,A, P,R, γ) where S is a finite state space,A is a finite action space, P : S×A×S → [0, 1] is the transition probability, R is the random variable of rewards in [−Rmax, Rmax], and γ ∈ [0, 1) is the discount factor. We define a stochastic policy π(·|s) which is a conditional distribution overA given state s. For a fixed policy π, we denote Zπ(s, a) as a random variable of return distribution of state-action pair (s, a) following the policy π. We attain Zπ(s, a) = ∑∞ t=0 γ
tR(St, At), where St+1 ∼ P (·|St, At), At ∼ π(·|St) and S0 = s, A0 = a. Then, we define an action-value function as Qπ(s, a) = E[Zπ(s, a)] in [−Vmax, Vmax] where Vmax = Rmax/(1−γ). For regularity, we further notice that the space of actionvalue distributions Z has the first moment bounded by Vmax:
Z = { Z : S ×A →P(R) ∣∣ E[|Z(s, a)|] ≤ Vmax,∀(s, a)} . In distributional RL, the return distribution for the fixed π can be computed via dynamic programming with the distributional Bellman operator defined as (Bellemare et al., 2017),
τ πZ(s, a) D= R(s, a) + γZ(S′, A′), S′ ∼ P (·|s, a), A′ ∼ π(·|S′)
where D= denotes that both random variables share the same probability distribution. We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as,
τZ(s, a) D= R(s, a) + γZ(S′, a∗), S′ ∼ P (·|s, a), a∗ = argmax a′ EZ [Z(S′, a′)].
Bellemare et al. (2017) have shown that τ π is a contraction in a maximal form of the Wasserstein metric but τ is not a contraction in any metric. Combining with the expectation operator, Eτ is a contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value, while the convergence of a return distribution itself is not guaranteed.

heading: 2.2 EXPLORATION ON DISTRIBUTIONAL REINFORCEMENT LEARNING
text: In this section, we will briefly describe the main practical algorithm for DRL with a deep neural network, and explain the derived exploration method. To combine with deep RL, a parametric distribution Zθ is used to learn a return distribution by using τ . Dabney et al. (2018b) have employed a quantile regression to approximate the full distribution by letting Zθ(s, a) = 1N ∑N i=1 δθi(s,a) where the parameter θ represents the locations of a mixture of N Dirac delta functions. Each θi represents the value where the cumulative probability is τi = iN . Then, by using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as a minimization problem as,
θ = argmin θ′ D (Zθ′(st, at), τ Zθ−(st, at)) := N∑ i=1 N∑ j=1 1 N [ρκτ̂i(rt + γθ − j (st+1, a ′)− θ′i(st, at))]
where (st, at, rt, st+1) is a given transition pair, τ̂i = τi−1+τi 2 , a ′ := argmaxa′ EZ [Zθ(st+1, a′)], ρκτ̂i(x) := |τ̂i−δ{x<0}|Lκ(x), andLκ(x) := x 2/2 for |x| ≤ κ andLκ(x) := κ(|x|− 12κ), otherwise. Based on the quantile regression, Dabney et al. (2018b) have proposed a quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach (Bellemare et al., 2017) since the quantile regression does not restrict the bounds for return. As deep RL typically did, QR-DQN adjusts ϵ-greedy schedule, which selects the greedy action with probability 1− ϵ and otherwise selects random available actions uniformly. The majority of QR-DQN variants (Dabney et al., 2018a; Yang et al., 2019) rely on the same exploration method. However, such approaches do not put aside inferior actions from the selection list and thus suffers from a loss (Osband et al., 2019). Hence, selecting a statistically plausible action is crucial for efficient exploration.
In recent studies, Mavrin et al. (2019) modifies the criterion of selecting an action for efficient exploration in the face of uncertainty. Using left truncated variance as a bonus term to estimate optimistic way and decaying ratio ct to suppress the intrinsic uncertainty, DLTV was proposed as an uncertainty-based exploration in DRL without using ϵ-greedy exploration. At timestep t, the action selection of DLTV can be described as:
a∗ = argmax a′
( EP [Z(s′, a′)] + ct √ σ2+(s ′, a′) ) , ct = c √ log t
t , σ2+ =
1
2N N∑ i=N2 (θN 2 − θi)2,
where θi’s are the values of quantile level τi. DLTV shows that a constant schedule degrades the performance significantly compared to a decaying schedule.

heading: 2.3 RISK IN DISTRIBUTIONAL REINFORCEMENT LEARNING
text: Instead of an expected value, risk-sensitive RL tries to maximize a risk measure such as MeanVariance (Zhang et al., 2020), Value-at-Risk (VaR) (Chow et al., 2017), or Conditional Value-at-Risk (CVaR) (Rockafellar et al., 2000; Rigter et al., 2021), which result in different classes of optimal policy. Especially, Dabney et al. (2018a) interprets risk measures as the expected utility function of the return, i.e., EZ [U(Z(s, a))]. Under this interpretation, risk-sensitive RL can be formulated as the maximization problem with various types of utility functions. If the utility function U is linear, the policy obtained under such risk measure is called risk-neutral. If U is concave or convex, the resulting policy is termed as risk-averse or risk-seeking, respectively. In general, a distortion risk measure is a generalized expression of risk measure generated from the distortion function. Definition 1. Let h : [0, 1]→ [0, 1] be a distortion function such that h(0) = 0, h(1) = 1 and nondecreasing. Given a probability space (Ω,F ,P) and a random variable Z : Ω → R, a distortion risk measure ρh corresponding to a distortion function h is defined by:
ρh(Z) := Eh(P)[Z] = ∫ ∞ −∞ z ∂ ∂z (h ◦ FZ)(z)dz,
where FZ is the cumulative distribution function of Z.
In fact, non-decreasing property of h makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF. Note that the concavity and the convexity of distortion function
also imply risk-averse or risk-seeking behavior, respectively. Dhaene et al. (2012) showed that any distorted expectation can be expressed as weighted averages of quantiles. In other words, generating a distortion risk measure is equivalent to choosing a reweighting distribution.
Fortunately, distributional RL has a suitable configuration to apply those uncertainty-based approaches that could naturally expand the class of policies. Chow et al. (2015) and Stanko & Macek (2019) considered risk-sensitive RL with a CVaR objective, where risk is related to robust decision making. Dabney et al. (2018a) expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets τ . Unlike the usual risk-sensitive RL, DLTV applied the risk measure only on action selection, while it keeps the standard objective to obtain a risk-neutral optimal policy. Our analysis shows that risk-based exploration can utilize risk measures in two different ways: (1) selecting action and (2) evaluating the value function by using distorted (perturbed) expectation.

heading: 3 PERTURBATION IN DISTRIBUTIONAL REINFORCEMENT LEARNING

heading: 3.1 MOTIVATION
text: Distribution-based OFU exploration (Moerland et al., 2018; Keramati et al., 2020) was proposed to give a bonus for the uncertainty that can be extracted from the distribution. However, we found that keeping optimism on uncertainty tends to select sub-optimal behaviors over a long exploration. For example, suppose we choose a criterion based on mean-standard deviation with coefficient ct. Consider two actions a1, a2 with mean µ1, µ2 and variance σ1, σ2 respectively, under the following conditions: µ1 ≥ µ2, σ1 ≤ σ2, and µ1 + ctσ1 ≤ µ2 + ctσ2. Then, the agent prefers to select a2 based on OFU. To change the decision towards the true optimal action a1, the following steps need to be spent:
η = min { t′ > t : ct′ ≤
µ1 − µ2 σ2 − σ1
} − t.
Hence, if there is a bias in the criterion itself, such stuckness often occurs and degrades the performance since the agent does not have experience with the optimal policy during that period.
To demonstrate such shortcomings of OFU exploration in distributional RL framework, we build a representative environment that is easy to interpret intuitively among the cases in which intrinsic uncertainty exists. We experiment on the stochastic variant of N-Chain environment used in Osband et al. (2016) as a toy experiment. A schematic diagram of the N-Chain environment is shown in Figure 1. The reward is only given in the leftmost and rightmost states and the game terminates when one of the reward states is reached. We set the leftmost reward as N (10, 0.12) and the rightmost reward as 12N (5, 0.1 2)+ 12N (13, 0.1 2) which has a lower mean as 9 but higher variance. The agent always starts from the middle state s2 and should move toward the leftmost state s0 to achieve the greatest expected return. For each state, the agent can take one of six available actions: left, right, and 4 no-op actions. The optimal policy with respect to mean is to move left twice from the start. Despite the simple configuration, the possibility to obtain a higher reward in the suboptimal state than the optimal state makes the agent difficult which policy is optimal until it experiences enough to detect the characteristics of each distribution. Thus, the goal of our toy experiment is to evaluate how quickly each algorithm could find a risk-neutral optimal policy.
Rather than applying a decaying schedule to suppress intrinsic uncertainty, we naively modify the optimism into a randomized risk criterion and named the perturbed variant as p-DLTV. We compare QR-DQN and DLTV with our randomized variants of two algorithms PQR and p-DLTV to examine the effect of randomized risk criteria. In short, p-DLTV is a straightforward modification of DLTV where randomness is given to the coefficient ct through normal distribution. We will describe our main algorithm, PQR, in detail in Section 3.4.
In Figure 2(a), DLTV fails to estimate the true optimal return distribution of action a1. Due to the erroneous estimation, the agent takes longer to recognize its error. Hence, the deterministic selection based on a fixed criteria could mislead toward exploitation rather than exploration. This indicates that DLTV may not gather experiences well in a complex environment that requires deep exploration. In Figure 2(b), we plot the point when the optimal policy was actually performed for each algorithm to show stuckness. Since the optimal policy consists of only the same index a1, we plot the total count of performing the optimal action with 10 different seeds. The slope indicates the ratio of performing the optimal policy on average. Hence, the interval with a slope of 1 implies that the optimal policy was performed every time. From the slope of each line, it is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed. Although the mean return of a1 (move left) is estimated to be superior, the agent only selects a2 (move right) during training due to its consistent optimism on uncertainty. Even if DLTV has spent enough number of time steps to choose the true optimal policy, the remaining procedure is already close to greedy selection as it starts from a decreased coefficient.
Surprisingly, p-DLTV alleviates the stuckness early by randomly choosing a plausible action. We also propose a theoretically guaranteed algorithm to be converged, PQR which shows a much steeper line by quickly obtaining the optimal policy. In Section 3.2, we derive the first theoretical sufficient condition for the convergence of exploration method for DRL, which implies that PQR has the same unique fixed point as the standard distributional Bellman equation.

heading: 3.2 PERTURBED DISTRIBUTIONAL BELLMAN OPTIMALITY OPERATOR
text: To choose statistically plausible actions which may be maximal for certain criteria, we are interested in generating a distortion risk measure involved in a pre-defined constraint set called an ambiguity set. The ambiguity set, originated from distributionally robust optimization (DRO) literature, is a family of distribution characterized by a certain statistical distance such as ϕ-divergence or Wasserstein distance (Esfahani & Kuhn, 2018; Shapiro et al., 2021). In this paper, we will examine the ambiguity set defined by the discrepancy between distortion risk measure and expectation. We say the sampled reweighting distribution ξ as (distributional) perturbation and define it as follows:
Definition 2. (Perturbation, Perturbation Gap, and Ambiguity Set) Given a probability space (Ω,F ,P), let X : Ω→ R be a random variable and Ξ = { ξ : ξ(w) ≥ 0, ∫ w∈Ω ξ(w)P(w)dw = 1 }
be a set of probability density functions. For a given constraint set U ⊂ Ξ, we say ξ ∈ U as a (distributional) perturbation from U and denote the ξ−weighted expectation of X as follows:
Eξ[X] := ∫ w∈Ω X(w)ξ(w)P(w)dw,
which can be interpreted as the expectation of X under perturbed probability distribution ξP. We further define d(X; ξ) = |E[X]− Eξ[X]| as perturbation gap of X with respect to ξ. Then, for a given constant ∆ ≥ 0, we define the ambiguity set with the bound ∆ as
U∆(X) = { ξ ∈ Ξ : d(X; ξ) ≤ ∆ } .
For brevity, we omit the input w from a random variable unless confusing. Since ξ is a probability density function, Eξ[X] is an induced risk measure with respect to a reference measure P. Intuitively, ξ(w) can be viewed as a distortion to generate a different probability measure and allow to vary the risk tendency. The aspect of using distortion risk measures looks similar to IQN (Dabney et al., 2018a). However, instead of changing the sampling distribution of quantile level τ implicitly, we reweight each quantile from the ambiguity set. This allows us to control the maximum allowable distortion with bound ∆, whereas in IQN the risk measure does not change throughout learning. In Section 3.4, we will suggest a practical method to construct the ambiguity set.
Now, we characterize perturbed distributional Bellman optimality operator τ ξ for a fixed perturbation ξ ∈ U∆(Z) written as below:
τ ξZ(s, a) D= R(s, a) + γZ(S′, a∗(ξ)), S′ ∼ P (·|s, a) , a∗(ξ) = argmax a′ Eξ,P [Z(s′, a′)].
Notice that ξ ≡ 1 corresponds to a base expectation, i.e., Eξ,P = EP , which recovers the standard distributional Bellman optimality operator τ . To understand the progress of PDBOO intuitively, we add the pipeline figure in Appendix A.4.
Specifically, updating via τ ξ can be described in two steps. 1) Select the superior action in terms of an arbitrary distortion risk measure instead of mean. We specify the distortion risk measure by sampling the perturbation ξ from the ambiguity set. 2) Compute the return distribution through dynamic programming which follows the standard Bellman optimality equation. In risk-sensitive DRL or distributionally robust RL, the Bellman optimality equation is reformulated for a pre-defined risk measure (Chow et al., 2015; Smirnova et al., 2019; Yang, 2020). In this sense, PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy while randomizing the risk criterion at every step.
If we consider the time-varying bound of ambiguity set, scheduling ∆t is a key ingredient to determine whether PDBOO will efficiently explore or converge. The following lemma plays an important role to guarantee the convergence of PDBOO. Lemma 3. If ξt converges to 1 uniformly on Ω, then Eτ ξt also converges to Eτ uniformly on Z for all s ∈ S and a ∈ A.
Intuitively, if an agent continues to sample the distortion risk measure from a fixed ambiguity set with a constant ∆, there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence. Hence, it will be crucial to schedule a constraint of ambiguity set properly at each action selection to guarantee convergence.
Based on the quantile model Zθ, our algorithm can be summarized into two parts. First, we aim to minimize the expected discrepancy between Zθ and τ ξZθ− where ξ is sampled from ambiguity set U∆. To clarify notation, we write Eξ[·] as a ξ−weighted expectation and Eξ∼P(U∆)[·] as an expectation with respect to ξ which is sampled from U∆. Then, our goal is to minimize the perturbed distributional Bellman objective with sampling procedure P:
min θ′ Eξt∼P(U∆t )[D(Zθ′(s, a), τ ξtZθ−(s, a))] (1) where we use the Huber quantile loss as a discrepancy on Zθ′ and τ ξZθ− at timestep t. It is different from DRO which performs the worst-case optimization by using a minimax objective. By using expectation instead of max operator, we investigate risk-neutral exploration that can avoid overly
pessimistic policies. Second, considering a sequence ξt which converges uniformly to 1 so that τ ξt converges uniformly to original τ , we derive a sufficient condition of ∆t that the expectation of iterated operator Eτ ξn:1 = Eτ ξnτ ξn−1 · · ·τ ξ1 has a unique fixed point with the same solution as the standard solution.

heading: 3.3 CONVERGENCE OF THE PERTURBED DISTRIBUTIONAL BELLMAN OPTIMALITY OPERATOR
text: In this section, we will provide the theoretical result of PDBOO about its convergence through E[Z(n)] where the iteration procedure is denoted as Z(n+1) := τ ξn+1Z(n) and Z(0) = Z for each timestep n > 0. Theorem 4. For a sequence of bound ∆n, let Ū∆n(Z(n−1)) := ⋂ s,a U∆n ( Z(n−1)(s, a) ) . If we
sample ξn from Ū∆n(Z(n−1)) for every iteration and ∑∞
n=1 ∆n < ∞ holds, then, the expectation of iterated operator τ ξn:1 has a fixed point E[Z∗]. Moreover, the following bound holds,
sup s,a ∣∣∣E[Z(n)(s, a)]− E[Z∗(s, a)]∣∣∣ ≤ ∞∑ k=n ( 2γk−1Vmax + 2 k∑ i=1 γi(∆k+2−i +∆k+1−i) ) .
Theorem 4 states that τ ξn:1 converges while it does not satisfy the γ-contraction property. Note that the fixed point E[Z∗] is not yet guaranteed to be unique for any Z ∈ Z . Fortunately, we could show that E[Z∗] is, in fact, the solution of the standard Bellman optimality equation which is already well known to have a unique solution. Theorem 5. If {∆n} follows the assumption in Theorem 4, then E[Z∗] is the unique solution of Bellman optimality equation.
From two theoretical results, PDBOO is guaranteed to have the same unique fixed point as the standard Bellman operator under weaker contraction property. Unlike other distribution-based or risk-sensitive approaches, PDBOO is a novel operator having compatibility for obtaining a riskneutral optimal policy. Also, it is also observed that this randomized method is more efficient than the OFU-based exploration by quickly alleviating from sub-optimal policies.

heading: 3.4 PERTURBED QUANTILE REGRESSION FOR DEEP Q-NETWORK
text: We propose a perturbed quantile regression (PQR) that is a practical algorithm for distributional reinforcement learning. We model a return distribution using quantile regression and update the quantile model by minimizing the objective function (1) induced by PDBOO. To compute the target distribution of (1), we propose a sampling method of ξ from ambiguity set U∆. Since we employ a quantile model, sampling a reweight function ξ can be reduced into sampling an N -dimensional weight vector ξ := [ξ1, · · · , ξN ] where ∑N i=1 ξi = N and ξi ≥ 0 for all i ∈ {1, · · · , N}. Based on
the QR-DQN setup, note that the condition ∫ w∈Ω ξ(w)P(w)dw = 1 turns into ∑N i=1 1 N ξi = 1 since the quantile level is set as τi = iN .
A key issue is how to construct an ambiguity set with bound ∆t and then sample ξ. A natural class of distribution for practical use is the symmetric Dirichlet distribution with concentration β, which
Algorithm 1: Perturbed QR-DQN (PQR) Input: transition (s, a, r, s′), discount γ ∈ [0, 1), timestep t > 0, epsilon ϵ > 0, concentration β
Initialize ∆0 > 0 ∆t ← ∆0t−(1+ϵ) ξ ← max ( 1N +∆t(Nx− 1N ), 0 )
where x ∼ Dir(β) # Sample ξ ∼ Ū∆t(Z(t)) ξ ← Nξ/ ∑ ξi a∗ ← argmaxa′ Eξ[Z(s′, a′)] # Select greedy action with distorted expectation τ θj ← r + γθj(s′, a∗), ∀j t← t+ 1
Output: ∑N
i=1 Ej [ρκτ̂i(τ θj − θi(s, a))]
represents distribution over distributions. (i.e. x ∼ Dir(β).) If β is small, then, most of the mass is concentrated on a few elements. Otherwise, all elements are similar to each other and produce evenly distributed weight. By using the Dirichlet distribution, we sample a random vector, x ∼ Dir(β), and define the reweight distribution as ξ := 1N + α(Nx − 1N ). From the construction of ξ, we have 1−α ≤ ξi ≤ 1+α(N − 1) for all i and it follows that |1− ξi| ≤ α(N − 1) for all i. By controlling α, we can bound the deviation of ξi from 1 and bound the perturbation gap as
sup s,a |E[Z(s, a)]− Eξ[Z(s, a)]| = sup s,a ∣∣∣∣∫ w∈Ω Z(w; s, a)(1− ξ(w))P(w)dw ∣∣∣∣
≤ sup w∈Ω |1− ξ(w)| sup s,a E[|Z(s, a)|] ≤ sup w∈Ω |1− ξ(w)|Vmax ≤ α(N − 1)Vmax.
Hence, letting α ≤ ∆(N−1)Vmax is sufficient to obtain d(Z; ξ) ≤ ∆ in the quantile setting. We set β = 0.05 · 1N to generate a constructive perturbation ξn which gap is close to the bound ∆n. To satisfy the condition stated in Theorem 4, we set ∆t = ∆0t−(1+ϵ) where ∆0 is a hyperparameter. The detailed procedure of the proposed method is summarized in Algorithm 1.

heading: 4 EXPERIMENTAL RESULTS AND DETAILS
text: In this section, we evaluate the performance on N-Chain, LunarLander and Atari games to demonstrate the efficiency of randomized criteria, comparing with QR-DQN (ϵ-greedy) and DLTV (OFU).
N-Chain. As the mean of each return is designed to be similar, it is useful to examine the learning behavior of the empirical return distribution for each algorithm. Figure 3 shows the empirical PDF of return distribution by using Gaussian kernel density estimation. PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation. Although the optimal policy was performed, QR-DQN overestimates the optimal Q-value of (s2, a1) as µ̂ = 9.33 while the ground truth is computed as µ = 10γ2 = 8.1. PQR estimated the target relatively better compared to QR-DQN as the agent often chooses an alternative action from a risk-neutral perspective. Only by chainging from optimism to a randomized scheme, p-DLTV made a much better estimate than DLTV.
LunarLander-v2. Next, we evaluate the performance on LunarLander-v2 environment where its input is given by 8-dimensional coordinates with 4 different discrete actions. The goal of the agent is to reach the landing pad with a given threshold of 200 points without crashing. Without obtaining the extra point of +100, the agent cannot reach the threshold.
Figure 4(a) shows that p-DLTV and PQR have reached the threshold 200 points faster than the other two algorithms. Significantly, p-DLTV successfully reached the goal just by randomizing the criterion from OFU while DLTV did not learn to obtain the extra point. The result shows that onesided tendency on risk leads to learning failure of an agent. We note that the randomized risk criterion could alleviate the stuckness within an affordable time budget.
Atari Games. Lastly, we evaluate the performance in Atari games, including multiple environments, each of which contained intrinsic uncertainty in different ways.
We plot the average performance of each algorithm in the evaluation step on 200M frames with 3 random seeds. In Figure 5, PQR achieves the shortest time to hit the asymptotic highest performance in various Atari games. Moreover, our proposed method is stable with relatively low variance in all four environments. As the bound of perturbation gap converges to 0, the plausible action is chosen from the shrinking ambiguity set, and thus the agent gradually performs only the optimal policy. Further experimental details are in Appendix D
In all experiments, just adding randomness to the coefficient ct shows the significant improvement supporting that the randomized risk criterion was superior to OFU in distributional RL. In the majority of environments where intrinsic uncertainties exist, OFU has difficulty in making a decision that matches risk-neutral purpose, because two uncertainties are intertwined during learning. We observe that PQR outperforms the baseline by showing faster convergence without degrading the performance. All experimental results support that the randomizing scheme is more effective than OFU in distributional RL.

heading: 5 CONCLUSIONS
text: In this paper, we proposed a general framework of risk-based exploration which captures the characteristics of a return distribution. Without resorting to a pre-defined risk criterion, we revealed and resolved the stuckness where one-sided tendency on risk can lead to biased action selection. To our best knowledge, this paper is the first attempt in DRL to integrate risk-sensitivity and exploration by using time-varying Bellman objective with theoretical analysis. We prove that PDBOO is theoretically guaranteed for convergence, and has a unique fixed point in a weaker condition than contraction. By obtaining the same fixed point as the standard Bellman optimality operator, we can obtain a risk-neutral optimal policy even if the agent explored with distortion risk measure. Our method shows better empirical results compared to recent exploration methods for distributional RL with various environments.

heading: A PROOF

heading: A.1 TECHNICAL LEMMA
text: Before proving our theoretical results, we present two inequalities for supremum to clear the description.
1. sup x∈X |f(x) + g(x)| ≤ sup x∈X |f(x)|+ sup x∈X |g(x)|
2. ∣∣∣sup x∈X f(x)− sup x′∈X g(x′) ∣∣∣ ≤ sup x,x′∈X |f(x)− g(x′)|
Proof of 1. Since |f(x) + g(x)| ≤ |f(x)|+ |g(x)| holds for all x ∈ X ,
sup x∈X |f(x) + g(x)| ≤ sup x∈X (|f(x)|+ |g(x)|) ≤ sup x∈X |f(x)|+ sup x∈X |g(x)|
■
Proof of 2. Since ∣∣∣∥a∥ − ∥b∥∣∣∣ ≤ ∥a− b∥ for any norm ∥ · ∥ and for a large enough M ,
sup x,x′∈X |f(x)− g(x′)| ≥ sup x∈X |f(x)− g(x)|= sup x∈X |(f(x) +M)− (g(x) +M)|
≥ ∣∣∣ sup x∈X (f(x) +M)− sup x∈X (g(x) +M) ∣∣∣
= ∣∣∣ sup x∈X f(x)− sup x′∈X g(x′) ∣∣∣
■

heading: A.2 PROOF OF THEOREM 3
text: Theorem 3. If ξt converges to 1 uniformly on Ω, then Eτ ξt also converges to Eτ uniformly on Z for all s ∈ S and a ∈ A. Proof. Recall that Z = { Z : S ×A →P(R)
∣∣ E[|Z(s, a)|] ≤ Vmax,∀(s, a)}. Then for any Z ∈ Z and ξ ∈ Ξ,
E[|τ ξZ|] ≤ Rmax + γ Rmax 1− γ = Rmax 1− γ = Vmax.
which implies PDBOO is closed in Z , i.e. τ ξZ ∈ Z for all ξ ∈ Ξ. Hence, for any sequence ξt, Z(n) = τ ξn:1Z ∈ Z for any n ≥ 0. Since ξt converges to 1 uniformly on Ω, there exists T such that for any t > T ,
sup w∈Ω |ξt(w)− 1| ≤ ϵ.
For any Z ∈ Z , s ∈ S, a ∈ A, and t > T , by using Hölder’s inequality,
sup Z∈Z sup s,a |Eξt [Z(s, a)]− E[Z(s, a)]| = sup Z∈Z sup s,a ∣∣∣∣∫ w∈Ω (1− ξt(w))Z(s, a, w)P(w)dw ∣∣∣∣
≤ sup w∈Ω |ξt(w)− 1| sup Z∈Z sup s,a ∣∣∣∣∫ w∈Ω |Z(s, a, w)|P(w)dw ∣∣∣∣ ≤ ϵVmax
which implies that Eξt converges to E uniformly on Z for all s, a.
By using A.1, we can get the desired result.
sup Z∈Z sup s,a |E[τ ξtZ(s, a)]− E[τZ(s, a)]|
≤ sup Z∈Z sup s,a |E[τ ξtZ(s, a)]− Eξt [τ ξtZ(s, a)]|+ sup Z∈Z sup s,a |Eξt [τ ξtZ(s, a)]− E[τZ(s, a)]|
≤ ϵVmax + γ sup Z∈Z sup s,a
Es′ [∣∣∣∣sup
a′ Eξt [Z(s′, a′)]− sup a′′ E[Z(s′, a′′)] ∣∣∣∣] ≤ ϵVmax + γ sup
Z∈Z sup s′,a′
|Eξt [Z(s′, a′)]− E[Z(s′, a′)]|
≤ ϵVmax + γϵVmax = (1 + γ)ϵVmax.
■

heading: A.3 PROOF OF THEOREM 4
text: Theorem 4. For a sequence of bound ∆n, let Ū∆n(Z(n−1)) := ⋂ s,a U∆n ( Z(n−1)(s, a) ) . If we
sample ξn from Ū∆n(Z(n−1)) for every iteration and ∑∞
n=1 ∆n < ∞ holds, then, the expectation of iterated operator τ ξn:1 has a fixed point E[Z∗]. Moreover, the following bound holds,
sup s,a ∣∣∣E[Z(n)(s, a)]− E[Z∗(s, a)]∣∣∣ ≤ ∞∑ k=n ( 2γk−1Vmax + 2 k∑ i=1 γi(∆k+2−i +∆k+1−i) ) .
Proof. We denote a∗i (ξn) = argmax a′ Eξn [Z (n−1) i (s ′, a′)] as the greedy action of Z(n−1)i under perturbation ξn. Also, we denote sup s,a | · | which is the supremum norm over s and a as ∥ · ∥sa.
Before we start from the term ∥∥E[Z(k+1)]− E[Z(k)]∥∥
sa , for a given (s, a),∣∣∣E[Z(k+1)(s, a)]− E[Z(k)(s, a)]∣∣∣
≤ γ sup s′ ∣∣∣E[Z(k)(s′, a∗(ξk+1))]− E[Z(k−1)(s′, a∗(ξk))]∣∣∣ ≤ γ sup
s′
( ∣∣∣E[Z(k)(s′, a∗(ξk+1)]−max a′ E[Z(k)(s′, a′)] ∣∣∣+ ∣∣∣max a′ E[Z(k)(s′, a′)]
−max a′
E[Z(k−1)(s′, a′)] ∣∣∣+ ∣∣∣max
a′ E[Z(k−1)(s′, a′)]− E[Z(k−1)(s′, a∗(ξk))] ∣∣∣ ) ≤ γsup
s′,a′ ∣∣∣E[Z(k)(s′, a′)]− E[Z(k−1)(s′, a′)]∣∣∣+ γ k∑ i=k−1 sup s′ (∣∣∣E[Z(i)(s′, a∗(ξi+1))] −max
a′ E[Z(i)(s′, a′)] ∣∣∣) ≤ γ ∥∥∥E[Z(k)]− E[Z(k−1)]∥∥∥ sa + γ k∑ i=k−1 sup s′
(∣∣∣E[Z(i)(s′, a∗(ξi+1))] − Eξi+1 [Z(i)(s′, a∗(ξi+1))] ∣∣∣+ ∣∣∣max a′ Eξi+1 [Z(i)(s′, a′)]−max a′′ E[Z(i)(s′, a′′))] ∣∣∣ )
≤ γ ∥∥∥E[Z(k)]− E[Z(k−1)]∥∥∥
sa + 2γ k∑ i=k−1 sup s′,a′ (∣∣∣E[Z(i)(s′, a′)]− Eξi+1 [Z(i)(s′, a′)]∣∣∣)
≤ γ ∥∥∥E[Z(k)]− E[Z(k−1)]∥∥∥
sa + 2γ k∑ i=k−1 ∆i+1
where we use A.1.1 in third and fifth line and A.1.2 in sixth line.
Taking a supremum over s and a, then for all k > 0,∥∥∥E[Z(k+1)]− E[Z(k)]∥∥∥ sa ≤ γ ∥∥∥E[Z(k)]− E[Z(k−1)]∥∥∥ sa + 2 k∑ i=k−1 γ∆i+1
≤ γ2 ∥∥∥E[Z(k−1)]− E[Z(k−2)]∥∥∥
sa + 2 k−1∑ i=k−2 γ2∆i+1 + 2 k∑ i=k−1 γ∆i+1
...
≤ γk ∥∥∥E[Z(1)]− E[Z]∥∥∥
sa + 2 k∑ i=1 γi(∆k+2−i +∆k+1−i)
≤ 2γkVmax + 2 k∑
i=1
γi(∆k+2−i +∆k+1−i)
Since ∑∞
i=1 γ i = γ1−γ <∞ and ∑∞ i=1 ∆i <∞ by assumption, we have
k∑ i=1 γi∆k+1−i → 0
which is resulted from the convergence of Cauchy product of two sequences {γi} and {∆i}. Hence, {E[Z(k)]} is a Cauchy sequence and therefore converges for every Z ∈ Z .
Let E[Z∗] be the limit point of the sequence {E[Z(n)]}. Then,∥∥∥E[Z∗]− E[Z(n)]∥∥∥ sa = lim l→∞ ∥∥∥E[Z(n+l)]− E[Z(n)]∥∥∥ sa
≤ ∞∑
k=n
∥∥∥E[Z(k+1)]− E[Z(k)]∥∥∥ sa
= ∞∑ k=n ( 2γkVmax + 2 k∑ i=1 γi(∆k+2−i +∆k+1−i) ) .
■

heading: A.4 PROOF OF THEOREM 5
text: Theorem 5. If {∆n} follows the assumption in Theorem 4, then E[Z∗] is the unique solution of Bellman optimality equation.
Proof. The proof follows by linearity of expectation. Denote the Q-value based operator as τ̄ . Note that ∆n converges to 0 with regularity of Z implies that ξn converges to 1 uniformly on Ω. By Theorem 3, for a given ϵ > 0, there exists a constant K = max(K1,K2) such that for every k ≥ K1,
sup Z∈Z ∥τ̄ ξkE[Z]− τ̄E[Z]∥sa ≤
ϵ 2 .
Since τ̄ is continuous, for every k ≥ K2, ∥τ̄E[Z(k)]− τ̄E[Z∗]∥sa ≤ ϵ
2 .
Thus, it holds that ∥τ̄ ξk+1E[Z(k)]− τ̄E[Z∗]∥sa ≤ ∥τ̄ ξk+1E[Z(k)]− τ̄E[Z(k)]∥sa + ∥τ̄E[Z(k)]− τ̄E[Z∗]∥sa
≤ sup Z∈Z ∥τ̄ ξk+1E[Z]− τ̄E[Z]∥sa + ∥τ̄E[Z(k)]− τ̄E[Z∗]∥sa ≤ ϵ 2 + ϵ
2 = ϵ.
Therefore, we have E[Z∗] = lim
k→∞ E[Z(k)] = lim k→∞ E[Z(k+1)] = lim k→∞ E[τ ξk+1Z(k)] = lim k→∞ τ̄ ξk+1E[Z(k)] = τ̄E[Z∗]
Since the standard Bellman optimality operator has a unique solution, we derived the desired result. ■

heading: B ALGORITHM PIPELINE
text: Figure 6 shows the pipeline of our algorithm. With the schedule of perturbation bound {∆n}, the ambiguity set U∆n(Zn−1) can be defined by previous Zn−1. For each step, (distributional) perturbation ξn is sampled from U∆n(Zn−1) by the symmetric Dirichlet distribution and then PDBOO τ ξn can be performed.
C IMPLEMENTATION DETAILS
Except for each own hyperparameter, our algorithms and DLTV shares the same hyperparameter and network architecture with QR-DQN. Also, we set up p-DLTV by only multiplying a gaussian noise N (0, 1) to the coefficient of DLTV. We do not combine any additional improvements such as double Q-learning, dueling network, prioritized replay, and n-step update. Experiments on LunarLander-v2 and Atari games were performed with 3 random seeds.

heading: C.1 N-CHAIN
text: For hyperparameter settings, we initialize all agents with a random policy for 500 steps and then train by 20K steps with 10 random seeds. ϵ−greedy policy which is only executed on QR-DQN annealed linearly from 1 to 0.01 over the first 2500 steps. The batch size was 64 and the discount factor was γ = 0.9. We update the network every 1 step and the number of steps to update targets was 25 steps.

heading: C.2 LUNARLANDER-V2
text: The hyperparameters of QR-DQN were followed by the settings reported in Raffin et al. (2019) for a fair comparison. Our experiments used 2 layers of MLP with 256 hidden units. We used the experience replay with batch size 128 and the buffer size 1 × 105. The number of quantiles N was 170 and γ = 0.995. As a stochastic gradient optimizer, we adopt Adam with a learning rate 1.5 × 10−3 with a linear decaying schedule. ϵ-greedy schedule was only performed on QR-DQN. For the rest, we used ct = 0.05, and ∆ = 5 × 104. We evaluated each algorithm for every 10K training steps by averaging 5 episodes.

heading: C.3 ATARI GAMES
text: For a fair comparison, our hyperparameter setting is aligned with Dabney et al. (2018b). The number of quantile fraction N is 200. We set γ = 0.99 and ct = 50 which refers to Mavrin et al. (2019) and ∆ = 1× 106. We use ϵ-greedy with threshold ϵ = 0.01 at training stage and ϵ = 0.001 at test stage.

heading: C.4 PSEUODOCODE OF P-DLTV
text: Algorithm 2: Perturbed DLTV (p-DLTV) Input: (s, a, r, s′), γ ∈ [0, 1) Q(s′, a′) = 1N ∑ j θj(s ′, a′)
ct ∼ c N (0, ln tt ) # Randomize the coefficient a∗ ← argmaxa′(Q(s′, a′) + ct √ σ2+(s ′, a′))
τ θj ← r + γθj(s′, a∗), ∀j Output: ∑N i=1 Ej [ρκτ̂i(τ θj − θi(s, a))]

heading: D FURTHER EXPERIMENTAL RESULTS OF ATARI GAMES
text: We test our algorithm under 30 no-op settings to align with previous works. We compare our baseline results with results from the DQN Zoo framework (Quan & Ostrovski, 2020), which provides the full benchmark results on 57 Atari games at 50M and 200M frames. We report the average of the best scores over 5 seeds for each baseline algorithms up to 50M frames. However, recent studies tried to follow the setting proposed by Machado et al. (2018) for reproducibility, where they recommended using sticky actions. Hence, we provide all human normalized scores results across 55 Atari games including previous report of Yang et al. (2019), Dopamine and DQN Zoo framework to help the follow-up researchers as a reference. We exclude Defender and Surround which is not reported on Yang et al. (2019) because of relialbility issues in the Dopamine framework.
Table 1 provides the mean and median human normalized scores across 55 Atari games. Due to the high computational cost, our algorithm was evaluated on 50M frames to provide results over as many environments as possible. It is observed that PQR shows better performance in terms of both mean and median metrics than QR-DQN. Since our method is based on QR-DQN, we would expect that PDBOO can be combined with IQN (Dabney et al., 2018a) or techniques in Rainbow (Hessel et al., 2018) as an efficient exploration method, and the performance can be further improved. Since ourmethod is based on QR-DQN, PDBOO can be combined with IQN (Dabney et al., 2018a) or the techniques in Rainbow (Hessel et al., 2018), such as double q-learning, n-step updates, dueling networks and prioritized experience replay. We would expect that PQR for efficient exploration would benefit from the additional improvement of performance as IQN and Rainbow outperform QR-DQN.

Review summaries:
The paper proposes a novel distributional RL algorithm that is neither risk-seeking (exploratory) or risk-averse (exploitatory) -- but rather risk-neutral. The paper motivates the need for risk neutrality from the family of works applying OFU (optimism in the face of uncertainty) to distributional RL -- it states that prior works in this space induce a one-sided risk tendency (risk-seeking or risk-averse) which is undesirable as it leads to “biased exploration”. An N-chain environment is initially studied where it is shown that prior works (DLTV, QRDQN)  are not able to identify the optimal action as fast as the proposed method (PQR) -- the prior works are misled by the low probability high reward side of the N-chain environment as opposed to the optimal side which provides maximum averaged reward. The paper then proposes a method for perturbing the risk-measure in risk-sensitive RL such that the resulting perturbed distributional bellman operator (PDBOO) converges to the same fixed point as the standard bellman operator (with some weak assumptions) and hence, produces a risk-neutral policy. The paper presented results in the LunarLander-v2 and Atari domain (4 Atari games selected), demonstrating that prior works with one-sided risk tendencies do not perform as well as the proposed risk-neutral method (PQR).

This paper proposes a novel strategy for performing exploration based on distributional reinforcement learning. They highlight the fact that existing approaches cannot distinguish between epistemic and aleatoric uncertainty. Then, they propose an alternative approach that leverages a random perturbation to the distribution, but then takes this perturbation to zero. They prove that this approach converges. Finally, they validate their approach in their experiments. 

This paper considers distributional reinforcement learning with the goal of seeking risk-neutral optimal policy. Prior approaches often have a convergence issue due to either being just risk-seeking or risk-averse. Towards resolving the problem, the authors propose a simple, yet effective approach by randomizing the risk criterion during the training process. On the theoretical front, the authors prove that the method converges to an optimal return distribution under certain conditions. Several experimental results also confirm the validity of the method.