Distributional Perturbation for Efficient Exploration in Distributional Reinforcement Learning
Authors: Taehyun Cho
Seungyub Han
Heesoo Lee
Kyungjae Lee
Jungwoo Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: distributional reinforcement learning
perturbation
exploration

Abstract: Distributional reinforcement learning aims to learn distribution of return under stochastic environments. Since the learned distribution of return contains rich information about the stochasticity of the environment, previous studies have relied on descriptive statistics, such as standard deviation, for optimism in face of uncertainty. These prior works are divided into risk-seeking or averse methods, which can be considered as having a one-sided tendency on risk. Unexpectedly, such approaches hinder convergence. In this paper, we propose a novel distributional reinforcement learning that explores by randomizing the risk criterion to reach a risk-neutral optimal policy. First, we provide a perturbed distributional Bellman optimality operator by distorting the risk measure in action selection. Second, we prove the convergence and optimality of the proposed method by using weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and converges to an optimal return distribution. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari games.

Under review as a conference paper at ICLR 2022
DISTRIBUTIONAL PERTURBATION FOR EFFICIENT
EXPLORATION IN DISTRIBUTIONAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Distributional reinforcement learning aims learn distribution of return under stochastic environments. Since the learned distribution of return contains rich in- formation about the stochasticity the environment, previous studies have relied descriptive statistics, such as standard deviation, for optimism in the face of uncertainty. However, using the uncertainty from an empirical distribution can hinder convergence and performance when exploring with the certain criterion that has an one-sided tendency on risk in these methods. In this paper, we pro- pose novel distributional reinforcement learning that explores by randomizing risk criterion to reach risk-neutral optimal policy. First, we provide a perturbed distributional Bellman optimality operator by distorting the risk measure in ac- tion selection. Second, we prove the convergence and optimality of the proposed method by using the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge an optimal return distribution. Finally, we empirically show that our method outperforms other existing distribution-base algorithms in various environments including Atari games.
INTRODUCTION
Distributional reinforcement learning (DRL.) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in several benchmark tasks. Its model generates the approximated distribution of returns, where the mean value implies the traditional Q- value (Bellemare et al. 2017: Dabney et all 2018b Choi et al. 2019). Learning procedure with stochasticity through return distribution is represented by parametric (epistemic) uncertainty, which due to insufficient or inaccurate data, and intrinsic (aleatoric) uncertainty, which is inherently possessed randomness in the environment. (Chow et al. 2015 Dabney et al. (018a) The learned stochasticity gives rise to the notion of risk-sensitivity, and some distributional reinforcement learn- ing algorithms distort the learned distribution to create a risk-averse risk-seeking policy.
Another way to employ the uncertainty is to design an efficient exploration method which essential find an optimal behavior with few number of trials. Optimism in the face of uncertainty (OFU) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set. Most OFU algorithms select an action with the highest upper-con confidecnn bound (UCB) of uncertainty which can be considered as the optimistic decision at the moment (Chen al. 2017 Ciosek al.) 2019). deep RL, several OFU studies often model the uncertainty explicitly through the Bayesian posterior, which is estimated by using neural networks. However, learning the representation of high-dimensional sate-action space and Bellman update simultancously Icads to unstable propagation Yang al. 2021).
On the other hand, DRL can provide more statistical information during control such as mode, me- dian, variance by addressing full characteristics of the return distribution. Despite the richness of information for return distribution, only a few DRL methods have tried to employ the benefits of distributional perspective for exploration (Tang& & Agrawal 2018 Mavrin et al. 2019 Clements jet al., 2019). To estimate the uncertainty from distributional outputs, descriptive statistics that is composed of mixture of intrinsic and parametric uncertainty can be utilized. Unfortunately, sepa-

---

Under review as a conference paper at ICLR 2022
rating these two types of uncertainty during learning is not trivial task. Mavrin al. (0119) a distribution-base OFU exploration that schedules a decaying bonus rate to suppress the effect of intrinsic uncertainty, which unintentionally induces risk-secking policy. Although OFU based ap- proaches try to reduce parametric uncertainty by revisiting the state with high uncertainty, there exists the side effect that the criteria unfortunately forces the agent to chase the intrinsic uncertainty (risk) simultaneously during updates. Relying on specific criteria causes an one-sided tendency on risk and makes an agent consistently select certain actions during exploration degrading perfor- mance. We call this phenomenon as stuckness.
propose
In this paper, we introduce perturbed quantile regression (PQR) which perturbs the criterion on un- certainty by randomizing the risk criterion in action selection. First, the distributional perturbation on return distribution to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight. Unlike the typical worst-case approach in risk-sensitive settings or OFU based approaches, we instead randomly sample a risk measure from an ambiguity set, which represents that the risk setting is ambiguous when the characteristics of given environment are unknown. We empirically demonstrate on the stochastic variant of N-Chain environment (Osband et al. 2016) that randomized scheme is more effective than OFU to alleviate the sub- optimaltll problem of tendency to obtain risk-ngeking policies. Second, any risk-measure with some time-varytn varying perturbation con- straint allows us average over all possible risk-sensitive behaviors achieve the optimal policy. We provide a sufficient condition for the convergence of return distribution the weaker contrac- tion property. Our method covers the full elements of the reinforcement learning with distributional perspective from the novel exploration method with distributional perturbation to the theoretically guaranteed convergence of return distribution, which has the same fixed-point with the standard Bellman optimality operator. The proposed algorithm is based on QR-DQN, which is a baseline of DRL architecture learn return distribution, and we show that PQR outperforms QR-DQN and other DRL baselines in several benchmark environments, LunarLander-v2 and Atari games.
2 BACKGROUNDS & RELATED WORKS
2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING
We consider a Markov decision process (MDP) which is defined as tuple (S,A, R. where
a finite state space, A is finite action space, P: S AxS - [0, 1] is the transition probability. R is the random variable of rewards in [-R Rmax]. and y E [0, 1) is the discount factor. We define a stochastic policy (-(ss) which is conditional distribution over A given state For fixed policy n,
we denote Zs(ss, a) as a random variable of return distribution of state-action pair (s, a) following the policy We attain Zd (s. a) = > y Ac), where S P(- A,), A, x-1(,) and So
Ao = a. Then, we define an action-value function as QT a) = E[2"(s, a)] in [-Vmax, m Viman where Vmax Rmax/ (11 -7). For regularity, we further notice that the space of action- value distributions Z has the first moment bounded by Vmax:
= (Z 1x:SxA1 P (R) Z a)l] S VIux, V(s, a)}.
In distributional RL, the return distribution for the fixed can be computed via dynamic program- ming with the distributional Bellman operator defined as (Bellemare et al. 2017).
T=)(s, 2 R(s.a)+ +7 ,A), S' ~' P((ssa),, A' *(|S')
(
where 2 denotes that both random variables share the same probability distribution. We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as,
T (Z) D R(s,a) + 7Z(S" aa), S' P(-ls, a" = arg argmax max Ez|Z(S',, .²)]-
a'
Bellemare et al. (2017) have shown that T F is contraction in a maximal form of the Wasserstein metric but T is not contraction in any metric. Combining with the expectation operator, ET is contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value, while the convergence of return distribution itself is not guaranteed.
2

---

Under review as a conference paper at ICLR 2022
2.2 EXPLORATION ON DISTRIBUTIONAL REINFORCEMENT LEARNING
In this section, we will briefly describe the main practical algorithm for DRL with a deep neural network, and explain the derived exploration method. To combine with deep RL, a parametric dis- tribution Zo is used to learn a return distribution by using T_ Dabney et al. (2018)) have employed a quantile regression to approximate the full distribution by letting Zp(s, a) NV E Ziit dg.(s.a) where the parameter represents the locations of mixture of N Dirac delta functions. Each 0, represents the value where the cumulative probability is T; N. Then, by using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as a minimiza- tion problem
min D (Zt, ae), T Zg- (st, az))
Σ
Σ
N |0 (re (re 100007078880000 (st+1, d')
(st, at))]
where (st, Tr, St+1) is given transition pair, F3 I=-1 a' max, Ez ( a
p (x) A ( (x.) and (x) := 2 for S and (z) := x( - (,), otherwise. Based on the quantile regression, Dabney et al. 201 8b) have proposed quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach (Belle- mare et al. 2017) since the quantile regression does not restrict the bounds for return. As deep RL typically did, QR-DQN adjusts e-greedy schedule, which selects the greedy action with probability 1 1-c and otherwise selects random available actions uniformly. The majority of QR-DQN variants Dabney et al. 2018a Yang et al. 2019) rely on the same exploration method. However, such ap- proaches do not put aside inferior actions from the selection list and thus suffers from loss (Osband et al. 2019). Hence, selecting a statistically plausible action is crucial for efficient exploration.
In recent studies, Mavrin al. 2019)) modifies the criterion of selecting an action for efficient exploration the face of uncertainty. Using left truncated variance as a bonus term to estimate optimistic way and decaying ratio e, to suppress the intrinsic uncertainty, DLTV was proposed as an uncertainty-basse exploration in DRL without using e-greedy exploration. At timestep t, the action selection of DLTV can be described as:
a" = arg max (EPlZ(s, a (o((()))) = ev
logt
N
(0 - 0,)
2N
where 0,'s 's are the values of quantile level т, DLTV shows that a constant schedule degrades the performance significantly compared to a decaying schedule.
2.3 RISK IN DISTRIBUTIONAL REINFORCEMENT LEARNING
Instead of expected value, risk-sensitive RL tries to maximize risk measure such Mean- Variance (Zhang et al. (020), Value-at- Ca-uee-t-Rissse (VaR) (Chow et al. or Conditional Value-at-Rine (CVaR) (Rockafellar al. 2000 Rigter al. 2021), which result different classes of optimal policy. Especially, Dabney et (018a) interprets risk measures as the expected utility function of the return, i.e., Ez (s,, (.,aa)) J. Under this interpretation, risk-sensitive RI can be formulated as the maximization problem with various types of utility functions. If the utility function U is linear, the policy obtained under such risk measure called risk-neutral. If U is concave or convex, the resulting policy is termed as risk-averse or risk-seeking. respectively. In general, a distortion risk measure is generalized expression of risk measure generated from the distortion function.
Definition 1. Let h [0, + [0, 1] be distortion function such that h(0) 0, h(1) 1 and non- decreasing. Given probability space (Ω, (2,FFPP) and a random variable Z Ω + IR a distortion risk measure ph. corresponding to a distortion function h is defined by:
Pi ==
(h (h)o2))
0z
where Fz is the cumulative distribution function of Z.
In fact, decreesinn property of f makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF. Note that the concavity and the convexity of distortion function
3

---

Under review as a conference paper at ICLR 2022
Start
$3
no-op
0 dg
o (oplimal)
r= parcs,o1') (0113,
Figure I: Illustration the N-Chain environment starting from state s2. To emphasize the stochas- ticity, the reward of state 94 set as mixture model composed of two Gaussian distributions. Blue arrows indicate the risk-neutral optimal policy in this MDPs.
also imply risk-averse or risk-seeking behavior, respectively, Dhaene et al. (2012) showed that any distorted expectation can be expressed as weighted averages of quantiles. In other words, generating a distortion risk measure is equivalent to choosing reweighting distribution.
Fortunately, distributional RI. has suitable configuration to apply those uncertainty-based ap- proaches that could naturally expand the class of policies. [Cow et al. (2015) and Stankoo Macek (019)) considered risk-sensitive RL with CVaR objective, where risk is related to robust decision making. Dabney et al. (2018a) 8a) expanded the class of policies on arbitrary distortion risk measures and investigated the effects of distinet distortion risk measures by changing the sampling distri- bution for quantile targets т. Unlike the usual risk-sensitive RL, DLTV applied the risk measure only action selection, while it keeps the standard objective obtain risk-neutral optimal policy. Our analysis shows that risk-based exploration can utilize risk measures in two different ways: (1) selecting action and (2) evaluating the value function by using distorted (perturbed) expectation.
3 PERTURBATION IN DISTRIBUTIONAL REINFORCEMENT LEARNING
3.1 MOTIVATION
Distribution- OFU exploration (Moerland al. 2018 Keramati al. 2020) was proposed to give a bonus for the uncertainty that can be extracted from the distribution. However, we found that keeping optimism on uncertainty tends to select sub-optimal behaviors over a long exploration. mean-standard deviation with coefficient c. Consider two actions a1, a2 with mean μ4 P2 and variance ai, d2 respectively, under the following conditions: μn 2 12, S 02, and fi C,O1 S #2 + C:02. Then, the agent prefers to select as based on OFU. To change the decision towards the true optimal action d1, the following steps need to be spent:
For example, suppose we choose a criterion based
= min (1 (1>L:cH (i S
111 u12
Hence, if there is bias in the criterion itself, such stuckness often occurs and degrades the perfor- mance since the agent does not have experience with the optimal policy during that period.
To demonstrate such shortcomings of OFU exploration in distributional RL framework, build a representative environment that is easy to interpret intuitively among the cases in which intrinsic uncertainty exists. We experiment on the stochastic variant of N-Chain environment used in Osband et al. (2016) as a toy experiment. A schematic diagram of the N-Chain environment is shown in Figure The reward is only given in the leftmost and rightmost states and the game terminates when one of the reward states is reached. We set the leftmost reward as A A(10, 0.1²) and the rightmost reward as [5, 0. + (A3313, 0.1 0.12) which has lower mean as 9 but higher variance. The agent always starts from the middle state $2 and should move toward the leftmost state 8o to achieve the greatest expected return. For each state, the agent can take one of six available actions: left, right, and 4 no-op actions. The optimal policy with respect to mean is to move left twice from the start. Despite the simple configuration, the possibility to obtain a higher reward in the suboptimal state than the optimal state makes the agent difficult which policy is optimal until it experiences enough to detect the characteristics of each distribution. Thus, the goal of our toy experiment to evaluate how quickly each algorithm could find risk-neutral optimal policy.
4

---

Under review as a conference paper at ICLR 2022
POP of 2032
2.00
20000
QRDON
17500
DLTV
seep
P-DLTV
Total Count
PDF of 2032
12500
10000
L5K depa
7500
5000
menn 2000 1.00
20K
5000
10000
15000
20000
0.
Steps
14
(b)
Figure 2: (a) Empirical return distribution of DLTV during training in N-Chain environment. The dashed lines denote the exact mean, and the dots on the x-axis denote the perturbed mean of each action. No-op actions are not shown for visibility. (b) Total count of performing true optimal action. The oracle (dashed line) to perform the true optimal action from start end.
Rather than applying decaying schedule to suppress intrinsic uncertainty, we naively modify the optimism into a randomized risk criterion and named the perturbed variant as P-DLTV. We compare QR-DQN and DLTV with our randomized variants of two algorithms PQR and p-DLTV examine the effect of randomized risk criteria. In short, P-DLTV is a straightforward modification of DLTV where randomness is given to the coefficient Cr through normal distribution. We will describe our main algorithm, PQR, in detail in Section 3.4
In Figure aa... DLTV fails estimate the true optimal return distribution of action d1 Due the erroneous estimation, the agent takes longer to recognize its error. Hence, the deterministic selection based on a fixed criteria could mislead toward exploitation rather than exploration. This indicates that DLTV may not gather experiences well in a complex environment that requires deep exploration. In Figure 2 b). we plot the point when the optimal policy was actually performed for each algorithm to show stuckness. Since the optimal policy consists of only the same index @1, we plot the total count of performing the optimal action with different seeds. The slope indicates the ratio of performing the optimal policy on average. Hence, the interval with slope of 1 implies that the optimal policy was performed every time. From the slope of each line, it is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed. Although the mean return of 01 (move left) is estimated to be superior, the agent only selects d2 (move right) during training due its consistent optimism on uncertainty. Even if DLTV has spent enough number of time steps to choose the true optimal policy, the remaining procedure is already close to greedy selection as it starts from a decreased coefficient.
Surprisingly, P-DLTV alleviates the stuckness early by randomly choosing a plausible action. We also propose a theoretically guaranteed algorithm to be converged, PQR which shows a much steeper line by quickly obtaining the optimal policy. In Section 3.2 we derive the first theoretical sufficient condition for the convergence of exploration method for DRL, which implies that PQR has the same unique fixed point as the standard distributional Bellman equation.
3.2 PERTURBED DISTRIBUTIONAL BELLMAN OPTIMALITY OPERATOR
To choose statistically plausible actions which may be maximal for certain criteria. are interested in generating a distortion risk measure involved in pre-defined constraint set called an ambiguity set. The ambiguity set, originated from distributionally robust optimization (DRO) literature, is family of distribution characterized by certain statistical distance such as ó-divergence or Wasser- stein distance (Esfahani & Kuhn 2018 Shapiro al., 2021). In this paper, will examine the ambiguity set detined by the discrepancy between distortion risk measure and expectation. We say the sampled reweighting distribution & as (distributional) perturbation and define it as follows:
Definition 2. (Perturbation, Perturbation Gap, and Ambiguity Set) Given a probability space (,,F, PP), ler X Ω. R. be random variable and = (E ((w) 0, Saee ((w)dw
5

---

Under review as a conference paper at ICLR 2022
be set of probability density functions. For given constraint set U E. say U as (distributional) perturbation from and denote the & w-eightted expectation of X as follows:
/ MED
EE [
((w){(u))((w)dd
which can be interpreted as the expectation of X under perturbed probability distribution &T. We
further define E |E[X] [x(XX] as perturbation gap of X with respect to &. Then, for
given constant A N O, we define the ambiguity set with the bound A as
Ua(x) = (5 :(::)) $ A
=
For brevity, we omit the input from random variable unless confusing. Since & is probability density function, Eg [X] is an induced risk measure with respect to a reference measure P. Intuitively, {(w) can be viewed as a distortion to generate different probability measure and allow to vary the risk tendency. The aspect of using distortion risk measures looks similar to IQN (Dabney et al.) 2018ap. However, instead of changing the sampling distribution of quantile level implicitly, we reweight each quantile from the ambiguity set. This allows to control the maximum allowable distortion with bound A, whereas in IQN the risk measure does not change throughout learning. In Section 3.4 will suggest practical method to construct the ambiguity set.
Now, we characterize perturbed distributional Bellman optimality operator T for a fixed perturba- tion & (Z) written as below:
T Ts 2 R(s,a) ZZS (E))), S'~ P (l)s,,), a" aE( arg Eg.p[Z(s', a')].
a'
Notice that f = corresponds to a base expectation, i.e., Eg, = Ep, which recovers the standard distributional Bellman optimality operator T. To understand the progress of PDBOO intuitively, we add the pipeline figure in Appendix A.4
Specifically, updating via Tg can be described in steps. Select the superior action in terms of an arbitrary distortion risk measure instead of mean. We specify the distortion risk measure by sampling the perturbation from the ambiguity set. 2) Compute the return distribution through dynamic programming which follows the standard Bellman optimality equation. In risk-sensitive DRL or distributionally robust RL, the Bellman optimality equation is reformulated for pre-defined risk measure (Chow et al.) 2015; Smirnova et al., 2019, Yang 2020). In this sense, PDBOO has a significant distinction that performs dynamic programming that adheres to the rist-neutral optimal policy while randomizing the risk criterion at every step.
If we consider the tine-varying bound of ambiguity set, scheduling A, is a key ingredient to deter- mine whether PDBOO will efficiently explore converge. The following lemma plays important role to guarantee the convergence of PDBOO.
Lemma 3. If &, converges to uniformly on Ω then ET also converges to ET uniformly on Z for all s E S and a E A.
Intuitively, an agent continues to sample the distortion risk measure from a fixed ambiguity set with constant A, there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence. Hence, will be crucial to schedule a constraint of ambiguity set properly each action selection guarantee convergence.
Based on the quantile model Za, our algorithm can be summarized into two parts. First, we aim to minimize the expected discrepancy between Ze and T 7 where & is sampled from ambiguity set Us. To clarify notation, we write Ee[-] as a w-weighted expectation and (g as an expectation with respect E which is sampled from Us. Then, our goal minimize the perturbed distributional Bellman objective with sampling procedure G::
min Ee (8, a), T Zo- [s)
(s,
where the Huber quantile loss discrepancy Zø. and T TeZo- at timestep t. It different from DRO which performs the worst-case optimization by using a minimax objective. By using expectation instead of max operator, investigate risk-neutral exploration that can avoid overly

---

Under review as a conference paper at ICLR 2022
pessimistic policies. Second, considering sequence & which converges uniformly to 1 so that T converges uniformly to original T, we derive a sufficient condition of A, that the expectation of iterated operator ET En:1 ET &n T. E--1 ... TT. &i has unique fixed point with the same solution as the standard solution.
3.3 CONVERGENCE OF THE PERTURBED DISTRIBUTIONAL BELLMAN OPTIMALITY OPERATOR
In this section, we will provide the theoretical result of PDBOO about its convergence through [])(")] where the iteration procedure denoted as z(++1) == Ts.+1 Z(n) and Z = Z for each timestep > O.
Theorem 4. For sequence of bound An, let Us (z(n-1)) = Na,a Us.. (Z( (s.)). If we
sample En from U.D, (Z(n-1)) for every iteration and Ex=1 An < Do holds, then, the expectation of iterated operator T E.n has a fixed point EE Moreover, the following bound holds,
sup 8,0
E B (m)
(s. a)] - E| a)l &
Σ
2- max + + +2Σ A(A4221 ( AOTL-)).
Theorem states that T Enc converges while it does not satisfy the y-contraction property. Note that the fixed point EE[2*] is not yet guaranteed to be unique for any Z E Z. Fortunately, we could show that E 2 is, fact, the solution of the standard Bellman optimality equation which is already well known to have unique solution.
Theorem 5. If } follows the assumption Theorem 4 then E[Z*] is the unique solution of Bellman optimality equation.
From two theoretical results, PDBOO is guaranteed to have the same unique fixed point as the standard Bellman operator under weaker contraction property. Unlike other distribution-based or risk-sensitive approaches, PDBOO is a novel operator having compatibility for obtaining risk- neutral optimal policy. Also, it is also observed that this randomized method is more efficient than the OaU-based exploration by quickly alleviating from sub-optimal policies.
3.4 PERTURBED QUANTILE REGRESSION FOR DEEP Q-NETWORK
We propose perturbed quantile regression (PQR) that is practical algorithm for distributional reinforcement learning. We model a return distribution using quantile regression and update the quantile model by minimizing the objective function 4 induced by PDBOO. To compute the target distribution of (T). we propose sampling method of & from ambiguity set Us. Since we employ a quantile model. sampling reweight function can be reduced into sampling an N N-dimensional
weight vector & := [EI. ... Ex] where Σ & N and & 2 0 for all {1. ,N.. Based on the QR-DQN setup, note that the condition Swen ((w)P(w)ddr the quantile level is set as T; N
turns into M 1 since
A key issue is how to construct an ambiguity set with bound A and then sample &. A natural class of distribution for practical use the symmetric Dirichlet distribution with concentration B, which
Algorithm 1: Perturbed QR-DQN (PQR)
Input: transition (s, a, r., 8'), discount 7 E [0, 1), timestep >> 0, epsilon e > 0, concentration B Initialize Ao > 0
A, < Oot (1+c)
t max N + A.Nx - where Dir(B) # Sample & &~ Us, (Z(+))
І + E;
a² + arg max [z d'))
Select greedy action with distorted expectation
T0, &- r r (0,0(8'000 Vj
++++1
Output:
(TO, (,(s
7

---

Dir(33, and Under review as a conference paper at ICLR 2022
202
250,
A
Ar
(A (9)
(0 mg)
QR DR-DON
1.23
0.01
DLIV
-1.02
1.01
VCP 275.
P-DLTV
0.38
(OR(s
-003
40.01
(a) QR-DQN
(b) P-DLTV
(c) (QR()
Figure 3: (Left) Empirical return distribution plot in N-Chain environment. Since QR-DQN does not depend on other criterion, the dots omitted. (Right) Mean and standard-deviation difference between each algorithm and ground truth distribution.
represents distribution over distributions. (i.e. x (3)( If 3 is small, then, most of the mass is concentrated on a few elements. Otherwise, all elements are similar to each other and produce evenly distributed weight. By using the Dirichlet distribution,
sample random vector,
:= 1º a(Nx )) From the construction of we have 1-a a-a I & S 1 + (N - 1) for all and it follows that |1 - Eil I a(N - 1) for all i. By controlling a, we can bound the deviation of &, from 1 and bound the perturbation gap as
define the reweight distribution
sup 8,a
[[ a)] E& (s, a)]l = =sup
Z(w; 8, a) - E(w))P(w)ddu
8,0
S supp |1 [(w) sup (s) < sup |1 - [(wx)Vmmx S a(N - 1) Vmxx.
WER
8.
wes
[N- is sufficient to obtain d(Z;E) < A in the quantile setting. We set 33 = 0.05 1N to generate a constructive perturbation En which gap is close to the bound An. To satisfy the condition stated in Theorem 4 we set A, = Aut (1+++) where Ag is a hyperparameter. The detailed procedure of the proposed method is summarized in Algorithm
Hence, letting
4 EXPERIMENTAL RESULTS AND DETAILS
In this section, we evaluate the performance on N-Chain, LunarLander and Atari games to demon- strate the efficiency of randomized criteria, comparing with QR-DQN (e-greedy) and DLTV (OFU).
N-Chain. As the mean of each return is designed to be similar, it is useful to examine the learning behavior of the empirical return distribution for each algorithm. Figure 3 shows the empirical PDF of return distribution by using Gaussian kernel density estimation. PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation. Although the op- timal policy was performed, QR-DQN overestimates the optimal Q-value of (82,a1) jú 9.33 while the ground truth is computed as = 10,2 = 8.1. PQR estimated the target relatively better compared QR-DQN as the agent often chooses an alternative action from a risk-neutral per- spective. Only by chainging from optimism to a randomized scheme, P-DLTV made much better estimate than DLTV.
LunarL Lander-v2. Next, we evaluate the performance on Ladder22 environment where its input is given by 8-dimensional coordinates with different discrete actions. The goal of the agent is to reach the landing pad with given threshold of 200 points without crashing. Without obtaining the extra point of +100, the agent cannot reach the threshold.
Figure 4(a) shows that P-DLTV and PQR have reached the threshold 200 points faster than the other two algorithms. Significantly, P-DLTV successfully reached the goal just by randomizing the criterion from OFU while DLTV did not learn to obtain the extra point. The result shows that one- sided tendency on risk leads to learning failure of an agent. We note that the randomized risk criterion could alleviate the stuckness within affordable time budget.
8

---

Under review as a conference paper at ICLR 2022
LunarLander-v2
Hitting Time (M)
QR-DQN
0.23
102
DLTV
N.A
GRDEN
P-DLTV
0.18
-100
5.LTTV PQR
PQR(ours)
0.16
622
0.10
Sceps (Million)
(a)
(b)
Figure 4: (a) Evaluation curves on LunarLander-v2. All curves are the average of three random seeds and the shaded area represents the standard deviation. We smoothed the curve over consecutive steps. (b) Hitting time to reach a given threshold.
Atari Games. Lastly, we evaluate the performance in Atari games, including multiple environ- ments, each of which contained intrinsic uncertainty in different ways.
Gravitar-va
Kangaroo-vd
Pong-vd
Seaquest-vd
Averggt Score
1740
Avera Score
3120K DLTV
1000
60LTVV
15
200
Training (Millon)
Traniing
50
Uhe
Figure 5: Evaluation curves Atari games. We smoothed all curves over 10 consecutive steps with three random seeds. case Pong-v4, we resize the x-axis, since it can easily obtain the optimal policy with few interactions due to its environmental simplicity.
We plot the average performance of each algorithm in the evaluation step on 200M frames with 3 random seeds. In Figure PQR achieves the shortest time to hit the asymptotic highest performance in various Atari games. Moreover, proposed method is stable with relatively low variance in all four environments. As the bound perturbation gap converges to O, the plausible action chosen from the shrinking ambiguity set, and thus the agent gradually performs only the optimal policy. Further experimental details are Appendix
In all experiments, just adding randomness to the coefficient C, shows the significant improvement supporting that the randomized risk criterion was superior to OFU in distributional RL. In the ma- jority of environments where intrinsic uncertainties exist, OFU has difficulty in making a decision that matches risk-neutral purpose, because two uncertainties are intertwined during learning. We observe that PQR outperforms the baseline by showing faster convergence without degrading the performance, All experimental results support that the randomizing scheme is more effective than OFU in distributional RL.
CONCLUSIONS
In this paper, we proposed general framework of risk-based exploration which captures the char- acteristics of return distribution. Without resorting to pre-defined risk criterion, we revealed and resolved the stuckness where one-sided tendency on risk can lead to biased action selection. To our best knowledge, this paper is the first attempt in DRL to integrate risk-sensitivity and explo- ration by using time-varying Bellman objective with theoretical analysis. We prove that PDBOO is theoretically guaranteed for convergence, and has a unique fixed point in weaker condition than contraction. By obtaining the same fixed point as the standard Bellman optimality operator, we can obtain risk-neutral optimal policy even if the agent explored with distortion risk measure. Our method shows better empirical results compared to recent exploration methods for distributional RI. with various environments.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, pp. 449-458, PMLR, 2017.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.
Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http://
06110
Richard Y Chen, Szymon Sidor, Pieter Abbeel. and John Schulman. Ucb exploration via q-
ensembles. arXiv preprint 10x.:1706.0100, 2017.
Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional deep reinforcement learning with mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9791-9797. IEEE, 2019.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. arXiv preprint arXiv:1506.0211 2015.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained
inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critie. arXiv preprint 191..v:111..12000 2019.
William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09633. 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distri- butional reinforcement learning. In International conference on machine learning, pp. 1096-1105, 105. PMLR, 2018a.
Will Dabney, Mark Rowland. Mare Bellemare, and Rémi Munos. Distributional reinforcement learn- ing with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018b.
Jan Dhaene, Alexander Kukush, Daniël Linders, and Qihe Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal, 2(2):319-322 2012.
Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization
using the wasserstein metrie: Performance guarantees and tractable reformulations. Mathematical Programming, 171(1)111-1166 2018.
Matteo beessel,Jssepe Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1436-4443 2020.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning, pp. 4424-44444 4434. PMLR, 2019.
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. The potential of the return distribu- tion for exploration in rl. arXiv preprint arXiv: 806.0424,, 2018.
10

---

Under review as a conference paper at ICLR 2022
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances neural information processing systems. 19:4026-40344 2016.
Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., --2, 2019.
John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020.
Antonin Raffin, Ashley Hill, Maximilian Emestts, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3, https: -tt.ccooo///eeeeee RM/sttalleeeeeee wtt.aa/////ttbbeeaaaaaeaeee wawllle-bbaeiineees 2019.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. arXiv preprint arXiv:2102.05762, 2021.
Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21 42, 2000.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. programming: modeling and theory. SIAM, 2021.
Lectures on stochastic
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcementt learning. arXiv preprint arXiv: www::1902.000 1902.08708, 2019.
Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: evar opti-
mization approach. In IJCCI, pp. 412-423, 2019.
Yunhao Tang and Shipra Agrawal. Exploration by distributional reinforcement learning. arXiv
preprint Xiv: 10XXx:1800..0. 888.00559777 2018.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie- Yan Liu. Fully parameterized quan- tile function for distributional reinforcement learning. Advances in neural information processing systems, 32:6193-6202, 2019.
Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions on Automatic Control, 2020.
Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, and Peng Liu. Exploration in deep reinforcement learning: A comprehensive survey. arXiv preprint arXiv:2109.06668, 2021.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. arXiv preprint Xiv:20000 10888 2020.

---

Under review as a conference paper at ICLR 2022
A PROOF
A.1 TECHNICAL LEMMA
Before proving our theoretical results, we present two inequalities for supremum to clear the de- scription.
1. supif(a) + g()) s (ap)f(=)) + suplg(x))
=eX
zEX
#EX
2. sup (xp((x)= - sup
sup g(x')
sup x,,'Ex
(f(z) - g(r')l
=ex x'EX
Proof of 1. Since (s(z) + g(x)و < [((x) + [g(rو|
holds for all x X.
sup
[f(z) g(0هو
sup sup()f()) ls()))) sup [f(z)] sup [g(x)]
rex
=ex
mex mex
Proof 2. Since ||a|| - ||a|| S ||a - bil for any norm
|| || and for a large enough M,
sup [f(x) - g(x')) 2 xex
sup f(x) -(x))= sup (f/())(()))))))))))( M) (g(x) M)I
x
=ex
sup (f + M)
sup (0(x) + M)
=ex
=ex
sup f(x) - sup g(x')
=EX
x'EX
A.2 PROOF OF THEOREM 3
Theorem 3. If & converges uniformly on f2 then ET e, also converges to ET uniformly on Z for all 8 E S and E A.
Proof. Recall that Z =Z: (R) E Z(s, s Vmax, v(s,a)). Then for any Z Z
and & E
Rmax
Ryas
Ei[T [ < Rmax + + 1-7 1
Vmax-
which implies PDBOO is closed Z, i.c. TgZ E Z for all g E. Hence, for any sequence &.. (n)) = T Z E Z for any n 2 0.
Since converges uniformly Ω there exists 7 such that for any >> T,
sup (w) S e.
weR
For any Z, s E S, E A, and >> T. by using Hölder's inequality,
/ WEB
sup sup [Eg, [Z(s E[Z(s, = sup sup ZEZ 5.0
(1 - (((w))Z(ss a, w)))w
zez 8.4
If
< sup [5((w) - 1| sup sup weft
17 w) (P)
ZEZ a.a
TEe
< eVina
which implies that E&g converges to I. uniformly 2 for all 8, a.
12

---

Under review as a conference paper at ICLR 2022
By using A.1
we can get the desired result.
sup sup ZEZ s,a
[
[
I sup sup ZEZ a,
| e. [[, a)] - Ee,, (T [s, a)]l + sup sup [T (T. (s, [(T2(ss0)]
ZE2 s,a
<Vmax + suu sup Eg ZEZ 5.a
sup Eg, [Z(s', ad'] - E[Z a')
a'
S max + ssu sup ZEZ 8'
EEg, Fz.lz a')l (T
I emmax + +7eVmax
(1 + Vmax
A.3 PROOF OF THEOREM 4
Theorem For a sequence of bound An, let Us, (Z(n-1)) Ne,a Us. (z, a)). we sample E. from Us, (Z()-1) for every iteration and Σm= An < oo holds, then, the expectation of iterated operator T Te.. has a fixed point B[Z*]. Moreover, the following bound holds,
2 AATI-))
E[ -.a)-BBB2 E(
27 2 Vmax + 2
sup 8.0
k=n
1=1
Proof. We denote at (Sn) = arg max E,,, [2 (2(11)) (8', (5', a')] as the greedy action of 2!
z((-1) under
a'
which is the supremum norm over s and a as || Ilsa- perturbation &n. Also, we denote supl
8,8
Before we start from the term
[[ |[ sa* for given (s,
E[z (1+)) (s, a)] (s a)]
$ sus
a E|Z (s', a (GA-1)) - (( (s',a"(Ex)) (Ex)))
( (8, a (6x+1) - max E (k) (k) (5) .a>]] + max EE (Z a')]
-max - (5', n')l
max | (5' a')] (|z(k-1) (5' a (Ex))
suss ([
(s', d')] E|2(k 135 (')5'a))] a')l
+7 Σ sup
(18ZZ a" (E++1))
s'
xax B|Z (1) (5', d')
S 7 B[ 11
sup +7 Σ
s'
1=
- Eg, 10 (5', a² (Ex+1))]
+ max Egg (2(0) (s',a') - max E (
(s', |)]
a"
<> B[ ([ 13|
+ +2 Σ sup
( (s',a') Eg.,, [2 a²) a')l)
d'
i=
<> B[Z(N]] BiZ 11
+ 2y
A141
i=
where we use A.1 in third and fifth line and A.1 2 in sixth line.
13

---

Under review as a conference paper at ICLR 2022
Taking supremum over and a, then for all lo > 0,
k +2 Σ
s < [||11|/011 -
>4++1
an
i=k-1
k-1
53² E| - Elz i 2)1
+ Σ
22
A +2
Σ
A
h=k-2
k=
:
+2 +2Σ k
~ 11220] - -F|
(A A
< Vmax + 2
AA + A++1-))
1=1
Since E Ea 7 xo and E A oc by assumption, we have
Σ
7'A41-:
which is resulted from the convergence of Cauchy product of two sequences {7²) and (A:). Hence, [E[ (+)) is a Cauchy sequence and therefore converges for every Z E 2.
Let E [ be the limit point of the sequence {E[Z
Then,
E l - E
lim
E (1
(x)
10000
E [[z E[Z(())
k=n
2 + 2
Σ 7 A Aut--i)).
k
1=1
A.4 PROOF OF THEOREM 5
Theorem 5 If (An) follows the assumption in Theorem 4 then R[Z*] is the unique solution of Bellman optimality equation.
Proof. The proof follows by linearity of expectation. Denote the Q-value based operator as T. Note
that An converges with regularity of Z implies that &n converges to 1 uniformly Q. By Theorem 3 for a given > O, there exists a constant K = max(K, K1, K2) such that for every k > K1.
sup | c. -[ -TEZ) ZEZ
sa S 2
Since T is continuous, for every t 2 K2,
|| [ -
2
Thus, it holds that
|| 1 [[
T < T Ex 1 B[2(k)] J112 AM + |
=0 sup IT eut, E 7E/Z111 AM [T TE| 11
-
Z22
2 +
2
14

---

Under review as a conference paper at ICLR 2022
Therefore, have [|Z"] lim k+000 Since the standard Bellman optimality operator has unique solution, we derived the desired result.
1 lim [Tz( ビ-v00
lim T k-000
Ex-1 2
lim T ccعs k--oc
TE
B ALGORITHM PIPELINE
Zo
11
2n-1
Tea
Zn
Z
f.
Ma (Zn)
A
Us. (2n-1)
Figure Pipeline of PDBOO.
Figure shows the pipeline of algorithm. With the schedule of perturbation bound {An): ) the ambiguity set Us (7n-1) can be defined by previous Zn-1- 1. For each step, (distributional) perturba- tion En is sampled from Us., (Zn-1) by the symmetric Dirichlet distribution and then PDBOO Te.. can be performed.
C IMPLEMENTATION DETAILS
Except for each own hyperparameter, our algorithms and DLTV shares the same hyperparameter and network architecture with QR-DQN, Also, we set up P-DLTV by only multiplying a gaussian noise (0, 1) the coefficient of DLTV. We do not combine any additional improvements such double Q-learning, dueling network, prioritized replay, and n-step update. Experiments on LunarLander-v2 and Atari games were performed with 3 random seeds.
C.1 N-CHAIN
For hyperparameter settings, we initialize all agents with a random policy for 500 steps and then train by 20K steps with 10 random seeds. c-greddy policy which is only executed on QR-DQN annealed linearly from to 0.01 over the first 2500 steps. The batch size was 64 and the discount factor was 7 = 0.9. We update the network every step and the number steps to update targets was 25 steps.
C.2 LUNARLANDER-V2
The hyperparameters of QR-DQN were followed by the settings reported Raffin et al. (2019) for fair comparison. Our experiments used 2 layers of MLP with 256 hidden units. We used the experience replay with batch size 128 and the buffer size 10 The number of quantiles N was 170 and y 0.995. As stochastic gradient optimizer, 1.5 x 10 33 with a linear decaying schedule, e-greedy schedule was only performed on QR-DQN. For the rest, used training steps by averaging 5 episodes.
adopt Adam with learning rate
0.05, and A 5 x 10 We evaluated each algorithm for every 10K
C.3 ATARI GAMES
For fair comparison, hyperparameter setting aligned with Dabney al. (008b)). The number of quantile fraction 200. We set y = 0.99 and c, = 50 which refers to Mavrin et al, (2019) and 106. We use e-greddy with threshold 0.01 training stage and 0.001 test stage.
A
15

---

Under review as a conference paper at ICLR 2022
C.4 PSEUODOCODE OF P-DLTV
Algorithm 2: Perturbed DLTV (p-DLTV)
Input: (8, a, r')), 10 Q(s', = N Σj (s',
Ct NN b
Randdmiize the coefficient
a² + arg max (Q(s', a') + aay (+)
T0 f- r +00 (b'()',))), Vj
Output: E/= Bylog (TO, - (s,,)))
D FURTHER EXPERIMENTAL RESULTS OF ATARI GAMES
We test our algorithm under 30 no-op settings to align with previous works. We compare our baseline results with results from the DQN Zoo framework (Quan & Ostrovski (020). which provides the full benchmark results on 57 Atari games at 50M and 200M frames. We report the average of the best scores over 5 seeds for each baseline algorithms up to 50M frames. However, recent studies tried to follow the setting proposed by Machado et al. 2018 for reproducibility, where they recommended using sticky actions. Hence,
provide all human normalized scores results across 55 Atari games including previous report of Yang et al. (2019) Dopamine and DQN Zoo framework to help the follow-up researchers reference. We exclude Defender and Surround which is not reported on Yang et al. 2019) because of relialbility issues in the Dopamine framework.
Mean
Median
D0N-dopamine(50NN
401% %
51%
DON~zoo(50M)
507%
69%
D0N-zoo/200M)
804%
84%
DQN(200
221%
79%
NR-DDO--dopaminee5500
562%
93%
OR-DON-zoo(50M))
707%
119%
200D)
1714%
174%
QR-DQN(200M)
902%
193%
IQN-dopaminc(50M)
940%
124%
IQN-200(50M)
1057%
143%
IQN-2o0(200M)
2070%
229%
IQN(200M)
1112%
218%
RAINBOW- MANNBOW-dopamine(500
965%
123%
RAINBOW-zoo
1366%
154%
RAINBOW-zoo/ 200M)
2115%
246%
RAINBOW(2WMM
1213%
230%
PQR(50M)
1121%
124%
Table 1:
Mean and median of best scores across 55 Atari games on 50M frames, measured as per-
centages of human baseline (Castro et al. 2018 Yang et al. 2019 Quan Ostrovskil 2020)
Table provides the mean and median human normalized scores across 55 Atari games. Due to the high computational cost, our algorithm was evaluated on 50M frames to provide results over as many environments as possible. mean and median metrics than QR-DQN. Since our method is based on QR-DQN, we would expect that PDBOO can be combined with IQN (Dabney al.) 2018a) or techniques in Rainbow (Hessel et al. 2018) as efficient exploration method, and the performance can be further improved. Since ourmethod based on QR-DQN, PDBOO can be combined with IQN (Dabney et al.) (0T8a) or the techniques in Rainbow (Hessel al. 2018), such double q-learning, n-step updates, ducling networks and prioritized experience replay. We would expect that PQR for efficient exploration would benefit from the additional improvement of performance IQN and Rainbow outperform QR-DQN.
observed that PQR shows better performance in terms of both
16

---

Under review as a conference paper at ICLR 2022
GAMES
RANDOM
HUMAN
DQN(OM)
QR-DON(SIM)
(QN(50M)
RAINHOW(5OM)
PQR(SOM)
Alien
227.8
7127.7
1633,9
1891.7
1991.0
4507.8
2455.8
Amidar
5.8
1719.5
344.6
761.2
853.6
2649.4
938.4
Assanlt
222.4
742.0
3744.7
13951 4
18910.2
11921.0
10759.2
210.0
Astemids
719.1
47388.7
1590.9
2113.3
2305.9
1917.8
1662.0
Atlantis
12850.0
29028.1 1
383243.6
962130.0
907430.0
954790.0
897640.0
BankHeist
14.2
753.1
477.1
1267.9
1298,1
1089.8
1038.8
BattleZone
2360.0
37187.5
22167.1
31242.0
33382.4
36797.6
28470.5
BeamRider
363.9
16926.5
8276.5
14710.0
23600.8
17280,9
10224.9
Berzerk
123.7
2630.4
644.5
936.0
960.9
2191.4
137873.1
Bowling
49.2
58.5
64.9
Boxing
0.1
12.1
90.2
99.7
99.6
99.9
97.1
Breakout
1.7
350.0
478.6
556.8
349.4
380.3
Centipede
2090.9
12017.0
6912.0
9334.4
8124.0
6055.8
7291.2
ChopperCammand
811.0
7387.8
1081.7
1453.6
2054.5
6277.6
1300.0
CrazyClimber
10780.5
3
109112.5
111783.3
1344666.5
168808.9
84390.9
DemonAAiack
152.1
1971.0
9695.5
116973.3
115749.8
98203.8
73794.0
DoubleDunk
-7.0
-7.6
-7.5
Enduro
0.0
860.5
850.5
2238.6
2345.7
2350.7
2341.2
Derby
91.7
-38.7
15.2
33.4
27.0
38.5
31.7
Freeway
0.0
29.6
24.6
33.5
33.8
34,0
34.0
Frostbite
65.2
4334.7
614.8
4346.2
6187.6
9902.2
4148.2
Gopher
257.6
7425.6
34520.2
37955.5
47054_5
Gravitar
173.0
3351.4
271.2
617.6
535.9
2253.4
635.8
Hero
1027.0
30826.4
10821.5
11823.4
13628.9
12579.2
lceHockey
-11.2
0.9
4.1
-1.0
-2.6
2.5
-1.4
Jamesbond
29.0
302.8
524.3
1411.5
1336.8
15191.
2121.8
Kanguron
52.0
3035.0
8146.9
14885.9
14504.8
14670.3
14617.1
Krull
1598.0
2665.5
10626.8
11004.2
10028.9
9055.3
9746.1
KungFuMasee
258.5
22736.3
25251.6
43258.6
MontezumaRevenge
0.0
4753.3
0.4
0.0
0.2
80.3
0.0
MsPacman
307.3
6951.6
2427.6
3385
3075,4
3894.8
2928.9
Name ThinGane
2292.3
8049.0
7260.8
12595.6
12538.4
11609.4
10298.2
Phoemix
761.4
7242.6
12646.5
37830.1
37153.7
61410.6
20453.8
Pitfall
-29.44
6463.7
0.0
Pong
20.77
14.6
19.2
21.0
21.0
21.0
21.0
PrivateFyye
69571.3
372.4
Qbert
163.9
13455.0
654
14885 8
1568 1.3
26067.9
15267 4
Riverraid
1338.5
17118.0
7467.2
10019.4
14311.3
17785.7
11175.3
RoadRunner
11.5
7845.0
32714.0
55506.2
55873,9
54265.5
50854.7
Robotank
2.2
11.9
35.7
$6.6
55.5
67.6
60.3
Seaquest
68.4
42054.7
3239.3
8990.9
18943.3
3057.5
19652.5
Skiing
117098.
4336.9
13445,9
9198.1
9310.5
14901.8
9299.3
Solaris
12326.7
2640.0
SpaceInvaders
148.0
1668.7
1314.2
1960.3
3058.2
2746.3
1749.4
Star Gunner
664.0
10250.0
30631.9
63950.3
81730.5
111760.9
62920.6
Tennis
-23.8
.8.3
0.0
0.0
0.0
0.0
-1.0
TimePilot
3568.0
5229.2
3439.9
7835.8
6971.4
12731.9
6506.4
Tutankbam
11.4
167.6
192.8
229.2
209.0
231.3
UpNDowwn
533.4
11693.2
11 11192.4
45653.5
65773,2
77524.5
36008. 1
Venture
154.2
17.5
54,1
993.3
VidcoPinball
16256.9
17667.9
258626.0
182452.1
416941.4
589122.4
465578.3
Wiz
563.5
4756.5
4914.8
19040.8
15892.2
13362.6
6132.8
YarsRevenge
3092.9
54576.9
18540.6
23385.8
23115.6
69092.9
27674.4
Zaxxon
32.5
9173.3
4890.3
9632.6
8686.6
23620.4
10806.6
Table 2: Raw scores for single seed across all 55 games, starting with 30 no-op actions. We report the best scores for DQN, QR-DQN, IQN and Rainbow on 50M frames, averaged by 5 seeds. Refer- values were provided by DQN Zoo framework (Quan & Ostrovski 2020)
17

---

Under review as a conference paper at ICLR 2022
GAMES
RANDOM
HUMAN
DQNGOM)
QR-DON(SIM)
(QN(50M)
RAINHOW(5OM)
PQR(SOM)
Alien
227.8
7127.7
1688.1
2754,2
4016.3
2076.2
2455.8
Amidar
5.8
1719.5
888.2
841.6
1642.8
1669.6
938.4
Assanlt
222.4
742
1615.9
2233.1
4305.6
2535.9
10759.2
210
Astemids
719.1
47388.7
$28. 2
1333
1336.3
1345.
1662.0
Atlantis
12850.0
29028.1 1
388466.7
879022.0
897558.0
870896.0
897640.0
BankHeist
14.2
753.1
720.2
964.1
1082.8
1104.9
1038.8
BattleZone
2360.0
37187.5
15110.3
25845.66
29959.7
32862.1
28470.5
BeamRider
343.9
16926.5
4771.3
7143.0
7113.7
6331.9
10224.9
Berzerk
123.7
2630.4
529.2
603.2
627.3
697.8
137873.1
Bowling
Boxing
0.1
12.1
80.0
96.6
97.8
96.3
97.1
Breakout
113.5
40.7
164.4
69.8
380.3
Centipede
2090.9
12017.0
3403.7
3562.5
3746.1
5087.6
7291.2
ChopperCammand
811
7387.8
1615.3
1600.3
6654,
5982.0
1044.0
CrazyClimber
10780.5
3
111493.8
108493.9
131645.8
84390.9
DemonAAiack
152.1
1971.0
4396.7
3182.6
7715.5
6346.4
73794.0
DoubleDunk
-16.7
17.4
-7.5
Enduro
0
860.5
799.5
2062.5
2268.1
2255.6
2341.2
Derby
91.7
-38.7
12.3
48.4
41.9
37.6
31.7
Freeway
0
29.6
25.8
33.5
33,5
33.2
34.0
Frostbite
65.2
4334.7
760.2
8022.8
7824,9
5697.2
4148.2
Gopher
257.6
3495.8
3917.1
11192.6
47054_5
Gravitar
173.0
3351.4
250.7
821.3
1083.5
926.2
635.8
Hero
30826.4
12316.4
14980.0
18754.0
8
12579.2
lceHockey
-11.2
0.9
-6.7
-4.5
0.0
2.3
-1.4
Jamesbond
29.0
302.8
500.0
802.3
1118.8
656.7
2121.8
Kanguron
52.0
3035.0
6768.2
4727.3
11385,4
13133.1
14617.1
Krull
1598
2665.5
6181.1
8073.9
8661.7
6292.5
9746.1
KungFuMaser
258.5
22736.3
20418.8
33099.9
26707.0
43258.6
MontezumaRevenge
0.0
4753.3
2.6
300.5
0.7
501.2
0.0
MsPacman
307.3
6951.6
2727.2
3313.9
3406.4
2928.9
Name ThinGame
2292.3
8049.0
5697.3
7307.9
9432.8
9389.5
10298.2
Phoemix
761.4
7245.6
5833.7
4641.1
$147.2
8272.9
20453.8
Pitfall
-29.44
6463.7
-16.8
-0.4
Pong
20.77
14.6
13.2
19.2
19.9
19.4
21.0
PrivateFyye
372.4
Qbert
163.9
13455.0
8216.2
17228
15045,5
17121.4
15267 4
Riverraid
1338.5
17118.0
9077.8
13389.4
14868.6
15748.9
11175.3
RoadRunner
11.5
7845.0
39703.1
44619.2
50534.1
51442.4
50854.7
Robotank
2.2
11.9
25.8
$3.6
65.9
63.6
60.3
Seaquest
68.4
42054.7
1585,9
4667.9
20081.3
3916.2
19652.5
Skiing
117098.
4336.9
17038.2
14401 .6
-13755.6
17960.
9299.3
Solaris
12326.7
2361.7
2234.5
2640.0
SpaceInvaders
148.0
1668.7
1361.1
940.2
3115.0
1908
1749.4
Star Gunner
664.0
10250.0
1676.5
23593.3
60090.0
39456.3
62920.6
Tennis
-23.8
9.3
0.1
19.2
3,5
0.0
-1.0
TimePilot
3568.0
5229.2
3200.9
6622.88
9820.6
9324.4
6506.4
Tutankbam
11.4
167.6
138.8
250.4
231.3
UpNDowwn
533.4
11693.2
10405.6
29890.1
44327.6
18790.7
36008. 1
Venture
1187.5
50.8
1488.9
993.3
VidcoPinball
16256.9
17667.9
216042.7
250650.0
486111.5
536364.4
465578.3
Wiz
563.5
4756.5
2664.9
2841.8
6791.4
7562.7
6132.8
YarsRevenge
3092.9
54576.9
20375.7
66055.9
57960.3
31864.4
27674.4
Zaxxon
32.5
9173.3
1928.6
8177.2
12048.6
14117.5
10806.6
Table 3: Raw scores for single seed across all 55 games. We report the best scores for DQN, QR- DQN, IQN, and Rainbow on 50M frames, averaged by 5 seeds. Reference values were provided by Dopamine framework (Castro et al. 2018).
18

---

Under review as a conference paper at ICLR 2022
GAMES
RANDOM
HUMAN
DQN(200M)
(200Q
IQN(200M)
PQR(50M)
Alien
227.8
7127.7
1620.0
4871.0
7022.0
2455.8
Amidar
1719.5
978.0
16410
2946.0
938.4
Assault
222.4
742.0
4280.4
22012.0
29091.0
10759.2
Asterix
210.0
8503.3
4359.0
261025.0
342016.0
Asteroids
719.1
47388.7
1364.5
4226.0
2898.0
1662.0
Atlantis
12850.0
29028.1
279987.0
971850.0
978200.0
897640.0
BankHeist
14.2
753.1
455.0
1249.0
1416.0
1038.8
BattleZone
2360.0
37187.5
29900.0
39268.0
42244.0
28470.5
BeamRider
363.9
16926.5
8627.5
34821.0
42776.0
10224,9
Berzerk
123.7
2630.4
585.6
3117.0
1053.0
137873.1
Bowling
23.1
160.7
50.4
77.2
86.5
86.9
Boxing
0.1
12.1
88.0
99.8
97.1
Breakout
1.7
30.5
385.5
742.0
734.0
380.3
Centipede
2090.9
12017.0
4657.7
12447.0
11561.0
7291.2
ChopperCommand
811.0
7387.8
6126.0
14667.0
16836.0
1044.0
CrazyClimber
10780.5
35829.4
110763.0
161196.0
179082.0
84390.9
DemonAttack
152.1
1971.0
12149.4
1215510
128580.0
73794.0
DoubleDunk
-18.6
-16.4
-6.6
21.9
5.6
-7.5
Enduro
0.0
860.5
729.0
2355.0
2359.0
2341.2
FishingDerby
91.7
-38.7
-4.9
39.0
33.8
31.7
Freeway
0.0
29.6
30.8
34.0
34.0
34.0
Frostbite
65.2
4334.7
797.4
4384.0
4324.0
4148,2
Gopher
257.6
2412.5
8777.4
113585.0
118365.0
47054.5
Gravitar
173.0
3351.4
473.0
995.0
911.0
635.8
Hero
1027.0
30826.4
20437.8
21395.0
28386.0
12579.2
IceHockey
-11.2
0.9
-1.9
-1.7
0.2
-1.4
Jamesbond
29.0
302.8
768.5
4703.0
35108.0
2121.8
Kangaroo
$2.0
3035.0
7259.0
15356.0
15487.0
14617.1
Krull
1598.0
2665.5
8422.3
11447.0
10707.0
9746.1
KungFuMaster
258.5
22736.3
26059.0
76642.0
73512.0
43258.6
MontezumaRevenge
0.0
4753.3
0.0
0.0
0.0
0.0
MsPacman
307.3
6951.6
3085.6
5821.0
6349.0
2928.9
Name ThisGame
2292.3
8049.0
8207.8
21890.0
22682.0
10298.2
Phoenix
761.4
7242.6
8485.2
16585.0
56599.0
20453.8
Pitfall
-229.4
6463.7
-286.1
0.0
0.0
0.0
Pong
-20.7
14.6
19.5
21.0
21.0
21.0
PrivateEye
24.9
69571.3
146.7
350.0
200.0
372.4
Qbert
163.9
13455.0
13117.3
572510.0
25750.0
15267.4
Riverraid
1338.5
17118.0
7377.6
17571.0
17765.0
11175.3
RoadRunner
7845.0
39544.0
64262.0
57900.0
50854.7
Robotank
2.2
11.9
63.9
59.4
60.3
Seaquest
42054.7
5860.6
8268.0
30140.0
19652.5
Skiing
-17098.1
-336.99
-13062.3
9932.00
-9289.0
-9299,3
Solaris
1236.3
12326.7
3482.8
6740.0
8007.0
2640.0
SpaceInvaders
148.0
1668.7
1692.3
20972.0
28888.0
1749.4
StarGunner
664.0
10250.0
54282.0
77495.0
74677.0
60920.6
Tennis
-23.8
-9.3
12.2
23.6
23.6
-1.0
TimePilot
3568.0
5229.2
4870.0
10345.0
12236.0
6506.4
Tutankham
11.4
167.6
68.1
297.0
293.0
231.3
UpNDown
533.4
11693.2
9989.9
71260.0
88148.0
36008.1
Venture
0.0
1187.5
163.0
43,9
1318.0
993.3
VideoPinball
16256.9
17667.9
196760.4
705662.0
698045.0 00
465578.3
WizardOfWor
563.5
4756.5
2704.0
25061.0
31190.0
6132.8
YarsRevenge
3092.9
54576.9
18098.9
26447.0
28379.0
2764.4
Zaxxon
32.5
9173.3
$363.0
13113.0
21772.0
10806.6
Table Raw scores for a single seed across all 55 games, starting with 30 no-op actions. Note that PQR was evaluated 50M frames. We report the published scores for DQN, QR-DQN, and IQN on 200M frames. Reference values from Yang et al. (2019).
19


Reviews:
[{"review_id": "OGspts0Wu2B", "reviewer": "Reviewer_61i6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that the overall idea is intuitive and makes sense to me. Since committing to specific criterion induces a one-sided tendency on risk for the agent, it seems natural to instead randomize the risk criterion so as to reduce the bias and gradually shrink the ambiguity set to obtain converged results. The paper is to some extend \"complete\", with both theoretical analysis showing the convergence property as well as a few examples demonstrating the performance of the method.\n\nAdditional comments:\n1. The writing can be improved. There are many small typos/grammatical errors here and there which seem to indicate that the paper was rushed. Just to name a few: definition 2, line 3, missing \".\"; P8, \"let define a reweigh'; P8, N-Chain, \"We more plot the empirical\". Please carefully go through the paper and correct the mistakes. Further, the theoretical part contains complicated notation, and the authors could try to see if there is a better way to present the technical content with more intuitive explanations.\n2. How were the four Atari games selected? Were they cherry-picked? The authors mention that \"To achieve high performance, the agent should take an optimistic approach in some games and a pessimistic approach in others\". Perhaps, a more detailed explanation and comparison should be provided so that the readers can better understand the behavior of the method. More thorough experiments and ablations studies on hyper-parameters would greatly improve the paper.", "summary_of_the_paper": "This paper considers distributional reinforcement learning with the goal of seeking risk-neutral optimal policy. Prior approaches often have a convergence issue due to either being just risk-seeking or risk-averse. Towards resolving the problem, the authors propose a simple, yet effective approach by randomizing the risk criterion during the training process. On the theoretical front, the authors prove that the method converges to an optimal return distribution under certain conditions. Several experimental results also confirm the validity of the method.", "summary_of_the_review": "Overall, I think the main idea is clearly articulated. While there is room to improve, the theoretical and empirical contributions seem to be relevant and of interest. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NCQZpALE3wx", "reviewer": "Reviewer_Ywdd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "According to the authors, the shortcoming of existing approaches is that they do not separate epistemic and aleatoric uncertainty. This problem appears well motivated (though their example is a bit simplistic; see discussion below). However, I’m not sure I understand how the proposed approach solves the problem. As far as I can tell, their main strategy is to use sampling from a distributional perturbation rather than optimism. The main reason their approach converges appears to be the fact that they require that the size of the perturbation goes to zero over time. Wouldn’t taking the optimism to zero over time achieve the same effect?\n\nFurthermore, their example in Section 3.1 is for a toy algorithm; it is unclear if existing techniques suffer from this issue (though I suspect that they do). As discussed above, it also appears that their algorithm primarily avoids this issue by taking \\Delta_n to zero over time; in their example, it appears that you can achieve the same effect by taking c_t to zero over time.\n\nAlong the same lines, their theoretical guarantee does not appear to be very strong. For example, as far as I can tell, it applies to traditional Bellman operator (i.e., the one that does not perform any exploration); indeed, this is the case \\Delta_n=0 for all n. Thus, their theoretical analysis primarily results from the fact that they are taking \\Delta_n to be small sufficiently fast, rather than from any feature of their particular exploration strategy. I would have expected the theory to support the benefits of using their exploration strategy instead of an alternative.\n\nFinally, their experiments also appear to be fairly weak. They only compare to a limited number of baselines; for instance, why not compare to prior approaches such as Keramati et al. 2020? Furthermore, they do not appear to outperform the baselines by very much in most of their empirical results.\n\nMinor comments\n- Does Theorem 4 apply for any choice of U_\\Delta in Theorem 4? This is not clear.\n- The proposed algorithm feels very similar to Thompson sampling; I wonder if some connection can be drawn here.\n- The authors do not clearly distinguish that their goal is to use distributional RL, but eventually learn a risk-neural policy. Several of the papers they cite have the goal of learning a risk-aware policy; the concerns regarding epistemic vs. aleatoric uncertainty do not apply to these approaches. It would be helpful if the authors clarify this distinction.\n", "summary_of_the_paper": "This paper proposes a novel strategy for performing exploration based on distributional reinforcement learning. They highlight the fact that existing approaches cannot distinguish between epistemic and aleatoric uncertainty. Then, they propose an alternative approach that leverages a random perturbation to the distribution, but then takes this perturbation to zero. They prove that this approach converges. Finally, they validate their approach in their experiments.\n", "summary_of_the_review": "Pros\n- Important problem\n\nCons\n- Unclear whether their approach solves the stated problem\n- Limited theoretical guarantee\n- Weak experiments\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ChNFEqGHMwN", "reviewer": "Reviewer_GS3Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n\n1. The proposed method is a novel application of results from DRO (Distributionally Robust Optimization) literature to formulate a perturbed bellman optimality operator (PDBOO) that is theoretically well-behaved -- it is shown to converge under a “weak contraction” property and to the same unique fixed point as the standard bellman operator.\n\n2. The pedagogical N-chain environment setting clearly demonstrates the benefit of the proposed risk-neutral method in converging to the optimal action despite the existence of sub-optimal actions that have a low probability of higher reward. In the LunarLander and Atari environments, a similar result is obtained with baselines including DLTV, a perturbed variant of DLTV and QR-DQN.\n\n## Weaknesses\n\n1. The paper is empirically weak, demonstrating results in only 4 Atari environments. It would be good to see a full empirical analysis of all Atari environments. The paper would also benefit from diversity in the choice of environments -- given that it proposes risk-neutral exploration, what about continuous control simulation tasks such as locomotion or manipulation (in MuJoCo for example), or the b-suite family of environments (https://github.com/deepmind/bsuite)?\n\n1. The paper has issues with writing. Grammatical errors are prevalent throughout the paper and the related works, motivation and presentation of the proposed method all seem to be lacking in clarity.\n\n", "summary_of_the_paper": "The paper proposes a novel distributional RL algorithm that is neither risk-seeking (exploratory) or risk-averse (exploitatory) -- but rather risk-neutral. The paper motivates the need for risk neutrality from the family of works applying OFU (optimism in the face of uncertainty) to distributional RL -- it states that prior works in this space induce a one-sided risk tendency (risk-seeking or risk-averse) which is undesirable as it leads to “biased exploration”. An N-chain environment is initially studied where it is shown that prior works (DLTV, QRDQN)  are not able to identify the optimal action as fast as the proposed method (PQR) -- the prior works are misled by the low probability high reward side of the N-chain environment as opposed to the optimal side which provides maximum averaged reward. The paper then proposes a method for perturbing the risk-measure in risk-sensitive RL such that the resulting perturbed distributional bellman operator (PDBOO) converges to the same fixed point as the standard bellman operator (with some weak assumptions) and hence, produces a risk-neutral policy. The paper presented results in the LunarLander-v2 and Atari domain (4 Atari games selected), demonstrating that prior works with one-sided risk tendencies do not perform as well as the proposed risk-neutral method (PQR).", "summary_of_the_review": "Overall, the paper provides valuable theoretical contributions in the form of their analysis of the perturbed distributional bellman optimality operator (PDBOO) and its successful application in toy environments (N-chain) as well as some non-trivial ones (LunarLander, Atari). However, the paper lacks in quality of writing and empirical analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies efficient algorithms for distributional reinforcement learning. The motivation stems from the need of risk neutrality, since other existing approaches might have one-sided risk tendencies. The algorithms proposed in this paper are based on sampling from a distributional perturbation rather than using optimism in the face of uncertainty. Both theoretical guarantees and empirical results have been provided to validate the effectiveness of the proposed algorithms. While this is certainly an important and interesting direction, I agree with the reviewers that it is unclear from the theory in this paper why distributional perturbation is helpful.
Decision: Reject