Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Authors: Taehyun Cho
Seungyub Han
Heesoo Lee
Kyungjae Lee
Jungwoo Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: distributional reinforcement learning
risk

Abstract: Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning that selects actions by randomizing risk criterion without losing the risk-neutral objective. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure. Also,we prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Taehyun Cho', Seungyub Han Heesoo Lee', Kyungjae Lee", Jungwoo Lee
Scoul National University, 2 Chung-Ang University, Hodoo Al Labs (talium, seungyubhan, algo17240, junglee) ac. (kyungjae.. lee) kc.
Abstract
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning algorithm that selects actions by randomizing risk criterion to avoid one-sided tendency on risk. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure and prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and guaranteed to converge to an optimal return. Finally, empirically show that our method outperforms other existing distribution-bast based algorithms in various environments including Atari 55 games.
Introduction
Distributional reinforcement learning (DistRL) learns the stochasticity of returns in the rein- forcement learning environments and has shown remarkable performance in numerous bench- mark tasks. DistRL agents model the ap- proximated distribution of returns, where the mean value implies the conventional Q-value
Low Risk
High Risk
Optimistic Decision
AUA
Figure Illustrative example of why a biased risk criterion (naīve optimism) can degrade per- formance. Suppose two actions have similar ex- pected returns, but different variances (intrinsic uncertainty). (Left) If an agent does not specify the risk criterion at the moment, the probability of selecting each action should be similar. (Right) As OFU principle encourages to decide uncertain behaviors, the empirical variance from quantiles was used an estimate of uncertainty. [19 21 23
13 25 and provides more statistical in- formation (e.g., mode, median, variance) for control. Precisely, DistRI aims to capture intrin- sic (aleatoric) uncertainty which an inherent and irreducible randomness in the environment. Such learned uncertainty gives rise to the no- tion of risk-sensitivity and several distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse risk-seeking decision making 8 [2).
2
6
However, optimistic decision based on empirical variance inevitably leads risk-seeking behavior, which causes biased action selection.
Despite the richness of risk-sensitive informa- tion from return distribution, only a few DistRI.
methods [ 21 26 35| (2) have tried to employ its benefits for exploration strategies which is essential in deep RL to find an optimal behavior within a few trials. The main reason is that the explo- ration strategies so far is based parametric (epistemic) uncertainty which arise from insufficient
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

list actions be added buffer
e-greedy replay buffer
PQR replay buffer
DLTV replay buffer
WV4
X
X
Number of candidate actions
An illustrative example of proposed algorithm (PQR). Each distribution represents the empirical PDF of return. PQR benefits from excluding inferior actions and promoting unbiased selection with regards high intrinsic uncertainty through randomized risk criterion.
Figure 2:
inaccurate data. In particular, Optimism in the face of uncertainty (OFU) one of fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and construct confidence set. In bandit or tabular MDP settings, OFU-based algorithms select action with the highest upper-confidence bound (UCB) of parametric uncertainty which can be considered as the optimistic decision at the moment [0].
However, in deep RL, it is hard to trivially estimate the parametric uncertainty accurately due to the black-box nature of neural networks and high- dimeeniioonlltty of state-action space. Without further computational task, the estimated variance from distribution is extracted as mixture of two types of uncertainty, making difficult decompose either component. For example, DLTV [2] was proposed as a distribution-bast of intrinsic uncertainty, which unintentionally induces a risk-seeking policy. Although DLTVV the first attempt to introduce OFU in distRI, we found that consistent optimism on the uncertainty of the estimated distribution still leads to biased exploration. We will refer to this sidc-effect as one-sided tendency on risk, where selecting action based fixed risk criterion degrades learning performance. Section 4 we will demonstrate the one-sided tendency on risk through toy experiment and show that proposed randomized approach is effective avoid this side effect.
OFU exploration that decays bonus rate to suppress the effect
In this paper, we introduce Perturbed Distributional Bellman Optimality Operator (PDBOO) to address the issue of biased exploration caused by one-sided tendency on risk in action selection. We define the distributional perturbation on return distribution re-evaluate the estimate return by distorting the learned distribution with perturbation weight. Too facilitate deep RI. algortihm, we present Perturbed Quantile Regression (PQR) and test Atari 55 games comparing with other distributional RL algorithms that have been verified for reproducibility by official platforms [4] 29
In summary, our contributions are as follows.
A randomized approach called perturbed quantile regression (PQR) is proposed without sacrificing the original nrisk-neutral) optimality and improves over naïve optimistic strategies.
A sufficient condition for convergence of the proposed Bellman operator is provided without satisfying the conventional contraction property.
Backgrounds & Related works
2.1 Distributional RL
We consider Markov decision process (MDP) which is defined a tuple (S,A,P, R, where S is
finite state space, AA is a finite action space, P S A x S + [0. 1] is the transition probability, R the random variable of rewards [-R Rmuax Rmmx) and y E [0, 1) the discount factor. We define stochastic policy (-|ss) which is conditional distribution over .A given state s. For fixed policy
we denote z" (s, a) as a random variable of return distribution of state-action pair (s, a) following
the policy We attain n" (8,8.00 a) > R(S,, At), where S P(. At). A, (-)).) and So 8, Ao = a. Then, we define an action-value function as Q* a) E[
(2, (s,a) a)) in
[- -max Vmax. where Vmax Rmax/ 7). For regularity, further notice that the space of

---

action-value distributions 2 has the first moment bounded by Vinx:
= (Z x ( ( a)l] - Vmux, as).
In distributional RL, the return distribution for the fixed can be computed via dynamic programming with the distributional Bellman operator defined as,
T Z(s R(s,a) + +2(("' A), P(-ssa)) A'~ *(1S")
where 2 denotes that both random variables share the same probability distribution. We compute the optimal return distribution by using the distributional Bellman optimality operator defined as,
T [ o) 2 R(s,a) + a²), S' P(-s,a)) a), a' argmax Ez|Z(S', a'))
Bellemare et al. [2] have shown that T is a contraction in a maximal form of the Wasserstein metric but T is not a contraction in any metric. Combining with the expectation operator, ET is contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value. Another notable difference is that the convergence of return distribution is not generally guaranteed to be unique, unless there is total ordering < on the set of greedy policies.
2.2 Exploration on Distributional RL
To combine with deep RL, a parametric distribution Zg is used to learn a return distribution. Dab- ney et al. 13 have employed a quantile regression to approximate the full distribution by letting 7o a) N Ei (e..) where / represents the locations of mixture of N Dirac delta functions. Each 8; represents the value where the cumulative probability is т, & By using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as minimization problem as,
e (re + 707 (8441, (8,, a:))
arg min D (Zo 0z); T (st,0e)) arg min 8'
N
where (St, Qe, Ft, S1+1) is given transition pair, 5i 14 a' := argmax Ez Zo (s ad) d')],
(2) = f. ( L (x), and (x) == x² for |2| S K and (x(x) = <(<=| 1t), otherwise.
Based on the quantile regression, Dabney et al. [33 have proposed a quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach 2 since the quantile regression does not restrict the bounds for return.
As deep RL typically did, QR-DQN adjusts e-greedy schedule, which selects the greedy action with probability 1 and otherwise selects random available actions uniformly. The majority of QR-DQN
variants [2 38 rely on the same exploration method, However, such approaches do not put aside
inferior actions from the selection list and thus suffers from loss 28]. Hence, designing a schedule to select statistically plausible action is crucial for efficient exploration.
In recent studies, Mavrin et al. [21] modifies the criterion of action selection for efficient exploration based on optimism in the face of uncertainty. Using left truncated variance as a bonus term and decaying ratio e suppress the intrinsic uncertainty, DLTV was proposed an uncertainty-based exploration in DistRL without using e-greedy schedule. The criterion of DLTV is described as:
a' = arrgmax + @V/2(5500)) a = OV
logt t
0²
Σ
N
(0 - 0,)
2N
where 0,'s 's are the values of quantile level Ti.
2.3 Risk in Distributional RL
Instead of an expected value, risk-sensitive RL is to maximize a pre-defined risk measure such as
Mean- Variance 41) Value-at-Risk (VaR) [9], or Conditional Value-at-Riss Risk (CVaR) (30 31 and
results in different classes of optimal policy. Especially, Dabney al. [2] interprets risk measures
3

---

as the expected utility function of the return, i.e., (z If the utility function U is linear, the policy obtained under such risk measure called risk-neutral. If U is concave or convex, the resulting policy is termed as risk-averse or risk-seeking, respectively. In general, a distortion risk measure is generalized expression of risk measure which generated from the distortion function.
Definition 2.1. Let h [0, - [0, 1] be distortion function such that A(O) non-decreasing. Given a probability space F2, P) and a random variable Z : S + TR, distortion risk measure Pa corresponding
h(1) 1 and
distortion function is defined by:
a (ho (z)(z) 02
Ph (z) = B
where Fz is the cumulative distribution function of Z.
In fact, nen-decreaging property of h makes it possible to distort the distribution of while satisfying the fundamental property of CDF. Note that the concavity or the convexity of distortion function also implies risk-averse or seeking behavior, respectively. Dhaene et al. [14] showed that any distorted expectation can be expressed as weighted averages of quantiles. In other words, generating a distortion risk measure is equivalent to choosing a reweighting distribution.
Fortunately, DistRL has a suitable configuration for sisk-sensitive decision making by using distortion risk measure. Chow al. [8] and Stanko and Macek [34 considered risk-sensitive RL with CVaR objective for robust decision making. Dabney et al. [12] expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets Zhang and Yao [40] have suggested QUOTA which derives different policies corresponding to different risk levels and considers them as options. Moskovitz et al. (24] have proposed TOP-TD3, an ensemble technique of distributional critics that balances between optimism and pessimism for continuous control.
Perturbation in Distributional RL
3.1 Perturbed Distributional Bellman Optimality Operator
To choose statistically plausible actions which may be maximal for certain risk criterion, we will generate distortion risk measure involved in pre-defined constraint set, called ambiguity ser. The ambiguity set, originated from distributionally robust optimization (DRO) literature, is a family of distribution characterized by certain statistical distance such é-divergence
Wasserstein
distance 15 32)- In this paper, we will examine the ambiguity set defined by the discrepancy between distortion risk measure and expectation. We say the sampled reweighting distribution & as (distributional) perturbation and define it as follows:
Definition 3.1. (Perturbation Gap, Ambiguity Set) Given probability space F2, IP), let X :Ω
be a random variable and = { 0 S {(w) oc, Swest ((w) = 1} be a set of probability density functions. For given constraint set 2/ E, we say U (distributional) perturbation from 24 and denote the Ç - weighted expectation of X as follows:
J
[x[X]
JUES X(w){(w)F(ddw),
which can be interpreted the expectation of X under some probability measure Q, where & dQ/dP is the Radon-Nikodym derivative of Q with respect to P. We further define di ((X:E) = [X Ex[X]] perturbation gap of X with respect to Ç. Then, for given constant A 2 0, the ambiguity set with the bound A is defined as
Ua(x) = f ( d(x:))
For brevity, we omit the input from a random variable unless confusing. Since & is probability density function, Eg [X] an induced risk measure with respect to a reference measure P. Intuitively, {(w) can be viewed as distortion to generate different probability measure and vary the risk tendency. The aspect of using distortion risk measures looks similar to IQN [12]. However, instead of changing the sampling distribution of quantile level implicitly, we reweight each quantile from the

---

ambiguity set. This allows us to control the maximum allowable distortion with bound A, whereas the risk measure in IQN does not change throughout learning. In Section 3.3 method to construct the ambiguity set.
we suggest a practical
Now, we characterize perturbed distributional Bellman optimality operator (PDBOO) T for fixed perturbation & E (Z) written as below:
[s a) 1 R(s,a) + 7 a (E)), S' ~ P(-s,a)) a²({) argmax Eg.p|Z(s', a')].
Notice that { = corresponds to a base expectation, i.e., Eg.p= Ep, which recovers the standard distributional Bellman optimality operator T. Specifically, PDBOO perturbs the estimated distribu- tion only to select the optimal behavior, while the target is updated with the original (unperturbed) return distribution.
we consider the time- vare-ng bound of ambiguity set, scheduling A, a key ingredient to determine whether PDBOO will efficiently explore or converge. Intuitively, if an agent continues to sample the distortion risk measure from a fixed ambiguity set with a constant A, there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence. Hence, scheduling a constraint of ambiguity set properly at each action selection is crucial to guarantee convergence.
Based on the quantile model Ze, our work can be summarized into two parts. First, we aim to minimize the expected discrepancy between Zo and 7 where & is sampled from ambiguity set MA. To clarify notation, we write [g[[]] as a weightted expectation and (e)))( [1] as an expectation with respect to & which is sampled from M/. Then, our goal is to minimize the perturbed distributional Bellman objective with sampling procedure P:
min RE 8'
(D (s, a), 7g-
(I)
where we use the Huber quantile loss discrepancy on Zor and T 57 at timestep t. In typical risk-sensitive RL or distributionally robust RL, the Bellman optimality equation is reformulated for pre-defined risk measure 88 33 (3]. In contrast, PDBOO has significant distinction in that it performs dynamic programming that adheres to the risk-ncutral optimal policy while randomizing the risk criterion at every step. By using expeexxtctotion instead of min-max operator, we suggest unbiased exploration that can avoid leading to overly pessimistic policies. Furthermore, considering sequence &, which converges to in probability, expectation of any composition of the operators ET E.:, ET = ET &.. T Ta_: ... Te, has the same unique fixed point the standard. These results are remarkable that can apply the diverse variations of distributional Bellman operators for learning.
derive a sufficient condition of A, that the
3.2 Convergence of the perturbed distributional Bellman optimality operator
Unlike conventional convergence proofs, PDBOO is time-varying and not a contraction, so it covers wider class of Bellman operators than before. Since the infinite composition of time-varying Bellman operators does not necessarily converge or have the same unique fixed point, we provide the sufficient
7(a), (()) 7 for each condition in this section. We denote the iteration as 2(++)) = T
timestep n > 0, and the intersection of ambiguity set Us, (Z(--1) = Ne.o Us. (Z (s, a))))
Assumption 3.2. Suppose that Em=1 An < Do and En, is uniformly bounded.
Theorem 3.3. (Weaker Contraction Property) Let En be sampled from ús. (Z for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators ET gna converges, i.e., IET sm. + B[Z*). Moreover, the following bound holds,
k- 11 Viaas + 2 Σ ( ( ABm1..))
sup 8.0
B BlZin) (2)
-
< Σ
k=n
Practically, satisfying Assumption 3.2 is not strict to characterize the landscape of scheduling. Theorem 3.3 states that even without satisfying 7-contraction property, we can show that [[[2]] is the fixed point for the operator ET s.... However, E[2*| is not yet guaranteed to be "unique" fixed point for any 2. Nevertheless, we can show that E. [ is, in fact, the solution of the standard Bellman optimality equation, which is already known have unique solution.
5

---

Algorithm 1 Perturbed QR-DQN (PQR)
Input: (s, a, (,),, [0, 1), timestep 0, 0, concentration
Initialize An >
A, < Dot- (1+e).
I/ Assumption
3.2
& + max I + ANNNx 1NNNN= 11)) where x ~ Dir(3)
I/ Sample & ~ Us, (2())
Ç + E
Refine as weighting function // Select greedy action with perturbed return Target update with unperturbed distribution
a" + argmax, E (g(Z(s ,a)]
T0 + -r r ,0"), Vj
++
Output: Ei E3 $ (TO, (,(s,))
N
Theorem 3.4. If Assumption holds, BiZ*) is the unique fixed point of Bellman optimality equation for any Z E Z.
As result, PDBOO generally achieves the unique fixed point of the standard Bellman operator. Unlike previous distribution-base or risk-sensitive approaches, PDBOO has the theoretical compati- bility obtain risk-neutral optimal policy even if the risk measure randomly sampled during training procedure. For proof, see Appendix A
3.3 Practical Algorithm with Distributional Perturbation
In this section, we propose a perturbed quantile regression (PQR) that a practical algorithm for distributional reinforcement learning. Our quantile model is updated by minimizing the objective function ①T induced by PDBOO. Since we employ a quantile model, sampling a reweight function can be reduced into sampling N N-dimensional weight vector E == [s,,... .EN] where Σ N and &, 2 for all E {1.... .N.. Based on the QR-DQN setup. note that the condition Swea ((w) = turns into ΣN C = 1, since the quantile level set as Ti = ;
N
A key issue is how construct an ambiguity set with bound A, and then sample &. A natural class of distribution for practical use is the symmetric Dirichlet distribution with concentration B, which represents distribution over distributions. (i.c. x Dir(3))) We sample random vector, æ Dir( and define the reweight distribution as & := 1 + a(Nx 1N). From the construction of &, we have 1- & < & < 1 + a(N - 1) for all and it follows that |1 &al I a(N - 1). By controlling we can bound the deviation &, from and bound the perturbation gap as
sup E[Z(s m)] Beg (s, = sup 9,4
(wu; 8, a) ((wo)
wesl
S supp {(w) sup El a)]l wen
sup
((w) S a(N 1)
wen
Hence, letting a 1N is sufficient to obtain d(Z;E) < A in the quantile setting. We set e = 0.05 N to generate a constructive perturbation &n which gap is close to the bound A,. For Assumption 3.2 our default schedule is set as A, = Apt (1++) where e = 0.001.
4
Experiments
Our experiments aim to investigate the following questions.
1. Does randomizing risk criterion successfully escape from the biased exploration in stochastic environments?
2. Can PQR accurately estimate return distribution?
3. Can a perturbation- based exploration perform sucessfully as a behavior policy for the full Atari benckmark?

---

010.0.19)
Start
:0g
di
$3
no-op :Q2 0g
:a (optimal))
W5. 0.12) 3M
Figure 3: Illustration of the N-Chain environment 27 with high uncertainty starting from state s2- To emphasize the intrinsic uncertainty, the reward of state 84 was set as a mixture model composed of two Gaussian distributions. Blue arrows indicate the risk-neutral optimal policy in this MDPs.
masss Ealk L.b
dezz
PCF
wne 133, 04 223
20K Meg
24 Agg
20K 403s
(a) QR-DQN
(b) DLTV
(c) P-DLTV
(d) PQR (Ours)
Figure Empirical return distribution plot in N-Chain environment. The ground truth of each distribution is 7 (10, 0.1²) and >2 N 0.1²) + [13,,, 0.13)]. Each dot represents an indicator for choosing action. Since QR-DQN does not depend on other criterion, the dots are omitted.
4.1 Learning on Stochastic Environments with High Intrinsic Uncertainty
For intuitive comparison between optimism and randomized criterion, we design p- D-DLTV, a per- turbed variant DLTV, where coefficient 0t is multiplied by a normal distribution (0, 12). Every experimental setup, pseudocodes, and implementation details can be found in Appendix C
N-Chain with high intrinsic uncertainty. We extend N-Chain environment 27 with stochastic reward to evaluate action selection methods. A schematic diagram the stochastic N-Chain environ- ment is depicted in Figure The reward is only given in the leftmost and rightmost states and the game terminates when one of the reward states reached. We set the leftmost reward as NV10,, 0.1²) but higher
and the rightmost reward [5, 0.1 + [W 0.12) which has lower mean variance. The agent always starts from the middle state and should move toward the leftmost state 5o achieve the greatest expected return. For each state, the agent can take one of six available actions: left, right, and 4 no-op actions. The optimal policy with respect to mean is to move left twice from the start. We the discount factor y 0.9 and the coefficient c 50.
Despite the simple configuration, the possibility to obtain higher reward in suboptimal state than the optimal state makes it difficult for an agent to determine which policy is optimal until experi- ences enough discern the characteristies of each distribution. Thus, the goal of our toy experiment is to evaluate how rapidly each algorithm could find rist-nlutral optimal policy. The results of varying the size of variance are reported in Appendix D.I
20000
QRDQN
17500
DLTV
15600
D-DLTV
Total Count
PQR
12500
7500
Analysis of Experimental Results. As we design the mean of each return is intended be similar, examining the learning behavior of the empirical return distribution for each algorithm can provide fruitful insights. Figure 4 shows the empirical PDF of return distribution by using Gaussian kernel density estimation. In Figure 4bb), DLTV fails to estimate the true optimal return distribution. While the return of (*2, right) (red line) is correctly estimated toward the ground truth, (82, left) (blue line) does not capture the shape and mean due to the lack of experience. At 20K
2500
5000
1,0000
35000
20000
Steps
Figure 5: Total count of perform- ing true optimal action. The or- acle (dashed line) is perform the optimal action from start to end.
7

---

timestep, the agent begins to see other actions. but the monotonic scheduling already makes the decision like exploitation. Hence, decaying schedule of optimism not a way to solve the underlying problem. Notably, P-DLTV made a much better estimate than DLTV only by changing from optimism to a randomized scheme. In comparison, PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation
Figure 5 shows the number of timesteps when the optimal policy was actually performed see the interference of biased criterion. Since the optimal policy consists of the same index d1, we plot the total count of performing the optimal action with 10 seeds. From the slope of each line, is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed. In contrast, p p-DLTV avoids getting stuck by randomizing criterion and eventually finds the true optimal policy. The experimental results demonstrate that randomizing the criterion is simple but effective way for exploration on training process.
typerparemett smeter
Hyperparamater
Pyperparameter
12
Hyperparameter 50
GRDON
DCTV
p-u
POR
Timesdapa
Timestpps
Timrecaaaa
Hypersorameter 100
Hyperparameter
Hoperparameter
Hyperparameter 5000
Timestepa
Timestogo
Timestopp
Timestopa
Figure 6: 2- 2-Wasserstein distance between the empirical return distribution and the ground truth 0.0811). We use QR-DQN with a fixed setting of c-greedy as reference baseline, because the hyperparameter of e-greedy is not related to the scale of Q-values.
Hyperparameter Sensitivity. In Figure 6 we compute the 2- Wasserstein distance from the ground truth return distribution 2 (0.1>²)* Except for QR-DQN each initial hyperparameter {c, A was implemented with grid search on [1, 5, 10, 50, 100, 500, 1000, 5000] in 5 different seeds. As the hyperparameter decreases, each agent is likely to behave exploitation. One interesting aspect is that, while it may be difficult for DLTV and P-DLTV to balance the scale between the return and bonus term, PQR shows robust performance to the initial hyperparameter. This is because the distorted return is bounded by the support of return distribution, so that PQR implicitly tunes the scale of exploration. In practice, we set Ao to be sufficiently large. See Table in Appendix C.
4.2 Full Atari Results
We compare our algorithm to various DistRL baselines, which have demon- strated good performance RL bench- marks. In [ we evaluated 55 Atari results, averaging over 5 different seeds at 50M frames. We compared with the published score of QR-DQN [13] IQN 11). and Rainbow [66 via the report of
Table 1: Mean and median of best scores across 55 Atari games, measured as percentages of human baseline. Refer- ence values from Quan and Ostrovski 29 and Castro et al.
50M Performance
Mean
Median
human
DQN
DON-200 (nu-ops) -opommmaamine (sticky)
314%
55%
18
401%
51%
15
0
DQN-Zoo (29 and Dopamine 4 bench- mark for reliability. This comparison particularly noteworthy since our pro- posed method only applys perturbation- based exploration strategy and outper- forms advanced variants of QR-DQN.
OR-DON-Zoe (no-ops)
559%
118%
29
47
OR-DON (sticky)
562%
934
IQN-200 (no-ops) IQN-dopamine (sticky)
902%
131%
21
50
940%
124%
32
RAINBOW- (ho-opss
1160%
154%.
37
HAINBOW-dopaniin (sticky)
965%
123%
35
53
PQR-z00 (no-ops)
1121%
121%
33
53
POR-dopumine (sticky)
962%
123%
35
'In Dopamine framework, IQN was implemented with n rtstep updates with = 3, which improves performance.
8

---

Ansiar
Accouit
roddic
$290
6000
2000
1000
60x-
2009
3000
502-
2000
Taaiing (romer
Trannng frome
Traing
Tholnn fromat
Battlnzane
Reamrider
Berzerk
Beoeling
00.00
12000
375-1
1010
20000
ands
300
1090
2005
Figure 7: Evaluation curves 8 Atari games with random seeds for 50 million frames following sticky actions protocol 20 Reference values are from Castro et al.
No-ops Protocol. First, we follow the evaluation protocol of [ 22 on full set of Atari games implemented in OpenAl's Gym 3 Even if it is well known that the no-ops protocol does not provide enough stochasticity to avoid memorization, intrinsic uncertainty still exists due to the random frume skipping [20]. While PQR cannot enjoy the environmental stochasticity by the deterministic dynamics of Atari games, PQR achieved 562% performance gain in the mean numan-normalizer score over QR-DQN, which is comparable results to Rainbow. From the raw scores of 55 games, PQR wins 39 games against QR-DQN and 34 games against IQN.
Sticky actions protocol. To prevent the deterministic dynamics of Atari games, Machado et al. (20 proposes injecting stochasticity scheme, called sticky actions, by forcing to repeat the previous action with probability p- 0.25. Sticky actions protocol prevents agents from relying and allows robust evaluation. In Figure 7 PQR shows steeper learning curves, even without any support of advanced schemes, such ntstep updates for Rainbow or IQN. In particular, PQR dramatically improves over IQN and Rainbow in ASSAULT, BATTLEZONE, BEAMRIDER, BERZERK and BOWLING. In Table 1 PQR shows robust median score against the injected stochasticity.
memorization
should be noted that IQN benefits from the generalized form of distributional outputs, which reduces the approximation error from the number of quantiles output. Compare to IQN, PQR does not rely on prior distortion risk measure such as CVaR 77. Wang [37 or CPW [36], but instead randomly samples the risk measure and evaluates it with risk-neutral criterion. Another notable difference that PQR shows the better or competitive performance solely through its exploration strategies, compared to e-greedy baselines, such as QR-DQN, IQN, and especially Rainbow. Note that Rainbow enjoys a combination of several orthogonal improvements such as double Q-learning, prioritized replay, dueling networks, and n-step updates.
5 Related Works
Randomized or perturbation-based exploration has been focused due to its strong empirical perfor- mance and simplicity. In tabular RL, Osband et al. (28) proposed randomized least-squares value iteration (RLSVI) using random perturbations for statistically and computationally efficient explo- ration. Ishfaq al. [7 leveraged the idea into optimistic reward sampling by perturbing rewards and regularizers. However, existing perturbation-ban -based methods requires tuning of the hyperparameter for the variance of injected Gaussian noise and depend on well-crafted feature vectors in advance. On the other hand, PDBOO does not rely on the scale of rewards or uncertainties due to the built-in scaling mechanism of risk measures. Additionally, we successfully extend PQR to deep RL scenarios in distributional lens, where feature vectors are not provided, but learned during training.
6 Conclusions
In this paper, we proposed a general framework of perturbation in distributional RL which is based on the characteristics of return distribution. Without resorting pre-defined risk criterion,

---

we revealed and resolved the underlying problem where one-sided tendency on risk can lead to biased action selection under the stochastic enviromment. To our best knowledge, this paper the first attempt to integrate risk-sensitivity and exploration by using time-varying Bellman objective with theoretical analysis. In order validate the effectiveness of PQR, we evaluate on various environments including 55 Atari games with several distributional RL baselines. Without separating the two uncertainties, the results show that perturbing the risk criterion is an effective approach to resolve the biased exploration. We believe that PQR can be combined with other distributional RL or risk-sensitive algorithms as a perturbation- exploration method without sacrificing their original objectives.
Acknowledgements
This work was supported in part by National Research Foundation of Korea (NRF, 2021R (02IRIA2C20155042229 and 202MM3FA2400337990000 in part by Institute of Informa- tion & communications Technology Planning Evaluation (IIP)) grant funded by the Ministry of Science and ICT (MSIT) 2021-0-00100615%% 0021-0-02068(1544 0021-0001800 15%). and 0021-0-0134(155%)) and in part by AI Graduate School Program, CAU, INMAC and BK21 FOUR
program
References
[1] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[2] Mare G Bellemare, Will Dahney, and Rémi Munos. A distributional perspective reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1.X.vv1606.0155000 2016.
[4] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
1rX1v:1812.001100 2018.
[5] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via g-ensembles. arXiv preprint 1706.0100,. 2017.
[6] Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional deep reinforcement learning with mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA), pages 9791-9797. IEFE, 2019.
[7] Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems, 27, 2014.
[8] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: cvar optimization approach. arXiv preprint arXiv:1506.022 02888888008..... 2015.
[9] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 1 (8(())6000-611)) 2017.
[10] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic. arXiv preprint 1910. 12807, 2019.
[11] William R Clements, Bastien Van Delft, Benott-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint 19X.v:1905.09638. 2019.
10

---

[12] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018.
[13] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[14] Jan Dhaene, Alexander Kukush, Daniël Linders, and Qihe Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal, (2):319-3283 2012.
[15] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathemati- cal Programming. 171(1):115-666, 2018.
[16] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney. Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
[17] Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang. Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning, pages 4607-4616. PMLR, 2021.
[18] Chi Jin, Zeyuan Allen-Zhi Zhu, Sebastien Bubeck, and Michael Jordan. q-leaming provably efficient? Advances in neural information processing systems, 31, 2018.
[19] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4436 4443, 2020.
[20] Marlos Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Reseurch, 61:523-562, 2018.
[21] Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning, pages 4424 4434. PMLR, 2019.
[22] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-55 2015.
[23] Thomas Moerland, Joost Broekens, and Catholijn M Jonker. The potential of the return distribution for exploration in rl. arXiv preprint 1rXiv:1806.0000 04242, 2018.
[24] Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tac- tical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems, 144992844-1286,, 2021.
[25] Thanh Nguyen- Tang, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning via moment matching. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 9144-9152 2021.
[26] Jihwan Oh, Joonkee Kim, and Youn Yun. Risk perspective exploration in distributional reinforcement learning. arXiv preprint arXiv:2206. /4/70, 2022.
[27] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4034 2016.
[28] Ian Oshand, Benjamin Van Roy, Daniel Russo, Zheng Wen, et Deep exploration via randomized value functions. J. Mach. Learn. Res., -06, 2019.
11

---

[29] John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN- based agents, 2020. URL BtRL:htp:///gg thub. com hoom//depmmndddgn....
[30] Mare Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning, arXiv preprint arXiv:2102.05762, 2021.
[31] R Tymell Rockafellar, Stanislav Uryasev, al. Optimization of conditional value-at-risk. Journal of risk, 2:21 2000.
[32] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezynski. Lectures on stochastic programming: modeling and theory. SIAM, 2021.
[33] Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint 10XI::1902208708,, 2019.
[34] Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A cvar optimization approach. IJCCI, pages 412-423, 2019.
[35] Yunhao Tang and Shipra Agrawal. Exploration by distributional reinforcement learning. arXiv preprint arXiv: 0100.000009777 2018.
[36] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992.
[37] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
[38] Derek Yang. Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie- Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems, 32:6193 12:6193-6002, 2019.
[39] Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions Automatic Control, 2020.
[40] Shangtong Zhang and Hengshuai Yao. Quota: The quantile option architecture for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5797-5804, 2019.
[41] Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. arXiv preprint arXiv:2004. 10888, 2020.
[42] Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. arXiv preprint arXiv:2105.06696 2021.
12

---

A Proof
A.1 Technical Lemma
Before proving our theoretical results, we present two inequalities for supremum to clear the descrip- tion.
1. (x)ls())+ +
S sup [x))] + suplg(z)) =eX
=EX
=ex
2. sup fx=)
sup sup g(x))
sup
lf(x) - g('')l
=eX z'EX
Proof of 1. Since (sx)) + s(x) s [((x) + |-(وو
holds for all x E X,
sup
[f(x) + g(Φ)l - =ex
(x)] + lo(x)))
=eX
S sup
sup [g(x) sex xeX
[f(=))+
Proof of 2. Since
||a|| - ||a||
S ||a - bll for any norm || and for a large enough M,
sup xx,''
[xx) g(x'') > supls(s) g(x))
=ex
supl(f(x) M) - (g(x) + M))
xex
z sfp(f(x) + M) - sup (g(x) + M) #ex
zEX
sup f(x) sup g(z') x'eX
=EX
A.2 Proof of Theorem
A.3
Theorem A.3 If & converges to 1 in probability on Ω2 then ET s. converges ET uniformly Z for all s and a e A.
Proof. Recall that 2 3:8xx x 9 [ 2(s,,)))) S Vmax, v(s,a)). Then for any Z E Z
and { E
Rmax
Rmax
T Rmax + 7 1 1-7
Vmax-
which implies PDBOO is closed in Z, i.e. T E Z for all & E E. Hence, for any sequence & z(n) = T Since & converges in probability on Ω there exists T such that for any c, and t
s... Z Z for any n 2 0.
P().)= =P (fwr E Ω: sup (w)) - 1| 2 (22) }) 5 wen
For any 7 E 2, 8 S, E A, and > T, by using Hölder's inequality.
sup sup Zez 8.0
[E [2(, a)) - E (8, a)] sup sup Zez saa
(1- (I-E(w))2(s,,, w))
men
sup sup ZeZ 8.a
(1- & (w)) (w)))()))))))))))) a, w) P((w) +
(1 (1=((w))Z(s,,)) a. w)
wEfh
P(2)) sup ((((w) 1 Vina + P(L)()) w
sup
(6((w) - 1| Vnax
wEf,
< 8 - 1| max + e eVimx
13

---

which implies that Eg, converges to E uniformly on Z for all 8, a.
By using
A.I we can get the desired result.
sup sup
E T c. (s,
E ((,
zez
S sup sup
| & ((,,a)] - EE, (T (8,a)] + sup sup |R T e, as) ElT ((x ZEZ
ZEZ
S Bg - mmxx + <Vimax + sus sup Ey ZEZ 8.0
sup Bg [Z a')l - sup E|z(s' [((s'''''
d" 1 <Vmax)
o' [R (s',, a')] ([Z(s',a')]
S Bg 1|V + +eV <Vima) + ssu sup ZEZ ,'
Bg 1/V mmnx)) + (88
= (1 + ()) B - 1 mmax + c mmma)).
A.3 Proof of Theorem
Theorem 3.3 Let En be sampled from Us, (Z(n-1)) for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators ET e... ET..... converges, i.e. ET e.. [2] - E. 2*1 Moreover, the following bound holds,
2 Sp + 2 k
sup B B[ (s,a)] - (s, - Σ
+
r 2-A
8,0
k=n
Rg,, 12, -- (s', the greedy action of z(==1) under
Proof. We denote (E.) argmax
a'
perturbation En. Also, we denote supl which is the supremum norm over s and as || Ilsa.
8.a
Before we start from the term
| ||
saa for given (s,a),
E (++ a)
ssu BLZ (E - (s' a(x)))
(111 dhss
((++ max BiZ (5', a'''l
+ max (4 max E (k-
1) (5', ,a)]
+ max (5', [')o (1 (s) (Ex)))
B (s', d')] - B[z(k-1) (6 a')] + sup E ( (s',a' (G++1) max E (0 'a'']
duss
8',''
k +7 Σ
$ B[z(a]] ElZ(A-11) (1-111)) 171
sup B| (0) (s', a (((+4))) - Es (s a (Ex+1))]
i=
+ max BeC (2(0 (s',a'))
max E (1) (s', a [
k Σ
$ >> [() E[Z(2-1)) 17]
+ 2y
sup
E|Z (1) (('''') - Egu., Bs (s, a'))
s'
i=
<> B[ (R]
+2y
A++1
i=--1
where we use A.I in third and fifth line and A. in sixth line.
14

---

Taking supremum over s and a, then for all k > 0.
k + Σ
E| [[z B[Z(k]] s0
S < S7| [
B|Z 11
A
i=k-1
k-1
&
53² E1 - 112 21
+2
Σ
22
A (1
+2
Σ
7
i=k-2
1=
k +
1 5* 1 - E||2]
1=1
s Vmax + 2
7 (4222 + Apt1-)
1=1
Since Ei 72 1-g oo and Ei= Ai oc by assumption, we have
b Σ
which is resulted from the convergence of Cauchy product of two sequences (7*) and (Ai}. Hence, [ is a Cauchy sequence and therefore converges for every Z E Z.
Let I [| be the limit point of the sequence [E|Z
1]. Then,
E |z - E E[
lim
( - E[Z(n)]
[ E|2(00)
k=
27 2 max + 2 Σ 7 (A A + Akt1-i).
A.4 Proof of Theorem 3.4
Theorem 3.4. If {A,} } follows the assumption in Theorem 3.3 then IE[Z"] is the unique solution of
Bellman optimality equation.
Proof. The proof follows by linearity of expectation. Denote the Q-value based operator T. Note that An converges to 0 with regularity of 2 implies that &., converges to 1 in probability on Q i.e.,
/
lim sup a 00000
(n)) (wr 8, (1 - E. (w)))P(( = 0
8.0
WEB
lim IP (fw E Ωt |1 En (w) 2 2 =
By Theorem A.3 for given 0, there exists constant K max (K1, K2) such that for every
k 2 K1,
sup zez
| c.. B[2] TB|Z sa S 2
Since T is continuous, for every k 2 K2
C < 2 7 -
15

---

Thus, it holds that
| cx+1 B[ T
S | B|Z(k) TE|2(4)] + ||
-
sup TT Ex+1 [iZ]
T
PM || -
ZEZ
+
2 2
Therefore, we have
E|Z'] lim E[Z(0]] lim (
lim T Ex-1
2
lim T scti Te E[zk)] TE[Z" k--no
8-000
Since the standard Bellman optimality operator has unique solution, we derived the desired result.
B Algorithm Pipeline
Figure 8 shows the pipeline of our algorithm. With the schedule of perturbation bound { J the ambiguity set Ua., -1) can be defined by previous Zn-1- For each step, (distributional) perturbation &n is sampled from Us (Zn-1) by the symmetric Dirichlet distribution and then PDBOO e.. can be performed.
Za
Teu
Zi
Z
T
Zn
Z
&a
Ma, (Zo)
Us
(Z
An
Figure 8: Pipeline of PDBOO.
16

---

C
Implementation details
Except for each own hyperparameter, our algorithms and DLTV shares the same hyperparameter and network architecture with QR-DQN [3 for fair comparison. Also, we set up p-DLTV by only multiplying a gaussian noise (O, 1) to the coefficient of DLTV. We do not combine any additional improvements of Rainbow such double Q-learning, dueling network, prioritized replay, and n-step update. Experiments on LunarLander- v2 and Atari games were performed with random seeds. The training process is 0-2% slower than QR-DQN due to the sampling & and reweighting procedures.
C.1 Hyperparameter Setting
We report the hyperparameters for each environments we used in our experiments.
Table 2: Table of hyperparameter setting
Hyperparameters
N-Chain
LunarLander
Atari Games
Batch size
Number of quantiles
200
170
200
st-p updates
Network optimizer
Adam
Grid search[0.05, 0.5, 1] x x1"
Memory size
le6
le5
le6
Learning rate
5e-5
1.5e-3
5e-5
0.9
0.99
0.99
Update interval
Target update interval
25
le4
Start steps (train)
le4
LinearAnnealer(1 + 1c-2)
(test)
le-3
le-3
le-3
decay steps Coefficient
2.5e3
le5
2.5e5
Grid search[1eO, 5e0, lel, 5e1, le2, 5e2, 1e3, Se3)
Ag
5e2
5e4
le6
Number of seeds
C.2 Pseuodocode of P-DLTV
Algorithm 2 Perturbed DLTV (D-DLTV)
Input: transition (s, a, 8"),, discount y E [0, 1) Q(s', d') a/4 E, (s'ss a')
a NO, Int
// Randomize the coefficient
a² argmax, /. (Q(8', a') + a Le (s(),
T0; + +r++00,(s (5','''''''') a²), Vj
Output: Σmn E (TO, (s,
D Further experimental results & Discussion
D.I N-Chain
To explore the effect of intrinsic uncertainty, we run multiple experiments with various reward settings for the rightmost state as keeping their mean at 9. As the distance between two Gaussians was increased, the performance results. The result implies the interference of one-sided tendency on risk is proportional to the magnitude of the intrinsic uncertainty and the randomized criterion effective escaping from the issue.
DLTV decrease gradually, while other algorithms show consistent
17

---

Table
Total counts of performing true optimal action with different seeds.
Total Count
(8,10)
(7.11)
(6.12)
(5.13)
(4.14)
(3.15)
(2.16)
(1.17)
QR-DQN
12293
11381
11827
12108
10041
11419
9696
11619
DLTV
9997
9172
9646
9251
7941
6964
7896
7257
P-DLTV
14344
14497
13769
15507
14469
14034
14068
13404
PQR
14546
15018
14693
13142
15361
13859
14602
14354
D.2
LunarLander-v2
action stochasticity
LunarLander-v2
random initial force
200
extreme reward system
QRDON
DLTV
-100
P-DLTV
+100
0.05
0.20
0.15
0.20
0.25
0.30
Steps (Million)
Figure 9: (Left) Three main environmental factors causing high intrinsic uncertainty on LunarLander- v2. (Right) Performance on LunarLander-vv
To verify the effectiveness of the proposed algorithm in the complex environment with high intrinsic uncertainty, we conduct the experiment on LunarLander-v2. We have focused on three main factors that increase the intrinsic uncertainty from the structural design of LunarLander environment:
Random initial force: The lander starts at the top center with an random initial force.
Action stochasticity: The noise of engines causes different transitions with same action.
Extreme reward system: If the lander crashes, it receives -100 points. If the lander comes rest, receives +100 points.
Therefore, several returns with fixed policy have a high variance. As previously discussed about the fixedness from N-Chain environment, we can demonstrate that randomized approaches, PQR and P-DLTV, outperform other baselines in L.ndrr.ander-v2
D.3 tari games
Gravitar-vd
Kangaroo-v4
Pong-v4
Seaquest-v4
1000
12008
4300
Avarage Scote
L2548
Avverage Sccom
-000e
Average
600
GADON
CLTV
DLTV
POR
Training
310
330
330
50
150
33
30
Training
Ut0 253
Figure 10: Evaluation curves Atari games. All curves smoothed over 10 consecutive steps with three random seeds. In case of Pong-v4, we resize the x-axis, since it can easily obtain the optimal policy with few interactions due environmental simplicity.
We test our algorithm under 30 no-op settings to align with previous works. We compare our baseline results with results from the DQN Zoo framework [29]. which provides the full benchmark results on 55 Atari games at 50M and 200M frames. We report the average of the best scores over seeds for each baseline algorithms up 50M frames.
18

---

However, recent studies tried to follow the setting proposed by Machado et al. 200 for reproducibility. where they recommended using sticky actions. Hence, we provide all human normalized scores results across 55 Atari games for 50M frames including previous report of Dopamine and DQN 7o0 framework to help the follow-up researchers
reference, We exclude Defender and Surround which is not reported on Yang et al. [38 because of relialbility issues in the Dopamine framework. In summary,
DQN Zoo framework corresponds to 30 no-op settings (version v4).
Dopamine framework corresponds to sticky actions protocol (version v0).
D.3.1
Ablation on PQR schedules
Ablation PQR schedules
20
10
-10
PQR
OPT
-20
10
35
Training Frames (Million)
Figure 11: Evaluation curves on Pong-v4 environments.
To investigate the effect of the schedule of A., we run the experiment on Pong-v4 and set up several baselines follows:
1/t0 A fixed size ambiguity set. A, = 0(1)
PQR Our main algorithm. A, = (1)
OPT We fix the output vector, sampled from the Dirichlet distribution, as [0, o.... 1], forcing the agent to estimate only optimistically.
Vlogt/i t/t We imitate the schedule of P-DLTV (which does not satisfy the sufficient condition presented). A, 0 logt/t)
Vlogt/t t/t OPT We imitate the schedule of DLTV (which does satisfy the sufficient condition we presented). We fixed the output vector, sampled from the Dirichlet distribution, as [0, 0, o,..,...11 1), ,..,,,,], forcing the agent to estimate only optimistically. A, = 10 Tog l/t)
In this experiment, our proposed PQR is the only method that stably achieves the maximum score with low variance. In the case of optimism (purple, orange curve), the agent learns quickly in the early stages, but converges without reaching the maximum score. In the of fixed ambiguity set (blue curve), converges to suboptimal and eventually shows low performance. This result implies the necessity of time-varying schedule of A. Finally, when imitating the schedule of P-DLTV (green curve), the performance also degrades implying that the proposed sufficient condition is quite tight.
19

---

D.4 Comparison with QUOTA
Zhang and Yao [40] have proposed Quantile Option Architetture(UUOTA) which derives different policies corresponding to different risk levels and consider them as options. By using an option- based framework, the agent learns a high-level policy that adaptively selects pessimistic or optimistic exploration strategy. While QUOTA has a similar approach in high-level idea, PQR gives lot of improvements in both theoretical analysis and experimental results.
Theoretical guarantees of convergence toward risk-neutrality.
Since the agent selects via randomized risk criterion, the natural question "How should we control the injected randomness without sacrificing the original purpose of risk-neutrality?". In this work, we provide the sufficient condition for convergence without sacrificing risk- neutral perspective. Although QUOTA explores by using optimism pessimism of value distribution, there is no discussion whether the convergence is guaranteed toward a risk-neutral objective.
Explaining the effectiveness of randomized strategy.
QUOTA tested on two Markov chains to illustrate the inefficiency of expectation-based RL. It assumed that each task has an inherent, but unknown, preferred risk strategy, so agents should learn hidden preference. In contrast, we point out that the amount of inherent (intrinsic) uncertainty causes the inefficiency of fixed optimism or pessimism based exploration.
Significant performance difference in experimental results.
QUOTA is based on option-bacc based learning which requires an additional option- value net- work. While QUOTA aims to control risk-sensitivity by transforming into option O the introduction of an option-value network requires the agent to explore an action space O1 [A]. This opposes the idea of efficient exploration as factor that increases the complexity of learning. In contrast, PQR does not require a additional network and explores over the original action space. In addition, PQR does not artificially discretize the ambiguity set of risk measurement. Another main reason is that PQR does not depend on an greedy schedule which is well-known for inefficient exploration strategies in tabular episodic MDP (18). PQR solely explores its own strategies which is a simple yet effective approach. However, QUOTA depends greedy schedule in both quantile and option networks.
D.5 Reproducibility issues on DLTV
For the expected concerns about the comparison with DLTV, we address some technical issues to correct misconceptions of their performance. Before reproduce the empirical results of DLTV, Mavrin et al. [21] did not report each raw scores of Atari games, but only the relative performance with cumulative rewards comparing with QR-DQN. While DLTV reported to have cumulative reward 4.8 times greater than QR-DQN, such gain mainly comes from VENTURE which is evaluated as 22,700% from their metric (i.e., 463% performance gain solely). However, the approximate raw score of VENTURE was 900 which is lower than our score of 993.3. Hence, the report with cumulative rewards causes a severe misconception that can be overestimated where the human-normalized score is commonly used for evaluation metrics. For a fair comparison, we computed based on mean and median of human-normalized scores and obtained results of 603.66% and 109.90% Due to the absence of public results, however, DLTV was inevitably excluded from the comparison with numan--ormallln score in the main paper for reliability. In Table and we report our raw scores and human- normalized score of DLTV based on QR-DQN zoo performance.
Table 4: Performance comparison on QUOTA, DLTV, and PQR.
QUOTA SQR-DQN Zhang
QR-DQN SQR-DQN Zhang
SQUOTA
SQR-DQN Zhang
SDLTV
30
34
42
42
39
Ang HIN Score QR- zoo)
Avg HIN Score(QR-DOV DON Zhang)
Arg HIN QUOT)(
Avg HIN Score(DLTV)
Avg HIN Score(PRR)
463.17
383.70
1078.00
Med HN QR-NN 200)
Med HN ScorecQR-DDO Zhang)
Med HN QUOVEA)
Med HN Soore(DL.V)
Med HN Seore(PQR)
120
78.07
91. 08
109 9K)
129.25
20

---

Table Raw scores across all 55 games, starting with no-op actions. We report the best scores for DQN, QR-DQN. IQN and Rainbow were provided by DQN Zoo framework (29].
50M frames, averaged by 5 seeds. Reference values
Bold are wins against DQN, QR-DQN and IQN, and
#asterisk wins over Rainbow.
GAMES
RANDOM
HUMAN
DQN(SOM)
QR-DQN(SOM)
IQN(5OM)
KAINBOW(5OM) FQR(S0M)
Alien
227.8
7127.7
1541.5
1645.7
1769.2
4356.9
2455.8
Amidar
5.8
1719.5
324.2
6834
799.2
2549.2
938.4
Assault
222.4
742.0
2387.8
11684.2
15152.4
9737.0
10759.2
Asterix
210.0
8503.3
5249.5
18373.4
32598.2
33378.6
10490.5
Astemids
719.1
1825.4
Atlantis
12850.0
29028.1 1
283392.2
937275 0
865360.0
941740.0
897640.0
BankHeist
14.2
753.1
389.0
1223.9
1266.8
1081.7
1038.8
BattleZone
2360.0
37187.5
19092.4
26325.0 ,
30253,9
35467.1
28470.5
BeamRider
363.9
16926.5
7133.1
12912.0
19251.4
15421.9
10224.9
Berzerk
123.7
2630.4
577.4
826.5
918.9
2061.6
137873.11
Bowling
23.1
160.7
34.4
45.4
41.5
54.7
86..
Boxing
87.2
99.6
99.8
97.1
Breakout
1.7
30.5
316.8
426.5
468.0
335.3
380.3
Centipede
12017.0
7124.0
ChopperCommann
811.0
7387.8
974.2
1187.8
1.549.0
5525.1
1300.0
CrazyClimber
10780.5
35829.4
96939.0
93499.1
127156.5
160757.7
84390.9
DemonAttack
152.1
1971.0
8325.6
106401
110773,
85776.5
73794.0
DoubleDunk
-18.6
-16.4
-15.7
-10.5
-12.1
-0.3
-7.5
Enduro
860.5
750.6
2105.7
2280.6
2318.3
#2341.
Derhh
91.77
-38.7
8.2
25.7
23.4
35.5
31.7
Freeway
24.4
33.3
34.0
34,0
Prostbite
65.2
4334.7
408.2
3859.2
5650.8
9672.6
4148.2
Gopher
257.6
2412.5
3439.4
6561.9
26768.9
32081.3
44705..
Gravitar
173.0
335
180.9
548.1
470.2
2236.8
635.88
Hero
1027.0
30826.4
9948.3
9909.8
1249
38017.9
12579.2
IceHockey
-11.2
0.9
-11.4
-4.2
-1.4
Jamesbond
29.0
302.8
486.4
1163.8
1058.0
14415.5
2121.8
Kangaroo
14558.2
14256.0
14383.6
*14617.1
Krull
1598.0
2665.5
7130.5
9612.5
9616.7
8328.5
+9746.1
KungeuMaster
258.5
22736.3
21330.9
27764.3
39450.1
30506.9
43228.6
MontezunaRevengg
0.0
4753.3
0.3
0.0
0.2
80.0
0.0
MsPacman
307.3
6951.6
2362.9
2877.5
2737.4
3703.4
2928.9
Name ThimGame
2292.3
8049.0
6328.0
11843.3
11582.2
11341.5
10298.2
Phoenix
761.4
7242.6
10153.6
35128.6
29138.9
49138.8
20453.8
-229.4
.9.5
Pong
-20.7
14,6
18.7
20.9
20.9
21.0
21.0
PrivateEye
24.9
69571.3
266.6
100.0
100.0
160.0
$372.4
Qbert
163.9
13455.0
$567.9
12808.4 .
15101.8
24484.9
15267.4
Riveraid
1338.5
17118.0
6782.8
9721.9
13555,9
17522.9
11175.3
RoadRunner
11.5
7845.0
29137.5
54276.3
53850.9
52222.6
50854.7
Robotank
2.2
11.9
31.4
$4.5
53.8
64.5
60.3
Seaquest
68.4
42054.7
2525.8
7608.2
17085.6
3048.9
#19652.5
Skiing
17799.1
4336.9
13930.8
14589.7
19191.
15232.3
..929.3
Solaris
12326.7
1857.3
1301.5
2522.6
IpaceInvaders
148.0
1668.7
1179.1
1753.2
2906.7
2715.3
1749.4
Star Gunner
664.0
10250.0
24532.5
63717_3
78503.4
107177.8
62920.6
Tennis
-23.8
-8.3
00.9
0.0
0.0
0.0
-1.0
TimePilox
3568.0
5229.2
2091.8
6266.8
6379.
12082.1
6506.4
Tutankham
11.4
167.6
138.7
210.2
204.4
194.3
233.3
UpNDowa
533.4
11693.2
6724.5
27311.3
35797.6
65174.2
36008.1
Venture
53.3
12.5
VidcoPinball
16256.9
17667.9
140528.4
104405 8
341767.5
465636.5
465578.3
WizardOrWor
563.5
4756.5
3459.9
14370.2
10612.1
12056.1
6132.8
YarsRevenge
3092.9
54576.9
16433.7
21641.4
21645.0
67893.3
27674.4
Zaxxon
32.5
9173.3
3244.9
9172.1
8205.2
22045.8
1080666
21

---

Table 6: Raw scores across 55 games. We report the best scores for DQN, QR-DQN, IQN*, and Rainbow on 50M frames, averaged by 5 seeds. Reference values were provided by Dopamine framework 4 Bolds are wins against DQN, QR-DQN, and "asterisk are wins over IQN and Rainbow. Note that IQN and Rainbow implemented in Dopamine framework applied 7-step updates with n 3 which improves performance.
GAMES
RANDOM
HUMAN
DQN(50M)
(R-DON(5OM)
(5ND)
RAIN
PQR(50M)
Alien
227.8
7127.7
1688.
2754.2
4016.3
2076.2
3173.9
Amidar
5.8
1719.5
888.2
841.6
1642.8
1669.6
*2814.7
Assault
222.4
742.0
1615.9
2233.1
4305.6
2535.
#8456.5
Asterix
210.0
8503.3
3326.1
3540.1
7038.4
5862.3
* 19004.66
Asteroids
719.1
47388.7
828.
1333.4
1336.3
1345
8518
Atlantis
12850.0
29028.1
388466.7
879022.0
897558.0
870896.0
880303.7
BankHeist
14.2
753.1
720.2
964.
1082.8
1104.9
1050.1
BattleZone
2360.0
37187.5
15110.3
25845.6
29959.7
32862.1
*61494,4
BeamRider
343.9
16926.5
4771.3
7143.0
7113.7
6331.9
$12217.6
Bowling
23.1
160.7
38.5
55.3
33.6
55.0
*174.1
Boxing
0.1
12.1
80.0
96.6
97.8
96.3
96.7
Breakout
1.7
30.5
113.5
40.7
164.4
69.8
48.5
Centipede
2090.9
12017.0
3403.7
3562.5
3746.1
5087.6
33009,
ChopperCommand
811.0
7387.8
1615.3
1600.3
6654.1
$982.0
4653,9
CrazyClimber
10780.5
35829.4
111493.8
108493.9
131645.8
135786.1
105526.0
DemonAttack
152.1
1971.0
4396.7
3182.6
7715.5
6346.4
O 19530.2
DoubleDunk
-18.6
-16.4
-16.7
7.4
20.2
17.4
15.0
Enduro
2268.1
2062.5
766.5
2255.6
1765.5
FerbyngDerby
-91.7
-38.7
12.3
48.4
41.9
37.6
46.8
Freeway
0.0
29.6
25.8
33.5
33.2
33.0
Frustbite
65.2
4334.7
760.2
8022.8
7824.9
$697.2
"8401.5
Gopher
257.6
2412.5
3495.8
7102.1
12252.9
Gravitar
173.0
3351.4
250.7
821.3
1083.5
926.2
703.5
Hero
1027.0
30826.4
12316.4
14980.0
18754.0
31254.8
15655.8
IceHockey
-11.2
0.9
6.7
4.5
0.0
2.3
Jamesbond
29.0
302.8
500.0
802.3
1118.8
656.7
#1454.9
Kangaroo
52.0
3035.0
6768.2
4727.3
11385.4
13133.1
$13894.0
Krull
1598
2665.5
6181
8073.9
8661.7
6292.5
331977.4
KungFuMaster
258.5
22736.3
20418.8
20988.3
33099.9
26707.0
22040.4
Montezunakevenge
0.0
4753.3
2.6
300.5
0.7
501.2
0.0
MsPaeman
307_3
2727.2
3406.4
#5426.5
SomeeThisGame
2292.3 3
8049.0
$697.3
7307.9
9432.8
9389.5
1_81
761.4
S833.7
Pitfall
-29.44
6463.7
-16.8
-3.4
-0.4
0.0
*0.0
Pong
-20.7
19.4
19.7
PrivateEye
24.9
69571.3
1884.6
680.7
1287.3
4298.8
$12806.1
Qbert
163.9
13455
8216.2
17228.0
15045.5
17121
15806.9
Riverrid
1338.5
17.
9077.8
13389.4
14868.6
15748.9
14101_3
RoadRunner
11.5
7845.0
39703.1
44619.2
50534.
51442.4
48339.7
Robotank
2.2
11.9
25.8
53.6
65.9
63.6
48.7
Seaquest
68.4
42054.7
1585.9
4667.9
20081.3
3916.2
5038.
Skiing
-17098.1
-4336.9
-17038.2
-14401.6
-13755.6
-17960.1
90221
Solaris
1236.3
12326.7
2029.5
2361.7
2234.5
2922.2
#7145.3
Spacelnvaders
148.0
1361.1
StarGunner
664.0
10250.0
1676.5
23593.3
60090.0
39456.3
594044,6
Tennis
-23.8
-9.3
19.2
$15.4
TimePilot
3568.0
$229.2
3200.9
6622.8
9820.6
9324.4
$597.0
Tutankham
167.6
138.8
209.9
250.4
252.2
147.3
UpNDown
533.4
11693.2
10405.6
29890.1
44327.66
18790.7
32155.5
Venture
0.0
1187.5
50.8
1099.6
1134.5
1488.9
1000.0
VideoPinhull
16256.9
17667.9
216042.7
250650.0
48611
536364.4
46086009
WizardOWWor
563.5
4756.5
2664.9
2841.8
679144
7562.7
$738.2
ReveRevenge
3092.9
54576.9
20375.7
66055.9
$7960.3
31864.4
967545.8
Zaxxon
32.5
9173.3
1928.6
8177.2
12048.6
14117.5
9531.8
22

---

Table 7: Raw scores across all 49 games, starting with 30 no-op actions. We report the best
scores for QR-DQN 29). QR-DQN Zhang 40 (implemented by QUOTA to evaluate the relative
improvement) for fair comparison and QUOTA 40), DLTV 40M frames, averaged by seeds. Bold are wins against QUOTA and DLTV.
Games
Random
Human
QR-DQN 20o(40M)
QR-DON Zhangg40MM
QUOTA(4MM)
DLTV(40V)
PQRC4NN5
Amidar
5.8
1719.5
552.9
567.9
571A
1042.7
644.1
Assmlt
222.4
742
9880.4
3308.7
3511.1
$896.2
10759.2
Asterix
210
8503.3
13157.2
6176.0
6112.1
6336.6
8431.
Asteroids
719.1
47388.7
1503.9 9
1305.
1497.6
1268.7
1416.00
Atlantis
750190.
978385.3
845324.9
897640.0
BankHeist
14.2
753.1
1146.1
644.7
735.2
1183.7
1038.8 8
BattleZone
2360
37187.5
17788.4
22725.0
25321.6
23315.8
28470.5
BeamRider
16926.5
5007.8
Rowling
23.1
160.7
44.3
27.6
34.0
29.8
86.9
Boxing
0.1
12.1
98.2
95.0
96.1
112.8
97.1
Breakout
322.1
316.7
Centipede
2090.9
12017.0
6633.0
4130.3
3537.9
4676.7
6803.6
Chopper
811.0
7387.8
3421.
3793.0
1500
Crimbere
10780.5
35829.4
93499.1
107371.6
113051.7
92769.1
83900.0
DemonAttak
152.1
1971.0
98063.6 6
80026.6
61005
146928.9
73794.
DoubleDunk
Enduro
0.0
860.5
2105.7
1220.0
1162.3
5665.9
2252.8
FishingDerby
-9.77
-38.7
25.7
9.66
-9.00
-8.2
31.7
Freeway
31.0
Frustbite
65.2
4334.7
3822.7
2046.3
2208.5
3867.6
4051.2
Gopher
257.6
2412.5
4191.2
9443.8
6824.3
10199.4
470545
Gravitar
173.0
3351.4
477.4
414.3
457.6
357.9
583.6
loeHockey
-11.2
0.9
-2.4
-9.8
-9.9
-14.3
2211
Jamesbond
Kanguroo
52.0
3035
14171
2364.6
2555.8
4596.7
14385.
Krull
1598.0
2665,5
9618.2
7725.4
7747.5
10012.21
9537.0
Kungl MunseuMasttr
27576.5
23078.4
Moviezunaaevenge
0.0
4753.3
0.0
0.0
0.0
0.0
0.0
MsPacman
307.3
6951.6
2561.
2273.3
2423.5
3191.7
2895.6
Name ThisGame
2292.3
17700.0
7748.2
7327.5
Pitfall
-229.4
6463.7
0.0
3329
-30.7
0.0
Pong
20.9
19.6
20.0
21.0
PrivateEye
24.9
69571.3
100.0
419.3
114.1
1358.6
372.4
Obert
163.9
13455.0
8348.2
10875.3
11790.2
15856.2
14593 0
Riverraid
17118.0
10487.3
RoadRunner
11.5
7845.0
52575.7
27640.7
27872.2
49255.7
44341.0
Robotank
2.2
11.9
50.4
45.1
37.6
58.4
$3.9
Seaquest
42054.7
1690.5
Spucelnvaders
148.0
1668.7
1281.8
1387.6
1553.8
1498.6
1562.6
StarGunner
664.0
10250.0
53624.7
49286.6 6
52920.0
53229.5
55475.0
Temnis
-23.8
-8.3
-22.7
-23.7
-18.4
1.0
TimePilor
3568.0
$229.2
6243.4
6417.7
$125,
6931.1
6506.4
Tutankham
11.4
UpNDown
533.4
11693.2
22248.8
30443.6
24912.7
443866.
33786.3
Venture
0.0
1187.5
12.5
5,3
26.5
1305.0
0.0
VideoPinhall
104227.2
123425.4
44919.1
WizardOrWar
$63.5
4756.5
13133.8
$219.0
4582.0
9582.0
6132.8
Zaxxon
32.5
9173.3
7222.7
6855.1
8252.8
6293.0
10250.0
23


Reviews:
[{"review_id": "8gnY06NUQN", "reviewer": "Reviewer_B5Gh", "summary": "This paper addresses exploration via distributional RL which has, to date, been most recently performed under an _optimism in the face of uncertainty_ (OFU) paradigm. This paper loosely characterizes this OFU approach as problematic as it results in biased exploration, resulting in sub-optimal value distributions. To remedy this pitfall, this paper proposes a stochastic perturbation of the distributional Bellman operator. Clear theoretical development of the proposed operator gives way to practical algorithm development building on top of quantile regression. Empirically, the proposed algorithm PQR is demonstrated to have clear performance benefits.", "questions": "Is the guarantee of risk-neutrality found in expectation? E.g. is it a response to evenly choosing a risk-averse obj vs a risk-seeking obj?\n\nThe writing in lines 41-45 is pretty difficult to follow. It’s apparent that the authors are taking care to concisely convey their thoughts but fail to do so clearly.\n\nLine 46: It’s not clear what “side-effect” is being referred to here. Additionally, it would be helpful to perhaps directly reference what section in the paper is being referred to in this sentence as the authors indicate that they’ve explored the effects of OFU in deep RL and the effects it has on handling the two types of uncertainty.\n\nLine 61: Could you call PQR a “stochastic optimization” approach to help situate the work among other prior work? This isn’t a major concern of mine. I’m just thinking that it could be an easier way to describe the approach rather than calling it a “randomized” approach.\n\nLine 93: The loss function here is simply the Huber loss, no? It’s not overly important to include this much detail to be honest unless the paper heavily develops further insight directly from these equations (applies to Section 2 in it’s entirety). Otherwise this just feels like a regurgitation of information that’s already established in the prior literature.\n\nLine 128-129: Why is it important to have an interpretation of different risk levels as constructing an options framework? How does this help build the proposed methods introduced in this paper?\n\nLine 178: Does “the standard” here mean the standard distributional Bellman operator?\n\nLine 186: What is $B_\\xi$? How does this relate to the symmetric Dirichlet distribution used to construct each ambiguity set? This seems to be addressed in Lines 210-212.", "limitations": "Several technical statements that crucial to the dvelopment of the proposed methods/contributions are vague and are not precisely justified. For example, it would be really helpful (and would help me agree/accept the claims) in Lines 170-174 if it were shown precisely how Eqt 1 maintains risk-neutral policy performance. As currently written we have to take the authors’ word. Additionally, the statement of using a min-expectation vs. a min-max operator should be developed and justified. This is a new concept that hasn’t been discussed previously.\n\nIt’s not clear how the time-varying amiguity sets are decoupled from a changing estimate of the return distribution Z. As formulated, PDBOO seems to ignore that $Z_\\theta$ is changing through time.\n\nThe paper omits at least one relevant paper, Moskovitz, et al (NeurIPS 2021) where an ensemble of distributional crtiics are used to balance between optimism and pessmism for continous control. It would be informative to hear from the authors why this paper, and its corresponding approach, should or shouldn’t be included as a comparative baseline.\n\n>Moskovitz, Ted, et al. \"Tactical optimism and pessimism for deep reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 12849-12863.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper sets up a really interesting juxaposition between the advantages of the distRL paradigm in accounting for aleatoric uncertainty but attempting to resolve epistemic uncertainty through exploration. As set out in the introduction of the paper, this presents an appealing motivation to perhaps determine more effective exploration strategies that are cognizant of this apparent mismatch.\n\nThe methods presented in this paper to address the one-sided tendency on risk, resulting in a practical algorithm, are reasonable and seem to neatly satisfy the intended contributions set out in the paper.\n\nI was impressed at the extent by which the authors provide sufficient background in distributional RL in Section 2. Many papers take shortcuts here however this paper lays out relevant concepts from which the remaining development of the methods introduced in the paper are easier to understand from this grounding.\n\nThe technical formulation of the PDBOO is clear and well written. The definitions help to outline important concepts that motivate the further development of the methods.\n\nI appreciate how the authors approach Section 4, clearly laying out the objectives of their empirical study. This helps to frame the impressive experimental performance achieved by the introduced PQR algorithm.\n", "weaknesses": "It’s not clear what is meant by “without losing the risk-neutral objective” (mentioned in the abstract)…\n\nIn line 43, it could be helpful to again list out what the two types of uncertainty are to more directly connect this sentence to what has been introduced conceptually earlier in the paper.\n\nThe authors seem to use “deep” RL interchangably with “distributional” RL in the introduction. This makes the framing of the work a little challenging to follow initially. Better clarification of prior works in the non-distributional deep RL from distributional deep RL would help make this easier to understand and would strengthen the beginning sections of the paper.\n\nThe statements made in lines 50-53 are pretty strong, as in they seem to be a result of some analysis or in the worst case are derived from opinionated intuition. Some additional justification with formal explanation (whether through a toy example or reference to later section or citation to a paper) would help make the writing easier to accept and be persuaded by. This is especially important because the “one-sided tendency on risk” appears to be a major foundation for the proposed PDBOO.\n\nIt would be helpful if the authors expanded more on their discussion at the end of Section 2.2 since DLTV appears to introduce biased exploration that is addressed in this work. A couple of sentences addressing the limitations of DLTV and how the work presented in this paper addresses those would be great. The sentence that ends the preceding paragraph (Line 100-101) is a decent example of setting up a clear understanding of the contributions set forth in the paper.\n\nMany of the claims about why PQR performs well on n-chain and in Atari are speculative. The paper would be significantly improved if more rigorous analysis connecting the theoretical treatise in Sections 3.2 and 3.3 to empirical performance and how it differs from the baselines would be great.\n\nThe discussion around the empirical results is somewhat disappointing. It would additionally be interesting to see any ablations that could be done on PQR to help tease apart the effect of the different components of the algorithm have on performance. I suppose that this is is partially addressed with the hyperparameter plots in Figure 6 but I think it would be helpful to see how sensitive the resulting policy is to the scope of the ambiguity sets used.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we appreciate the time and effort you put into our paper. We will revise our paper based on all feedback to improve the quality of the paper. We have responded to the key comments below.\n\n## Clarity of “Without losing the risk-neutral objective”.\n\nPrevious work has focused on learning variations of existing objective functions such as maximum entropy or risk-sensitive. To emphasize that our work does not modify the objective of the existing distributional Bellman update, we refer to it as a risk-neutral objective. However, this does not seem to convey the meaning well, so we will change “without losing the risk-neutral objective” to “to avoid one-sided risk tendency” for clarity.\n\n## Strong statements in line 50-53, unclear what ‘side-effect’ is being referred to here.\nThanks for pointing this out for readability. We will revise the paper as follows: \nIn Section 4, although DLTV is the first attempt to introduce OFU in distRL, we found that consistent optimism on the uncertainty of the estimated distribution still leads to biased exploration. We will refer to this side-effect as \"one-sided tendency on risk\", where selecting an action based on a fixed risk criterion degrades learning performance.\n\n## Connecting the theory to empirical performance and further ablations.\nWhile DLTV and p-DLTV do not satisfy our sufficient conditions, **we additionally experiment with different forms of $\\Delta_t$ scheduling for PQR in global response**, such as $1/t^0$ and $\\sqrt{ \\log t /t}$, for a more robust comparison. We will also provide a table that shows the performance of the resulting policy as the hyperparameter varies.\n\n## Guarantee of risk-neutrality\nRisk neutrality is guaranteed by a sufficient condition on $\\Delta_t$ and updating the target with an unperturbed return of selected action. For example, DLTV is a well-motivated distributional RL method with the unperturbed return updates, but it does not theoretically guarantee the risk neutrality because the bonus decay schedule is heuristically defined by the Hoeffding inequality.  \nEmpirically, selecting behaviors based on randomized risk criteria plays an important role in eliminating the biased exploration of traditional OFUs, where behaviors with high intrinsic uncertainty are explored only because of the consistent positive bonuses (we call this \"one-sided tendency on risk\"). In our experiments, we confirm the above claim by showing that p-DLTV with only Gaussian noise injection outperforms DLTV with fixed optimism.\n\n## Interpreting as a stochastic optimization.\nThere is a slight difference. Stochastic optimization aims to obtain a robust solution from a fixed ambiguity set, while we shrink the ambiguity set to avoid one-sided risk tendency, so our goal is different. The word \"randomized\" is a common expression in the exploration literature [1,2], and we think it is an appropriate description because our approach is aligned with the existing works. We will explain these differences in more detail in lines 170-179 of the revised version.\n\n[1] Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning, pages 4607–4616. PMLR, 2021.\n\n[2] Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20(124):1–62, 2019.\n\n## Relation with $B_{\\xi}$ and the construction of ambiguity set\n $B_{\\xi}$ is a constant that describes uniform boundedness (such as $L$-Lipschitz, smoothness) and we just use it for theoretical rigor. Since we consider $\\xi$ to be the weight for $N$ quantiles, $B_{\\xi}$ is always less than or equal to $N$. It plays no direct role in constructing the ambiguity set. To avoid confusion, we will use this expression only in the appendix for the proof and remove it from the main text.\n\n## How does Eq1 maintain risk-neutral policy performance? More discussion on min-expectation vs min-max operator.\nIf we use a min-max operator with respect to risk, the solution is always more conservative (pessimistic) than the standard solution as it always considers the worst-case scenario. On the other hand, we considered the average-case which has the possibility to maintain risk-neutrality. Specifically, we aimed to find a sufficient condition to ensure risk-neutrality by scheduling the ambiguity set of risks. As such, we only share the definition of ambiguity set by the DRO literature, but with a different goal of risk-neutrality.\nPDBOO seems to ignore that $Z_\\theta$ is changing through time.\nLine 156 defines PDBOO for a fixed xi, but we extend it for a time-varying PDBOO below, denoted $\\xi_t$. The ambiguity set is to sample a perturbation $\\xi$ that yields the same weighted expectation of the return distribution $Z$ for all actions. \n\n[Moskovitz, et al.] as a comparative baseline.\nThanks for suggesting a novel related work. As I understand it, their work controls pessimism/optimism based on the Exponentially Weighted Average Forecasting algorithm in the MAB literature. In this manner, this work has a similar point in terms of changing the risk criteria but it is questionable whether the EWAF algorithm is still valid in deep RL where the agent updates the target and uses a function approximation.\nIt is difficult to use TOP-TD3 as a baseline because our work is based on a value-based algorithm targeting a discrete action space, while TOP-TD3 experimented with continuous control using an actor-critical architecture. Note that our main baselines QR-DQN, DLTV, IQN, and RAINBOW were all experimented on the Atari environment, which is a discrete action space. However, since TOP-TD3 has a similar goal to address intrinsic(aleatoric) uncertainty in exploration, we will add it to our related work.\n"}, {"review_id": "Nl2ky9n7FR", "reviewer": "Reviewer_u65W", "summary": "In this work, the authors address the issue of biased exploration caused by a one-sided tendency on risk in action selection by proposing the method perturbed quantile regression (PQR). PQR selects actions by randoming risk criterion while retaining a risk-neutral objective. The authors also derive a sufficient condition for the convergence of the proposed Bellman operator without satisfying the conventional contraction property. Results are demonstrated in an extended N-Chain environment and the Atari suite showing improved performance.\n", "questions": "I am unsure if I am correctly interpreting the results in Figure 4. Is the lower standard deviation around the dotted line for the mean a1 considered to be a better estimation result? Given that this experimental setting is new (the authors indicate that they adapt the N-Chain environment and cite work - DQN - which presents very different result metrics), substantially clearer explanation is required about the connection between the presentation and the claims. ", "limitations": "The paper does not discuss limitations or broader impact. This discussion should be added.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper was written very well in a pedagogical manner which provided intuition and understanding of the challenges in the field surrounding the contribution of the method. Deep technical detail was provided with accessible explanations.\n\nThe motivation for the method was well grounded intuitively. The experiment section also directly confirmed the intended contributions, organizing each experiment and result around specific questions directly aligned to the motivation.\n\nThe background section was thorough and clear. Since the authors pulled together several topics which many researchers do not study all of, this presentation was very important and added a clear path of accessibility to the technical detail in the rest of the paper.\n\nThe supplementary material provided sufficient information to reproduce results including code. The authors also included substantial supporting theoretical information in proofs and a large amount of additional convincing experiments.\n\nThe contribution to the community regarding managing risk in exploration strategies is important and interesting.", "weaknesses": "\nThe authors claim limitations about epistemic and parametric uncertainty metrics as well as optimism in the face of uncertainty approaches which I thought were not backed up in the paper strongly enough given that these claims are the premise of the entire work. The paragraph in the introduction which presents the limitations describes it as a broad issue, yet only one paper is cited as an example. To provide more context, it would help to give at least more citations here to back up the importance of this challenge. The deeper explanation of the DLTV paper is not necessary for all additional citations.\n\nPlease provide more comments in Algorithm 1 (like the current text “Select greedy action…”). The algorithm is fairly clear but with the density of notation, recalling the variable references would be much easier for the reader with textual reminders in the algorithm block.\n\nMy primary concerns regarding this work are related to the experiments in Section 4.1 which require clarification. Figure 4 is quite hard to read since the text is very small. Reformatting could make these plots readable when the paper is printed. Furthermore, the results in Figure 4 are very hard to understand due not only to the legibility but also due to insufficient explanation in the experimental results section and figure. See Questions for more information. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the time and effort in reviewing our paper. Please find below our response to the main points raised in the review.\n\n## Provide more context to back up the importance of this challenge.\nWe will add a few more papers in line 50 that attempt to use distributions for exploration following DLTV.\n\n- Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a cvar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4436–4443, 2020.\n\n## Provide more comments in Algorithm 1\nThank you for your consideration. **In the global response, we add more detailed explanations in Algorithm 1**, including comments about scheduling for $\\Delta_t$ and refinement to a weighted function.\n\n## Reformatting and more explanation about Figure 4\nThanks for the details. We will try to make the font size as large as possible.\nAlso, you are right that a smaller standard deviation around $a_1$, the blue curve, means a better estimate. Regarding the explanation of Fig. 4, we will add information about the ground truth return distribution of $s_0$ and $s_4$ to the caption of Fig. 3 to make it easier to check whether the estimate of the red/blue curve is close to the ground truth.\n\n- Ground truth return of $s_0$ : $\\gamma^2 \\mathcal{N}(10, 0.1^2)$\n- Ground truth return of $s_4$ : $\\gamma^2 [\\frac{1}{2} \\mathcal{N}(5, 0.1^2) + \\frac{1}{2} \\mathcal{N}(13, 0.1^2)]$\n"}, {"review_id": "vgkpxHqeC7", "reviewer": "Reviewer_qLSL", "summary": "This paper investigates the exploration problem in distributional reinforcement learning (DRL). It proposes the Perturbed Distributional Bellman Operator (PDBOO) as an extension of the distributional Bellman operator, which introduces non-directional noise to the target return distribution. This extension originates from the problem that a one-sided risk tendency can cause biased action selection, which might lead to unsuitable behavior depending on tasks. To address this issue, the PDBOO adds non-directional noise as a perturbation to the return distribution targeted for learning, ensuring risk neutrality. The perturbation term is sampled at each time step. This approach allows for the selection of diverse actions while maintaining risk neutrality. The authors provide a theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to asymptotically converge to the unique fixed point of the Bellman optimality equation. Then, the PDBOO is applied to the Quantile Regression Deep Q Network (QR-DQN) to propose a method called Perturbed Quantile Regression (PQR). The effectiveness of PQR is evaluated in a 4-states chain MDP and 55 Atari games and compared with several DQN variants.", "questions": "- How exactly does the proposed PQR select actions? Does it use an epsilon-greedy or softmax policy? More importantly, how does it compute the values of the actions which will be the input to the policy?  Does it sample the perturbation \\xi, as PDBOO does, and skew the distribution for calculating the mean? I think there are two timings at which the distribution can be perturbed by introducing ξ: during the parameter θ update, and during the action selection. This paper only seems to discuss the former, but considering that p-DLTV performs well similarly to the proposed method, wouldn't it also be valid to perturb the mean for the action selection?\n- What is the type of policy for each baseline used in the experiments? Is it epsilon-greedy or softmax policy? I do not see it stated.\n- Regarding the N-Chain with high intrinsic uncertainty experiment, the difference in the decaying rate of \\Delta and c between PQR and DLTV or p-DLTV might also be significant. What are the authors' thoughts? It would be best to align the experiment conditions as much as possible.\n- The authors used the N-chain task where risk-based exploration like DLTV obviously does not work well, so the presented results are reasonable. On the other hand, in a task where DLTV works well (e.g., both s_0 and s_4 have the same reward variance), I think it is essential to experiment to see how the proposed method performs to compare the proposed method fairly.", "limitations": "- The paper lacks a thorough discussion of the limitations of the proposed PDBOO and PQR. The authors should provide more insight into scenarios where the method might not be applicable or efficient.\n- As pointed out above, why the proposed method is effective (for instance, as opposed to p-DLTV or simply increasing the temperature during action selection) remains unclear. It seems that the questions are not adequately addressed in the experiments. The authors should consider addressing these points to provide a more complete view of their method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces the Perturbed Distributional Bellman Operator (PDBOO), a novel extension of the distributional Bellman operator. This approach enables diverse exploration while maintaining risk neutrality in distributional reinforcement learning. The authors also develop Perturbed Quantile Regression (PQR), a DQN variant that estimates quantiles based on PDBOO. The proposal represents a new and interesting direction of the study in distributional RL.\n- The authors provide a solid theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to achieve the unique fixed point of the Bellman optimality equation asymptotically. This result provides confidence in their proposed methods and valuable insights for algorithm development.\n- The proposed PQR method is evaluated in a 4-states chain MDP and 55 Atari games. While the results showed room for improvement, the evaluation confirms the potential practical applicability of PQR and its ability to address the exploration problem while maintaining risk neutrality.", "weaknesses": "- The paper lacks an explanation as to why the proposed approach is more effective compared to other methods. It is crucial to discuss under what tasks PDBOO is effective and when it might not be suitable to understand its value.\n   - For example, PDBOO would add random noise (perturbation) to the target return distribution for learning. Depending on the perspective, this could also have the side effect of making it more challenging to learn the return distribution due to the existence of noise. I would like to know if there are situations where PDBOO should not be applied.\n   - Also, it is not clear why the approach of PDBOO is necessary, as it seems that what the authors want to achieve with PDBOO would not be the perturbation of the target return distribution for learning but instead could also be accomplished by perturbing the estimated return distribution at the time of action selection. This is somewhat supported by the results of p-DLTV in the N-chain MDP experiment.\n\n- The overall experimental results seem weak and do not resolve the abovementioned questions. Furthermore, the improved performance of PQR is observed in a subset of games, and the improvement seems modest. There is no comparison with p-DLTV, which showed similar good performance in the N-chain MDP experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for taking the time to review our submission. Contrary to your concerns, we want to emphasize that we've already designed PDBOO in the direction you're thinking, and we have provided detailed answers below.\n\n## Misconceptions about how PDBOO is designed\nWe will address this concern first, as it is the one that has caused the most misunderstanding. We want to emphasize that, as in p-DLTV, PDBOO only uses the perturbed return to determine the action that maximizes it, and updates the target by the “unperturbed” return distribution of that action. This is mentioned in lines 156-157 and in the pseudocode. To learn in the way you are concerned, the target must be updated by the perturbed pdf of $Z$ as $Q(dw) = \\xi(w)P(dw)$. Specifically, our distributional Bellman update is written as\n$T_\\xi Z(s,a) = R(s,a) + \\gamma Z(S', a^*(\\xi))$, not\n$T_\\xi Z(s,a) = R(s,a) + \\gamma (\\xi Z)(S', a^*(\\xi))$.\nThe latter is probably what you interpreted.\n\nWe are deeply concerned that this misunderstanding has had a significant impact on the evaluation of the paper, and we would like to add the following sentence to line 158 to avoid confusion. \n\n- “Specifically, PDBOO perturbs the estimated distribution only to select the optimal behavior, while the target is updated with the original (unperturbed) return distribution.”\n\n**In the global response, we add the comment that the target is updated with unperturbed return distribution.**\n\n## When is PDBOO effective/not effective?\nPDBOO is more effective in environments with high intrinsic uncertainty, such as stochasticity from transitions, rewards, or observations, and less effective in near-deterministic environments. (However, this is not necessarily a disadvantage.) As an experiment, Appendix D.1 shows the results of varying the amount of intrinsic uncertainty, and we can see that p-DLTV and PQR give consistent results, while DLTV shows a gradual degradation in performance as intrinsic uncertainty increases. This efficiency is crucial to achieve a better Bellman update because the experimental results show that choosing particular sub-optimal actions repeatedly contaminates the return distribution due to its incorrect policy evaluation.\n\n## Performance improvements in a subset of games and marginal improvements.\nWe respectfully disagree with your comment. To prevent performance exploitation from a subset of games, we have already provided 4 different performance metrics including the median of human normalized score following the evaluation protocol of [1] in Table 1. Furthermore, the metrics compared to human and DQN scores support that PQR outperforms IQN and is competitive with RAINBOW. As mentioned in lines 307-309, RAINBOW applied several improvement techniques, while PQR only changed its exploration strategy, which cannot be considered a marginal performance.\n\n[1] Bellemare, Marc G., et al. \"The arcade learning environment: An evaluation platform for general agents.\" Journal of Artificial Intelligence Research 47 (2013): 253-279.\n\n## The type of policy for each baseline\nOnly DLTV, p-DLTV, and PQR explored in their own way without using an epsilon-greedy schedule. DLTV's exploration method is mentioned in lines 102-106. It is noteworthy that PQR, like soft q-learning, includes the exploration process in its Bellman operator. We mentioned in lines 97-99 and 307-308 that the remaining baselines QR-DQN, IQN, and RAINBOW all use epsilon-greedy schedules.\nAlign the experiment conditions as much as possible.\nDLTV and p-DLTV have their own schedules, so we don't think it's appropriate to change them. We also set the scheduling of PQR to be close to $\\frac{1}{t}$ based on theory. However, we think your concerns are valid, and it would be interesting to experiment with scheduling PQR like DLTV, even if it's not consistent with theory. We'll try to explain the various scheduling aspects in more detail.\n\n## No experiment where DLTV works well.\nThe N-Chain environment we design has been tested in scenarios where the optimal action of the predefined risk measure  is not necessarily the risk-neutral optimal solution (action). Our experiments on the standard benchmarks, 55 Atari games and Lunarlander-v2 have different reward designs, which confirms the generalizability of our action selection method. In other words, to design an environment where DLTV performs better, we need to make the optimal solution of the mean-variance risk measure equal to the risk-neutral optimal solution. This is an artificial design that exploits the property of optimal return distributions that should be unknown. \n\nDespite the above issue of fair comparison, we can answer your comment, “compare the proposed method fairly” in the standard benchmark setting as follows. \nIn Figure 10 in Appendix D.3, we demonstrated the comparison on Atari with 30 no-ops protocol. Among the injected stochasticities, it is known that the 30 no-ops has small randomness, because this protocol only has an effect on the beginning of the episode. By the nature of Atari games themselves, Pong has less intrinsic uncertainty because the opponent has the identical policy even if our agent is trained, but Seaquest has different transitions over time as its manual notes \n\n*“so after each round, take a breath - enemy subs and sharks will increase in speed”*. \n\nWe now know that Pong has small intrinsic uncertainty, but Seaquest has larger intrinsic uncertainty (different transitions on a given state over time). In Figure 10, DLTV works well in Pong (low intrinsic uncertainty) but shows a lower performance in Seaquest because of the high intrinsic uncertainty.\n\nWe want to emphasize that DLTV is a decent algorithm, but our proposed method outperforms in the cases of high intrinsic uncertainty as shown in Figure 7,9,10 and all result tables.\n"}, {"review_id": "7vy6J5qcwo", "reviewer": "Reviewer_jYGR", "summary": "The authors propose a new exploration strategy that is more principled than epsilon-greedy or DLTV of Mavrin et al. (based on truncated variance). The proposed exploration strategy is to based on a novel perturbed distributional Bellman optimality operator, where the goal (Eqn. 1) is to minimize the distributional Bellman objective subject to the perturbation being sampled from an uncertainty set; the uncertainty set is defined as importance weighting the next state's return distribution in the Bellman operator, where the radius of the set decreases with time. The authors show that by appropriately decaying the set's radius, the algorithm can learn the optimal value function, and provide experimental validation on N-chain and Atari, showing competitive performance to IQN and Rainbow.", "questions": "1. Typically, distributionally robust optimization (DRO) optimizes the worst-case performance. However, in eqn. 1, you sample from the uncertainty set instead of taking a supremum. Can you give some intuition on why you care about the average performance over the set, rather than worst case? \n2. Is the perturbation gap related/inspired by the dual form of dynamic risk with coherent risk measures? \n3. Theorem 3.3 shows some kind of convergence of the learned $E Z^{(n)}$ to the true $Q*$ , but it seems that as n -> infinity, the bound does not necessarily to go zero. Doesn't this mean there will be some bias in the proposed approach?\n4. To be clear, does your implementation only use QR-DQN + the proposed PQR exploration strategy? Do you make use of target networks, double Q-learning, prioritized replay, or any of the bag of tricks that Rainbow uses?\n5. In Section 4.2, what is meant by \"intrinsic uncertainty\" of Atari? Without sticky actions, Atari is fully deterministic and there is no aleatoric uncertainty (only epistemic uncertainty), right?\n", "limitations": "Please see weakness/questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Perturbing the expectation with a density ratio is a simple but neat idea.\n2. Experimental results seem pretty promising, given that QR-DQN's performance significantly improves with the proposed strategy, sometimes even surpassing IQN.", "weaknesses": "1. I am not fully sold that perturbing the Bellman operator is a principled way to modeling epistemic uncertainty of our estimates. In particular, the paper seems to suggest that optimism is actually biased and unprincipled, while we know that in both theory and practice, optimism in the face of uncertainty is principled for online RL and minimax optimal in most well-known settings. \n1b. I think the paper would benefit from a clearer and more proper explanation of how prior settings are biased / unbiased, and how the proposed approach is better/worse than prior exploration strategies.\n2. if I understand correctly, the proposed uncertainty set seems to be (s,a)-independent, which seems wrong. (in particular, if some parts of the state space was not visited often, that uncertainty should be high, while uncertainty should be low for frequented states).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the time and effort in reviewing our paper. Please find below our response to the main points raised in the review.\n\n## How does the perturbed Bellman operator is a principled way to modeling epistemic uncertainty?\nWe are not trying to model epistemic uncertainty through perturbations; rather, we are trying to point out that using the OFU without considering the entanglement of intrinsic uncertainty (commonly referred to as risk) leads to biased exploration and degrades performance. \n\nIn principle, OFU works well for most online RL tasks where epistemic uncertainty (e.g., the number of visits) can be modeled. However, as mentioned in lines 41-47, in the deep reinforcement learning case, it is difficult to separate epistemic uncertainty because learning a feature representation of the high-dimensional state-action space and updating the bellman target occur simultaneously. There have been several attempts to solve this problem, and we have proposed a simple yet effective exploration method that does not separate the two uncertainties.\n\n## Proposed uncertainty set seems to be $(s,a)$-independent, which seems wrong.\nThat's a great point. First, we would like to note that our method is not exploration based on parametric uncertainty. The ambiguity set we define is for sampling perturbations xi, which is a weight applied equally to each return distribution, where the perturbation gaps are different for each $(s,a)$ depending on the shape of the distribution. Therefore, we believe that $(s,a)$-dependent elements like bonus correspond to perturbation gaps, but not to perturbation gaps or ambiguity sets.\nAlso, note that $\\bar{U}_\\Delta$ is the definition that the upper bound of the perturbation interval for all $(s,a)$ is less than $\\Delta$, which does not mean that all $(s,a)$ have the “same” amount of perturbation gap. Thus, for uncertain state-action which will have high parametric uncertainty, the estimated distribution will also have a high variance, so the perturbation gap will be large and still be chosen frequently.\n\n## Give some intuition on why we care about the average performance, rather than the worst case.\nIf we use a min-max operator with respect to risk, the solution is always more conservative (pessimistic) than the standard solution as it always considers the worst-case scenario. On the other hand, we considered the average-case which has the possibility to maintain risk-neutrality. Specifically, we aimed to find a sufficient condition to ensure risk-neutrality by scheduling the ambiguity set of risks. As such, we only share the definition of ambiguity set by the DRO literature, but with a different goal of risk-neutrality.\n\n## Relation/Inspiration by the dual form of coherent risk measures?\nCoherent risk measures and perturbation gaps seem to be related, since we try to construct ambiguity sets according to the DRO literature. However, we were not inspired by the dual form and have not yet attempted to make the connection. The definition of the perturbation gap is a natural consequence of obtaining the upper bound of Theorem 3.3. Because it is a tractable and simple form, we were able to design a practical PQR by defining the perturbation gap as shown in the paper. It would be interesting to define the perturbation gap in a more sophisticated way.\n\n## The bound in Theorem 3.3 does not necessarily go zero.\nWe cannot guarantee that delta_t will always go to zero if it is unconstrained. Hence, in Assumption 3.2, we gave sufficient conditions to ensure that the upper bound always goes to zero. Please make sure that the subscript \"$k=n$\" in the summation of the right terms is not misinterpreted as \"$k=1$\".\n\n## Does our work only use the proposed PQR exploration strategy?\nWe implemented PQR as you would expect, and we believe it's the right way to make a fair comparison. We did not use any improvement techniques of RAINBOW to verify the effectiveness of our method alone. Please note that our baseline comparison, IQN-dopamine, also used $n=3$ multi-steps and still PQR achieved higher performance.\n\n## What is meant by “intrinsic uncertainty” of Atari?\nWe apologize for not explaining this in more detail. While it's true that the Atari game itself (without no-op or sticky actions) is completely deterministic. However, in v0 and v4, the versions we have experimented with, opengym additionally introduced random frame skipping, so there's another source of intrinsic uncertainty. We will revise the sentence to make this clear."}, {"review_id": "QsnxZe4ss9", "reviewer": "Reviewer_rSdX", "summary": "This paper proposes a distributional RL algorithm, PQR, that uses distributional return estimates for exploration by updating with a greedy distributional Bellman operator for time-varying, random risk measures. The paper demonstrates that, under a simple concentration condition on the random risk measures over time, the means of return distributions learned by PQR converge to the optimal Q-values. It is argued that the use of random risk measures (as opposed to static risk measures) produces updates that are less biased away from the optimal Q-values. It is shown that PQR demonstrates favorable exploration performance in a simple chain MDP and in Atari.", "questions": "L172 says \"PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy...\" -- what does it mean to \"adhere to the risk-neutral optimal policy\"? The PDBOO updates are still performing greedy updates w.r.t. risk-sensitive policies.\n\nAs mentioned above, convergence of return distributions is not really established at all under the PDBOO update -- we only know about the convergence of their means. But if the dynamics of the other statistics of the return distributions are essentially unknown, how can you be sure that the random risk measures are actually estimating anything meaningful?\n\nIs there any correspondence between the PDBOO targets and the \"Thompson Sampling\" approach introduced by Riou and Honda (2020) http://proceedings.mlr.press/v117/riou20a/riou20a.pdf? There may be an interesting connection there, which could perhaps help add precision to the statement about \"adhering to the risk-neutral optimal policy\".", "limitations": "Limitations are addressed adequately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a well-motivated algorithm for leveraging return distributions for exploration in (deep) RL. The idea of using random risk measures in greedy distributional Bellman updates to veer less from the optimal Q-values is novel and interesting. Furthermore, the empirical results of the PQR algorithm appear to be quite good.", "weaknesses": "The proposed algorithm itself may not be substantially novel -- I suspect similar results would hold for QR-DQN with a dynamic distortion risk measure that tends to the identity sufficiently quickly. \n\nThe theoretical results are fairly weak. Firstly, convergence of the return distributions is only established with respect to their first moments. Moreover, the upper bound equation in Theorem 3.3 is difficult to interpret, and I am not sure what one should conclude from this upper bound.\n\nAdditionally, the influence of the schedule $(\\Delta_t)_{t\\geq 0}$ and the Dirichlet parameter $\\beta$ is not really studied -- I suspect, at least in theory, that there are more optimal data-based schedules.\n\n## Grammatical / non-major issues\n\nL4 \"present a novel distributional reinforcement learning that selects...\" -> \"present a novel distributional reinforcement learning **algorithm** that selects...\"\n\nLast sentence in the abstract cites [6] as a risk-sensitive distributional RL algorithm, but this is not really a distributional RL algorithm.\n\nL144 mentions the perturbed probability distribution $\\xi\\mathbb{P}$, but this doesn't really type-check. $\\mathbbf{P}$ is a probability measure (maps measurable sets to probabilities) and $\\xi$ is a probability density function (mapping elements of the sample space to reals), so the product $\\xi\\mathbb{P}$ is not exactly correct.\n\nTheorem 3.4 needs to be made more precise. It makes a claim about the fixed point of the Bellman optimality operator, but I think you meant the PDBOO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the time and effort in reviewing our paper. Please find below our response to the main points raised in the review.\n\n## The proposed algorithm itself may not be substantially novel.\nOf course, any scheduling that is sufficiently close to the QR-DQN can yield a risk-neutral optimal solution, but this is not sufficient for exploration. Since our theory provides sufficient conditions for satisfying the risk-neutrality, we have made it as exploratory as possible by choosing scheduling that satisfies non-trivial boundary conditions. We also believe that our proof of convergence for the time-varying bellman operator is sufficiently novel.\n\n## Convergence of the return distributions is only established with respect to their first moments.\nI think this is a very important question. In general, the return distribution in distRL does not have a unique fixed distribution. This is because there can be multiple greedy policies, since they are defined only for expectations and do not focus on the shape of the distribution. If we could define a total ordering on the set of greedy policies,(e.g., select the action with the lowest variance if expected values are equal) then there is a unique fixed distribution [1,2]. \n\nThe same is true for the situation we face, and the total ordering is likewise required for convergence at the distribution level. If the total ordering is well-defined so that the uniqueness of the fixed distribution is guaranteed, then our fixed distribution is the same as the solution of the standard distributional Bellman optimality equation. We will add more details in the revised version in Section 3.2.\n\n[1] Bellemare, Marc G., Will Dabney, and Rémi Munos. \"A distributional perspective on reinforcement learning.\" International conference on machine learning. PMLR, 2017.\n\n[2] Bellemare, Marc G., Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.\n\n## The influence of the schedule $\\Delta_t$ and $\\beta$\nWe have analyzed the effect of learning as a function of the size of the initial value of $\\Delta_t$ and PQR shows consistent results unlike other algorithms. **In the global response, we additionally experiment with different forms of scheduling for PQR.**\n\nFor the hyperparameter $\\beta$, we chose a weight of the form $c * \\textbf{1}^N$ to draw symmetrically from the simplex. If the value of $c$ is too large, the sampled weights will be around the center of the simplex, which is not much different from the original mean operator. So we have experimented with the setting $c=grid search[0.05, 0.1, 0.5, 1]$. We then chose a small value so that the perturbation gap is large enough by sampling $\\xi$ in the form of spikes at the edge of the simplex.\n\n## Reference [6] is not really a distributional RL algorithm\nThank you for the careful review. We'll fix the citation in that paragraph and include another reference.\n\n## Expression issue of $\\xi\\mathcal{P}$\nThis was a sloppy expression. We'll fix it in the main text as follows,\n- line 144: which can be interpreted as the expectation of $X$ under perturbed probability distribution $Q(dw) = \\xi(w)P(dw)$.\n\n## Fixed point of the Bellman optimality operator in Theorem 3.4 seems to mean the PDBOO.\nSince Theorem 3.4 shows that the fixed point of PDBOO is the same as that of a standard DBOO, this statement reflects our meaning well. We state this once more in lines 193-194 and 197.\n\n## Connection between the PDBOO targets and the Thompson Sampling?\nWe agree that the suggested method is similar to our approach. Thanks for suggesting the reference, it seems to be quite an important paper. The proposed nonparametric TS algorithm seems to have many similarities with our approach in that it uses a weighted sum of Dirichlet distributions for action selection. Their situation is different from ours in that they record all visits and reward histories, but perhaps extending their work to value function approximation approach would deepen the connection. The improved version of nonparametric TS algorithms controls the Dirichlet parameter, while PQR controls the upper bound of the perturbation gap, which seems to have a different approach but similar effect.\n"}]

Author rebuttal:
# Global Response from Author

We thank all the reviewers for their valuable comments on our paper.

First, we added a revised **description of the pseudocode** and **additional experimental results for the schedule of $\Delta_t$** in the PDF, based on feedback from some reviewers.

The additional experiments were conducted in Pong-v4 where the maximum score is 21.0 and the baseline is described as follows:

- PQR : Our main algorithm, $\Delta_t$ = O(1/t^{1+\epsilon} )

- $\frac{1}{t^0}$ : $\Delta_t = O(1)$ to check the results for a fixed size ambiguity set, and it can be seen that it affects the convergence.

- $\sqrt{ \frac{\log t}{t}} $: $\Delta_t = O(\sqrt{\log t/t})$ as in the scheduling of DLTV, which theoretically does not correspond to the sufficient condition we presented.

- OPT : The output vector sampled from the Dirichlet distribution is fixed to $[0,0,...,1]$, forcing the agent to estimate only optimistically.

- $\sqrt{ \frac{\log t}{t}} $ + OPT : A hybrid of the two methods above.

Pong-v4 is a simple and easy environment where a maximum score of 21.0 can easily be achieved for many distRL algorithms(QR-DQN, IQN, RAINBOW).
The game's description for Pong-v4 is as follows:

*A player or team scores one point when the opponent hits the ball out of bounds or misses a hit.  The first player or team to score 21 points wins the game.*

In this experiment, our proposed PQR is the only method that stably achieves the maximum score with very low variance. 
In the case of optimism, we can see that it performs quickly in the early stages of learning, but converges without reaching the maximum score, which is similar to the behavior of N-Chain. 
In the case of fixed ambiguity, it converges to suboptimal and shows very low performance, showing the necessity of a time-varying schedule. Finally, when mimicking a schedule of p-DLTV is applied to PQR, the performance also degrades. From this, we believe that the proposed sufficient condition is quite tight.





Also, We would like to summarize and answer some common issues raised by several reviewers.

## Definition of a risk-neutral objective
Several RL papers propose new frameworks with different roles of optimal solutions by modifying the original objective functions. (e.g., maximum entropy RL and risk-sensitive RL.) In this paper, we propose the novel Bellman operator that **even if the objective function is modified, the optimal solution remains the same as before**, but only the exploration performance is improved. Since we want to obtain the same optimal solution as the original, even if we change the objective, we call it a *risk-neutral objective*, which tries to avoid one-sided risk tendency.

## Why the randomized approach is more effective than OFU in deep RL
While OFU is a principled criterion for exploration in bandit or tabular RL, it is already known that randomized approaches, including Thompson sampling, empirically outperform OFU, while there are not many papers that theoretically analyze the exact reason why.
 
Meanwhile, in deep RL, there have been attempts to use the information for exploration from distRL, which estimates the return distribution. DistRL tries to capture intrinsic (aleatoric) uncertainty, but the variance of the estimated distribution is mixed with parametric (epistemic) uncertainty. 

We argue that using the estimated variance as the OFU is problematic and can be mitigated by a randomized approach. 
The reason is that **the intrinsic uncertainty that remains in the estimated variance is irreducible during learning, and applying the OFU will bias the exploration towards risk-seeking behaviors.** We demonstrate this biased exploration phenomenon and its significant performance degradation. We also show in numerous experimental settings that the randomized approach is highly effective in avoiding this bias. Furthermore, we provide a sufficient condition for PDBOO to converge to the original optimal solution while applying a randomized risk criterion.



Meta-review: This paper contributes a method for exploration in RL using distributional reinforcement learning. Initial reviews for this paper were mixed. Several reviewers found the paper clear, with good background on relevant topics within distributional RL given. Many reviewers also found the contributions novel, and the theoretical analysis solid. Some concerns mentioned in the initial reviews included the strength of the main theoretical results (Reviewer rSdX), and the potential for further explanation/examination of the empirical results obtained in the Atari suite of environments.

After detailed discussion with the authors, several reviewers have updated their scores and we now have the majority of reviewers recommending acceptance. Reviewer jYGR's recommendation of "borderline reject" remains, although we have not heard from this reviewer since the initial review. Their review raises several specific questions, such as how principled the perturbations of the DBO are, and the nature of the uncertainty sets considered. They also ask several more minor questions. In my view, the authors address these queries satisfactorily in their rebuttal. The authors also addressed several more minor concerns raised regarding the theory in their rebuttal. The recommendation is therefore to accept the paper. I would encourage the authors to take the feedback of the reviewers into account when preparing the camera-ready version.
Decision: Accept (poster)