Title: Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributional reinforcement learning

risk
Abstract: Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning that selects actions by randomizing risk criterion without losing the risk-neutral objective. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure. Also,we prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion

Abstract

Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning algorithm that selects actions by randomizing risk criterion to avoid one-sided tendency on risk. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure and prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.

1 Introduction

Distributional reinforcement learning (DistRL) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in numerous benchmark tasks. DistRL agents model the approximated distribution of returns, where the mean value implies the conventional Q-value [2, 6, 13, 25] and provides more statistical information (e.g., mode, median, variance) for control. Precisely, DistRL aims to capture intrinsic (aleatoric) uncertainty which is an inherent and irreducible randomness in the environment. Such learned uncertainty gives rise to the notion of risk-sensitivity, and several distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking decision making [8, 12].

Despite the richness of risk-sensitive information from return distribution, only a few DistRL

Figure 1: Illustrative example of why a biased risk criterion (naïve optimism) can degrade performance. Suppose two actions have similar expected returns, but different variances (intrinsic uncertainty). (Left) If an agent does not specify the risk criterion at the moment, the probability of selecting each action should be similar. (Right) As OFU principle encourages to decide uncertain behaviors, the empirical variance from quantiles was used as an estimate of uncertainty. [19, 21, 23]. However, optimistic decision based on empirical variance inevitably leads a risk-seeking behavior, which causes biased action selection.

methods [11, 21, 26, 35, 42] have tried to employ its benefits for exploration strategies which is essential in deep RL to find an optimal behavior within a few trials. The main reason is that the exploration strategies so far is based on parametric (epistemic) uncertainty which arise from insufficient or

PQR replay buffer

𝝐𝝐

-greedy replay buffer

DLTV replay buffer

: list of actions to be added to the buffer

Number of candidate actions

inaccurate data. In particular, Optimism in the face of uncertainty (OFU) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set. In bandit or tabular MDP settings, OFU-based algorithms select an action with the highest upper-confidence bound (UCB) of parametric uncertainty which can be considered as the optimistic decision at the moment [5, 10].

However, in deep RL, it is hard to trivially estimate the parametric uncertainty accurately due to the black-box nature of neural networks and high-dimensionality of state-action space. Without further computational task, the estimated variance from distribution is extracted as a mixture of two types of uncertainty, making it difficult to decompose either component. For example, DLTV [21] was proposed as a distribution-based OFU exploration that decays bonus rate to suppress the effect of intrinsic uncertainty, which unintentionally induces a risk-seeking policy. Although DLTV is the first attempt to introduce OFU in distRL, we found that consistent optimism on the uncertainty of the estimated distribution still leads to biased exploration. We will refer to this side-effect as one-sided tendency on risk , where selecting an action based on a fixed risk criterion degrades learning performance. In Section 4, we will demonstrate the one-sided tendency on risk through a toy experiment and show that our proposed randomized approach is effective to avoid this side effect.

In this paper, we introduce Perturbed Distributional Bellman Optimality Operator (PDBOO) to address the issue of biased exploration caused by a one-sided tendency on risk in action selection. We define the distributional perturbation on return distribution to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight. To facilitate deep RL algortihm, we present Perturbed Quantile Regression (PQR) and test in Atari 55 games comparing with other distributional RL algorithms that have been verified for reproducibility by official platforms [4, 29].

In summary, our contributions are as follows.

A randomized approach called perturbed quantile regression (PQR) is proposed without sacrificing the original (risk-neutral) optimality and improves over naïve optimistic strategies.

A sufficient condition for convergence of the proposed Bellman operator is provided without satisfying the conventional contraction property.

2 Backgrounds & Related works

2.1 Distributional RL

We consider a Markov decision process (MDP) which is defined as a tuple ( S , A , P, R, γ ) where S is a finite state space, A is a finite action space, P : S × A × S → [0 , 1] is the transition probability, R is the random variable of rewards in [ -R max , R max ] , and γ ∈ [0 , 1) is the discount factor. We define a stochastic policy π ( ·| s ) which is a conditional distribution over A given state s . For a fixed policy π , we denote Z π ( s, a ) as a random variable of return distribution of state-action pair ( s, a ) following the policy π . We attain Z π ( s, a ) = ∑ ∞ t =0 γ t R ( S t , A t ) , where S t +1 ∼ P ( ·| S t , A t ) , A t ∼ π ( ·| S t ) and S 0 = s, A 0 = a . Then, we define an action-value function as Q π ( s, a ) = E [ Z π ( s, a )] in [ -V max , V max ] where V max = R max / (1 -γ ) . For regularity, we further notice that the space of

action-value distributions Z has the first moment bounded by V max:

In distributional RL, the return distribution for the fixed π can be computed via dynamic programming with the distributional Bellman operator defined as,

where D = denotes that both random variables share the same probability distribution. We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as,

Bellemare et al. [2] have shown that τ π is a contraction in a maximal form of the Wasserstein metric but τ is not a contraction in any metric. Combining with the expectation operator, E τ is a contraction so that we can guarantee that the expectation of Z converges to the optimal state-action value. Another notable difference is that the convergence of a return distribution is not generally guaranteed to be unique, unless there is a total ordering ≺ on the set of greedy policies.

2.2 Exploration on Distributional RL

To combine with deep RL, a parametric distribution Z θ is used to learn a return distribution. Dabney et al. [13] have employed a quantile regression to approximate the full distribution by letting Z θ ( s, a ) = 1 N ∑ N i =1 δ θ i ( s,a ) where θ represents the locations of a mixture of N Dirac delta functions. Each θ i represents the value where the cumulative probability is τ i = i N . By using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as a minimization problem as,

Based on the quantile regression, Dabney et al. [13] have proposed a quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach [2], since the quantile regression does not restrict the bounds for return.

As deep RL typically did, QR-DQN adjusts ϵ -greedy schedule, which selects the greedy action with probability 1 -ϵ and otherwise selects random available actions uniformly. The majority of QR-DQN variants [12, 38] rely on the same exploration method. However, such approaches do not put aside inferior actions from the selection list and thus suffers from a loss [28]. Hence, designing a schedule to select a statistically plausible action is crucial for efficient exploration.

In recent studies, Mavrin et al. [21] modifies the criterion of action selection for efficient exploration based on optimism in the face of uncertainty. Using left truncated variance as a bonus term and decaying ratio c t to suppress the intrinsic uncertainty, DLTV was proposed as an uncertainty-based exploration in DistRL without using ϵ -greedy schedule. The criterion of DLTV is described as:

where θ i 's are the values of quantile level τ i .

2.3 Risk in Distributional RL

Instead of an expected value, risk-sensitive RL is to maximize a pre-defined risk measure such as Mean-Variance [41], Value-at-Risk (VaR) [9], or Conditional Value-at-Risk (CVaR) [30, 31] and results in different classes of optimal policy. Especially, Dabney et al. [12] interprets risk measures

as the expected utility function of the return, i.e., E Z [ U ( Z ( s, a ))] . If the utility function U is linear, the policy obtained under such risk measure is called risk-neutral . If U is concave or convex, the resulting policy is termed as risk-averse or risk-seeking , respectively. In general, a distortion risk measure is a generalized expression of risk measure which is generated from the distortion function.

Definition 2.1. Let h : [0 , 1] → [0 , 1] be a distortion function such that h (0) = 0 , h (1) = 1 and non-decreasing. Given a probability space (Ω , F , P ) and a random variable Z : Ω → R , a distortion risk measure ρ h corresponding to a distortion function h is defined by:

where F Z is the cumulative distribution function of Z .

In fact, non-decreasing property of h makes it possible to distort the distribution of Z while satisfying the fundamental property of CDF. Note that the concavity or the convexity of distortion function also implies risk-averse or seeking behavior, respectively. Dhaene et al. [14] showed that any distorted expectation can be expressed as weighted averages of quantiles. In other words, generating a distortion risk measure is equivalent to choosing a reweighting distribution.

Fortunately, DistRL has a suitable configuration for risk-sensitive decision making by using distortion risk measure. Chow et al. [8] and Stanko and Macek [34] considered risk-sensitive RL with a CVaR objective for robust decision making. Dabney et al. [12] expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets τ . Zhang and Yao [40] have suggested QUOTA which derives different policies corresponding to different risk levels and considers them as options. Moskovitz et al. [24] have proposed TOP-TD3, an ensemble technique of distributional critics that balances between optimism and pessimism for continuous control.

3 Perturbation in Distributional RL

3.1 Perturbed Distributional Bellman Optimality Operator

To choose statistically plausible actions which may be maximal for certain risk criterion, we will generate a distortion risk measure involved in a pre-defined constraint set, called an ambiguity set . The ambiguity set, originated from distributionally robust optimization (DRO) literature, is a family of distribution characterized by a certain statistical distance such as ϕ -divergence or Wasserstein distance [15, 32]. In this paper, we will examine the ambiguity set defined by the discrepancy between distortion risk measure and expectation. We say the sampled reweighting distribution ξ as (distributional) perturbation and define it as follows:

Definition 3.1. (Perturbation Gap, Ambiguity Set) Given a probability space (Ω , F , P ) , let X : Ω → R be a random variable and Ξ = { ξ : 0 ≤ ξ ( w ) < ∞ , ∫ w ∈ Ω ξ ( w ) P ( dw ) = 1 } be a set of probability density functions. For a given constraint set U ⊂ Ξ , we say ξ ∈ U as a (distributional) perturbation from U and denote the ξ -weighted expectation of X as follows:

which can be interpreted as the expectation of X under some probability measure Q , where ξ = d Q /d P is the Radon-Nikodym derivative of Q with respect to P . We further define d ( X ; ξ ) = | E [ X ] -E ξ [ X ] | as perturbation gap of X with respect to ξ . Then, for a given constant ∆ ≥ 0 , the ambiguity set with the bound ∆ is defined as

For brevity, we omit the input w from a random variable unless confusing. Since ξ is a probability density function, E ξ [ X ] is an induced risk measure with respect to a reference measure P . Intuitively, ξ ( w ) can be viewed as a distortion to generate a different probability measure and vary the risk tendency. The aspect of using distortion risk measures looks similar to IQN [12]. However, instead of changing the sampling distribution of quantile level τ implicitly, we reweight each quantile from the

ambiguity set. This allows us to control the maximum allowable distortion with bound ∆ , whereas the risk measure in IQN does not change throughout learning. In Section 3.3, we suggest a practical method to construct the ambiguity set.

Now, we characterize perturbed distributional Bellman optimality operator (PDBOO) τ ξ for a fixed perturbation ξ ∈ U ∆ ( Z ) written as below:

Notice that ξ ≡ 1 corresponds to a base expectation, i.e., E ξ,P = E P , which recovers the standard distributional Bellman optimality operator τ . Specifically, PDBOO perturbs the estimated distribution only to select the optimal behavior, while the target is updated with the original (unperturbed) return distribution.

If we consider the time-varying bound of ambiguity set, scheduling ∆ t is a key ingredient to determine whether PDBOO will efficiently explore or converge. Intuitively, if an agent continues to sample the distortion risk measure from a fixed ambiguity set with a constant ∆ , there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence. Hence, scheduling a constraint of ambiguity set properly at each action selection is crucial to guarantee convergence.

Based on the quantile model Z θ , our work can be summarized into two parts. First, we aim to minimize the expected discrepancy between Z θ and τ ξ Z θ -where ξ is sampled from ambiguity set U ∆ . To clarify notation, we write E ξ [ · ] as a ξ -weighted expectation and E ξ ∼ P ( U ∆ ) [ · ] as an expectation with respect to ξ which is sampled from U ∆ . Then, our goal is to minimize the perturbed distributional Bellman objective with sampling procedure P :

where we use the Huber quantile loss as a discrepancy on Z θ ′ and τ ξ Z θ -at timestep t . In typical risk-sensitive RL or distributionally robust RL, the Bellman optimality equation is reformulated for a pre-defined risk measure [8, 33, 39]. In contrast, PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy while randomizing the risk criterion at every step. By using min-expectation instead of min-max operator, we suggest unbiased exploration that can avoid leading to overly pessimistic policies. Furthermore, considering a sequence ξ t which converges to 1 in probability, we derive a sufficient condition of ∆ t that the expectation of any composition of the operators E τ ξ n :1 := E τ ξ n τ ξ n -1 · · · τ ξ 1 has the same unique fixed point as the standard. These results are remarkable that we can apply the diverse variations of distributional Bellman operators for learning.

3.2 Convergence of the perturbed distributional Bellman optimality operator

Unlike conventional convergence proofs, PDBOO is time-varying and not a contraction, so it covers a wider class of Bellman operators than before. Since the infinite composition of time-varying Bellman operators does not necessarily converge or have the same unique fixed point, we provide the sufficient condition in this section. We denote the iteration as Z ( n +1) := τ ξ n +1 Z ( n ) , Z (0) = Z for each timestep n > 0 , and the intersection of ambiguity set as ¯ U ∆ n ( Z ( n -1) ) := ⋂ s,a U ∆ n ( Z ( n -1) ( s, a ) ) .

Assumption 3.2. Suppose that ∑ ∞ n =1 ∆ n < ∞ and ξ n is uniformly bounded.

Theorem 3.3. (Weaker Contraction Property) Let ξ n be sampled from ¯ U ∆ n ( Z ( n -1) ) for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators E τ ξ n :1 converges, i.e., E τ ξ n :1 [ Z ] → E [ Z ∗ ] . Moreover, the following bound holds,

Practically, satisfying Assumption 3.2 is not strict to characterize the landscape of scheduling. Theorem 3.3 states that even without satisfying γ -contraction property, we can show that E [ Z ∗ ] is the fixed point for the operator E τ ξ n :1 . However, E [ Z ∗ ] is not yet guaranteed to be 'unique' fixed point for any Z ∈ Z . Nevertheless, we can show that E [ Z ∗ ] is, in fact, the solution of the standard Bellman optimality equation, which is already known to have a unique solution.

Algorithm 1 Perturbed QR-DQN (PQR)

Theorem 3.4. If Assumption 3.2 holds, E [ Z ∗ ] is the unique fixed point of Bellman optimality equation for any Z ∈ Z .

As a result, PDBOO generally achieves the unique fixed point of the standard Bellman operator. Unlike previous distribution-based or risk-sensitive approaches, PDBOO has the theoretical compatibility to obtain a risk-neutral optimal policy even if the risk measure is randomly sampled during training procedure. For proof, see Appendix A.

3.3 Practical Algorithm with Distributional Perturbation

In this section, we propose a perturbed quantile regression (PQR) that is a practical algorithm for distributional reinforcement learning. Our quantile model is updated by minimizing the objective function (1) induced by PDBOO. Since we employ a quantile model, sampling a reweight function ξ can be reduced into sampling an N -dimensional weight vector ξ := [ ξ 1 , · · · , ξ N ] where ∑ N i =1 ξ i = N and ξ i ≥ 0 for all i ∈ { 1 , · · · , N } . Based on the QR-DQN setup, note that the condition ∫ w ∈ Ω ξ ( w ) P ( dw ) = 1 turns into ∑ N i =1 1 N ξ i = 1 , since the quantile level is set as τ i = i N .

A key issue is how to construct an ambiguity set with bound ∆ t and then sample ξ . A natural class of distribution for practical use is the symmetric Dirichlet distribution with concentration β , which represents distribution over distributions. (i.e. x ∼ Dir ( β ) .) We sample a random vector, x ∼ Dir( β ), and define the reweight distribution as ξ := 1 N + α ( N x -1 N ) . From the construction of ξ , we have 1 -α ≤ ξ i ≤ 1 + α ( N -1) for all i and it follows that | 1 -ξ i | ≤ α ( N -1) . By controlling α , we can bound the deviation of ξ i from 1 and bound the perturbation gap as

Hence, letting α ≤ ∆ ( N -1) V max is sufficient to obtain d ( Z ; ξ ) ≤ ∆ in the quantile setting. We set β = 0 . 05 · 1 N to generate a constructive perturbation ξ n which gap is close to the bound ∆ n . For Assumption 3.2, our default schedule is set as ∆ t = ∆ 0 t -(1+ ϵ ) where ϵ = 0 . 001 .

4 Experiments

Our experiments aim to investigate the following questions.

Does randomizing risk criterion successfully escape from the biased exploration in stochastic environments?

Can PQR accurately estimate a return distribution?

Can a perturbation-based exploration perform sucessfully as a behavior policy for the full Atari benckmark?

𝑠𝑠0

𝑠𝑠1

𝑠𝑠2

𝑠𝑠3

Start

:

𝑎𝑎0

:

𝑎𝑎1

no-op

:

𝑎𝑎2

~

𝑎𝑎6

:

𝑎𝑎

∗

(optimal)

= N(10,0.12)

r

+N(13, 0.12)

r

1.0

PDF of Z(s2, a)

1.0

PDF of Z(s2, a)

1.0

PDF of Z(s2, a)

1.0

PDF of Z(s2, a)

Aisuaa

0.5

a0 : mean 8.22, std 2.65

a1 : mean 7.65, std 1.85

Aisuaa

0.5

== a0 : mean 7.37, std 2.43

a1 : mean 5.58, std 2.03

Aisuaa

0.5

a0 : mean 6.71, std 1.79

a1 : mean 7.93, std 0.75

Aisuaa

0.5

a0 : mean 7.44, std 2.27

a1 : mean 8.08, std 0.23

10K steps

10K steps

10K steps

10K steps

0.0

2

4

6

8

10

12

14

16

18

20

0.0

2

6

8

10

12

14

16

1820

0.0

Z

4

6

8

10

12

14

16

18

20

0.0

Z

4

6

8

10

12

14

16

18

20

PDF of Z(s2, a)

PDF of Z(s2, a)

PDF of Z(s2, a)

PDF of Z(s2, a)

1.0

1.0

1.0

1.0

a0 : mean 8.17, std 2.21

==a0 : mean 6.79, std 2.03

== a0 : mean 6.60, std 1.95

a0 : mean 7.26, std 2.15

Density

0.5

al : mean 9.03, std 0.25

Aisuaa

0.5

al : mean 5.16, std 1.54

Density

0.5

a1 : mean 7.75, std 0.66

Density

0.5

a1 : mean 8.03, std 0.06

15K steps

15K steps

15K steps

15K steps

0.0

4

6

8

PDF of Z(s2, a)

10

12

14

16

18

20

0.0

6

8

PDF of Z(s2, a)

10

12

14

16

18

20

0.0

Z

8

PDF of Z(s2, a)

10

12

14

16

18

20

0.0

Z

4

8

PDF of Z(s2, a)

10

12

14

16

18

20

1.0

1.0

1.0

1.0

a0 : mean 8.09, std 2.57

a0 : mean 7.22, std 2.29

a0 : mean 6.91, std 1.92

a0 : mean 7.17, std 2.21

Aisuaa

0.5

a1 : mean 9.33, std 0.09

Aisuaa

0.5

a1 : mean 7.08, std 1.09

Aisuaa

0.5

a1 : mean 8.12, std 0.46

Density

0.5

a1 : mean 8.07, std 0.07

20K steps

20K steps

20K steps

20K steps

0.0

2

4

6

8

10

12

14

16

18

20

0.0

Z

4

6

8

10

12

14

16

18

20

0.0

4

6

8

10

12

14

16

18

20

0.0

2

4

8

10

12

14

16

18

20

(a) QR-DQN

(b) DLTV

(c) p-DLTV

(d) PQR (Ours)

4.1 Learning on Stochastic Environments with High Intrinsic Uncertainty

For intuitive comparison between optimism and randomized criterion, we design p-DLTV , a perturbed variant of DLTV, where coefficient c t is multiplied by a normal distribution N (0 , 1 2 ) . Every experimental setup, pseudocodes, and implementation details can be found in Appendix C.

N-Chain with high intrinsic uncertainty. We extend N-Chain environment [27] with stochastic reward to evaluate action selection methods. A schematic diagram of the stochastic N-Chain environment is depicted in Figure 3. The reward is only given in the leftmost and rightmost states and the game terminates when one of the reward states is reached. We set the leftmost reward as N (10 , 0 . 1 2 ) and the rightmost reward as 1 2 N (5 , 0 . 1 2 ) + 1 2 N (13 , 0 . 1 2 ) which has a lower mean as 9 but higher variance. The agent always starts from the middle state s 2 and should move toward the leftmost state s 0 to achieve the greatest expected return. For each state, the agent can take one of six available actions: left, right, and 4 no-op actions. The optimal policy with respect to mean is to move left twice from the start. We set the discount factor γ = 0 . 9 and the coefficient c = 50 .

Despite the simple configuration, the possibility to obtain higher reward in suboptimal state than the optimal state makes it difficult for an agent to determine which policy is optimal until it experiences enough to discern the characteristics of each distribution. Thus, the goal of our toy experiment is to evaluate how rapidly each algorithm could find a risk-neutral optimal policy. The results of varying the size of variance are reported in Appendix D.1.

Analysis of Experimental Results. As we design the mean of each return is intended to be similar, examining the learning behavior of the empirical return distribution for each algorithm can provide fruitful insights. Figure 4 shows the empirical PDF of return distribution by using Gaussian kernel density estimation. In Figure 4(b), DLTV fails to estimate the true optimal return distribution. While the return of ( s 2 , right ) (red line) is correctly estimated toward the ground truth, ( s 2 , left ) (blue line) does not capture the shape and mean due to the lack of experience. At 20K

0

5000

10000

15000

20000

Steps

0

2500

5000

7500

10000

12500

15000

17500

20000

Total Count

DLTV

p-DLTV

PQR

timestep, the agent begins to see other actions, but the monotonic scheduling already makes the decision like exploitation. Hence, decaying schedule of optimism is not a way to solve the underlying problem. Notably, p-DLTV made a much better estimate than DLTV only by changing from optimism to a randomized scheme. In comparison, PQR estimates the ground truth much better than other baselines with much closer mean and standard-deviation.

Figure 5 shows the number of timesteps when the optimal policy was actually performed to see the interference of biased criterion. Since the optimal policy consists of the same index a 1 , we plot the total count of performing the optimal action with 10 seeds. From the slope of each line, it is observed that DLTV selects the suboptimal action even if the optimal policy was initially performed. In contrast, p-DLTV avoids getting stuck by randomizing criterion and eventually finds the true optimal policy. The experimental results demonstrate that randomizing the criterion is a simple but effective way for exploration on training process.

Hyperparameter=1

Hyperparameter=5

Hyperparameter=10

Hyperparameter=50

8

6

DLTV

Error

p-DLTV

4

PQR

2

0

5K

10K

15K

20K

5K

10K

15K

20K

5K

10K

15K

20K

5K

10K

15K

20K

Timesteps

Timesteps

Timesteps

Timesteps

Hyperparameter=100

Hyperparameter=500

Hyperparameter=5000

5K

10K

15K

20K

5K

10K

15K

20K

5K

10K

15K

20K

5K

10K

15K

20K

Timesteps

Timesteps

Timesteps

Timesteps

Hyperparameter Sensitivity. In Figure 6, we compute the 2-Wasserstein distance from the ground truth return distribution N (10 γ 2 , (0 . 1 γ 2 ) 2 ) . Except for QR-DQN , each initial hyperparameter { c, ∆ 0 } was implemented with grid search on [1 , 5 , 10 , 50 , 100 , 500 , 1000 , 5000] in 5 different seeds. As the hyperparameter decreases, each agent is likely to behave as exploitation. One interesting aspect is that, while it may be difficult for DLTV and p-DLTV to balance the scale between the return and bonus term, PQR shows robust performance to the initial hyperparameter. This is because the distorted return is bounded by the support of return distribution, so that PQR implicitly tunes the scale of exploration. In practice, we set ∆ 0 to be sufficiently large. See Table2 in Appendix C.1.

4.2 Full Atari Results

We compare our algorithm to various DistRL baselines, which have demonstrated good performance on RL benchmarks. In Table 1, we evaluated 55 Atari results, averaging over 5 different seeds at 50M frames. We compared with the published score of QR-DQN [13], IQN [12], and Rainbow [16] via the report of DQN-Zoo [29] and Dopamine [4] benchmark for reliability. This comparison is particularly noteworthy since our proposed method only applys perturbationbased exploration strategy and outperforms advanced variants of QR-DQN. 2

0

10

20

30

40

50

Training Frames (Million)

1000

2000

3000

4000

5000

Average Return

Alien

PQR

C51

DQN

IQN

Rainbow

0

10

20

30

40

50

Training Frames (Million)

0

500

1000

1500

2000

2500

3000

3500

Average Return

Amidar

0

10

20

30

40

50

Training Frames (Million)

0

2000

4000

6000

8000

Average Return

Assault

0

10

20

30

40

50

Training Frames (Million)

200

400

600

800

1000

1200

1400

Average Return

Asteroids

0

10

20

30

40

50

Training Frames (Million)

0

10000

20000

30000

40000

50000

60000

70000

Average Return

Battlezone

0

10

20

30

40

50

Training Frames (Million)

0

2000

4000

6000

8000

10000

12000

14000

Average Return

0

10

20

30

40

50

Training Frames (Million)

0

1000

2000

3000

4000

5000

6000

Average Return

0

10

20

30

40

50

Training Frames (Million)

25

50

75

100

125

150

175

200

Average Return

Bowling

No-ops Protocol. First, we follow the evaluation protocol of [1, 22] on full set of Atari games implemented in OpenAI's Gym [3]. Even if it is well known that the no-ops protocol does not provide enough stochasticity to avoid memorization, intrinsic uncertainty still exists due to the random frame skipping [20]. While PQR cannot enjoy the environmental stochasticity by the deterministic dynamics of Atari games, PQR achieved 562% performance gain in the mean of human-normalized score over QR-DQN, which is comparable results to Rainbow. From the raw scores of 55 games, PQR wins 39 games against QR-DQN and 34 games against IQN.

Sticky actions protocol. To prevent the deterministic dynamics of Atari games, Machado et al. [20] proposes injecting stochasticity scheme, called sticky actions , by forcing to repeat the previous action with probability p = 0 . 25 . Sticky actions protocol prevents agents from relying on memorization and allows robust evaluation. In Figure 7, PQR shows steeper learning curves, even without any support of advanced schemes, such as n -step updates for Rainbow or IQN. In particular, PQR dramatically improves over IQN and Rainbow in ASSAULT, BATTLEZONE, BEAMRIDER, BERZERK and BOWLING. In Table 1, PQR shows robust median score against the injected stochasticity.

It should be noted that IQN benefits from the generalized form of distributional outputs, which reduces the approximation error from the number of quantiles output. Compare to IQN, PQR does not rely on prior distortion risk measure such as CVaR [7], Wang [37] or CPW [36], but instead randomly samples the risk measure and evaluates it with a risk-neutral criterion. Another notable difference is that PQR shows the better or competitive performance solely through its exploration strategies , compared to ϵ -greedy baselines, such as QR-DQN, IQN, and especially Rainbow. Note that Rainbow enjoys a combination of several orthogonal improvements such as double Q-learning, prioritized replay, dueling networks, and n -step updates.

5 Related Works

Randomized or perturbation-based exploration has been focused due to its strong empirical performance and simplicity. In tabular RL, Osband et al. [28] proposed randomized least-squares value iteration (RLSVI) using random perturbations for statistically and computationally efficient exploration. Ishfaq et al. [17] leveraged the idea into optimistic reward sampling by perturbing rewards and regularizers. However, existing perturbation-based methods requires tuning of the hyperparameter for the variance of injected Gaussian noise and depend on well-crafted feature vectors in advance. On the other hand, PDBOO does not rely on the scale of rewards or uncertainties due to the built-in scaling mechanism of risk measures. Additionally, we successfully extend PQR to deep RL scenarios in distributional lens, where feature vectors are not provided, but learned during training.

6 Conclusions

In this paper, we proposed a general framework of perturbation in distributional RL which is based on the characteristics of a return distribution. Without resorting to a pre-defined risk criterion,

we revealed and resolved the underlying problem where one-sided tendency on risk can lead to biased action selection under the stochastic environment. To our best knowledge, this paper is the first attempt to integrate risk-sensitivity and exploration by using time-varying Bellman objective with theoretical analysis. In order to validate the effectiveness of PQR, we evaluate on various environments including 55 Atari games with several distributional RL baselines. Without separating the two uncertainties, the results show that perturbing the risk criterion is an effective approach to resolve the biased exploration. We believe that PQR can be combined with other distributional RL or risk-sensitive algorithms as a perturbation-based exploration method without sacrificing their original objectives.

References

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning , pages 449-458. PMLR, 2017.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 , 2018.

Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502 , 2017.

Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional deep reinforcement learning with a mixture of gaussians. In 2019 International Conference on Robotics and Automation (ICRA) , pages 9791-9797. IEEE, 2019.

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. Advances in neural information processing systems , 27, 2014.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: a cvar optimization approach. arXiv preprint arXiv:1506.02188 , 2015.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120, 2017.

Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic. arXiv preprint arXiv:1910.12807 , 2019.

William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638 , 2019.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning , pages 1096-1105. PMLR, 2018.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Jan Dhaene, Alexander Kukush, Daniël Linders, and Qihe Tang. Remarks on quantiles and distortion risk measures. European Actuarial Journal , 2(2):319-328, 2012.

Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1):115-166, 2018.

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence , 2018.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607-4616. PMLR, 2021.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a cvar policy. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4436-4443, 2020.

Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562, 2018.

Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning , pages 4424-4434. PMLR, 2019.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. The potential of the return distribution for exploration in rl. arXiv preprint arXiv:1806.04242 , 2018.

Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems , 34:12849-12863, 2021.

Thanh Nguyen-Tang, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning via moment matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9144-9152, 2021.

Jihwan Oh, Joonkee Kim, and Se-Young Yun. Risk perspective exploration in distributional reinforcement learning. arXiv preprint arXiv:2206.14170 , 2022.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29:4026-4034, 2016.

Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res. , 20(124):1-62, 2019.

John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020. URL http://github.com/deepmind/dqn_zoo .

Marc Rigter, Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. arXiv preprint arXiv:2102.05762 , 2021.

R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk , 2:21-42, 2000.

Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures on stochastic programming: modeling and theory . SIAM, 2021.

Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv:1902.08708 , 2019.

Silvestr Stanko and Karel Macek. Risk-averse distributional reinforcement learning: A cvar optimization approach. In IJCCI , pages 412-423, 2019.

Yunhao Tang and Shipra Agrawal. Exploration by distributional reinforcement learning. arXiv preprint arXiv:1805.01907 , 2018.

Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty , 5(4):297-323, 1992.

Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance , pages 15-36, 2000.

Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie-Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems , 32:6193-6202, 2019.

Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions on Automatic Control , 2020.

Shangtong Zhang and Hengshuai Yao. Quota: The quantile option architecture for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5797-5804, 2019.

Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean-variance policy iteration for risk-averse reinforcement learning. arXiv preprint arXiv:2004.10888 , 2020.

Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. arXiv preprint arXiv:2105.06696 , 2021.

A Proof

A.1 Technical Lemma

Before proving our theoretical results, we present two inequalities for supremum to clear the description.

Proof of 1. Since | f ( x ) + g ( x ) | ≤ | f ( x ) | + | g ( x ) | holds for all x ∈ X ,

Proof of 2. Since ∣ ∣ ∣ ∥ a ∥ - ∥ b ∥ ∣ ∣ ∣ ≤ ∥ a -b ∥ for any norm ∥ · ∥ and for a large enough M ,

A.2 Proof of Theorem A.3

Theorem A.3. If ξ t converges to 1 in probability on Ω , then E τ ξ t converges to E τ uniformly on Z for all s ∈ S and a ∈ A .

Proof. Recall that Z = { Z : S × A → P ( R ) ∣ ∣ E [ | Z ( s, a ) | ] ≤ V max , ∀ ( s, a ) } . Then for any Z ∈ Z and ξ ∈ Ξ ,

which implies PDBOO is closed in Z , i.e. τ ξ Z ∈ Z for all ξ ∈ Ξ . Hence, for any sequence ξ t , Z ( n ) = τ ξ n :1 Z ∈ Z for any n ≥ 0 .

Since ξ t converges to 1 in probability on Ω , there exists T such that for any ϵ, δ > 0 and t > T ,

For any Z ∈ Z , s ∈ S , a ∈ A , and t > T , by using Hölder's inequality,

■

which implies that E ξ t converges to E uniformly on Z for all s, a .

By using A.1, we can get the desired result.

A.3 Proof of Theorem 3.3

Theorem 3.3. Let ξ n be sampled from ¯ U ∆ n ( Z ( n -1) ) for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators E τ ξ n :1 converges, i.e. E τ ξ n :1 [ Z ] → E [ Z ∗ ] Moreover, the following bound holds,

Proof. We denote a ∗ i ( ξ n ) = argmax a ′ E ξ n [ Z ( n -1) i ( s ′ , a ′ )] as the greedy action of Z ( n -1) i under perturbation ξ n . Also, we denote sup s,a | · | which is the supremum norm over s and a as ∥ · ∥ sa .

Before we start from the term ∥ ∥ E [ Z ( k +1) ] -E [ Z ( k ) ] ∥ ∥ sa , for a given ( s, a ) ,

where we use A.1.1 in third and fifth line and A.1.2 in sixth line.

■

Taking a supremum over s and a , then for all k > 0 ,

Since ∑ ∞ i =1 γ i = γ 1 -γ < ∞ and ∑ ∞ i =1 ∆ i < ∞ by assumption, we have

which is resulted from the convergence of Cauchy product of two sequences { γ i } and { ∆ i } . Hence, { E [ Z ( k ) ] } is a Cauchy sequence and therefore converges for every Z ∈ Z .

Let E [ Z ∗ ] be the limit point of the sequence { E [ Z ( n ) ] } . Then,

A.4 Proof of Theorem 3.4

Theorem 3.4. If { ∆ n } follows the assumption in Theorem 3.3, then E [ Z ∗ ] is the unique solution of Bellman optimality equation.

Proof. The proof follows by linearity of expectation. Denote the Q-value based operator as ¯ τ . Note that ∆ n converges to 0 with regularity of Z implies that ξ n converges to 1 in probability on Ω , i.e.,

By Theorem A.3, for a given ϵ > 0 , there exists a constant K = max( K 1 , K 2 ) such that for every k ≥ K 1 ,

Since ¯ τ is continuous, for every k ≥ K 2 ,

Thus, it holds that

Therefore, we have

Since the standard Bellman optimality operator has a unique solution, we derived the desired result. ■

B Algorithm Pipeline

Figure 8 shows the pipeline of our algorithm. With the schedule of perturbation bound { ∆ n } , the ambiguity set U ∆ n ( Z n -1 ) can be defined by previous Z n -1 . For each step, (distributional) perturbation ξ n is sampled from U ∆ n ( Z n -1 ) by the symmetric Dirichlet distribution and then PDBOO τ ξ n can be performed.

Zo

Z1

Z

Te

n-1

n

U△,(Zo)

E

sn

U△n(Zn-1)

C Implementation details

Except for each own hyperparameter, our algorithms and DLTV shares the same hyperparameter and network architecture with QR-DQN [13] for a fair comparison. Also, we set up p-DLTV by only multiplying a gaussian noise N (0 , 1) to the coefficient of DLTV. We do not combine any additional improvements of Rainbow such as double Q-learning, dueling network, prioritized replay, and n -step update. Experiments on LunarLander-v2 and Atari games were performed with 3 random seeds. The training process is 0-2% slower than QR-DQN due to the sampling ξ and reweighting procedures.

C.1 Hyperparameter Setting

We report the hyperparameters for each environments we used in our experiments.

C.2 Pseuodocode of p-DLTV

D Further experimental results & Discussion

D.1 N-Chain

To explore the effect of intrinsic uncertainty, we run multiple experiments with various reward settings for the rightmost state as keeping their mean at 9. As the distance between two Gaussians was increased, the performance of DLTV decrease gradually, while other algorithms show consistent results. The result implies the interference of one-sided tendency on risk is proportional to the magnitude of the intrinsic uncertainty and the randomized criterion is effective in escaping from the issue.

D.2 LunarLander-v2

action stochasticity

extreme reward system

random initial force

+100

0.05

0.10

0.15

0.20

0.25

0.30

Steps (Million)

200

100

0

100

200

300

Score

LunarLander-v2

DLTV

p-DLTV

PQR

To verify the effectiveness of the proposed algorithm in the complex environment with high intrinsic uncertainty , we conduct the experiment on LunarLander-v2. We have focused on three main factors that increase the intrinsic uncertainty from the structural design of LunarLander environment:

Random initial force: The lander starts at the top center with an random initial force.

Action stochasticity: The noise of engines causes different transitions with same action.

Extreme reward system: If the lander crashes, it receives -100 points. If the lander comes to rest, it receives +100 points.

Therefore, several returns with a fixed policy have a high variance. As previously discussed about the fixedness from N-Chain environment, we can demonstrate that randomized approaches, PQR and p-DLTV, outperform other baselines in LunarLander-v2.

D.3 Atari games

0

50

100

150

200

Training Frames (Million)

200

400

600

800

1000

Average Score

Gravitar-v4

DLTV

p-DLTV

PQR

0

50

100

150

200

Training Frames (Million)

0

2500

5000

7500

10000

12500

15000

Average Score

Kangaroo-v4

0

10

20

30

40

Training Frames (Million)

0

5

10

15

20

Average Score

Pong-v4

0

50

100

150

200

Training Frames (Million)

0

10000

20000

30000

40000

Average Score

Seaquest-v4

We test our algorithm under 30 no-op settings to align with previous works. We compare our baseline results with results from the DQN Zoo framework [29], which provides the full benchmark results on 55 Atari games at 50M and 200M frames. We report the average of the best scores over 5 seeds for each baseline algorithms up to 50M frames.

However, recent studies tried to follow the setting proposed by Machado et al. [20] for reproducibility, where they recommended using sticky actions. Hence, we provide all human normalized scores results across 55 Atari games for 50M frames including previous report of Dopamine and DQN Zoo framework to help the follow-up researchers as a reference. We exclude Defender and Surround which is not reported on Yang et al. [38] because of relialbility issues in the Dopamine framework. In summary,

DQN Zoo framework corresponds to 30 no-op settings (version v4 ).

Dopamine framework corresponds to sticky actions protocol (version v0 ).

D.3.1 Ablation on PQR schedules

0

5

10

15

20

25

30

35

40

Training Frames (Million)

20

10

0

10

20

Average Return

Ablation on PQR schedules

1

t

0

PQR

OPT

log

t

t

+ OPT

log

t

t

To investigate the effect of the schedule of ∆ t , we run the experiment on Pong-v4 and set up several baselines as follows:

1 /t 0 : A fixed size ambiguity set. ∆ t = O (1)

PQR : Our main algorithm. ∆ t = O (1 /t 1+ ϵ )

OPT : We fix the output vector, sampled from the Dirichlet distribution, as [0 , 0 , . . . , 1] , forcing the agent to estimate only optimistically.

√ log t/t : We imitate the schedule of p-DLTV (which does not satisfy the sufficient condition we presented). ∆ t = O ( √ log t/t )

√ log t/t + OPT : We imitate the schedule of DLTV (which does not satisfy the sufficient condition we presented). We fixed the output vector, sampled from the Dirichlet distribution, as [0 , 0 , . . . , 1] , forcing the agent to estimate only optimistically. ∆ t = | O ( √ log t/t ) |

In this experiment, our proposed PQR is the only method that stably achieves the maximum score with low variance. In the case of optimism (purple, orange curve), the agent learns quickly in the early stages, but converges without reaching the maximum score. In the case of fixed ambiguity set (blue curve), it converges to suboptimal and eventually shows low performance. This result implies the necessity of time-varying schedule of ∆ t . Finally, when imitating the schedule of p-DLTV (green curve), the performance also degrades implying that the proposed sufficient condition is quite tight.

D.4 Comparison with QUOTA

Zhang and Yao [40] have proposed Quantile Option Architecture(QUOTA) which derives different policies corresponding to different risk levels and consider them as options. By using an option-based framework, the agent learns a high-level policy that adaptively selects a pessimistic or optimistic exploration strategy. While QUOTA has a similar approach in high-level idea, PQR gives a lot of improvements in both theoretical analysis and experimental results.

· Theoretical guarantees of convergence toward risk-neutrality.

Since the agent selects via randomized risk criterion, the natural question is 'How should we control the injected randomness without sacrificing the original purpose of risk-neutrality?'. In this work, we provide the sufficient condition for convergence without sacrificing riskneutral perspective. Although QUOTA explores by using optimism or pessimism of a value distribution, there is no discussion whether the convergence is guaranteed toward a risk-neutral objective.

· Explaining the effectiveness of randomized strategy.

QUOTAtested on two Markov chains to illustrate the inefficiency of expectation-based RL. It assumed that each task has an inherent, but unknown, preferred risk strategy, so agents should learn hidden preference. In contrast, we point out that the amount of inherent (intrinsic) uncertainty causes the inefficiency of fixed optimism or pessimism based exploration.

· Significant performance difference in experimental results.

QUOTA is based on option-based learning which requires an additional option-value network. While QUOTA aims to control risk-sensitivity by transforming into an option O , the introduction of an option-value network requires the agent to explore an action space | O | × | A | . This opposes the idea of efficient exploration as a factor that increases the complexity of learning. In contrast, PQR does not require a additional network and explores over the original action space. In addition, PQR does not artificially discretize the ambiguity set of risk measurement. Another main reason is that PQR does not depend on an greedy schedule which is well-known for inefficient exploration strategies in tabular episodic MDP [18]. PQR solely explores its own strategies which is a simple yet effective approach. However, QUOTA depends on a greedy schedule in both quantile and option networks.

D.5 Reproducibility issues on DLTV

For the expected concerns about the comparison with DLTV, we address some technical issues to correct misconceptions of their performance. Before we reproduce the empirical results of DLTV, Mavrin et al. [21] did not report each raw scores of Atari games, but only the relative performance with cumulative rewards comparing with QR-DQN. While DLTV was reported to have a cumulative reward 4.8 times greater than QR-DQN, such gain mainly comes from VENTURE which is evaluated as 22,700% from their metric (i.e., 463% performance gain solely). However, the approximate raw score of VENTURE was 900 which is lower than our score of 993.3. Hence, the report with cumulative rewards causes a severe misconception that can be overestimated where the human-normalized score is commonly used for evaluation metrics. For a fair comparison, we computed based on mean and median of human-normalized scores and obtained results of 603.66% and 109.90%. Due to the absence of public results, however, DLTV was inevitably excluded from the comparison with human-normalized score in the main paper for reliability. In Table 4 and 7, we report our raw scores and human-normalized score of DLTV based on QR-DQN_zoo performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a distributional RL algorithm, PQR, that uses distributional return estimates for exploration by updating with a greedy distributional Bellman operator for time-varying, random risk measures. The paper demonstrates that, under a simple concentration condition on the random risk measures over time, the means of return distributions learned by PQR converge to the optimal Q-values. It is argued that the use of random risk measures (as opposed to static risk measures) produces updates that are less biased away from the optimal Q-values. It is shown that PQR demonstrates favorable exploration performance in a simple chain MDP and in Atari.\n\nSTRENGTHS:\nThe paper presents a well-motivated algorithm for leveraging return distributions for exploration in (deep) RL. The idea of using random risk measures in greedy distributional Bellman updates to veer less from the optimal Q-values is novel and interesting. Furthermore, the empirical results of the PQR algorithm appear to be quite good.\n\nWEAKNESSES:\nThe proposed algorithm itself may not be substantially novel -- I suspect similar results would hold for QR-DQN with a dynamic distortion risk measure that tends to the identity sufficiently quickly. \n\nThe theoretical results are fairly weak. Firstly, convergence of the return distributions is only established with respect to their first moments. Moreover, the upper bound equation in Theorem 3.3 is difficult to interpret, and I am not sure what one should conclude from this upper bound.\n\nAdditionally, the influence of the schedule $(\\Delta_t)_{t\\geq 0}$ and the Dirichlet parameter $\\beta$ is not really studied -- I suspect, at least in theory, that there are more optimal data-based schedules.\n\n## Grammatical / non-major issues\n\nL4 \"present a novel distributional reinforcement learning that selects...\" -> \"present a novel distributional reinforcement learning **algorithm** that selects...\"\n\nLast sentence in the abstract cites [6] as a risk-sensitive distributional RL algorithm, but this is not really a distributional RL algorithm.\n\nL144 mentions the perturbed probability distribution $\\xi\\mathbb{P}$, but this doesn't really type-check. $\\mathbbf{P}$ is a probability measure (maps measurable sets to probabilities) and $\\xi$ is a probability density function (mapping elements of the sample space to reals), so the product $\\xi\\mathbb{P}$ is not exactly correct.\n\nTheorem 3.4 needs to be made more precise. It makes a claim about the fixed point of the Bellman optimality operator, but I think you meant the PDBOO.\n\nQUESTIONS:\nL172 says \"PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy...\" -- what does it mean to \"adhere to the risk-neutral optimal policy\"? The PDBOO updates are still performing greedy updates w.r.t. risk-sensitive policies.\n\nAs mentioned above, convergence of return distributions is not really established at all under the PDBOO update -- we only know about the convergence of their means. But if the dynamics of the other statistics of the return distributions are essentially unknown, how can you be sure that the random risk measures are actually estimating anything meaningful?\n\nIs there any correspondence between the PDBOO targets and the \"Thompson Sampling\" approach introduced by Riou and Honda (2020) http://proceedings.mlr.press/v117/riou20a/riou20a.pdf? There may be an interesting connection there, which could perhaps help add precision to the statement about \"adhering to the risk-neutral optimal policy\".\n\nLIMITATIONS:\nLimitations are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new exploration strategy that is more principled than epsilon-greedy or DLTV of Mavrin et al. (based on truncated variance). The proposed exploration strategy is to based on a novel perturbed distributional Bellman optimality operator, where the goal (Eqn. 1) is to minimize the distributional Bellman objective subject to the perturbation being sampled from an uncertainty set; the uncertainty set is defined as importance weighting the next state's return distribution in the Bellman operator, where the radius of the set decreases with time. The authors show that by appropriately decaying the set's radius, the algorithm can learn the optimal value function, and provide experimental validation on N-chain and Atari, showing competitive performance to IQN and Rainbow.\n\nSTRENGTHS:\n1. Perturbing the expectation with a density ratio is a simple but neat idea.\n2. Experimental results seem pretty promising, given that QR-DQN's performance significantly improves with the proposed strategy, sometimes even surpassing IQN.\n\nWEAKNESSES:\n1. I am not fully sold that perturbing the Bellman operator is a principled way to modeling epistemic uncertainty of our estimates. In particular, the paper seems to suggest that optimism is actually biased and unprincipled, while we know that in both theory and practice, optimism in the face of uncertainty is principled for online RL and minimax optimal in most well-known settings. \n1b. I think the paper would benefit from a clearer and more proper explanation of how prior settings are biased / unbiased, and how the proposed approach is better/worse than prior exploration strategies.\n2. if I understand correctly, the proposed uncertainty set seems to be (s,a)-independent, which seems wrong. (in particular, if some parts of the state space was not visited often, that uncertainty should be high, while uncertainty should be low for frequented states).\n\nQUESTIONS:\n1. Typically, distributionally robust optimization (DRO) optimizes the worst-case performance. However, in eqn. 1, you sample from the uncertainty set instead of taking a supremum. Can you give some intuition on why you care about the average performance over the set, rather than worst case? \n2. Is the perturbation gap related/inspired by the dual form of dynamic risk with coherent risk measures? \n3. Theorem 3.3 shows some kind of convergence of the learned $E Z^{(n)}$ to the true $Q*$ , but it seems that as n -> infinity, the bound does not necessarily to go zero. Doesn't this mean there will be some bias in the proposed approach?\n4. To be clear, does your implementation only use QR-DQN + the proposed PQR exploration strategy? Do you make use of target networks, double Q-learning, prioritized replay, or any of the bag of tricks that Rainbow uses?\n5. In Section 4.2, what is meant by \"intrinsic uncertainty\" of Atari? Without sticky actions, Atari is fully deterministic and there is no aleatoric uncertainty (only epistemic uncertainty), right?\n\nLIMITATIONS:\nPlease see weakness/questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the exploration problem in distributional reinforcement learning (DRL). It proposes the Perturbed Distributional Bellman Operator (PDBOO) as an extension of the distributional Bellman operator, which introduces non-directional noise to the target return distribution. This extension originates from the problem that a one-sided risk tendency can cause biased action selection, which might lead to unsuitable behavior depending on tasks. To address this issue, the PDBOO adds non-directional noise as a perturbation to the return distribution targeted for learning, ensuring risk neutrality. The perturbation term is sampled at each time step. This approach allows for the selection of diverse actions while maintaining risk neutrality. The authors provide a theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to asymptotically converge to the unique fixed point of the Bellman optimality equation. Then, the PDBOO is applied to the Quantile Regression Deep Q Network (QR-DQN) to propose a method called Perturbed Quantile Regression (PQR). The effectiveness of PQR is evaluated in a 4-states chain MDP and 55 Atari games and compared with several DQN variants.\n\nSTRENGTHS:\n- The paper introduces the Perturbed Distributional Bellman Operator (PDBOO), a novel extension of the distributional Bellman operator. This approach enables diverse exploration while maintaining risk neutrality in distributional reinforcement learning. The authors also develop Perturbed Quantile Regression (PQR), a DQN variant that estimates quantiles based on PDBOO. The proposal represents a new and interesting direction of the study in distributional RL.\n- The authors provide a solid theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to achieve the unique fixed point of the Bellman optimality equation asymptotically. This result provides confidence in their proposed methods and valuable insights for algorithm development.\n- The proposed PQR method is evaluated in a 4-states chain MDP and 55 Atari games. While the results showed room for improvement, the evaluation confirms the potential practical applicability of PQR and its ability to address the exploration problem while maintaining risk neutrality.\n\nWEAKNESSES:\n- The paper lacks an explanation as to why the proposed approach is more effective compared to other methods. It is crucial to discuss under what tasks PDBOO is effective and when it might not be suitable to understand its value.\n   - For example, PDBOO would add random noise (perturbation) to the target return distribution for learning. Depending on the perspective, this could also have the side effect of making it more challenging to learn the return distribution due to the existence of noise. I would like to know if there are situations where PDBOO should not be applied.\n   - Also, it is not clear why the approach of PDBOO is necessary, as it seems that what the authors want to achieve with PDBOO would not be the perturbation of the target return distribution for learning but instead could also be accomplished by perturbing the estimated return distribution at the time of action selection. This is somewhat supported by the results of p-DLTV in the N-chain MDP experiment.\n\n- The overall experimental results seem weak and do not resolve the abovementioned questions. Furthermore, the improved performance of PQR is observed in a subset of games, and the improvement seems modest. There is no comparison with p-DLTV, which showed similar good performance in the N-chain MDP experiment.\n\nQUESTIONS:\n- How exactly does the proposed PQR select actions? Does it use an epsilon-greedy or softmax policy? More importantly, how does it compute the values of the actions which will be the input to the policy?  Does it sample the perturbation \\xi, as PDBOO does, and skew the distribution for calculating the mean? I think there are two timings at which the distribution can be perturbed by introducing ξ: during the parameter θ update, and during the action selection. This paper only seems to discuss the former, but considering that p-DLTV performs well similarly to the proposed method, wouldn't it also be valid to perturb the mean for the action selection?\n- What is the type of policy for each baseline used in the experiments? Is it epsilon-greedy or softmax policy? I do not see it stated.\n- Regarding the N-Chain with high intrinsic uncertainty experiment, the difference in the decaying rate of \\Delta and c between PQR and DLTV or p-DLTV might also be significant. What are the authors' thoughts? It would be best to align the experiment conditions as much as possible.\n- The authors used the N-chain task where risk-based exploration like DLTV obviously does not work well, so the presented results are reasonable. On the other hand, in a task where DLTV works well (e.g., both s_0 and s_4 have the same reward variance), I think it is essential to experiment to see how the proposed method performs to compare the proposed method fairly.\n\nLIMITATIONS:\n- The paper lacks a thorough discussion of the limitations of the proposed PDBOO and PQR. The authors should provide more insight into scenarios where the method might not be applicable or efficient.\n- As pointed out above, why the proposed method is effective (for instance, as opposed to p-DLTV or simply increasing the temperature during action selection) remains unclear. It seems that the questions are not adequately addressed in the experiments. The authors should consider addressing these points to provide a more complete view of their method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors address the issue of biased exploration caused by a one-sided tendency on risk in action selection by proposing the method perturbed quantile regression (PQR). PQR selects actions by randoming risk criterion while retaining a risk-neutral objective. The authors also derive a sufficient condition for the convergence of the proposed Bellman operator without satisfying the conventional contraction property. Results are demonstrated in an extended N-Chain environment and the Atari suite showing improved performance.\n\nSTRENGTHS:\nThe paper was written very well in a pedagogical manner which provided intuition and understanding of the challenges in the field surrounding the contribution of the method. Deep technical detail was provided with accessible explanations.\n\nThe motivation for the method was well grounded intuitively. The experiment section also directly confirmed the intended contributions, organizing each experiment and result around specific questions directly aligned to the motivation.\n\nThe background section was thorough and clear. Since the authors pulled together several topics which many researchers do not study all of, this presentation was very important and added a clear path of accessibility to the technical detail in the rest of the paper.\n\nThe supplementary material provided sufficient information to reproduce results including code. The authors also included substantial supporting theoretical information in proofs and a large amount of additional convincing experiments.\n\nThe contribution to the community regarding managing risk in exploration strategies is important and interesting.\n\nWEAKNESSES:\nThe authors claim limitations about epistemic and parametric uncertainty metrics as well as optimism in the face of uncertainty approaches which I thought were not backed up in the paper strongly enough given that these claims are the premise of the entire work. The paragraph in the introduction which presents the limitations describes it as a broad issue, yet only one paper is cited as an example. To provide more context, it would help to give at least more citations here to back up the importance of this challenge. The deeper explanation of the DLTV paper is not necessary for all additional citations.\n\nPlease provide more comments in Algorithm 1 (like the current text “Select greedy action…”). The algorithm is fairly clear but with the density of notation, recalling the variable references would be much easier for the reader with textual reminders in the algorithm block.\n\nMy primary concerns regarding this work are related to the experiments in Section 4.1 which require clarification. Figure 4 is quite hard to read since the text is very small. Reformatting could make these plots readable when the paper is printed. Furthermore, the results in Figure 4 are very hard to understand due not only to the legibility but also due to insufficient explanation in the experimental results section and figure. See Questions for more information.\n\nQUESTIONS:\nI am unsure if I am correctly interpreting the results in Figure 4. Is the lower standard deviation around the dotted line for the mean a1 considered to be a better estimation result? Given that this experimental setting is new (the authors indicate that they adapt the N-Chain environment and cite work - DQN - which presents very different result metrics), substantially clearer explanation is required about the connection between the presentation and the claims.\n\nLIMITATIONS:\nThe paper does not discuss limitations or broader impact. This discussion should be added.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses exploration via distributional RL which has, to date, been most recently performed under an _optimism in the face of uncertainty_ (OFU) paradigm. This paper loosely characterizes this OFU approach as problematic as it results in biased exploration, resulting in sub-optimal value distributions. To remedy this pitfall, this paper proposes a stochastic perturbation of the distributional Bellman operator. Clear theoretical development of the proposed operator gives way to practical algorithm development building on top of quantile regression. Empirically, the proposed algorithm PQR is demonstrated to have clear performance benefits.\n\nSTRENGTHS:\nThe paper sets up a really interesting juxaposition between the advantages of the distRL paradigm in accounting for aleatoric uncertainty but attempting to resolve epistemic uncertainty through exploration. As set out in the introduction of the paper, this presents an appealing motivation to perhaps determine more effective exploration strategies that are cognizant of this apparent mismatch.\n\nThe methods presented in this paper to address the one-sided tendency on risk, resulting in a practical algorithm, are reasonable and seem to neatly satisfy the intended contributions set out in the paper.\n\nI was impressed at the extent by which the authors provide sufficient background in distributional RL in Section 2. Many papers take shortcuts here however this paper lays out relevant concepts from which the remaining development of the methods introduced in the paper are easier to understand from this grounding.\n\nThe technical formulation of the PDBOO is clear and well written. The definitions help to outline important concepts that motivate the further development of the methods.\n\nI appreciate how the authors approach Section 4, clearly laying out the objectives of their empirical study. This helps to frame the impressive experimental performance achieved by the introduced PQR algorithm.\n\nWEAKNESSES:\nIt’s not clear what is meant by “without losing the risk-neutral objective” (mentioned in the abstract)…\n\nIn line 43, it could be helpful to again list out what the two types of uncertainty are to more directly connect this sentence to what has been introduced conceptually earlier in the paper.\n\nThe authors seem to use “deep” RL interchangably with “distributional” RL in the introduction. This makes the framing of the work a little challenging to follow initially. Better clarification of prior works in the non-distributional deep RL from distributional deep RL would help make this easier to understand and would strengthen the beginning sections of the paper.\n\nThe statements made in lines 50-53 are pretty strong, as in they seem to be a result of some analysis or in the worst case are derived from opinionated intuition. Some additional justification with formal explanation (whether through a toy example or reference to later section or citation to a paper) would help make the writing easier to accept and be persuaded by. This is especially important because the “one-sided tendency on risk” appears to be a major foundation for the proposed PDBOO.\n\nIt would be helpful if the authors expanded more on their discussion at the end of Section 2.2 since DLTV appears to introduce biased exploration that is addressed in this work. A couple of sentences addressing the limitations of DLTV and how the work presented in this paper addresses those would be great. The sentence that ends the preceding paragraph (Line 100-101) is a decent example of setting up a clear understanding of the contributions set forth in the paper.\n\nMany of the claims about why PQR performs well on n-chain and in Atari are speculative. The paper would be significantly improved if more rigorous analysis connecting the theoretical treatise in Sections 3.2 and 3.3 to empirical performance and how it differs from the baselines would be great.\n\nThe discussion around the empirical results is somewhat disappointing. It would additionally be interesting to see any ablations that could be done on PQR to help tease apart the effect of the different components of the algorithm have on performance. I suppose that this is is partially addressed with the hyperparameter plots in Figure 6 but I think it would be helpful to see how sensitive the resulting policy is to the scope of the ambiguity sets used.\n\nQUESTIONS:\nIs the guarantee of risk-neutrality found in expectation? E.g. is it a response to evenly choosing a risk-averse obj vs a risk-seeking obj?\n\nThe writing in lines 41-45 is pretty difficult to follow. It’s apparent that the authors are taking care to concisely convey their thoughts but fail to do so clearly.\n\nLine 46: It’s not clear what “side-effect” is being referred to here. Additionally, it would be helpful to perhaps directly reference what section in the paper is being referred to in this sentence as the authors indicate that they’ve explored the effects of OFU in deep RL and the effects it has on handling the two types of uncertainty.\n\nLine 61: Could you call PQR a “stochastic optimization” approach to help situate the work among other prior work? This isn’t a major concern of mine. I’m just thinking that it could be an easier way to describe the approach rather than calling it a “randomized” approach.\n\nLine 93: The loss function here is simply the Huber loss, no? It’s not overly important to include this much detail to be honest unless the paper heavily develops further insight directly from these equations (applies to Section 2 in it’s entirety). Otherwise this just feels like a regurgitation of information that’s already established in the prior literature.\n\nLine 128-129: Why is it important to have an interpretation of different risk levels as constructing an options framework? How does this help build the proposed methods introduced in this paper?\n\nLine 178: Does “the standard” here mean the standard distributional Bellman operator?\n\nLine 186: What is $B_\\xi$? How does this relate to the symmetric Dirichlet distribution used to construct each ambiguity set? This seems to be addressed in Lines 210-212.\n\nLIMITATIONS:\nSeveral technical statements that crucial to the dvelopment of the proposed methods/contributions are vague and are not precisely justified. For example, it would be really helpful (and would help me agree/accept the claims) in Lines 170-174 if it were shown precisely how Eqt 1 maintains risk-neutral policy performance. As currently written we have to take the authors’ word. Additionally, the statement of using a min-expectation vs. a min-max operator should be developed and justified. This is a new concept that hasn’t been discussed previously.\n\nIt’s not clear how the time-varying amiguity sets are decoupled from a changing estimate of the return distribution Z. As formulated, PDBOO seems to ignore that $Z_\\theta$ is changing through time.\n\nThe paper omits at least one relevant paper, Moskovitz, et al (NeurIPS 2021) where an ensemble of distributional crtiics are used to balance between optimism and pessmism for continous control. It would be informative to hear from the authors why this paper, and its corresponding approach, should or shouldn’t be included as a comparative baseline.\n\n>Moskovitz, Ted, et al. \"Tactical optimism and pessimism for deep reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 12849-12863.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}