SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taehyun Cho $^{1}$ , Seungyub Han $^{1,3}$ , Heesoo Lee $^{1}$ , Kyungjae Lee $^{2}$ , Jungwoo Lee $^{1*}$ $^{1}$  Seoul National University,  $^{2}$  Chung-Ang University,  $^{3}$  Hodoo AI Labs {talion, seungyubhan, algol7240, junglee}@snu.ac.kr {kyungjae.lee}@ai.cau.ac.kr
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning algorithm that selects actions by randomizing risk criterion to avoid one-sided tendency on risk. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure and prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.
Distributional reinforcement learning (DistRL) learns the stochasticity of returns in the reinforcement learning environments and has shown remarkable performance in numerous benchmark tasks. DistRL agents model the approximated distribution of returns, where the mean value implies the conventional Q-value [2, 6, 13, 25] and provides more statistical information (e.g., mode, median, variance) for control. Precisely, DistRL aims to capture intrinsic (aleatoric) uncertainty which is an inherent and irreducible randomness in the environment. Such learned uncertainty gives rise to the notion of risk-sensitivity, and several distributional reinforcement learning algorithms distort the learned distribution to create a risk-averse or risk-seeking decision making [8, 12].
Despite the richness of risk-sensitive information from return distribution, only a few DistRL
methods [11, 21, 26, 35, 42] have tried to employ its benefits for exploration strategies which is essential in deep RL to find an optimal behavior within a few trials. The main reason is that the exploration strategies so far is based on parametric (epistemic) uncertainty which arise from insufficient or
Figure 1: Illustrative example of why a biased risk criterion (naive optimism) can degrade performance. Suppose two actions have similar expected returns, but different variances (intrinsic uncertainty). (Left) If an agent does not specify the risk criterion at the moment, the probability of selecting each action should be similar. (Right) As OFU principle encourages to decide uncertain behaviors, the empirical variance from quantiles was used as an estimate of uncertainty. [19, 21, 23]. However, optimistic decision based on empirical variance inevitably leads a risk-seeking behavior, which causes biased action selection.
Number of candidate actions
Figure 2: An illustrative example of proposed algorithm (PQR). Each distribution represents the empirical PDF of return. PQR benefits from excluding inferior actions and promoting unbiased selection with regards to high intrinsic uncertainty through randomized risk criterion.
inaccurate data. In particular, Optimism in the face of uncertainty (OFU) is one of the fundamental exploration principles that employs parametric uncertainty to promote exploring less understood behaviors and to construct confidence set. In bandit or tabular MDP settings, OFU-based algorithms select an action with the highest upper-confidence bound (UCB) of parametric uncertainty which can be considered as the optimistic decision at the moment [5, 10].
However, in deep RL, it is hard to trivially estimate the parametric uncertainty accurately due to the black-box nature of neural networks and high-dimensionality of state-action space. Without further computational task, the estimated variance from distribution is extracted as a mixture of two types of uncertainty, making it difficult to decompose either component. For example, DLTV [21] was proposed as a distribution-based OFU exploration that decays bonus rate to suppress the effect of intrinsic uncertainty, which unintentionally induces a risk-seeking policy. Although DLTV is the first attempt to introduce OFU in distRL, we found that consistent optimism on the uncertainty of the estimated distribution still leads to biased exploration. We will refer to this side-effect as one-sided tendency on risk, where selecting an action based on a fixed risk criterion degrades learning performance. In Section 4, we will demonstrate the one-sided tendency on risk through a toy experiment and show that our proposed randomized approach is effective to avoid this side effect.
In this paper, we introduce Perturbed Distributional Bellman Optimality Operator (PDBOO) to address the issue of biased exploration caused by a one-sided tendency on risk in action selection. We define the distributional perturbation on return distribution to re-evaluate the estimate of return by distorting the learned distribution with perturbation weight. To facilitate deep RL algorithm, we present Perturbed Quantile Regression (PQR) and test in Atari 55 games comparing with other distributional RL algorithms that have been verified for reproducibility by official platforms [4, 29].
In summary, our contributions are as follows.
- A randomized approach called perturbed quantile regression (PQR) is proposed without sacrificing the original (risk-neutral) optimality and improves over naive optimistic strategies.
- A sufficient condition for convergence of the proposed Bellman operator is provided without satisfying the conventional contraction property.
2 Backgrounds & Related works
We consider a Markov decision process (MDP) which is defined as a tuple  $(S, \mathcal{A}, P, R, \gamma)$  where  $S$  is a finite state space,  $\mathcal{A}$  is a finite action space,  $P: S \times \mathcal{A} \times S \to [0,1]$  is the transition probability,  $R$  is the random variable of rewards in  $[-R_{\max}, R_{\max}]$ , and  $\gamma \in [0,1)$  is the discount factor. We define a stochastic policy  $\pi(\cdot|s)$  which is a conditional distribution over  $\mathcal{A}$  given state  $s$ . For a fixed policy  $\pi$ , we denote  $Z^{\pi}(s,a)$  as a random variable of return distribution of state-action pair  $(s,a)$  following the policy  $\pi$ . We attain  $Z^{\pi}(s,a) = \sum_{t=0}^{\infty} \gamma^{t} R(S_{t}, A_{t})$ , where  $S_{t+1} \sim P(\cdot|S_{t}, A_{t})$ ,  $A_{t} \sim \pi(\cdot|S_{t})$  and  $S_{0} = s$ ,  $A_{0} = a$ . Then, we define an action-value function as  $Q^{\pi}(s,a) = \mathbb{E}[Z^{\pi}(s,a)]$  in  $[-V_{\max}, V_{\max}]$  where  $V_{\max} = R_{\max} / (1 - \gamma)$ . For regularity, we further notice that the space of
: list of actions to be added to the buffer
action-value distributions  $\mathcal{Z}$  has the first moment bounded by  $V_{\mathrm{max}}$ :
$$
\mathcal {Z} = \left\{Z: \mathcal {S} \times \mathcal {A} \rightarrow \mathcal {P} (\mathbb {R}) \middle | \mathbb {E} [ | Z (s, a) | ] \leq V _ {\max }, \forall (s, a) \right\}.
$$
In distributional RL, the return distribution for the fixed  $\pi$  can be computed via dynamic programming with the distributional Bellman operator defined as,
$$
\mathcal {T} ^ {\pi} Z (s, a) \stackrel {{D}} {=} R (s, a) + \gamma Z (S ^ {\prime}, A ^ {\prime}), \quad S ^ {\prime} \sim P (\cdot | s, a), A ^ {\prime} \sim \pi (\cdot | S ^ {\prime})
$$
where  $\stackrel{D}{=}$  denotes that both random variables share the same probability distribution. We can compute the optimal return distribution by using the distributional Bellman optimality operator defined as,
$$
\mathcal {T} Z (s, a) \stackrel {{D}} {{=}} R (s, a) + \gamma Z (S ^ {\prime}, a ^ {*}), S ^ {\prime} \sim P (\cdot | s, a), a ^ {*} = \operatorname * {a r g m a x} _ {a ^ {\prime}} \mathbb {E} _ {Z} [ Z (S ^ {\prime}, a ^ {\prime}) ].
$$
Bellemare et al. [2] have shown that  $\mathcal{T}^{\pi}$  is a contraction in a maximal form of the Wasserstein metric but  $\mathcal{T}$  is not a contraction in any metric. Combining with the expectation operator,  $\mathbb{E}\mathcal{T}$  is a contraction so that we can guarantee that the expectation of  $Z$  converges to the optimal state-action value. Another notable difference is that the convergence of a return distribution is not generally guaranteed to be unique, unless there is a total ordering  $\prec$  on the set of greedy policies.
2.2 Exploration on Distributional RL
To combine with deep RL, a parametric distribution  $Z_{\theta}$  is used to learn a return distribution. Dabney et al. [13] have employed a quantile regression to approximate the full distribution by letting  $Z_{\theta}(s,a) = \frac{1}{N}\sum_{i = 1}^{N}\delta_{\theta_i(s,a)}$  where  $\theta$  represents the locations of a mixture of  $N$  Dirac delta functions. Each  $\theta_{i}$  represents the value where the cumulative probability is  $\tau_{i} = \frac{i}{N}$ . By using the quantile representation with the distributional Bellman optimality operator, the problem can be formulated as a minimization problem as,
$$
\theta = \arg \min _ {\theta^ {\prime}} D (Z _ {\theta^ {\prime}} (s _ {t}, a _ {t}), \mathcal {T} Z _ {\theta^ {-}} (s _ {t}, a _ {t})) = \arg \min _ {\theta^ {\prime}} \sum_ {i, j = 1} ^ {N} \frac {\rho_ {\hat {\tau} _ {i}} ^ {\kappa} (r _ {t} + \gamma \theta_ {j} ^ {-} (s _ {t + 1} , a ^ {\prime}) - \theta_ {i} ^ {\prime} (s _ {t} , a _ {t}))}{N}
$$
where  $(s_t, a_t, r_t, s_{t+1})$  is a given transition pair,  $\hat{\tau}_i = \frac{\tau_{i-1} + \tau_i}{2}$ ,  $a' := \operatorname{argmax}_{a'} \mathbb{E}_Z[Z_\theta(s_{t+1}, a')]$ ,  $\rho_{\tilde{\tau}_i}^\kappa(x) := |\hat{\tau}_i - \delta_{\{x < 0\}}| \mathcal{L}_\kappa(x)$ , and  $\mathcal{L}_\kappa(x) := x^2/2$  for  $|x| \leq \kappa$  and  $\mathcal{L}_\kappa(x) := \kappa(|x| - \frac{1}{2}\kappa)$ , otherwise.
Based on the quantile regression, Dabney et al. [13] have proposed a quantile regression deep Q network (QR-DQN) that shows better empirical performance than the categorical approach [2], since the quantile regression does not restrict the bounds for return.
As deep RL typically did, QR-DQN adjusts  $\epsilon$ -greedy schedule, which selects the greedy action with probability  $1 - \epsilon$  and otherwise selects random available actions uniformly. The majority of QR-DQN variants [12, 38] rely on the same exploration method. However, such approaches do not put aside inferior actions from the selection list and thus suffers from a loss [28]. Hence, designing a schedule to select a statistically plausible action is crucial for efficient exploration.
In recent studies, Mavrin et al. [21] modifies the criterion of action selection for efficient exploration based on optimism in the face of uncertainty. Using left truncated variance as a bonus term and decaying ratio  $c_{t}$  to suppress the intrinsic uncertainty, DLTV was proposed as an uncertainty-based exploration in DistRL without using  $\epsilon$ -greedy schedule. The criterion of DLTV is described as:
$$
a ^ {*} = \underset {a ^ {\prime}} {\mathrm {a r g m a x}} \left(\mathbb {E} _ {P} [ Z (s ^ {\prime}, a ^ {\prime}) ] + c _ {t} \sqrt {\sigma_ {+} ^ {2} (s ^ {\prime} , a ^ {\prime})}\right), c _ {t} = c \sqrt {\frac {\log t}{t}}, \sigma_ {+} ^ {2} = \frac {1}{2 N} \sum_ {i = \frac {N}{2}} ^ {N} (\theta_ {\frac {N}{2}} - \theta_ {i}) ^ {2},
$$
where  $\theta_{i}$ 's are the values of quantile level  $\tau_{i}$ .
2.3 Risk in Distributional RL
Instead of an expected value, risk-sensitive RL is to maximize a pre-defined risk measure such as Mean-Variance [41], Value-at-Risk (VaR) [9], or Conditional Value-at-Risk (CVaR) [30, 31] and results in different classes of optimal policy. Especially, Dabney et al. [12] interprets risk measures
as the expected utility function of the return, i.e.,  $\mathbb{E}_Z[U(Z(s,a))]$ . If the utility function  $U$  is linear, the policy obtained under such risk measure is called risk-neutral. If  $U$  is concave or convex, the resulting policy is termed as risk-averse or risk-seeking, respectively. In general, a distortion risk measure is a generalized expression of risk measure which is generated from the distortion function.
Definition 2.1. Let  $h:[0,1]\to [0,1]$  be a distortion function such that  $h(0) = 0$ ,  $h(1) = 1$  and non-decreasing. Given a probability space  $(\Omega ,\mathcal{F},\mathbb{P})$  and a random variable  $Z:\Omega \rightarrow \mathbb{R}$ , a distortion risk measure  $\rho_h$  corresponding to a distortion function  $h$  is defined by:
$$
\rho_ {h} (Z) := \mathbb {E} ^ {h (\mathbb {P})} [ Z ] = \int_ {- \infty} ^ {\infty} z \frac {\partial}{\partial z} (h \circ F _ {Z}) (z) d z,
$$
where  $F_{Z}$  is the cumulative distribution function of  $Z$ .
In fact, non-decreasing property of  $h$  makes it possible to distort the distribution of  $Z$  while satisfying the fundamental property of CDF. Note that the concavity or the convexity of distortion function also implies risk-averse or seeking behavior, respectively. Dhaene et al. [14] showed that any distorted expectation can be expressed as weighted averages of quantiles. In other words, generating a distortion risk measure is equivalent to choosing a reweighting distribution.
Fortunately, DistRL has a suitable configuration for risk-sensitive decision making by using distortion risk measure. Chow et al. [8] and Stanko and Macek [34] considered risk-sensitive RL with a CVaR objective for robust decision making. Dabney et al. [12] expanded the class of policies on arbitrary distortion risk measures and investigated the effects of a distinct distortion risk measures by changing the sampling distribution for quantile targets  $\tau$ . Zhang and Yao [40] have suggested QUOTA which derives different policies corresponding to different risk levels and considers them as options. Moskovitz et al. [24] have proposed TOP-TD3, an ensemble technique of distributional critics that balances between optimism and pessimism for continuous control.
3 Perturbation in Distributional RL
3.1 Perturbed Distributional Bellman Optimality Operator
To choose statistically plausible actions which may be maximal for certain risk criterion, we will generate a distortion risk measure involved in a pre-defined constraint set, called an ambiguity set. The ambiguity set, originated from distributionally robust optimization (DRO) literature, is a family of distribution characterized by a certain statistical distance such as  $\phi$ -divergence or Wasserstein distance [15, 32]. In this paper, we will examine the ambiguity set defined by the discrepancy between distortion risk measure and expectation. We say the sampled reweighting distribution  $\xi$  as (distributional) perturbation and define it as follows:
Definition 3.1. (Perturbation Gap, Ambiguity Set) Given a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ , let  $X: \Omega \to \mathbb{R}$  be a random variable and  $\Xi = \{\xi: 0 \leq \xi(w) < \infty, \int_{w \in \Omega} \xi(w) \mathbb{P}(dw) = 1\}$  be a set of probability density functions. For a given constraint set  $\mathcal{U} \subset \Xi$ , we say  $\xi \in \mathcal{U}$  as a (distributional) perturbation from  $\mathcal{U}$  and denote the  $\xi$ -weighted expectation of  $X$  as follows:
$$
\mathbb {E} _ {\xi} [ X ] := \int_ {w \in \Omega} X (w) \xi (w) \mathbb {P} (d w),
$$
which can be interpreted as the expectation of  $X$  under some probability measure  $\mathbb{Q}$ , where  $\xi = d\mathbb{Q} / d\mathbb{P}$  is the Radon-Nikodym derivative of  $\mathbb{Q}$  with respect to  $\mathbb{P}$ . We further define  $d(X; \xi) = |\mathbb{E}[X] - \mathbb{E}_{\xi}[X]|$  as perturbation gap of  $X$  with respect to  $\xi$ . Then, for a given constant  $\Delta \geq 0$ , the ambiguity set with the bound  $\Delta$  is defined as
$$
\mathcal {U} _ {\Delta} (X) = \left\{\xi \in \Xi : d (X; \xi) \leq \Delta \right\}.
$$
For brevity, we omit the input  $w$  from a random variable unless confusing. Since  $\xi$  is a probability density function,  $\mathbb{E}_{\xi}[X]$  is an induced risk measure with respect to a reference measure  $\mathbb{P}$ . Intuitively,  $\xi(w)$  can be viewed as a distortion to generate a different probability measure and vary the risk tendency. The aspect of using distortion risk measures looks similar to IQN [12]. However, instead of changing the sampling distribution of quantile level  $\tau$  implicitly, we reweight each quantile from the
ambiguity set. This allows us to control the maximum allowable distortion with bound  $\Delta$ , whereas the risk measure in IQN does not change throughout learning. In Section 3.3, we suggest a practical method to construct the ambiguity set.
Now, we characterize perturbed distributional Bellman optimality operator (PDBOO)  $\mathcal{T}_{\xi}$  for a fixed perturbation  $\xi \in \mathcal{U}_{\Delta}(Z)$  written as below:
$$
\mathcal {T} _ {\xi} Z (s, a) \stackrel {{D}} {=} R (s, a) + \gamma Z (S ^ {\prime}, a ^ {*} (\xi)), \quad S ^ {\prime} \sim P (\cdot | s, a), a ^ {*} (\xi) = \underset {a ^ {\prime}} {\operatorname {a r g m a x}} \mathbb {E} _ {\xi , P} [ Z (s ^ {\prime}, a ^ {\prime}) ].
$$
Notice that  $\xi \equiv 1$  corresponds to a base expectation, i.e.,  $\mathbb{E}_{\xi ,P} = \mathbb{E}_P$ , which recovers the standard distributional Bellman optimality operator  $\mathcal{T}$ . Specifically, PDBOO perturbs the estimated distribution only to select the optimal behavior, while the target is updated with the original (unperturbed) return distribution.
If we consider the time-varying bound of ambiguity set, scheduling  $\Delta_t$  is a key ingredient to determine whether PDBOO will efficiently explore or converge. Intuitively, if an agent continues to sample the distortion risk measure from a fixed ambiguity set with a constant  $\Delta$ , there is a possibility of selecting sub-optimal actions after sufficient exploration, which may not guarantee eventual convergence. Hence, scheduling a constraint of ambiguity set properly at each action selection is crucial to guarantee convergence.
Based on the quantile model  $Z_{\theta}$ , our work can be summarized into two parts. First, we aim to minimize the expected discrepancy between  $Z_{\theta}$  and  $\mathcal{T}_{\xi}Z_{\theta^{-}}$  where  $\xi$  is sampled from ambiguity set  $\mathcal{U}_{\Delta}$ . To clarify notation, we write  $\mathbb{E}_{\xi}[\cdot ]$  as a  $\xi$ -weighted expectation and  $\mathbb{E}_{\xi \sim \mathcal{P}(\mathcal{U}_{\Delta})}[\cdot ]$  as an expectation with respect to  $\xi$  which is sampled from  $\mathcal{U}_{\Delta}$ . Then, our goal is to minimize the perturbed distributional Bellman objective with sampling procedure  $\mathcal{P}$ :
$$
\min  _ {\theta^ {\prime}} \mathbb {E} _ {\xi_ {t} \sim \mathcal {P} \left(\mathcal {U} _ {\Delta_ {t}}\right)} [ D (Z _ {\theta^ {\prime}} (s, a), \mathcal {T} _ {\xi_ {t}} Z _ {\theta^ {-}} (s, a)) ] \tag {1}
$$
where we use the Huber quantile loss as a discrepancy on  $Z_{\theta'}$  and  $\mathcal{T}_{\xi}Z_{\theta^-}$  at timestep  $t$ . In typical risk-sensitive RL or distributionally robust RL, the Bellman optimality equation is reformulated for a pre-defined risk measure [8, 33, 39]. In contrast, PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy while randomizing the risk criterion at every step. By using min-expectation instead of min-max operator, we suggest unbiased exploration that can avoid leading to overly pessimistic policies. Furthermore, considering a sequence  $\xi_t$  which converges to 1 in probability, we derive a sufficient condition of  $\Delta_t$  that the expectation of any composition of the operators  $\mathbb{E}\mathcal{T}_{\xi_{n:1}} \coloneqq \mathbb{E}\mathcal{T}_{\xi_n}\mathcal{T}_{\xi_{n-1}}\dots \mathcal{T}_{\xi_1}$  has the same unique fixed point as the standard. These results are remarkable that we can apply the diverse variations of distributional Bellman operators for learning.
3.2 Convergence of the perturbed distributional Bellman optimality operator
Unlike conventional convergence proofs, PDBOO is time-varying and not a contraction, so it covers a wider class of Bellman operators than before. Since the infinite composition of time-varying Bellman operators does not necessarily converge or have the same unique fixed point, we provide the sufficient condition in this section. We denote the iteration as  $Z^{(n + 1)} \coloneqq \mathcal{T}_{\xi^{n + 1}}Z^{(n)}$ ,  $Z^{(0)} = Z$  for each timestep  $n > 0$ , and the intersection of ambiguity set as  $\bar{\mathcal{U}}_{\Delta_n}(Z^{(n - 1)}) \coloneqq \bigcap_{s,a} \mathcal{U}_{\Delta_n}\left(Z^{(n - 1)}(s,a)\right)$ .
Assumption 3.2. Suppose that  $\sum_{n=1}^{\infty} \Delta_n < \infty$  and  $\xi_n$  is uniformly bounded.
Theorem 3.3. (Weaker Contraction Property) Let  $\xi_{n}$  be sampled from  $\bar{\mathcal{U}}_{\Delta_n}(Z^{(n - 1)})$  for every iteration. If Assumption 3.2 holds, then the expectation of any composition of operators  $\mathbb{E}\mathcal{T}_{\xi_{n:1}}$  converges, i.e.,  $\mathbb{E}\mathcal{T}_{\xi_{n:1}}[Z]\to \mathbb{E}[Z^* ]$ . Moreover, the following bound holds,
$$
\sup  _ {s, a} \left| \mathbb {E} \left[ Z ^ {(n)} (s, a) \right] - \mathbb {E} \left[ Z ^ {*} (s, a) \right] \right| \leq \sum_ {k = n} ^ {\infty} \left(2 \gamma^ {k - 1} V _ {\max } + 2 \sum_ {i = 1} ^ {k} \gamma^ {i} \left(\Delta_ {k + 2 - i} + \Delta_ {k + 1 - i}\right)\right).
$$
Practically, satisfying Assumption 3.2 is not strict to characterize the landscape of scheduling. Theorem 3.3 states that even without satisfying  $\gamma$ -contraction property, we can show that  $\mathbb{E}[Z^{*}]$  is the fixed point for the operator  $\mathbb{E}\mathcal{T}_{\xi_{n:1}}$ . However,  $\mathbb{E}[Z^{*}]$  is not yet guaranteed to be "unique" fixed point for any  $Z \in \mathcal{Z}$ . Nevertheless, we can show that  $\mathbb{E}[Z^{*}]$  is, in fact, the solution of the standard Bellman optimality equation, which is already known to have a unique solution.
Algorithm 1 Perturbed QR-DQN (PQR)
Input:  $(s, a, r, s')$ ,  $\gamma \in [0,1)$ , timestep  $t > 0$ ,  $\epsilon > 0$ , concentration  $\beta$
Initialize  $\Delta_0 > 0$
$$
\Delta_ {t} \gets \Delta_ {0} t ^ {- (1 + \epsilon)}.
$$
$$
\boldsymbol {\xi} \leftarrow \max  \left(\mathbf {1} ^ {N} + \Delta_ {t} (N \boldsymbol {x} - \mathbf {1} ^ {N}), 0\right) \text {w h e r e} \boldsymbol {x} \sim \operatorname {D i r} (\boldsymbol {\beta})
$$
$$
\boldsymbol {\xi} \leftarrow N \boldsymbol {\xi} / \sum \xi_ {i}
$$
$$
a ^ {*} \leftarrow \operatorname {a r g m a x} _ {a ^ {\prime}} \mathbb {E} _ {\xi} [ Z (s ^ {\prime}, a ^ {\prime}) ]
$$
$$
\mathcal {T} \theta_ {j} \leftarrow r + \gamma \theta_ {j} \left(s ^ {\prime}, a ^ {*}\right), \quad \forall j
$$
$$
t \leftarrow t + 1
$$
Output:  $\sum_{i=1}^{N} \mathbb{E}_j[\rho_{\hat{\tau}_i}^\kappa(\mathcal{T}\theta_j - \theta_i(s, a))]$
// Assumption 3.2
// Sample  $\xi \sim \bar{\mathcal{U}}_{\Delta_t}(Z^{(t)})$
// Refine as a weighting function
// Select greedy action with perturbed return
// Target update with unperturbed distribution
Theorem 3.4. If Assumption 3.2 holds,  $\mathbb{E}[Z^{*}]$  is the unique fixed point of Bellman optimality equation for any  $Z\in \mathcal{Z}$ .
As a result, PDBOO generally achieves the unique fixed point of the standard Bellman operator. Unlike previous distribution-based or risk-sensitive approaches, PDBOO has the theoretical compatibility to obtain a risk-neutral optimal policy even if the risk measure is randomly sampled during training procedure. For proof, see Appendix A.
3.3 Practical Algorithm with Distributional Perturbation