SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dohyeong Kim<sup>1</sup>, Kyungjae Lee<sup>2</sup>, and Songhwai Oh<sup>1</sup>
<sup>1</sup>Dep. of Electrical and Computer Engineering and ASRI, Seoul National University
<sup>2</sup>Artificial Intelligence Graduate School, Chung-Ang University dohyeong.kim@rllab.snu.ac.kr, kyungjae.lee@ai.cau.ac.kr, songhwai@snu.ac.kr
In safety-critical robotic tasks, potential failures must be reduced, and multiple constraints must be met, such as avoiding collisions, limiting energy consumption, and maintaining balance. Thus, applying safe reinforcement learning (RL) in such robotic tasks requires to handle multiple constraints and use risk-averse constraints rather than risk-neutral constraints. To this end, we propose a trust region-based safe RL algorithm for multiple constraints called a safe distributional actor-critic (SDAC). Our main contributions are as follows: 1) introducing a gradient integration method to manage infeasibility issues in multi-constrained problems, ensuring theoretical convergence, and 2) developing a TD( $\lambda$ ) target distribution to estimate risk-averse constraints with low biases. We evaluate SDAC through extensive experiments involving multi- and single-constrained robotic tasks. While maintaining high scores, SDAC shows 1.93 times fewer steps to satisfy all constraints in multi-constrained tasks and 1.78 times fewer constraint violations in single-constrained tasks compared to safe RL baselines. Code is available at: https://github.com/rllab-snu/Safe-Distributional-Actor-Critic.
Deep reinforcement learning (RL) enables reliable control of complex robots [Merel et al., 2020, Peng et al., 2021, Rudin et al., 2022]. In order to successfully apply RL to real-world systems, it is essential to design a proper reward function which reflects safety guidelines, such as collision avoidance and limited energy consumption, as well as the goal of the given task. However, finding the reward function that considers all such factors involves a cumbersome and time-consuming process since RL algorithms must be repeatedly performed to verify the results of the designed reward function. Instead, safe RL, which handles safety guidelines as constraints, is an appropriate solution. A safe RL problem can be formulated using a constrained Markov decision process [Altman, 1999], where not only the reward but also cost functions are defined to provide the safety guideline signals. By defining constraints using expectation or risk measures of the sum of costs, safe RL aims to maximize returns while satisfying the constraints. Under the safe RL framework, the training process becomes straightforward since there is no need to search for a reward that reflects the safety guidelines.
While various safe RL algorithms have been proposed to deal with the safety guidelines, their applicability to general robotic applications remains limited due to the insufficiency in 1) handling multiple constraints and 2) minimizing failures, such as robot breakdowns after collisions. First, many safety-critical applications require multiple constraints, such as maintaining distance from obstacles, limiting operational space, and preventing falls. Lagrange-based safe RL methods [Yang et al., 2021, Zhang and Weng, 2022, Bai et al., 2022], which convert a safe RL problem into a dual problem and
update the policy and Lagrange multipliers, are commonly used to solve these multi-constrained problems. However, the Lagrangian methods are difficult to guarantee satisfying constraints during training theoretically, and the training process can be unstable due to the multipliers [Stooke et al., 2020]. To this end, trust region-based methods [Yang et al., 2020, Kim and Oh, 2022a], which can ensure to improve returns while satisfying the constraints under tabular settings [Achiam et al., 2017], have been proposed as an alternative to stabilize the training process. Still, trust region-based methods have a critical issue. Depending on the initial policy settings, there can be an infeasible starting case, meaning that no policy within the trust region satisfies constraints. To address this issue, we can sequentially select a violated constraint and update the policy to reduce the selected constraint [Xu et al., 2021]. However, this can be inefficient as only one constraint is considered per update. It will be better to handle multiple constraints at once, but it is a remaining problem to find a policy gradient that reflects several constraints and guarantees to reach a feasible policy set.
Secondly, as RL settings are inherently stochastic, employing risk-neutral measures like expectation to define constraints can lead to frequent failures. Hence, it is crucial to define constraints using risk measures, such as conditional value at risk (CVaR), as they can reduce the potential for massive cost returns by emphasizing tail distributions [Yang et al., 2021, Kim and Oh, 2022a]. In safe RL, critics are used to estimate the constraint values. Especially, to estimate constraints based on risk measures, it is required to use distributional critics [Dabney et al., 2018b], which can be trained using the distributional Bellman update [Bellemare et al., 2017]. However, the Bellman update only considers the one-step temporal difference, which can induce a large bias. The estimation bias makes it difficult for critics to judge the policy, which can lead to the policy becoming overly conservative or risky, as shown in Section 5.3. In particular, when there are multiple constraints, the likelihood of deriving incorrect policy gradients due to estimation errors grows exponentially. Therefore, there is a need for a method that can train distributional critics with low biases.
In this paper, we propose a trust region-based safe RL algorithm called a safe distributional actor-critic (SDAC), designed to effectively manage multiple constraints and estimate risk-averse constraints with low biases. First, to handle the infeasible starting case by considering all constraints simultaneously, we propose a gradient integration method that projects unsafe policies into a feasible policy set by solving a quadratic program (QP) consisting of gradients of all constraints. It guarantees to obtain a feasible policy within a finite time under mild technical assumptions, and we experimentally show that it can restore the policy more stably than the existing method [Xu et al., 2021]. Furthermore, by updating the policy using the trust region method with the integrated gradient, our approach makes the training process more stable than the Lagrangian method, as demonstrated in Section 5.2. Second, to train critics to estimate constraints with low biases, we propose a  $TD(\lambda)$  target distribution which can adjust the bias-variance trade-off. The target distribution is obtained by merging the quantile regression losses [Dabney et al., 2018b] of multi-step distributions and extracting a unified distribution from the loss. The unified distribution is then projected onto a quantile distribution set in a memory-efficient manner. We experimentally show that the target distribution can trade off the bias-variance of the constraint estimations (see Section 5.3).
We conduct extensive experiments with multi-constrained locomotion tasks and single-constrained Safety Gym tasks [Ray et al., 2019] to evaluate the proposed method. In the locomotion tasks, SDAC shows 1.93 times fewer steps to satisfy all constraints than the second-best baselines. In the Safety Gym tasks, the proposed method shows 1.78 times fewer constraint violations than the second-best methods while achieving high returns when using risk-averse constraints. As a result, it is shown that the proposed method can efficiently handle multiple constraints using the gradient integration method and effectively lower the constraint violations using the low-biased distributional critics.
Constrained Markov Decision Processes. We formulate the safe RL problem using constrained Markov decision processes (CMDPs) [Altman, 1999]. A CMDP is defined as  $(S, A, P, R, C_{1,\dots,K}, \rho, \gamma)$ , where  $S$  is a state space,  $A$  is an action space,  $P: S \times A \times S \mapsto [0,1]$  is a transition model,  $R: S \times A \times S \mapsto \mathbb{R}$  is a reward function,  $C_{k \in \{1,\dots,K\}}: S \times A \times S \mapsto \mathbb{R}_{\geq 0}$  are cost functions,  $\rho: S \mapsto [0,1]$  is an initial state distribution, and  $\gamma \in (0,1)$  is a discount factor. Given a policy  $\pi$  from a stochastic policy set  $\Pi$ , the discounted state distribution is defined as  $d^{\pi}(s) := (1 - \gamma) \sum_{t=0}^{\infty} \gamma^{t} \mathrm{Pr}(s_{t} = s|\pi)$ , and the return is defined as  $Z_{R}^{\pi}(s,a) := \sum_{t=0}^{\infty} \gamma^{t} R(s_{t},a_{t},s_{t+1})$ , where  $s_{0} = s$ ,  $a_{0} = a$ ,  $a_{t} \sim \pi(\cdot|s_{t})$ , and  $s_{t+1} \sim P(\cdot|s_{t},a_{t})$ . Then, the state value and state action
value functions are defined as:  $V_{R}^{\pi}(s) \coloneqq \mathbb{E}\left[Z_{R}^{\pi}(s,a)|a \sim \pi(\cdot|s)\right]$ ,  $Q_{R}^{\pi}(s,a) \coloneqq \mathbb{E}\left[Z_{R}^{\pi}(s,a)\right]$ . By substituting the costs for the reward, the cost value functions  $V_{C_k}^{\pi}(s)$  and  $Q_{C_k}^{\pi}(s,a)$  are defined. In the remainder of the paper, the cost parts will be omitted since they can be retrieved by replacing the reward with the costs. Then, the safe RL problem is defined as follows with a safety measure  $F$ :
$$
\operatorname {m a x i m i z e} _ {\pi} J (\pi) \mathbf {s}. \mathbf {t}. F \left(Z _ {C _ {k}} ^ {\pi} (s, a) | s \sim \rho , a \sim \pi (\cdot | s)\right) \leq d _ {k} \forall k, \tag {1}
$$
where  $J(\pi) \coloneqq \mathbb{E}\left[Z_R^\pi(s, a) | s \sim \rho, a \sim \pi(\cdot | s)\right] + \beta \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t H(\pi(\cdot | s_t)) | \rho, \pi, P\right]$ ,  $\beta$  is an entropy coefficient,  $H$  is the Shannon entropy, and  $d_k$  is a threshold of the  $k$ th constraint.
Trust-Region Method With a Mean-Std Constraint. Kim and Oh [2022a] have proposed a trust region-based safe RL method with a risk-averse constraint, called a mean-standard constraint. The definition of the mean-standard constraint function is as follows:
$$
F (Z; \alpha) := \mathbb {E} [ Z ] + (\phi (\Phi^ {- 1} (\alpha)) / \alpha) \cdot \operatorname {S t d} [ Z ], \tag {2}
$$
where  $\alpha \in (0,1]$  adjusts the risk level of constraints,  $\mathrm{Std}[Z]$  is the standard deviation of  $Z$ , and  $\phi$  and  $\Phi$  are the probability density function and the cumulative distribution function (CDF) of the standard normal distribution, respectively. In particular, setting  $\alpha = 1$  causes the standard deviation part to be zero, so the constraint becomes a risk-neutral constraint. Also, the mean-standard constraint can effectively reduce the potential for massive cost returns, as shown in Yang et al. [2021], Kim and Oh [2022a,b]. In order to calculate the mean-standard constraint, it is essential to estimate the standard deviation of the cost return. To this end, Kim and Oh [2022a] define the square value functions:
$$
S _ {C _ {k}} ^ {\pi} (s) := \mathbb {E} \left[ Z _ {C _ {k}} ^ {\pi} (s, a) ^ {2} | a \sim \pi (\cdot | s) \right], S _ {C _ {k}} ^ {\pi} (s, a) := \mathbb {E} \left[ Z _ {C _ {k}} ^ {\pi} (s, a) ^ {2} \right]. \tag {3}
$$
Since  $\mathrm{Std}[Z]^2 = \mathbb{E}[Z^2] - \mathbb{E}[Z]^2$ , the  $k$ th constraint can be written as follows:
$$
F _ {k} (\pi ; \alpha) = J _ {C _ {k}} (\pi) + \left(\phi \left(\Phi^ {- 1} (\alpha)\right) / \alpha\right) \cdot \sqrt {J _ {S _ {k}} (\pi) - J _ {C _ {k}} (\pi) ^ {2}} \leq d _ {k}, \tag {4}
$$
where  $J_{C_k}(\pi) \coloneqq \mathbb{E}\left[V_{C_k}^\pi (s)|s\sim \rho \right]$ ,  $J_{S_k}(\pi) \coloneqq \mathbb{E}\left[S_{C_k}^\pi (s)|s\sim \rho \right]$ . In order to apply the trust region method [Schulman et al., 2015], it is necessary to derive surrogate functions for the objective and constraints. These surrogates can substitute for the objective and constraints within the trust region. Given a behavioral policy  $\mu$  and the current policy  $\pi_{\mathrm{old}}$ , we denote the surrogates as  $J^{\mu ,\pi_{\mathrm{old}}}(\pi)$  and  $F_{k}^{\mu ,\pi_{\mathrm{old}}}(\pi ;\alpha)$ . For the definition and derivation of the surrogates, please refer to Appendix A.7 and [Kim and Oh, 2022a]. Using the surrogates, a policy can be updated by solving the following subproblem:
$$
\operatorname {m a x i m i z e} _ {\pi^ {\prime}} J ^ {\mu , \pi} \left(\pi^ {\prime}\right) \mathbf {s}. \mathbf {t}. D _ {K L} \left(\pi | | \pi^ {\prime}\right) \leq \epsilon , F _ {k} ^ {\mu , \pi} \left(\pi^ {\prime}, \alpha\right) \leq d _ {k} \forall k, \tag {5}
$$
where  $D_{\mathrm{KL}}(\pi ||\pi^{\prime})\coloneqq \mathbb{E}_{s\sim d^{\mu}}[D_{\mathrm{KL}}(\pi (\cdot |s)||\pi^{\prime}(\cdot |s))]$ ,  $D_{\mathrm{KL}}$  is the KL divergence, and  $\epsilon$  is a trust region size. This subproblem can be solved through approximation and a line search (see Appendix A.8). However, it is possible that there is no policy satisfying the constraints of (5). In order to tackle this issue, the policy must be projected onto a feasible policy set that complies with all constraints, yet there is a lack of such methods. In light of this issue, we introduce an efficient feasibility handling method for multi-constrained RL problems.
Distributional Quantile Critic. Dabney et al. [2018b] have proposed a method for approximating the random variable  $Z_R^\pi$  to follow a quantile distribution. Given a parametric model,  $\theta : S \times A \mapsto \mathbb{R}^M$ ,  $Z_R^\pi$  can be approximated as  $Z_{R,\theta}$ , called a distributional quantile critic. The probability density function of  $Z_{R,\theta}$  is defined as follows:
$$
\Pr \left(Z _ {R, \theta} (s, a) = z\right) := \sum_ {m = 1} ^ {M} \delta_ {\theta_ {m} (s, a)} (z) / M, \tag {6}
$$
where  $M$  is the number of atoms,  $\theta_{m}(s,a)$  is the  $m$ th atom,  $\delta$  is the Dirac function, and  $\delta_{a}(z) := \delta(z - a)$ . The percentile value of the  $m$ th atom is denoted by  $\tau_{m}$  ( $\tau_{0} = 0$ ,  $\tau_{i} = i / M$ ). In distributional RL, the returns are directly estimated to get value functions, and the target distribution can be calculated from the distributional Bellman operator [Bellemare et al., 2017]:  $\mathcal{T}^{\pi}Z_{R}(s,a):= R(s,a,s') + \gamma Z_{R}(s',a')$ , where  $s' \sim P(\cdot | s, a)$  and  $a' \sim \pi(\cdot | s')$ . The above one-step distributional operator can be expanded to the  $n$ -step one:  $\mathcal{T}_{n}^{\pi}Z_{R}(s_{0},a_{0}):= \sum_{t=0}^{n-1} \gamma^{t}R(s_{t},a_{t},s_{t+1}) + \gamma^{n}Z_{R}(s_{n},a_{n})$ , where  $a_{t} \sim \pi(\cdot | s_{t})$  for  $t = 1, \dots, n$ . Then, the critic can be trained to minimize the following quantile regression loss [Dabney et al., 2018b]:
$$
\mathcal {L} (\theta) = \sum_ {m = 1} ^ {M} \underbrace {\mathbb {E} _ {(s , a) \sim D} \left[ \mathbb {E} _ {Z \sim \mathcal {T} ^ {\pi} Z _ {R , \theta} (s , a)} \left[ \rho_ {\bar {\tau} _ {m}} \left(Z - \theta_ {m} (s , a)\right) \right] \right]} _ {=: \mathcal {L} _ {\mathrm {Q R}} ^ {\bar {\tau} _ {m}} \left(\theta_ {m}\right)}, \text {w h e r e} \rho_ {\tau} (x) = x \cdot \left(\tau - \mathbf {1} _ {\mathrm {x} <   0}\right), \tag {7}
$$
Figure 1: Left: Gradient integration. Each constraint is truncated to be tangent to the trust region indicated by the ellipse, and the dashed straight lines show the truncated constraints. The solution of (8) is indicated in blue, pointing to the nearest point in the intersection of the truncated constraints. If the solution crosses the trust region, parameters are updated by the clipped direction, shown in red. Right: Optimization paths of the proposed and naive method in a toy example. The description is presented in Appendix A.2. The contour graph represents the objective of the toy example. The optimization paths exhibit distinct characteristics due to the difference that the naive method reflects only one constraint and the proposed method considers all constraints at once.
$D$  is a replay buffer,  $\bar{\tau}_{m} := (\tau_{m-1} + \tau_{m}) / 2$ , and  $\mathcal{L}_{\mathrm{QR}}^{\bar{\tau}_{m}}(\theta_{m})$  denotes the quantile regression loss for a single atom. The distributional quantile critic can be plugged into existing actor-critic algorithms because only the critic modeling part is changed.
The proposed method comprises two key components: 1) a feasibility handling method required for multi-constrained safe RL problems and 2) a target distribution designed to minimize estimation bias. This section sequentially presents these components, followed by a detailed explanation of the proposed method.
3.1 Feasibility Handling For Multiple Constraints
An optimal safe RL policy can be found by iteratively solving the subproblem (5), but the feasible set of (5) can be empty in the infeasible starting cases. To address the feasibility issue in safe RL with multiple constraints, one of the violated constraints can be selected, and the policy is updated to minimize the constraint until the feasible region is not empty [Xu et al., 2021], which is called a naive approach. However, it may not be easy to quickly reach the feasible condition if only one constraint at each update step is used to update the policy. Therefore, we propose a feasibility handling method which reflects all the constraints simultaneously, called a gradient integration method. The main idea is to get a gradient that reduces the value of violated constraints and keeps unviolated constraints. To find such a gradient, the following quadratic program (QP) can be formulated by linearly approximating the constraints:
$$
g ^ {*} = \underset {g} {\operatorname {a r g m i n}} \frac {1}{2} g ^ {T} H g \quad \mathbf {s}. \mathbf {t}. g _ {k} ^ {T} g + c _ {k} \leq 0 \forall k, c _ {k} := \min  \left(\sqrt {2 \epsilon g _ {k} ^ {T} H ^ {- 1} g _ {k}}, F _ {k} \left(\pi_ {\psi}; \alpha\right) - d _ {k} + \zeta\right), \tag {8}
$$
where  $H$  is the Hessian of KL divergence between the previous policy and the current policy with parameters  $\psi$ ,  $g_{k}$  is the gradient of the  $k$ th constraint,  $c_{k}$  is a truncated threshold to make the  $k$ th constraint tangent to the trust region,  $\epsilon$  is a trust region size, and  $\zeta \in \mathbb{R}_{>0}$  is a slack coefficient. The reason why we truncate constraints is to make the gradient integration method invariant to the gradient scale. Otherwise, constraints with larger gradient scales might produce a dominant policy gradient. Finally, we update the policy parameters using the clipped gradient as follows:
$$
\psi^ {*} = \psi + \min  \left(1, \sqrt {2 \epsilon / \left(g ^ {* T} H g ^ {*}\right)}\right) g ^ {*}. \tag {9}
$$
Figure 1 illustrates the proposed gradient integration process. In summary, the policy is updated by solving (5); if there is no solution to (5), it is updated using the gradient integration method. Then, the policy can reach the feasibility condition within finite time steps.
Theorem 3.1. Assume that the constraints are differentiable and convex, gradients of the constraints are  $L$ -Lipschitz continuous, eigenvalues of the Hessian are equal or greater than a positive value  $R \in \mathbb{R}_{>0}$ , and  $\{\psi | F_k(\pi_{\psi}; \alpha) + \zeta < d_k, \forall k\} \neq \emptyset$ . Then, there exists  $E \in \mathbb{R}_{>0}$  such that if
$0 < \epsilon \leq E$  and a policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
Note that the first two assumptions of Theorem 3.1 are commonly used in multi-task learning [Liu et al., 2021, Yu et al., 2020, Navon et al., 2022], and the assumption on eigenvalues is used in most trust region-based RL methods [Schulman et al., 2015, Kim and Oh, 2022a], so the assumptions in Theorem 3.1 can be considered reasonable. We provide the proof and show the existence of a solution (8) in Appendix A.1. The provided proof shows that the constant  $E$  is proportional to  $\zeta$ . This means that the trust region size should be set smaller as  $\zeta$  decreases. Also, we further analyze the worst-case time to satisfy all constraints by comparing the gradient integration method and naive approach in Appendix A.3. In conclusion, if the policy update rule (5) is not feasible, a finite number of applications of the gradient integration method will make the policy feasible.
3.2 TD(λ) Target Distribution
The mean-standard constraints can be estimated using the distributional quantile critics. Since the estimated constraints obtained from the critics are directly used to update policies in (5), estimating the constraints with low biases is crucial. In order to reduce the estimation bias of the critics, we propose a target distribution by capturing that the  $\mathrm{TD}(\lambda)$  loss, which is obtained by a weighted sum of several losses, and the quantile regression loss with a single distribution are identical. A recursive method is then introduced so that the target distribution can be obtained practically. First, the  $n$ -step targets for the current policy  $\pi$  are estimated as follows, after collecting trajectories  $(s_t, a_t, s_{t+1}, \ldots)$  with a behavioral policy  $\mu$ :
$$
\hat {Z} _ {R} ^ {(n)} \left(s _ {t}, a _ {t}\right) := \underline {{D}} R _ {t} + \gamma R _ {t + 1} + \dots + \gamma^ {n - 1} R _ {t + n - 1} + \gamma^ {n} Z _ {R, \theta} \left(s _ {t + n}, \hat {a} _ {t + n}\right), \tag {10}
$$
where  $R_{t} = R(s_{t},a_{t},s_{t + 1})$ , and  $\hat{a}_{t + n}\sim \pi (\cdot |s_{t + n})$ . Note that the  $n$ -step target controls the bias-variance tradeoff using  $n$ . If  $n$  is equal to 1, the  $n$ -step target is equivalent to the temporal difference method that has low variance but high bias. On the contrary, if  $n$  increases to infinity, it becomes a Monte Carlo estimation that has high variance but low bias. However, finding proper  $n$  is another cumbersome task. To alleviate this issue, TD(λ) [Sutton, 1988] method considers the discounted sum of all  $n$ -step targets. Similar to TD(λ), we define the TD(λ) loss for the distributional quantile critic as the discounted sum of all quantile regression losses with  $n$ -step targets. Then, the TD(λ) loss for a single atom is approximated using importance sampling of the sampled  $n$ -step targets in (10) as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {Q R}} ^ {\bar {\tau} _ {m}} \left(\theta_ {m}\right) = (1 - \lambda) \sum_ {i = 0} ^ {\infty} \lambda^ {i} \mathbb {E} _ {\left(s _ {t}, a _ {t}\right) \sim D} \left[ \mathbb {E} _ {Z \sim \mathcal {T} _ {i + 1} ^ {\pi} Z _ {R, \theta} \left(s _ {t}, a _ {t}\right)} \left[ \rho_ {\bar {\tau} _ {m}} \left(Z - \theta_ {m} \left(s _ {t}, a _ {t}\right)\right) \right] \right] \\ = (1 - \lambda) \sum_ {i = 0} ^ {\infty} \lambda^ {i} \mathbb {E} _ {(s _ {t}, a _ {t}) \sim D} \left[ \mathbb {E} _ {Z \sim \mathcal {T} _ {i + 1} ^ {\mu} Z _ {R, \theta} (s _ {t}, a _ {t})} \left[ \prod_ {j = t + 1} ^ {t + i} \frac {\pi (a _ {j} | s _ {j})}{\mu (a _ {j} | s _ {j})} \rho_ {\bar {\tau} _ {m}} (Z - \theta_ {m} (s _ {t}, a _ {t})) \right] \right] \tag {11} \\ \approx (1 - \lambda) \sum_ {i = 0} ^ {\infty} \lambda^ {i} \mathbb {E} _ {(s _ {t}, a _ {t}) \sim D} \left[ \prod_ {j = t + 1} ^ {t + i} \frac {\pi (a _ {j} | s _ {j})}{\mu (a _ {j} | s _ {j})} \sum_ {m = 1} ^ {M} \frac {1}{M} \rho_ {\bar {\tau} _ {m}} (\hat {Z} _ {R, m} ^ {(i + 1)} (s _ {t}, a _ {t}) - \theta_ {m} (s _ {t}, a _ {t})) \right], \\ \end{array}
$$
where  $\lambda$  is a trace-decay value, and  $\hat{Z}_{R,m}^{(i)}$  is the  $m$ th atom of  $\hat{Z}_R^{(i)}$ . Since  $\hat{Z}_R^{(i)}(s_t,a_t)\stackrel {D}{=}R_t+$ $\gamma \hat{Z}_R^{(i - 1)}(s_{t + 1},a_{t + 1})$  is satisfied, (11) is the same as the quantile regression loss with the following single distribution  $\hat{Z}_t^{\mathrm{tot}}$ , called a  $TD(\lambda)$  target distribution: