SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} l _ {p} ^ {p} \left(\mathcal {T} _ {\lambda} ^ {\mu , \pi} \nu_ {1} (s, a), \mathcal {T} _ {\lambda} ^ {\mu , \pi} \nu_ {2} (s, a)\right) \\ = l _ {p} ^ {p} \left(\frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \lambda^ {i} w _ {s, a, r _ {0: i}} \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu_ {1} (s, a)\right), \right. \\ \left. \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \lambda^ {i} w _ {s, a, r _ {0: i}} \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu_ {2} (s, a)\right)\right) \\ \leq \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \frac {(1 - \lambda) \lambda^ {i} w _ {s , a , r _ {0 : i}}}{\mathcal {N}} l _ {p} ^ {p} \left(\left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} (\nu_ {1} (s, a))\right), \\ \left. \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) \# \left(\nu_ {2} (s, a)\right)\right) \tag {35} \\ \leq \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \frac {(1 - \lambda) \lambda^ {i} w _ {s , a , r _ {0 : i}}}{\mathcal {N}} l _ {p} ^ {p} \left(\left(f _ {\gamma^ {i + 1}, 0}\right) _ {\#} \left(\nu_ {1} (s, a)\right), \left(f _ {\gamma^ {i + 1}, 0}\right) _ {\#} \left(\nu_ {2} (s, a)\right)\right) \\ = \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \frac {(1 - \lambda) \lambda^ {i} w _ {s , a , r _ {0 : i}}}{\mathcal {N}} \gamma^ {i + 1} l _ {p} ^ {p} (\nu_ {1} (s, a), \nu_ {2} (s, a)) \\ \leq \sum_ {i = 0} ^ {\infty} \sum_ {s \in S} \sum_ {a \in A} \sum_ {r _ {0: i}} \frac {(1 - \lambda) \lambda^ {i} w _ {s , a , r _ {0 : i}}}{\mathcal {N}} \gamma^ {i + 1} \left(\bar {l} _ {p} \left(\nu_ {1}, \nu_ {2}\right)\right) ^ {p} \\ \leq \gamma (\bar {l} _ {p} (\nu_ {1}, \nu_ {2})) ^ {p}. \\ \end{array}
$$
Therefore,  $\bar{l}_p\left(\mathcal{T}_{\lambda}^{\mu,\pi}\nu_1,\mathcal{T}_{\lambda}^{\mu,\pi}\nu_2\right) \leq \gamma^{1/p}\bar{l}_p\left(\nu_1,\nu_2\right)$ .
By the Banach's fixed point theorem, the operator  $\mathcal{T}_{\lambda}^{\mu,\pi}$  has a unique fixed distribution. We now show that the fixed distribution is  $\nu_{R}^{\pi}$ .
Lemma A.6. The fixed distribution of the operator  $\mathcal{T}_{\lambda}^{\mu,\pi}$  is  $\nu_{R}^{\pi}$ .
Proof. From the definition of  $Z_R^\pi$ , the following equality holds [Rowland et al., 2018]:  $\nu_R^\pi(s, a) = \mathbb{E}_\pi[(f_{\gamma, r})_\#(\nu_R^\pi(s', a'))]$ . Then, it can be shown that  $\nu_R^\pi$  is the fixed distribution by applying the operator  $T_\lambda^{\mu, \pi}$  to  $\nu_R^\pi$ :
$$
\begin{array}{l} \mathcal {T} _ {\lambda} ^ {\mu , \pi} \nu_ {R} ^ {\pi} (s, a) = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \\ \times \mathbb {E} _ {\mu} \left[ \left(\prod_ {j = 1} ^ {i} \eta \left(s _ {j}, a _ {j}\right)\right) \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s _ {i + 1})} \left[ \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu_ {R} ^ {\pi} \left(s _ {i + 1}, a ^ {\prime}\right)\right) \right] \mid s _ {0} = s, a _ {0} = a \right] \tag {36} \\ = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \mathbb {E} _ {\pi} \left[ \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu_ {R} ^ {\pi} \left(s _ {i + 1}, a _ {i + 1}\right)\right) \right| s _ {0} = s, a _ {0} = a \Bigg ] \\ = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \nu_ {R} ^ {\pi} (s, a) = \nu_ {R} ^ {\pi} (s, a). \\ \end{array}
$$
$$
\sum_ {i = 0} ^ {\infty} \mathbb {E} _ {\mu} \left[ \lambda^ {i} \prod_ {j = 1} ^ {i} \frac {\pi \left(a _ {j} \mid s _ {j}\right)}{\mu \left(a _ {j} \mid s _ {j}\right) {} _ {a ^ {\prime}} \sim \pi \left(\cdot \mid s _ {i + 1}\right)} \mathbb {E} \left[ \Pr \left(\sum_ {t = 0} ^ {i} \gamma^ {t} R _ {t} + \gamma^ {i + 1} Z \left(s _ {i + 1}, a ^ {\prime}\right) = z\right) \right] \mid s _ {0} = s, a _ {0} = a \right]. \tag {13}
$$
Then, a sequence,  $Z_{k + 1}(s,a) = \mathcal{T}_{\lambda}^{\mu ,\pi}Z_{k}(s,a)\forall (s,a)$  converges to  $Z_R^\pi$
Proof. The operator  $\mathcal{T}_{\lambda}^{\mu,\pi}$  is  $\gamma^{1/p}$ -contractive under the distance  $\bar{l}_p$  according to Lemma A.5. Also, the fixed distribution of the operator is  $\nu_R^\pi$ , which is equivalent to  $Z_R^\pi$ , according to Lemma A.6. By the Banach's fixed point theorem, the sequence,  $Z_{k+1}(s,a) = \mathcal{T}_{\lambda}^{\mu,\pi}Z_k(s,a) \forall (s,a)$ , converges to the fixed distribution of the operator,  $Z_R^\pi$ .
A.5 Pseudocode of  $\mathbf{TD}(\lambda)$  Target Distribution
We provide the pseudocode for calculating  $\mathrm{TD}(\lambda)$  target distribution for the reward critic in Algorithm 2. The target distribution for the cost critics can also be obtained by simply replacing the reward part with the cost.
Algorithm 2 TD(λ) Target Distribution
Input: Policy network  $\pi_{\psi}$ , critic network  $Z_{\theta}^{\pi}$ , and trajectory  $\{(s_t, a_t, \mu(a_t|s_t), r_t, d_t, s_{t+1})\}_{t=1}^T$ . Sample an action  $a_{T+1}' \sim \pi_{\psi}(s_{T+1})$  and get  $\hat{Z}_T^{\mathrm{tot}} = r_T + (1 - d_T)\gamma Z_{\theta}^{\pi}(s_{T+1}, a_{T+1}')$ . Initialize the total weight  $w_{\mathrm{tot}} = \lambda$ .
for  $t = T$  to 1 do
Sample an action  $a_{t + 1}^{\prime}\sim \pi_{\psi}(s_{t + 1})$  and get  $\hat{Z}_t^{(1)} = r_t + (1 - d_t)\gamma Z_\theta^\pi (s_{t + 1},a_{t + 1}^{\prime}).$
Set the current weight  $w = 1 - \lambda$ .
Combine the two targets,  $(\hat{Z}_t^{(1)},w)$  and  $(\hat{Z}_t^{\mathrm{(tot)}}$ $w_{\mathrm{tot}})$ , and sort the combined target according to the positions of atoms.
Build the CDF of the combined target by accumulating the weights at each atom.
Project the combined target into a quantile distribution with  $M'$  atoms, which is  $\hat{Z}_t^{(\mathrm{proj})}$ , using the CDF (find the atom positions corresponding to each quantile).
Update  $\hat{Z}_{t - 1}^{\mathrm{(tot)}} = r_{t - 1} + (1 - d_{t - 1})\gamma \hat{Z}_t^{\mathrm{(proj)}}$  and  $w_{\mathrm{tot}} = \lambda \frac{\pi_{\psi}(a_t|s_t)}{\mu(a_t|s_t)} (1 - d_{t - 1})(1 - \lambda +w_{\mathrm{tot}})$
Return  $\{\hat{Z}_t^{(\mathrm{proj})}\}_{t = 1}^T$
A.6 Quantitative Analysis on  $\mathbf{TD}(\lambda)$  Target Distribution
We experiment with a toy example to measure the bias and variance of the reward estimation according to  $\lambda$ . The toy example has two states,  $s_1$  and  $s_2$ ; the state distribution is defined as an uniform; the reward function is defined as  $r(s_1) \sim \mathcal{N}(-0.005, 0.02)$  and  $r(s_2) \sim \mathcal{N}(0.005, 0.03)$ . We train parameterized reward distributions by minimizing the quantile regression loss with the TD( $\lambda$ ) target distribution for  $\lambda = 0, 0.5, 0.9$ , and 1.0. The experimental results are presented in the table below.
Table 1: Experimental results of the toy example.
5th iteration	10th iteration	15th iteration	20th iteration	25th iteration
λ = 0.0	4.813 (0.173)	4.024 (0.253)	3.498 (0.085)	3.131 (0.103)	2.835 (0.070)
λ = 0.5	4.621 (0.185)	3.688 (0.273)	2.925 (0.183)	2.379 (0.134)	2.057 (0.070)
λ = 0.9	4.141 (0.461)	2.237 (0.402)	1.389 (0.132)	1.058 (0.031)	0.923 (0.019)
λ = 1.0	2.886 (0.767)	1.733 (0.365)	1.509 (0.514)	1.142 (0.325)	1.109 (0.476)
The values in the table are the mean and standard deviation of the past five values of the Wasserstein distance between the true reward return and the estimated distribution. Looking at the fifth iteration, it is clear that the larger the  $\lambda$  value, the smaller the mean and the higher the standard deviation. At the 25th iteration, the run with  $\lambda = 0.9$  has the lowest mean and standard deviation, indicating that training has converged. On the other hand, the run with  $\lambda = 1.0$  has the biggest standard deviation, and the mean is greater than  $\lambda = 0.9$ , indicating that the significant variance hinders training. In conclusion, we measured bias and variance quantitatively through the toy example, and the results are well aligned with our claim that  $\lambda$  can trade off bias and variance.
A.7 Surrogate Functions
In this section, we introduce the surrogate functions for the objective and constraints. First, Kim and Oh [2022a] define a doubly discounted state distribution:  $d_2^\pi(s) \coloneqq (1 - \gamma^2)\sum_{t=0}^\infty \gamma^{2t}\operatorname{Pr}(s_t = s|\pi)$ . Then, the surrogates for the objective and constraints are defined as follows [Kim and Oh, 2022a]:
$$
J ^ {\mu , \pi} (\pi^ {\prime}) := \underset {s _ {0} \sim \rho} {\mathbb {E}} \left[ V ^ {\pi} (s _ {0}) \right] + \frac {1}{1 - \gamma} \Big (\beta \underset {d ^ {\pi}} {\mathbb {E}} \left[ H (\pi^ {\prime} (\cdot | s)) \right] + \underset {d ^ {\mu}, \pi^ {\prime}} {\mathbb {E}} [ Q _ {R} ^ {\pi} (s, a) ] \Big),
$$
$$
J _ {C _ {k}} ^ {\mu , \pi} (\pi^ {\prime}) := \underset {s _ {0} \sim \rho} {\mathbb {E}} \left[ V _ {C _ {k}} ^ {\pi} (s _ {0}) \right] + \frac {1}{1 - \gamma} _ {d ^ {\mu}, \pi^ {\prime}} \underset {1} {\mathbb {E}} \left[ Q _ {C _ {k}} ^ {\pi} (s, a) \right], \tag {37}
$$
$$
J _ {S _ {k}} ^ {\mu , \pi} (\pi^ {\prime}) := \underset {s _ {0} \sim \rho} {\mathbb {E}} \left[ S _ {C _ {k}} ^ {\pi} (s _ {0}) \right] + \frac {1}{1 - \gamma^ {2}} \underset {d _ {\mu} ^ {2}, \pi^ {\prime}} {\mathbb {E}} \left[ S _ {C _ {k}} ^ {\pi} (s, a) \right],
$$
$$
F _ {k} ^ {\mu , \pi} (\pi^ {\prime}; \alpha) := J _ {C _ {k}} ^ {\mu , \pi} (\pi^ {\prime}) + \frac {\phi (\Phi^ {- 1} (\alpha))}{\alpha} \sqrt {J _ {S _ {k}} ^ {\mu , \pi} (\pi^ {\prime}) - (J _ {C _ {k}} ^ {\mu , \pi} (\pi^ {\prime})) ^ {2}},
$$
where  $\mu, \pi, \pi'$  are behavioral, current, and next policies, respectively. According to Theorem 1 in [Kim and Oh, 2022a], the constraint surrogates are bounded by  $D_{\mathrm{KL}}(\pi, \pi')$ . We also show that the surrogate of the objective is bounded by  $D_{\mathrm{KL}}(\pi, \pi')$  in Appendix A.9. As a result, the gradients of the objective function and constraints become the same as the gradients of the surrogates, and the surrogates can substitute the objective and constraints within the trust region.
A.8 Policy Update Rule
To solve the constrained optimization problem (5), we find a policy update direction by linearly approximating the objective and safety constraints and quadratically approximating the trust region constraint, as done by Achiam et al. [2017]. After finding the direction, we update the policy using a line search method. Given the current policy parameter  $\psi_t \in \Psi$ , the approximated problem can be expressed as follows:
$$
x ^ {*} = \underset {x \in \Psi} {\operatorname {a r g m a x}} g ^ {T} x \quad \text {s . t .} \frac {1}{2} x ^ {T} H x \leq \epsilon , b _ {k} ^ {T} x + c _ {k} \leq 0 \forall k, \tag {38}
$$
where  $g = \nabla_{\psi}J^{\mu ,\pi}(\pi_{\psi})|_{\psi = \psi_t}$ ,  $H = \nabla_{\psi}^{2}D_{\mathrm{KL}}(\pi_{\psi_{t}}||\pi_{\psi})|_{\psi = \psi_{t}}$ ,  $b_{k} = \nabla_{\psi}F_{k}^{\mu ,\pi}(\pi_{\psi};\alpha)|_{\psi = \psi_{t}}$ , and  $c_{k} = F_{k}(\pi_{\psi};\alpha) - d_{k}$ . Since (38) is convex, we can use an existing convex optimization solver.
However, the search space, which is the policy parameter space  $\Psi$ , is excessively large, so we reduce the space by converting (38) to a dual problem as follows:
$$
\begin{array}{l} g (\lambda , \nu) = \min _ {x} L (x, \lambda , \nu) = \min _ {x} \{- g ^ {T} x + \nu (\frac {1}{2} x ^ {T} H x - \epsilon) + \lambda^ {T} (B x + c) \} \\ = \frac {- 1}{2 \nu} \left(\underbrace {g ^ {T} H ^ {- 1} g} _ {=: q} - 2 \underbrace {g ^ {T} H ^ {- 1} B ^ {T}} _ {=: r ^ {T}} \lambda + \lambda^ {T} \underbrace {B H ^ {- 1} B ^ {T}} _ {=: S} \lambda\right) + \lambda^ {T} c - \nu \epsilon \tag {39} \\ = \frac {- 1}{2 \nu} (q - 2 r ^ {T} \lambda + \lambda^ {T} S \lambda) + \lambda^ {T} c - \nu \epsilon , \\ \end{array}
$$
where  $B = (b_{1},\dots,b_{K}),c = (c_{1},\dots,c_{K})^{T}$ , and  $\lambda \in \mathbb{R}^K\geq 0$  and  $\nu \in \mathbb{R}\geq 0$  are Lagrange multipliers. Then, the optimal  $\lambda$  and  $\nu$  can be obtained by a convex optimization solver. After obtaining the optimal values,  $(\lambda^{*},\nu^{*}) = \operatorname{argmax}_{(\lambda ,\nu)}g(\lambda ,\nu)$ , the policy update direction  $x^{*}$  are calculated by  $\frac{1}{\nu^{*}} H^{-1}(g - B^{T}\lambda^{*})$ . Then, the policy is updated by  $\psi_{t + 1} = \psi_t + \beta x^*$ , where  $\beta$  is a step size, which can be found through a backtracking method (please refer to Section 6.3.2 of Dennis and Schnabel [1996]).
Before using the above policy update rule, we should note that the existing trust-region method with the risk-averse constraint [Kim and Oh, 2022a] and the equations (1, 37, 5) are slightly different. There are two differences: 1) the objective is augmented with an entropy bonus, and 2) the surrogates are expressed with Q-functions instead of value functions. To use the entropy-regularized objective in the trust-region method, it is required to show that the objective is bounded by the KL divergence. We present the existence of bound in Appendix A.9. Next, there is no problem using the Q-functions because it is mathematically equivalent between the original surrogates [Kim and Oh, 2022a] and the new ones expressed with Q-functions (37). However, we experimentally show that using the Q-functions in off-policy settings has advantages in Appendix A.10.
A.9 Bound of Entropy-Augmented Objective
In this section, we show that the entropy-regularized objective in (1) has a bound expressed by the KL divergence. Before showing the boundness, we present a new function and a lemma. A value difference function is defined as follows:
$$
\delta^ {\pi^ {\prime}} (s) := \mathbb {E} \left[ R (s, a, s ^ {\prime}) + \gamma V ^ {\pi} (s ^ {\prime}) - V ^ {\pi} (s) \mid a \sim \pi^ {\prime} (\cdot | s), s ^ {\prime} \sim P (\cdot | s, a) \right] = \underset {a \sim \pi^ {\prime}} {\mathbb {E}} \left[ A ^ {\pi} (s, a) \right],
$$
where  $A^{\pi}(s,a)\coloneqq Q^{\pi}(s,a) - V^{\pi}(s,a)$
Lemma A.7. The maximum of  $|\delta^{\pi'}(s) - \delta^{\pi}(s)|$  is equal or less than  $\epsilon_R \sqrt{2D_{\mathrm{KL}}^{\max}(\pi||\pi')}$ , where  $\epsilon_R = \max_{s,a} |A^\pi(s,a)|$ .
Proof. The value difference can be expressed in a vector form,
$$
\delta^ {\pi^ {\prime}} (s) - \delta^ {\pi} (s) = \sum_ {a} (\pi^ {\prime} (a | s) - \pi (a | s)) A ^ {\pi} (s, a) = \langle \pi^ {\prime} (\cdot | s) - \pi (\cdot | s), A ^ {\pi} (s, \cdot) \rangle .
$$
Using Hölder's inequality, the following inequality holds:
$$
\begin{array}{l} \left| \delta^ {\pi^ {\prime}} (s) - \delta^ {\pi} (s) \right| \leq | | \pi^ {\prime} (\cdot | s) - \pi (\cdot | s) | | _ {1} \cdot | | A ^ {\pi} (s, \cdot) | | _ {\infty} \\ = 2 D _ {\mathrm {T V}} (\pi^ {\prime} (\cdot | s) | | \pi (\cdot | s)) \max _ {a} A ^ {\pi} (s, a). \\ \Rightarrow | | \delta^ {\pi^ {\prime}} - \delta^ {\pi} | | _ {\infty} = \max  _ {s} | \delta^ {\pi^ {\prime}} (s) - \delta^ {\pi} (s) | \leq 2 \epsilon_ {R} \max  _ {s} D _ {\mathrm {T V}} (\pi (\cdot | s) | | \pi^ {\prime} (\cdot | s)). \\ \end{array}
$$
Using Pinsker's inequality,  $||\delta^{\pi '} - \delta^{\pi}||_{\infty}\leq \epsilon_{R}\sqrt{2D_{\mathrm{KL}}^{\max}(\pi||\pi')}$
Theorem A.8. Let us assume that  $\max_s H(\pi (\cdot |s)) < \infty$  for  $\forall \pi \in \Pi$ . The difference between the objective and surrogate functions is bounded by a term consisting of KL divergence as:
$$
\left| J \left(\pi^ {\prime}\right) - J ^ {\mu , \pi} \left(\pi^ {\prime}\right) \right| \leq \frac {\sqrt {2} \gamma}{(1 - \gamma) ^ {2}} \sqrt {D _ {\mathrm {K L}} ^ {\max } \left(\pi \mid \mid \pi^ {\prime}\right)} \left(\beta \epsilon_ {H} + \epsilon_ {R} \sqrt {2 D _ {\mathrm {K L}} ^ {\max } \left(\mu \mid \mid \pi^ {\prime}\right)}\right), \tag {40}
$$
where  $\epsilon_{H} = \max_{s}|H(\pi^{\prime}(\cdot |s))|$ ,  $D_{\mathrm{KL}}^{\max}(\pi ||\pi ') = \max_{s}D_{\mathrm{KL}}(\pi (\cdot |s)||\pi^{\prime}(\cdot |s))$ , and the equality holds when  $\pi^{\prime} = \pi$
Proof. The surrogate function can be expressed in vector form as follows:
$$
J ^ {\mu , \pi} (\pi^ {\prime}) = \langle \rho , V ^ {\pi} \rangle + \frac {1}{1 - \gamma} \left(\langle d ^ {\mu}, \delta^ {\pi^ {\prime}} \rangle + \beta \langle d ^ {\pi}, H ^ {\pi^ {\prime}} \rangle\right),
$$
where  $H^{\pi'}(s) = H(\pi'(\cdot|s))$ . The objective function of  $\pi'$  can also be expressed in a vector form using Lemma 1 from Achiam et al. [2017],
$$
\begin{array}{l} J (\pi^ {\prime}) = \frac {1}{1 - \gamma} \mathbb {E} \left[ R (s, a, s ^ {\prime}) + \beta H ^ {\pi^ {\prime}} (s) | s \sim d ^ {\pi^ {\prime}}, a \sim \pi^ {\prime} (\cdot | s), s ^ {\prime} \sim P (\cdot | s, a) \right] \\ = \frac {1}{1 - \gamma} _ {s \sim d ^ {\pi^ {\prime}}} \mathbb {E} \left[ \delta^ {\pi^ {\prime}} (s) + \beta H ^ {\pi^ {\prime}} (s) \right] + \mathbb {E} _ {s \sim \rho} [ V ^ {\pi} (s) ] \\ = \langle \rho , V ^ {\pi} \rangle + \frac {1}{1 - \gamma} \langle d ^ {\pi^ {\prime}}, \delta^ {\pi^ {\prime}} + \beta H ^ {\pi^ {\prime}} \rangle . \\ \end{array}
$$
By Lemma 3 from Achiam et al. [2017],  $||d^{\pi} - d^{\pi'}||_1 \leq \frac{\gamma}{1 - \gamma}\sqrt{2D_{\mathrm{KL}}^{\max}(\pi||\pi'))}$ . Then, the following inequality is satisfied:
$$
\begin{array}{l} \left. \left| (1 - \gamma) \left(J ^ {\mu , \pi} \left(\pi^ {\prime}\right) - J \left(\pi^ {\prime}\right)\right) \right| \right. \\ = \left| \left\langle d ^ {\pi^ {\prime}} - d ^ {\mu}, \delta^ {\pi^ {\prime}} \right\rangle + \beta \left\langle d ^ {\pi} - d ^ {\pi^ {\prime}}, H ^ {\pi^ {\prime}} \right\rangle \right| \\ \leq | \langle d ^ {\pi^ {\prime}} - d ^ {\mu}, \delta^ {\pi^ {\prime}} \rangle | + \beta | \langle d ^ {\pi} - d ^ {\pi^ {\prime}}, H ^ {\pi^ {\prime}} \rangle | \\ = \left| \left\langle d ^ {\pi^ {\prime}} - d ^ {\mu}, \delta^ {\pi^ {\prime}} - \delta^ {\pi} \right\rangle \right| + \beta \left| \left\langle d ^ {\pi} - d ^ {\pi^ {\prime}}, H ^ {\pi^ {\prime}} \right\rangle \right| \quad (\because \delta^ {\pi} = 0) \\ \leq \left| \left| d ^ {\pi^ {\prime}} - d ^ {\mu} \right| \right| _ {1} \left| \left| \delta^ {\pi^ {\prime}} - \delta^ {\pi} \right| \right| _ {\infty} + \beta \left| \left| d ^ {\pi} - d ^ {\pi^ {\prime}} \right| \right| _ {1} \left| \left| H ^ {\pi^ {\prime}} \right| \right| _ {\infty} \quad (\because \text {H o l d e r s i n e q u a l i t y}) \\ \leq \frac {2 \epsilon_ {R} \gamma}{1 - \gamma} \sqrt {D _ {\mathrm {K L}} ^ {\max } (\mu | | \pi^ {\prime}) D _ {\mathrm {K L}} ^ {\max } (\pi | | \pi^ {\prime})} + \frac {\beta \gamma \epsilon_ {H}}{1 - \gamma} \sqrt {2 D _ {\mathrm {K L}} ^ {\max } (\pi | | \pi^ {\prime})} \quad (\because \text {L e m m a A . 7}) \\ = \frac {\gamma}{1 - \gamma} \sqrt {D _ {\mathrm {K L}} ^ {\operatorname* {m a x}} (\pi | | \pi^ {\prime})} \left(\sqrt {2} \beta \epsilon_ {H} + 2 \epsilon_ {R} \sqrt {D _ {\mathrm {K L}} ^ {\operatorname* {m a x}} (\mu | | \pi^ {\prime})}\right). \\ \end{array}
$$
If  $\pi' = \pi$ , the KL divergence term becomes zero, so equality holds.
A.10 Comparison of Q-Function and Value Function-Based Surrogates
The original surrogate is defined as follows:
$$
J ^ {\mu , \pi} (\pi^ {\prime}) := J (\pi) + \frac {1}{1 - \gamma} \mathbb {E} _ {d ^ {\mu}, \mu} \left[ \frac {\pi^ {\prime} (a | s)}{\mu (a | s)} A ^ {\pi} (s, a) \right], \tag {41}
$$
where  $A^{\pi}(s,a)\coloneqq Q^{\pi}(s,a) - V^{\pi}(s,a)$ , and the surrogate is the same as that of OffTRPO [Meng et al., 2022] and OffTRC [Kim and Oh, 2022a]. An entropy-regularized version can be derived as:
$$
J ^ {\mu , \pi} (\pi^ {\prime}) = J (\pi) + \frac {1}{1 - \gamma} \left(\beta \underset {d ^ {\pi}} {\mathbb {E}} [ H (\pi^ {\prime} (\cdot | s)) ] + \underset {d ^ {\mu}, \mu} {\mathbb {E}} \left[ \frac {\pi^ {\prime} (a | s)}{\mu (a | s)} A ^ {\pi} (s, a) \right]\right). \tag {42}
$$
Then, the surrogate expressed by Q-functions in (37), called SAC-style version, can be rewritten as:
$$
J ^ {\mu , \pi} \left(\pi^ {\prime}\right) = J (\pi) + \frac {1}{1 - \gamma} \left(\beta \underset {d ^ {\pi}} {\mathbb {E}} [ H \left(\pi^ {\prime} (\cdot | s)\right) ] + \underset {d ^ {\mu}, \pi^ {\prime}} {\mathbb {E}} [ Q ^ {\pi} (s, a) ]\right). \tag {43}
$$
In this section, we evaluate the original, entropy-regularized, and SAC-style versions in the continuous control tasks of the MuJoCo simulators [Todorov et al., 2012]. We use neural networks with two hidden layers with (512, 512) nodes and ReLU for the activation function. The output of a value network is linear, but the input is different; the original and entropy-regularized versions use states, and the SAC-style version uses state-action pairs. The input of a policy network is the state, the output is mean  $\mu$  and std  $\sigma$ , and actions are squashed into  $\tanh (\mu +\epsilon \sigma)$ ,  $\epsilon \sim \mathcal{N}(0,1)$  as in SAC [Haarnoja et al., 2018]. The entropy coefficient  $\beta$  in the entropy-regularized and SAC-style versions are adaptively adjusted to keep the entropy above a threshold (set as  $-d$  given  $A\subseteq \mathbb{R}^d$ ). The hyperparameters for all versions are summarized in Table 2.
The training curves are presented in Figure 5. All methods are trained with five different random seeds. Although the entropy-regularized version (42) and SAC-style version (43) are mathematically
Table 2: Hyperparameters for all versions.
Parameter	Value
Discount factor γ	0.99
Trust region size ε	0.001
Length of replay buffer	105
Critic learning rate	0.0003
Trace-decay λ	0.97
Initial entropy coefficient β	1.0
β learning rate	0.01
(a) Ant-v3
(b) HalfCheetah-v3
(c) Hopper-v3
(d) Humanoid-v3
(e) Swimmer-v3
(f) Walker2d-v3
Figure 5: MuJoCo training curves.
equivalent, it can be observed that the performance of the SAC-style version is superior to the regularized version. It can be inferred that this is due to the variance of importance sampling. In the off-policy setting, the sampling probabilities of the behavioral and current policies can be significantly different, so the variance of the importance ratio is huge. The increased variance prevents estimating the objective accurately, so significant performance degradation can happen. As a result, using the Q-function-based surrogates has an advantage for efficient learning.
(a) Point goal.
(b) Car button.
(c) Cassie.
Figure 6: (a) and (b) are Safety Gym tasks. (c), (d), and (e) are locomotion tasks.
(d) Laikago.
(e) Mini-Cheetah.
Safety Gym. We use the goal and button tasks with the point and car robots in the Safety Gym environment [Ray et al., 2019], as shown in Figure 6a and 6b. The environmental setting for the goal task is the same as in Kim and Oh [2022b]. Eight hazard regions and one goal are randomly spawned at the beginning of each episode, and a robot gets a reward and cost as follows:
$$
\begin{array}{l} R (s, a, s ^ {\prime}) = - \Delta d _ {\text {g o a l}} + \mathbf {1} _ {d _ {\text {g o a l}} \leq 0. 3}, \\ C (s, a, s ^ {\prime}) = C _ {i n t r i b u t i o n} (1 0 - (0. 2 - 1)) \end{array} \tag {44}
$$
$$
C (s, a, s ^ {\prime}) = \operatorname {S i g m o i d} \left(1 0 \cdot \left(0. 2 - d _ {\text {h a z a r d}}\right)\right),
$$
where  $d_{\mathrm{goal}}$  is the distance to the goal, and  $d_{\mathrm{hazard}}$  is the minimum distance to hazard regions. If  $d_{\mathrm{goal}}$  is less than or equal to 0.3, a goal is respawned. The state consists of relative goal position, goal distance, linear and angular velocities, acceleration, and LiDAR values. The action space is two-dimensional, which consists of  $xy$ -directional forces for the point and wheel velocities for the car robot.