SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.1. For all  $k$ , the minimum value of  $m_{k}$  is positive.
Proof. Assume that there exist  $k \in \{1, \dots, K\}$  such that  $m_k$  is equal to zero at a policy parameter  $\psi^* \in \mathrm{OB}_k$ , i.e.,  $||\nabla F_k(\psi^*)|| = 0$ . Since  $F_k$  is convex,  $\psi^*$  is a minimum point of  $F_k$ ,  $\min_{\psi} F_k(\psi) = F_k(\psi^*) < d_k - \zeta$ . However,  $F_k(\psi^*) \geq d_k - \zeta$  as  $\psi^* \in \mathrm{OB}_k$ , so  $m_k$  is positive due to the contradiction. Hence, the minimum of  $m_k$  is also positive.
Lemma A.2. A solution of (15) always exists.
Proof. There exists a policy parameter  $\hat{\psi} \in \{\psi | F_k(\psi) + \zeta < d_k \text{ for } \forall k\}$  due to the assumptions. Let  $g = \psi - \psi_t$ . Then, the following inequality holds.
$$
\begin{array}{l} g _ {k} ^ {T} (\psi - \psi_ {t}) + c _ {k} \leq g _ {k} ^ {T} (\psi - \psi_ {t}) + F _ {k} (\psi_ {t}) + \zeta - d _ {k} \leq F _ {k} (\psi) + \zeta - d _ {k}. \quad (\because F _ {k} \text {i s c o n v e x .}) \\ \Rightarrow g _ {k} ^ {T} (\hat {\psi} - \psi_ {t}) + c _ {k} \leq F _ {k} (\hat {\psi}) + \zeta - d _ {k} <   0 \text {f o r} \forall k. \\ \end{array}
$$
Since  $\hat{\psi} -\psi_t$  satisfies all constraints of (15), the feasible set is non-empty and convex. Also,  $H$  is positive definite, so the QP has a unique solution.
Lemma A.2 shows the existence of solution of (15). Now, we show the convergence of the proposed gradient integration method in the case of varying step sizes.
Lemma A.3. If  $\sqrt{2\epsilon} M \leq \zeta$  and a policy is updated by  $\psi_{t+1} = \psi_t + \beta_t g_t$ , where  $0 < \beta_t < \frac{2\sqrt{2\epsilon} m R}{L||b_t||^2}$  and  $\beta_t \leq 1$ , the policy satisfies  $F_k(\psi) \leq d_k$  for  $\forall k$  within a finite time.
Proof. We can reformulate the step size as  $\beta = \frac{2\sqrt{2\epsilon}mR}{L||b_t||^2}\beta_t'$ , where  $\beta_t' \leq \frac{L||b_t||^2}{2\sqrt{2\epsilon}mR}$  and  $0 < \beta_t' < 1$ . Since the eigenvalues of  $H$  is equal to or bigger than  $R$  and  $H$  is symmetric and positive definite,
$\frac{1}{R} I - H^{-1}$  is positive semi-definite. Hence,  $x^{T}H^{-1}x \leq \frac{1}{R} ||x||^{2}$  is satisfied. Using this fact, the following inequality holds:
$$
\begin{array}{l} F _ {k} \left(\psi_ {t} + \beta_ {t} g _ {t}\right) - F _ {k} \left(\psi_ {t}\right) \leq \beta_ {t} \nabla F _ {k} \left(\psi_ {t}\right) ^ {T} g _ {t} + \frac {L}{2} \left\| \beta_ {t} g _ {t} \right\| ^ {2} \quad (\because \nabla F _ {k} \text {i s L - L i p s c h i t z c o n t i n u o u s .}) \\ = \beta_ {t} g _ {k} ^ {T} g _ {t} + \frac {L}{2} \beta_ {t} ^ {2} | | g _ {t} | | ^ {2} \\ = \beta_ {t} g _ {k} ^ {T} g _ {t} + \frac {L}{2} \beta_ {t} ^ {2} b _ {t} ^ {T} H ^ {- 1} b _ {t} \quad (\because g _ {t} = B ^ {- T} b _ {t}) \\ \leq - \beta_ {t} c _ {k} + \frac {L}{2 R} \beta_ {t} ^ {2} | | b _ {t} | | ^ {2}. \quad (\because g _ {k} ^ {T} g _ {t} + c _ {k} \leq 0) \\ \end{array}
$$
Now, we will show that  $\psi$  enters  $\mathrm{IB}_k$  in a finite time for  $\forall \psi \in \mathrm{OB}_k$  and that the  $k$ th constraint is satisfied for  $\forall \psi \in \mathrm{IB}_k$ . Thus, we divide into two cases, 1)  $\psi_t \in \mathrm{OB}_k$  and 2)  $\psi_t \in \mathrm{IB}_k$ . For the first case,  $c_k = \sqrt{2\epsilon} ||\bar{g}_k||$ , so the following inequality holds:
$$
\begin{array}{l} F _ {k} \left(\psi_ {t} + \beta_ {t} g _ {t}\right) - F _ {k} \left(\psi_ {t}\right) \leq \beta_ {t} \left(- \sqrt {2 \epsilon} | | \bar {g} _ {k} | | + \frac {L}{2 R} \beta_ {t} | | b _ {t} | | ^ {2}\right) \\ \leq \beta_ {t} \sqrt {2 \epsilon} \left(- \| \bar {g} _ {k} \| + m \beta_ {t} ^ {\prime}\right) \tag {16} \\ \leq \beta_ {t} \sqrt {2 \epsilon} m \left(\beta_ {t} ^ {\prime} - 1\right) <   0. \\ \end{array}
$$
The value of  $F_{k}$  decreases strictly with each update step according to (16). Hence,  $\psi_t$  can reach  $\mathrm{IB}_k$  by repeatedly updating the policy. We now check whether the constraint is satisfied for the second case. For the second case, the following inequality holds by applying  $c_k = F_k(\psi_t) - d_k + \zeta$ :
$$
\begin{array}{l} F _ {k} \left(\psi_ {t} + \beta_ {t} g _ {t}\right) - F _ {k} \left(\psi_ {t}\right) \leq \beta_ {t} d _ {k} - \beta_ {t} F _ {k} \left(\psi_ {t}\right) - \beta_ {t} \zeta + \frac {L}{2 R} \beta_ {t} ^ {2} | | b _ {t} | | ^ {2} \\ \Rightarrow F _ {k} \left(\psi_ {t} + \beta_ {t} g _ {t}\right) - d _ {k} \leq \left(1 - \beta_ {t}\right) \left(F _ {k} \left(\psi_ {t}\right) - d _ {k}\right) + \beta_ {t} \left(- \zeta + \sqrt {2 \epsilon} m \beta_ {t} ^ {\prime}\right). \\ \end{array}
$$
Since  $\psi_t \in \mathrm{IB}_k$ ,
$$
F _ {k} \left(\psi_ {t}\right) - d _ {k} \leq \sqrt {2 \epsilon} | | \bar {g} _ {k} | | - \zeta \leq \sqrt {2 \epsilon} M - \zeta \leq 0.
$$
Since  $m \leq M$  and  $\beta_t' < 1$ ,
$$
- \zeta + \sqrt {2 \epsilon} m \beta_ {t} ^ {\prime} <   - \zeta + \sqrt {2 \epsilon} M \leq 0.
$$
Hence,  $F_{k}(\psi_{t} + \beta_{t}g_{t})\leq d_{k}$ , which means that the  $k$ th constraint is satisfied if  $\psi_t\in \mathrm{IB}_k$ . As  $\psi_t$  reaches  $\mathrm{IB}_k$  for  $\forall k$  within a finite time according to (16), the policy can satisfy all constraints within a finite time.
Lemma A.3 shows the convergence to the feasibility condition in the case of varying step sizes. We introduce a lemma, which shows  $||b_t||$  is bounded by  $\sqrt{\epsilon}$ , and finally show the proof of Theorem 3.1, which can be considered a special case of varying step sizes.
Lemma A.4. There exists  $T\in \mathbb{R}_{>0}$  such that  $\| b_t\| \leq T\sqrt{\epsilon}$ .
Proof. Let us define the following sets:
$$
I := \left\{k \mid F _ {k} \left(\psi_ {t}\right) + \zeta - d _ {k} <   0 \right\}, O := \left\{k \mid F _ {k} \left(\psi_ {t}\right) + \zeta - d _ {k} > 0 \right\},
$$
$$
U := \left\{k \mid F _ {k} \left(\psi_ {t}\right) + \zeta - d _ {k} = 0 \right\}, C (\epsilon) := \left\{g \mid g _ {k} ^ {T} g + c _ {k} (\epsilon) \leq 0 \forall k \right\}, \tag {17}
$$
$$
I _ {G} := \left\{g \mid g _ {k} ^ {T} g + F _ {k} (\psi_ {t}) + \zeta - d _ {k} \leq 0 \forall k \in I \right\},
$$
where  $c_{k}(\epsilon) = \min (\sqrt{2\epsilon g_{k}^{T}H^{-1}g_{k}},F_{k}(\pi_{\psi}) - d_{k} + \zeta)$ . Using these sets, the following vectors can be defined:  $g(\epsilon)\coloneqq \mathrm{argmin}_{g\in C(\epsilon)}\frac{1}{2} g^{T}Hg$ $b(\epsilon)\coloneqq B^T g(\epsilon)$ . Now, we will show that  $||b(\epsilon)||$  is bounded above and  $||b(\epsilon)||\propto \sqrt{\epsilon}$  for sufficiently small  $\epsilon >0$ .
First, the following is satisfied for a sufficiently large  $\epsilon$ :
$$
C (\epsilon) = \left\{g \mid g _ {k} ^ {T} g + F _ {k} \left(\psi_ {t}\right) + \zeta - d _ {k} \leq 0 \forall k \right\}. \tag {18}
$$
Since  $\hat{\psi} -\psi_t\in C(\epsilon)$ , where  $\hat{\psi}$  is defined in Lemma A.2,  $||b(\epsilon)||\leq \frac{1}{2} (\hat{\psi} -\psi_t)^T H(\hat{\psi} -\psi_t)$  for  $\forall \epsilon$ . Therefore,  $||b(\epsilon)||$  is bounded above.
Second, let us define the following trust region size:
$$
\hat {\epsilon} := \frac {1}{2} \left(\min  _ {k \in O} \frac {F _ {k} \left(\psi_ {t}\right) + \zeta - d _ {k}}{\left| \left| \bar {g} _ {k} \right| \right|}\right) ^ {2} > 0. \tag {19}
$$
if  $\epsilon \leq \hat{\epsilon}$ , the following is satisfied:
$$
C (\epsilon) = I _ {G} \cap \left\{g \mid g _ {k} ^ {T} g + \sqrt {2 \epsilon} \left\| \bar {g} _ {k} \right\| \leq 0 \forall k \in O, g _ {k} ^ {T} g \leq 0 \forall k \in U \right\} \neq \phi . \tag {20}
$$
Thus,  $O_G(\epsilon) \coloneqq \{g|g_k^T g + \sqrt{2\epsilon} ||\bar{g}_k|| \leq 0 \forall k \in O, g_k^T g \leq 0 \forall k \in U\}$  is not empty. If we define  $\hat{g}(\epsilon) \coloneqq \operatorname*{argmin}_{g \in O_G(\epsilon)} \frac{1}{2} g^T H g$ , the following is satisfied:
$$
\hat {g} (\epsilon) = \sqrt {\frac {\epsilon}{\hat {\epsilon}}} \hat {g} (\hat {\epsilon}) \text {f o r} 0 \leq \epsilon \leq \hat {\epsilon}. \tag {21}
$$
Then, if  $\epsilon \leq \hat{\epsilon}\left(\min_{k\in I}(d_k - \zeta -F_k(\psi_t)) / g_k^T\hat{g} (\hat{\epsilon}))\right)^2$ ,  $\hat{g} (\epsilon) = g(\epsilon)$  since  $\hat{g} (\epsilon)\in I_G$ . Consequently, by defining a trust region size:
$$
\epsilon^ {*} := \hat {\epsilon} \cdot \min  \left(1, \left(\min  _ {k \in I} \frac {d _ {k} - \zeta - F _ {k} (\psi_ {t})}{g _ {k} ^ {T} \hat {g} (\hat {\epsilon})}\right) ^ {2}\right) > 0, \tag {22}
$$
$g(\epsilon) = \sqrt{\epsilon / \hat{\epsilon}}\hat{g} (\hat{\epsilon})$  for  $\epsilon \leq \epsilon^{*}$ . Therefore,  $||b(\epsilon)||\propto \sqrt{\epsilon}$  if  $\epsilon \leq \epsilon^{*}$ .
Finally, since  $||b(\epsilon)||$  is bounded above and proportional to  $\sqrt{\epsilon}$  for sufficiently small  $\epsilon$ , there exist a constant  $T$  such that  $||b(\epsilon)|| \leq T\sqrt{\epsilon}$ .
Theorem 3.1. Assume that the constraints are differentiable and convex, gradients of the constraints are  $L$ -Lipschitz continuous, eigenvalues of the Hessian are equal or greater than a positive value  $R \in \mathbb{R}_{>0}$ , and  $\{\psi | F_k(\pi_{\psi}; \alpha) + \zeta < d_k, \forall k\} \neq \emptyset$ . Then, there exists  $E \in \mathbb{R}_{>0}$  such that if  $0 < \epsilon \leq E$  and a policy is updated by the proposed gradient integration method, all constraints are satisfied within finite time steps.
Proof. The proposed step size is  $\beta_{t} = \min (1,\sqrt{2\epsilon} /||b_{t}||)$ , and the sufficient conditions that guarantee the convergence according to Lemma A.3 are followings:
$$
\sqrt {2 \epsilon} M \leq \zeta \text {a n d} 0 <   \beta_ {t} \leq 1 \text {a n d} \beta_ {t} <   \frac {2 \sqrt {2 \epsilon} m R}{L | | b _ {t} | | ^ {2}}.
$$
The second condition is self-evident. To satisfy the third condition, the proposed step size  $\beta_{t}$  should satisfy the followings:
$$
\frac {\sqrt {2 \epsilon}}{| | b _ {t} | |} <   \frac {2 \sqrt {2 \epsilon} m R}{L | | b _ {t} | | ^ {2}} \Leftrightarrow | | b _ {t} | | <   \frac {2 m R}{L}.
$$
If  $\epsilon < 4((mR) / (LT))^2$ , the following inequality holds:
$$
\sqrt {\epsilon} <   \frac {2 m R}{L T} \Rightarrow | | b _ {t} | | \leq T \sqrt {\epsilon} <   \frac {2 m R}{L}. \quad (\because \text {L e m m a A . 4 .})
$$
Hence, if  $\epsilon \leq E = \frac{1}{2}\min \left(\frac{\zeta^2}{2M^2},4\left(\frac{mR}{LT}\right)^2\right)$ , the sufficient conditions are satisfied.
A.2 Toy Example for Gradient Integration Method
The problem of the toy example in Figure 1 is defined as:
$$
\underset {x _ {1}, x _ {2}} {\text {m i n i m i z e}} \sqrt {(\sqrt {3} x _ {1} + x _ {2} + 2) ^ {2} + 4 (x _ {1} - \sqrt {3} x _ {2} + 4) ^ {2}} \quad \mathbf {s}. \mathbf {t}. x _ {1} \geq 0, x _ {1} - 2 x _ {2} \leq 0, \tag {23}
$$
where there are two linear constraints. The initial points for the naive and gradient integration methods are  $x_{1} = -2.5$  and  $x_{2} = -3.0$ , which do not satisfy the two constraints. We use the Hessian matrix for the trust region as identity matrix and the trust region size as 0.5 in both methods. The naive method minimizes the constraints in order from the first to the second constraint.
A.3 Analysis of Worst-Case Time to Satisfy All Constraints
To analyze the sample complexity, we consider a tabular MDP and use softmax policy parameterization as follows (for more details, see [Xu et al., 2021]):
$$
\pi_ {\psi} (a | s) := \frac {\exp \psi (s , a)}{\sum_ {a ^ {\prime}} \exp \psi \left(s , a ^ {\prime}\right)} \forall (s, a) \in S \times A. \tag {24}
$$
According to Agarwal et al. [2021], the natural policy gradient (NPG) update is as follows:
$$
\psi_ {t + 1} = \psi_ {t} + \beta A ^ {\pi_ {\psi_ {t}}}, \pi_ {\psi_ {t + 1}} (a | s) = \pi_ {\psi_ {t}} (a | s) \frac {\exp \left(\beta A ^ {\pi_ {\psi_ {t}}} (s , a)\right)}{Z _ {t} (s)}, \tag {25}
$$
where  $\beta$  is a step size,  $A^{\pi_{\psi_t}} \in \mathbb{R}^{|S||A|}$  is the vector expression of the advantage function, and  $Z_t(s) = \sum_a \pi_{\psi_t}(a|s) \exp(\beta A^{\pi_{\psi_t}}(s,a)) / (1 - \gamma)$ . Analyzing the sample complexity of trust region-based methods is challenging since their stepsize is not fixed, so we modify the gradient integration method to use the NPG as follows:
$$
g ^ {*} = \operatorname {a r g m i n} _ {g} \frac {1}{2} g ^ {T} H g \text {s . t .} g _ {k} ^ {T} g + c _ {k} \leq 0 \forall k \in \{k \mid F _ {k} \left(\pi_ {\psi_ {t}}; \alpha\right) > d _ {k} \}, \tag {26}
$$
$$
\psi_ {t + 1} = \psi_ {t} + \beta g ^ {*} / | | g ^ {*} | | _ {2}.
$$
In the remainder, we abbreviate  $\pi_{\psi_t}, A^{\pi_{\psi_t}}$ , and  $F_k(\pi_{\psi_t}; \alpha)$  as  $\pi_t, A^t$ , and  $F_k(\pi_t)$ , respectively. Since  $g^*$  always exists due to Lemma A.2, we can write the policy using Lagrange multipliers  $\lambda_k^t \geq 0$  as follows:
$$
\psi_ {t + 1} = \psi_ {t} - \beta \sum_ {k} \lambda_ {k} ^ {t} A _ {C _ {k}} ^ {t} / W _ {t}, W _ {t} := | | \sum_ {k} \lambda_ {k} ^ {t} A _ {C _ {k}} ^ {t} | | _ {2},
$$
$$
\pi_ {t + 1} (a | s) = \pi_ {t} (a | s) \exp \left(- \frac {\beta}{W _ {t}} \sum_ {k} \lambda_ {k} ^ {t} A _ {C _ {k}} ^ {t} (s, a)\right) / Z _ {t} (s), \tag {27}
$$
where  $Z_{t}(s)$  is a normalization factor, and  $\lambda_k^t = 0$  for  $F_{k}(\pi_{t})\leq d_{k}$ . The naive approach can also be written as above, except that  $\lambda^t$  is a one-hot vector, where  $i$ -th value  $\lambda_i^t$  is one only for corresponding to the randomly selected constraint. Then, we can get the followings:
$$
\begin{array}{l} \sum_ {k} \lambda_ {k} ^ {t} \left(F _ {k} \left(\pi_ {t + 1}\right) - F _ {k} \left(\pi_ {t}\right)\right) / W _ {t} = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d ^ {\pi_ {t + 1}}} \left[ \sum_ {a} \pi_ {t + 1} (a | s) \sum_ {k} \lambda_ {k} ^ {t} A _ {C _ {k}} ^ {t} (s, a) / W _ {t} \right] \\ = - \frac {1}{\beta (1 - \gamma)} \mathbb {E} _ {s \sim d ^ {\pi_ {t + 1}}} \left[ \sum_ {a} \pi_ {t + 1} (a | s) \log \frac {\pi_ {t + 1} (a | s) Z _ {t} (s)}{\pi_ {t} (a | s)} \right] \\ = - \frac {1}{\beta (1 - \gamma)} \mathbb {E} _ {s \sim d ^ {\pi_ {t + 1}}} \left[ D _ {\mathrm {K L}} \left(\pi_ {t + 1} (\cdot | s) \mid \mid \pi_ {t} (\cdot | s)\right) + \log Z _ {t} (s) \right] \tag {28} \\ \leq - \frac {1}{\beta (1 - \gamma)} \mathbb {E} _ {s \sim d ^ {\pi_ {t + 1}}} [ \log Z _ {t} (s) ] \quad (\because D _ {\mathrm {K L}} (\pi^ {\prime} | | \pi) \geq 0) \\ \leq - \frac {1}{\beta} \mathbb {E} _ {s \sim \rho} [ \log Z _ {t} (s) ], \quad (\because | | d ^ {\pi} / \rho | | _ {\infty} \geq 1 - \gamma) \\ \end{array}
$$
We can also get the followings by using the Lemma 7 in Xu et al. [2021]:
$$
\begin{array}{l} \sum_ {k} \lambda_ {k} ^ {t} (F _ {k} (\pi_ {*}) - F _ {k} (\pi_ {t})) / W _ {t} \geq - \frac {1}{\beta (1 - \gamma)} \mathbb {E} _ {s \sim d ^ {*}} [ D _ {\mathrm {K L}} (\pi_ {*} | | \pi_ {t}) - D _ {\mathrm {K L}} (\pi_ {*} | | \pi_ {t + 1}) ] \\ - \sum_ {k} \lambda_ {k} ^ {t} \frac {2 \beta C _ {\max}}{(1 - \gamma) ^ {2} W _ {t}}, \tag {29} \\ \end{array}
$$
where  $\pi_*$  is an optimal policy, and  $C_{\mathrm{max}}$  is the maximum value of costs. If  $\lambda_k^t > 0$ ,  $F_k(\pi_t) - F_k(\pi_*) > \zeta$ . Thus,  $\sum_k \lambda_k^t (F_k(\pi_*) - F_k(\pi_t)) \leq -\zeta \sum_k \lambda_k^t$ . If the policy does not satisfy the constraints until  $T$  step, the following inequality holds by summing the above inequalities from  $t = 0$  to  $T$ :
$$
\beta (1 - \gamma) (\zeta - \frac {2 \beta C _ {\operatorname* {m a x}}}{(1 - \gamma) ^ {2}}) \sum_ {t = 0} ^ {T} \sum_ {i} \lambda_ {i} ^ {t} / W _ {t} \leq \mathbb {E} _ {s \sim d ^ {*}} [ D _ {\mathrm {K L}} (\pi_ {*} | | \pi_ {0}) ]. \tag {30}
$$
Let denote  $\frac{1}{T}\sum_{t=0}^{T}\sum_{i}\lambda_{i}^{t}/W_{t}$  as  $\mathbb{E}_{t}[\sum_{i}\lambda_{i}^{t}/W_{t}]$ , and we can get  $W_{t} = ||\sum_{k}\lambda_{k}^{t}A_{C_{k}}^{t}||_{2} \leq \sum_{k}\lambda_{k}^{t}2|S||A|C_{\max}/(1-\gamma)$ . Then, the maximum  $T$  can be expressed as:
$$
T \leq \frac {D _ {\mathrm {K L}}}{\beta (1 - \gamma) (\zeta - \frac {2 \beta C _ {\max }}{(1 - \gamma) ^ {2}}) \mathbb {E} _ {t} \left[ \sum_ {i} \lambda_ {i} ^ {t} / W _ {t} \right]} \leq \frac {2 | S | | A | C _ {\max } D _ {\mathrm {K L}}}{\beta (1 - \gamma) ^ {2} \zeta - 2 \beta^ {2} C _ {\max }} =: T _ {\max }, \tag {31}
$$
where we abbreviate  $\mathbb{E}_{s\sim d^*}\left[D_{\mathrm{KL}}(\pi_*)|\pi_0)\right]$  as  $D_{\mathrm{KL}}$ . Finally, the policy can reach the feasible region within  $T_{\mathrm{max}}$  steps.
The worst-case time of the naive approach is the same as the above equation, except for the  $\lambda$  part. In the naive approach,  $\lambda^t$  is a one-hot vector, as mentioned earlier. In other words, only  $\mathbb{E}_t[\sum_i\lambda_i^t /W_t] = \mathbb{E}_t[\sum_i\lambda_i^t ] / ||\sum_i\lambda_i^t A_{C_k}^t ||_2]$  is different. Let us assume that the advantage vector follows a normal distribution. Then, the variance of  $\sum_{i}\lambda_{i}^{t}A_{C_{k}}^{t}$  is smaller for  $\lambda^t$  with distributed values than for one-hot values. Then, the reciprocal of the 2-norm becomes larger, resulting in a decrease in the worst-case time. From this perspective, the gradient integration method has a benefit over the naive approach as it reduces the variance of the advantage vector. Even though we cannot officially say that the worst-case time of the proposed method is smaller than the naive method because the advantage vector does not follow the normal distribution, we can deliver our insight on the benefit of gradient integration method.
In this section, we show that a sequence,  $Z_{k + 1} = \mathcal{T}_{\lambda}^{\mu ,\pi}Z_{k}$ , converges to the  $Z_R^\pi$ . First, we rewrite the operator  $\mathcal{T}_{\lambda}^{\mu ,\pi}$  for random variables to an operator for distributions and show that the operator is contractive. Finally, we show that  $Z_R^\pi$  is the unique fixed point.
Before starting the proof, we introduce useful notions and distance metrics. As the return  $Z_R^\pi(s, a)$  is a random variable, we define the distribution of  $Z_R^\pi(s, a)$  as  $\nu_R^\pi(s, a)$ . Let  $\eta$  be the distribution of a random variable  $X$ . Then, we can express the distribution of affine transformation of random variable,  $aX + b$ , using the pushforward operator, which is defined by Rowland et al. [2018], as  $(f_{a,b})_\#(\eta)$ . To measure a distance between two distributions, Bellemare et al. [2023] has defined the distance  $l_p$  as follows:
$$
l _ {p} \left(\eta_ {1}, \eta_ {2}\right) := \left(\int_ {\mathbb {R}} \left| F _ {\eta_ {1}} (x) - F _ {\eta_ {2}} (x) \right| ^ {p} d x\right) ^ {1 / p}, \tag {32}
$$
where  $F_{\eta}(x)$  is the cumulative distribution function. This distance is  $1 / p$ -homogeneous, regular, and  $p$ -convex (see Section 4 of Bellemare et al. [2023] for more details). For functions that map state-action pairs to distributions, a distance can be defined as [Bellemare et al., 2023]:  $\bar{l}_p(\nu_1,\nu_2)\coloneqq \sup_{(s,a)\in S\times A}l_p(\nu_1(s,a),\nu_2(s,a))$ . Then, we can rewrite the operator  $T_{\lambda}^{\mu ,\pi}$  for random variables in (13) as an operator for distributions as below.
$$
\begin{array}{l} \mathcal {T} _ {\lambda} ^ {\mu , \pi} \nu (s, a) := \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \\ \times \mathbb {E} _ {\mu} \left[ \left(\prod_ {j = 1} ^ {i} \eta \left(s _ {j}, a _ {j}\right)\right) \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s _ {i + 1})} \left[ \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu \left(s _ {i + 1}, a ^ {\prime}\right)\right) \right] \mid s _ {0} = s, a _ {0} = a \right], \tag {33} \\ \end{array}
$$
where  $\eta(s, a) = \frac{\pi(a|s)}{\mu(a|s)}$  and  $\mathcal{N}$  is a normalization factor. Since the random variable  $Z(s, a)$  and the distribution  $\nu(s, a)$  is equivalent, the operators in (13) and (33) are also equivalent. Hence, we are going to show the proof of Theorem 3.2 using (33) instead of (13). We first show that the operator  $\overline{T}_{\lambda}^{\mu, \pi}$  has a contraction property.
Lemma A.5. Under the distance  $\bar{l}_p$  and the assumption that the state, action, and reward spaces are finite,  $\mathcal{T}_{\lambda}^{\mu,\pi}$  is  $\gamma^{1/p}$ -contractive.
Proof. First, the operator can be rewritten using summation as follows.
$$
\begin{array}{l} \mathcal{T}_{\lambda}^{\mu ,\pi}\nu (s,a) = \frac{1 - \lambda}{\mathcal{N}}\sum_{i = 0}^{\infty}\lambda^{i}\sum_{a^{\prime}\in A}\sum_{(s_{0},a_{0},r_{0},\dots ,s_{i + 1})}\Pr_{\mu}(\underbrace{s_{0},a_{0},r_{0},\dots,s_{i + 1}}_{=: \tau})\left(\prod_{j = 1}^{i}\eta (s_{j},a_{j})\right) \\ \times \pi (a ^ {\prime} | s _ {i + 1}) \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t}}\right) _ {\#} \left(\nu \left(s _ {i + 1}, a ^ {\prime}\right)\right) \\ = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \sum_ {a ^ {\prime} \in A} \sum_ {\tau} \Pr_ {\mu} (\tau) \left(\prod_ {j = 1} ^ {i} \eta (s _ {j}, a _ {j})\right) \pi (a ^ {\prime} | s _ {i + 1}) \sum_ {s ^ {\prime} \in S} \mathbf {1} _ {s ^ {\prime} = s _ {i + 1}} \\ \times \sum_ {r _ {0: i} ^ {\prime}} \left(\prod_ {k = 0} ^ {i} \mathbf {1} _ {r _ {k} ^ {\prime} = r _ {k}}\right) \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t} ^ {\prime}}\right) \# (\nu \left(s ^ {\prime}, a ^ {\prime}\right)) \tag {34} \\ = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \lambda^ {i} \sum_ {a ^ {\prime} \in A} \sum_ {s ^ {\prime} \in S} \sum_ {r _ {0: i} ^ {\prime}} \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t} ^ {\prime}}\right) _ {\#} \left(\nu \left(s ^ {\prime}, a ^ {\prime}\right)\right) \\ \times \underbrace {\mathbb {E} _ {\mu} \left[ \left(\prod_ {j = 1} ^ {i} \eta \left(s _ {j} , a _ {j}\right)\right) \pi \left(a ^ {\prime} \mid s _ {i + 1}\right) \mathbf {1} _ {s ^ {\prime} = s _ {i + 1}} \left(\prod_ {k = 0} ^ {i} \mathbf {1} _ {r _ {k} ^ {\prime} = r _ {k}}\right) \right]} _ {=: w _ {s ^ {\prime}, a ^ {\prime}, r _ {0: i} ^ {\prime}}} \\ = \frac {1 - \lambda}{\mathcal {N}} \sum_ {i = 0} ^ {\infty} \sum_ {s ^ {\prime} \in S} \sum_ {a ^ {\prime} \in A} \sum_ {r _ {0: i} ^ {\prime}} \lambda^ {i} w _ {s ^ {\prime}, a ^ {\prime}, r _ {0: i} ^ {\prime}} \left(f _ {\gamma^ {i + 1}, \sum_ {t = 0} ^ {i} \gamma^ {t} r _ {t} ^ {\prime}}\right) _ {\#} (\nu (s ^ {\prime}, a ^ {\prime})). \\ \end{array}
$$
Since the sum of weights of distributions should be one, we can find the normalization factor  $\mathcal{N} = (1 - \lambda)\sum_{i = 0}^{\infty}\sum_{s\in S}\sum_{a\in A}\sum_{r_{0:i}}\lambda^{i}w_{s,a,r_{0:i}}$ . Then, the following inequality can be derived using the homogeneity, regularity, and convexity of  $l_p$ :