SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underset {q ^ {\pi} \in \mathcal {Q}} {\text {m a x i m i z e}} \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} \left\langle q ^ {\pi}, r + \lambda g \right\rangle = \underset {\lambda \in \Lambda} {\text {m i n i m i z e}} \underset {q ^ {\pi} \in \mathcal {Q}} {\text {m a x i m i z e}} \left\langle q ^ {\pi}, r + \lambda g \right\rangle .
$$
We notice that the game value keeps the same as  $V^{\star}$  that is achieved at  $(q^{\pi^{\star}},\lambda^{\star})$ , where  $q^{\pi^{\star}}$  is the occupancy measure under the policy  $\pi^{\star}$ . Let the set of occupancy measures associated with  $\Pi^{\star}$  be  $Q^{\star}$ . According to Lemma 26, we know that there exists a constant  $c > 0$  such that
$$
\max  _ {q ^ {\pi} \in \mathcal {Q}} \langle q ^ {\pi}, r + \lambda g \rangle - V ^ {\star} \geq c | \lambda - \mathcal {P} _ {\Lambda^ {\star}} (\lambda) | \tag {39a}
$$
$$
V ^ {\star} - \min  _ {\lambda \in \Lambda} \left\langle q ^ {\pi}, r + \lambda g \right\rangle \geq c \| q ^ {\pi} - \mathcal {P} _ {\mathcal {Q} ^ {\star}} \left(q ^ {\pi}\right) \|. \tag {39b}
$$
It is more straightforward to see (38a) from (39a) if we take  $\lambda = \hat{\lambda}_{t + 1}$  and  $c_{1} = c$ . We next show (38b) using (39b) by taking  $\pi^{\star}(\cdot |s)$  to be the policy associated with  $\mathcal{P}_{\mathcal{Q}^{\star}}(q^{\pi})$
$$
\begin{array}{l} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \pi (\cdot | s) - \pi^ {\star} (\cdot | s) \right\| \\ = \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \frac {q ^ {\pi} (s , \cdot)}{q ^ {\pi} (s)} - \frac {q ^ {\pi^ {\star}} (s , \cdot)}{q ^ {\pi^ {\star}} (s)} \right\| \\ \leq \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \frac {\left\| q ^ {\pi} (s , \cdot) - q ^ {\pi^ {\star}} (s , \cdot) \right\| q ^ {\pi^ {\star}} (s)}{q ^ {\pi} (s) q ^ {\pi^ {\star}} (s)} + \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \frac {\left\| q ^ {\pi^ {\star}} (s , \cdot) \right\| | q ^ {\pi} (s) - q ^ {\pi^ {\star}} (s) |}{q ^ {\pi} (s) q ^ {\pi^ {\star}} (s)} \\ \leq \sum_ {s} \frac {\left\| q ^ {\pi} (s , \cdot) - q ^ {\pi^ {\star}} (s , \cdot) \right\|}{q ^ {\pi} (s)} + \sum_ {s} \frac {\left| q ^ {\pi} (s) - q ^ {\pi^ {\star}} (s) \right|}{q ^ {\pi} (s)} \\ \leq \frac {1}{\rho_ {\min }} \sum_ {0} \left(\left\| q ^ {\pi} (s, \cdot) - q ^ {\pi^ {\star}} (s, \cdot) \right\| + | q ^ {\pi} (s) - q ^ {\pi^ {\star}} (s) |\right) \\ \leq \frac {1}{\rho_ {\min }} \left(\sqrt {| S |} \sqrt {\sum_ {s} \| q ^ {\pi} (s , \cdot) - q ^ {\pi^ {\star}} (s , \cdot) \| ^ {2}} + \sqrt {| S | | A |} \sqrt {\sum_ {s , a} | q ^ {\pi} (s , a) - q ^ {\pi^ {\star}} (s , a) | ^ {2}}\right) \\ = \frac {2 \sqrt {| S | | A |}}{\rho_ {\min }} \left\| q ^ {\pi} - q ^ {\pi^ {\star}} \right\| \\ = \frac {2 \sqrt {| S | | A |}}{\rho_ {\min }} \| q ^ {\pi} - \mathcal {P} _ {\mathcal {Q} ^ {\star}} (q ^ {\pi}) \| \\ \end{array}
$$
where the first inequality is due to triangle inequality, we use the fact:  $(1 - \gamma)q^{\pi^{*}}(s) = d_{\rho}^{\pi^{*}}(s)$ ,  $d_{\rho}^{\pi^{*}}(s) \leq 1$ , and  $\left\| q^{\pi^{*}}(s,\cdot)\right\| \leq 1$  in the second inequality, the third inequality is due to that  $q^{\pi}(s) \geq \rho_{\min}$ , and we apply Cauchy-Schwarz inequality in the last inequality. Hence, we can further lower bound (39b),
$$
\begin{array}{l} V ^ {\star} - \min _ {\lambda \in \Lambda} \langle q ^ {\pi}, r + \lambda g \rangle \geq c \| q ^ {\pi} - \mathcal {P} _ {\mathcal {Q} ^ {\star}} (q ^ {\pi}) \| \\ \geq \frac {c \rho_ {\operatorname* {m i n}}}{2 \sqrt {| S | | A |}} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \pi (\cdot | s) - \pi^ {\star} (\cdot | s) \| \\ \geq \frac {c \rho_ {\mathrm {m i n}}}{2 \sqrt {| S | | A |}} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \pi (\cdot | s) - \mathcal {P} _ {\Pi^ {\star}} (\pi (\cdot | s)) \| \\ \end{array}
$$
which yields (38b) using  $c_2 = \frac{c\rho_{\mathrm{min}}}{2\sqrt{|S||A|}}$ .
Finally, we combine all selected constants and take  $D_{\mathrm{max}} = 1 + \frac{1}{(1 - \gamma)\xi}$  to conclude the proof.
Lemma 22. For any policies  $\pi$  and  $\pi'$ , we have
$$
\left\| Q _ {r} ^ {\pi} (\cdot , \cdot) - Q _ {r} ^ {\pi^ {\prime}} (\cdot , \cdot) \right\| _ {\infty} \leq \frac {\gamma}{(1 - \gamma) ^ {2}} \max  _ {s} \| \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \| _ {1}
$$
$$
\left| V _ {g} ^ {\pi} (\rho) - V _ {g} ^ {\pi^ {\prime}} (\rho) \right| \leq \frac {\kappa_ {\rho}}{(1 - \gamma) ^ {3}} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \| _ {1}.
$$
Proof. By the Bellman equations, for each pair  $(s,a)$
$$
Q _ {r} ^ {\pi} (s, a) = r (s, a) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s ^ {\prime} \mid s, a\right) \pi \left(a ^ {\prime} \mid s ^ {\prime}\right) Q _ {r} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right)
$$
$$
Q _ {r} ^ {\pi^ {\prime}} (s, a) = r (s, a) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P (s ^ {\prime} \mid s, a) \pi^ {\prime} (a ^ {\prime} \mid s ^ {\prime}) Q _ {r} ^ {\pi^ {\prime}} (s ^ {\prime}, a ^ {\prime}).
$$
Hence, for each pair  $(s,a)$
$$
\begin{array}{l} \left| Q _ {r} ^ {\pi} (s, a) - Q _ {r} ^ {\pi^ {\prime}} (s, a) \right| \leq \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s ^ {\prime} \mid s, a\right) \left| \pi \left(a ^ {\prime} \mid s ^ {\prime}\right) Q _ {r} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) - \pi^ {\prime} \left(a ^ {\prime} \mid s ^ {\prime}\right) Q _ {r} ^ {\pi^ {\prime}} \left(s ^ {\prime}, a ^ {\prime}\right) \right| \\ \leq \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P \left(s ^ {\prime} \mid s, a\right) \left| \pi \left(a ^ {\prime} \mid s ^ {\prime}\right) - \pi^ {\prime} \left(a ^ {\prime} \mid s ^ {\prime}\right) \right| Q _ {r} ^ {\pi^ {\prime}} \left(s ^ {\prime}, a ^ {\prime}\right) \\ + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P (s ^ {\prime} \mid s, a) \pi^ {\prime} (a ^ {\prime} \mid s ^ {\prime}) \left| Q _ {r} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) - Q _ {r} ^ {\pi^ {\prime}} (s ^ {\prime}, a ^ {\prime}) \right| \\ \leq \frac {\gamma}{1 - \gamma} \max _ {s} \| \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \| _ {1} + \gamma \left\| Q _ {r} ^ {\pi} (\cdot , \cdot) - Q _ {r} ^ {\pi^ {\prime}} (\cdot , \cdot) \right\| _ {\infty} \\ \end{array}
$$
which yields the first inequality.
To show the second inequality, we employ the performance difference lemma to show that,
$$
\begin{array}{l} \left| V _ {g} ^ {\pi} (\rho) - V _ {g} ^ {\pi^ {\prime}} (\rho) \right| \leq \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi} (s) \left| \pi (a \mid s) - \pi^ {\prime} (a \mid s) \right| \left| Q _ {g} ^ {\pi^ {\prime}} (s, a) \right| \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi} (s) \| \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \| _ {1} \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi} (s) \| \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \| _ {1} \\ \end{array}
$$
where we replace  $d_{\rho}^{\pi}$  by  $d_{\rho}^{\pi^{*}}$  using the inequality  $\left\| d_{\rho}^{\pi} / d_{\rho}^{\pi^{*}}\right\|_{\infty}\leq \kappa_{\rho} / (1 - \gamma)$  for any policy  $\pi \in \Pi$
D.2 Proof of Theorem 6
Proof. From the non-increasing relation (29) in Lemma 19, we have
$$
\frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} + \frac {1}{2} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \leq \zeta_ {t} \leq \frac {1 6}{7} \Theta_ {t} \leq \frac {1 6}{7} \Theta_ {1}.
$$
Meanwhile, from Lemma 20, we obtain that
$$
\begin{array}{l} \zeta_ {t} \geq \frac {1}{4} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \right\| ^ {2} + \frac {1}{4} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \\ + \frac {1}{4} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} + \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ + \frac {1}{4} \left(\left(\hat {\lambda} _ {t + 1} - \lambda_ {t}\right) ^ {2} + \left(\lambda_ {t} - \hat {\lambda} _ {t}\right) ^ {2}\right) \\ \geq \frac {1}{4} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \right\| ^ {2} + \frac {1}{4} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \\ + \frac {\eta^ {2}}{3 6 \kappa_ {\rho , \gamma} ^ {2}} \sup  _ {\pi \in \Pi , \lambda \in \Lambda} \frac {\left[ V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho) \right] _ {+} ^ {2}}{\left(\max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| + | \lambda - \hat {\lambda} _ {t + 1} |\right) ^ {2}} \\ \geq \frac {1}{4} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} + \frac {1}{4} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \\ + \frac {\eta^ {2} C _ {\rho , \xi} ^ {\prime}}{3 6 \kappa_ {\rho , \gamma} ^ {2}} \left(\sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t + 1} (\cdot | s)) \| ^ {2} + (\hat {\lambda} _ {t + 1} - \mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t + 1})) ^ {2}\right) \\ \geq (1 - \gamma) \min  \left(2, \frac {4 \eta^ {2} C _ {\rho , \xi} ^ {\prime}}{9 \kappa_ {\rho , \gamma} ^ {2}}\right) \Theta_ {t + 1} \\ \end{array}
$$
where the third inequality is due to Lemma 21, the inequalities:  $(x + y)^{2} \geq x^{2} + y^{2}$  for  $x, y \geq 0$ , and  $d_{\rho}^{\pi^{\star}}(s) \geq (1 - \gamma)\rho_{\min}$ , and  $C_{\rho, \xi}' := (1 - \gamma)C_{\rho, \xi}^{2}\rho_{\min}$ . Hence, the relation (29) reduces into,
$$
\Theta_ {t + 1} \leq \Theta_ {t} - (1 - \gamma) \min  \left(\frac {7}{8}, \frac {7 \eta^ {2} C _ {\rho , \xi} ^ {\prime}}{3 6 \kappa_ {\rho , \gamma} ^ {2}}\right) \Theta_ {t + 1}
$$
which implies that  $\Theta_t$  is decreasing to zero at an exponential rate.
D.3 Proof of Corollary 7
Proof. According to Theorem 6, if we take the same stepsize  $\eta$ , then for any  $t = \Omega(\log \frac{1}{\epsilon})$ ,
$$
\frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t} (\cdot | s)) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} = O (\epsilon) \mathrm {a n d} \frac {1}{2} (\mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t}) - \hat {\lambda} _ {t}) ^ {2} = O (\epsilon).
$$
Let  $\hat{\pi}_t^\star (\cdot |s)\coloneqq \mathcal{P}_{\Pi^\star}(\hat{\pi}_t(\cdot |s))$  and  $\hat{\lambda}_t^\star \coloneqq \mathcal{P}_{\Lambda^\star}(\hat{\lambda}_t)$ . Because of the interchangeability of saddle points from Lemma 9,  $(\hat{\pi}_t^\star ,\hat{\lambda}_t^\star)$  is a saddle point in the set  $\Pi^{\star}\times \Lambda^{\star}$  for any  $t\geq 0$ .
$$
\begin{array}{l} V _ {r} ^ {\hat {\pi} _ {t} ^ {*}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\hat {\pi} _ {t} ^ {*}} (s) \left(\hat {\pi} _ {t} ^ {*} (a | s) - \hat {\pi} _ {t} (a | s)\right) Q _ {r} ^ {\pi_ {t}} (s, a) \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\hat {\pi} _ {t} ^ {\star}} (s) \| \hat {\pi} _ {t} ^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| _ {1} \\ \leq \frac {\sqrt {| A |}}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\hat {\pi} _ {t} ^ {\star}} (s) \| \hat {\pi} _ {t} ^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| \\ \leq \frac {\sqrt {| A |}}{(1 - \gamma) ^ {2}} \sqrt {\sum_ {s} d _ {\rho} ^ {\hat {\pi} _ {t} ^ {\star}} (s) \left\| \hat {\pi} _ {t} ^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \right\| ^ {2}} \\ \end{array}
$$
where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to  $\| x\| _1\leq \sqrt{d}\| x\| _2$  for  $x\in \mathbb{R}^d$ , which shows  $V_{r}^{\hat{\pi}_{t}^{\star}}(\rho) - V_{r}^{\hat{\pi}_{t}}(\rho)\leq O(\sqrt{\epsilon})$ . By the saddle-point property of  $(\hat{\pi}_t^\star ,\hat{\lambda}_t^\star)$ ,  $\hat{\pi}_t^\star$  is also an optimal constrained policy, i.e.,  $V_{r}^{\hat{\pi}_{t}^{\star}}(\rho) = V_{r}^{\pi^{\star}}(\rho)$ . Therefore,  $V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\hat{\pi}_{t}}(\rho)\leq O(\sqrt{\epsilon})$ .
$$
- V _ {g} ^ {\hat {\pi} _ {t}} (\rho) = \underbrace {- V _ {g} ^ {\hat {\pi} _ {t} ^ {*}} (\rho)} _ {\text {(i)}} + \underbrace {V _ {g} ^ {\hat {\pi} _ {t} ^ {*}} (\rho) - V _ {g} ^ {\hat {\pi} _ {t}} (\rho)} _ {\text {(i i)}}.
$$
Similar to bounding  $V_{r}^{\hat{\pi}_{t}^{\star}}(\rho) - V_{r}^{\hat{\pi}_{t}}(\rho)$ , we can show that (ii)  $\leq O(\sqrt{\epsilon})$ . By the saddle-point property of  $(\hat{\pi}_t^\star, \hat{\lambda}_t^\star)$ ,  $V_{g}^{\hat{\pi}_{t}^{\star}}(\rho) \geq 0$ . Therefore,  $-V_{g}^{\hat{\pi}_{t}}(\rho) \leq O(\sqrt{\epsilon})$ .
D.4 Zero constraint violation of OPG-PD (9)
Corollary 23 (Zero constraint violation). Let Assumption 1 hold and the optimal state visitation distribution be unique, i.e.,  $d_{\rho}^{\pi} = d_{\rho}^{\pi^{\star}}$  for any  $\pi \in \Pi^{\star}$ , and  $\rho_{\min} > 0$ . For small  $\epsilon$ , there exists  $\delta > 0$  such that if we instead use the conservative constraint  $V_{g'}^{\pi}(\rho) \geq 0$  for  $g' = g - (1 - \gamma)\delta$ , and take the stepsize  $\eta$  from Theorem 6, then the policy iterates of OPG-PD satisfy
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) \leq \epsilon a n d - V _ {g} ^ {\hat {\pi} _ {t}} (\rho) \leq 0 f o r a n y t = \Omega \left(\log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\Omega (\cdot)$  only omits some problem-dependent constant.
It is straightforward to see that Assumption 1 ensures that  $V_{g'}^{\pi_t}(\rho) \geq 0$  is strictly feasible for a new slack variable  $\xi' \coloneqq \xi - \delta$ . We now can apply OPG-PD (9) to a new Lagrangian,
$$
L ^ {\prime} (\pi , \lambda) := V _ {r + \lambda g ^ {\prime}} ^ {\pi} (\rho)
$$
and Corollary 7 holds if we replace  $g$  in OPG-PD by  $g'$ . Thus,
$$
V _ {r} ^ {\pi_ {\delta} ^ {*}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) \leq O (\epsilon) \text {a n d} - V _ {g ^ {\prime}} ^ {\hat {\pi} _ {t}} (\rho) \leq O (\epsilon) \text {f o r a n y} t = \Omega \left(\log^ {2} \frac {1}{\epsilon}\right)
$$
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r} ^ {\pi} (\rho) \quad \text {s u b j e c t t o} V _ {g} ^ {\pi} (\rho) - \delta \geq 0. \tag {40}
$$
We notice that the above  $\Omega(\cdot)$  has some problem-dependent constant  $\Upsilon > 0$ . Thus, we select  $\delta$  such that  $\delta \geq \epsilon \Upsilon$ , which is always possible for small enough  $\epsilon$ , for instance  $\delta = \frac{\xi}{2}$  and  $\xi' = \frac{\xi}{2}$ . Hence, if we take  $\delta = \frac{\xi}{2}$  and such small  $\epsilon$ , then,
$$
- V _ {g ^ {\prime}} ^ {\hat {\pi} _ {t}} (\rho) = - V _ {g} ^ {\hat {\pi} _ {t}} (\rho) + \delta \leq \epsilon \Upsilon \text {f o r a n y} t = \Omega \left(\log^ {2} \frac {1}{\epsilon}\right)
$$
which shows that  $V_{g}^{\tilde{\pi}_{t}}(\rho)\geq 0$  for some large  $t$ .
The rest is to show that  $V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\hat{\pi}_{t}}(\rho) \leq O(\epsilon)$ . We notice that  $\pi^{\star}$  is an optimal policy to Problem (40) when  $\delta = 0$ . Let  $q^{\star}$  and  $q_{\delta}^{\star}$  be associated occupancy measures of policies  $\pi^{\star}$  and  $\pi_{\delta}^{\star}$ . In the occupancy measure space, Problem (40) becomes a linear program and it has a solution  $q_{\delta}^{\star}$ . Thus, we can view  $q_{\delta}^{\star}$  as a  $\delta$ -perturbed solution of a convex optimization problem in which all functions are continuous differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution  $q_{\delta}^{\star}$  is continuous in  $\delta$ , which implies that for any
$\epsilon > 0$ , there exists  $\delta'$  such that  $|\langle r, q^{\star} \rangle - \langle r, q_{\delta}^{\star} \rangle| \leq O(\epsilon)$  for any  $\delta < \delta'$ . Thus,  $|V_r^{\pi^*}(\rho) - V_r^{\pi_{\delta}^*}(\rho)| \leq O(\epsilon)$  for small enough  $\epsilon$ . Therefore,
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) \leq V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) + | V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) | \leq O (\epsilon)
$$
E Other Computational Experiments
In this section, we report details of our experimental setup and additional experimental results that verify merits and effectiveness of our methods: RPG-PD (6) and OPG-PD (9). We implement RPG-PD in the form of NPG [49] and the restricted probability simplex  $\hat{\Delta}(A)$  by restraining policy parameter to be bounded.
To properly assess the convergence performance, our experiment is a tabular constrained MDP with a randomly generated transition kernel, a discount factor  $\gamma = 0.9$ , uniform rewards  $r \in [0,1]$  and utilities  $g \in [-1,1]$ , and a uniform initial state distribution  $\rho$ . The constraint is  $V_{g}^{\pi}(\rho) \geq 0$ . To check feasibility, we employ the standard policy iteration procedure to solve a standard MDP problem with respect to  $V_{g}^{\pi}(\rho)$ . If feasible, we then solve a linear program in occupancy-measure space to find the optimal reward value  $V_{r}^{\pi^{\star}}(\rho)$  at an optimal policy  $\pi^{\star}$  induced by the optimal occupancy measure. Throughout all experiments, the random seed is fixed and  $V_{r}^{\pi^{\star}}(\rho)$  takes the value 8.16 at an optimal policy  $\pi^{\star}$ .
We compare our methods RPG-PD and OPG-PD with three typical learning algorithms in the constrained MDP literature: (i) primal-dual methods [23, 29]; (ii) dual methods [28, 24, 34]; and (iii) primal method [77]. We have reported our comparison for primal-dual methods in Section 5 and Figure 1. We now report our comparison experiments on other two baseline methods: (ii) dual methods [28, 24, 34] and (iii) primal method [77], in Section E.1. In addition, we showcase the last-iterate zero constraint violation performance of our methods in Section E.2, and conduct sensitivity analysis of our methods to regularization and stepsize in Section E.3 and Section E.4, together with a variant of OPG-PD and policy-based ReLOAD [40]. All the experiments were conducted on an Apple MacBook Pro 2017 laptop equipped with a  $2.3\mathrm{GHz}$  Dual-Core Intel Core i5 in Jupyter Notebook.
E.1 Last-iterate convergence comparison with other baselines
We report our comparison for dual methods in Figure 4. We notice that dual methods [28, 24, 34] work in a double-loop fashion, where a (regularized) NPG subroutine is executed to perform the dual update. To make a fair comparison, the number of dual updates and the number of NPG steps are set to be 53 and 20 to ensure that dual methods take the same number of policy gradient updates: 1060, as RPG-PD and OPG-PD, and we evaluate all policy iterates inside the NPG subroutines of dual methods.
In Figure 4, RPG-PD  $(- - )$  and OPG-PD  $(- - )$  outperform PMD-PD [24]  $(- - )$ , AR-CPO [28]  $(- - )$ , and Accelerated Dual [34]  $(\dots)$  in several aspects. First, we see that the known oscillation behavior in primal-dual methods also shows up in these dual methods, which could result from that we can't exactly evaluate the search direction of the dual update via a NPG subroutine. Thus, dual methods can be viewed as an instantiation of two-time-scale methods in which the primal update performs faster than the dual update. Regarding this, RPG-PD and OPG-PD show outstanding performance in suppressing oscillation behavior. Second, since RPG-PD and OPG-PD are single-time-scale primal-dual methods, it is easy to tune algorithmic parameters compared with double-loop dual methods. For instance, in Accelerated Dual, it is relatively difficult to find a set of algorithmic parameters that avoid the sub-optimal performance in Figure 4. Third, OPG-PD reaches the maximum reward value 8.16 and RPG-PD converges to a slightly smaller reward value because of regularization, while both methods enjoy the utility constraint satisfaction in the last-iterate fashion, which seems to be difficult for dual methods because of oscillating utility values. Hence, using the same number of policy gradient updates, we have verified that OPG-PD and RPG-PD also can outperform several dual methods by yielding an optimal constrained policy in the last-iterate fashion.
Figure 4: Convergence performance of RPG-PD, OPG-PD, and dual methods. Learning curves of our RPG-PD  $(- - )$  and OPG-PD  $(- - )$ , and PMD-PD [24]  $(- - )$ , AR-CPO [28]  $(- - )$ , and Accelerated Dual [34]  $(\dots)$  methods. The horizontal axes represent the policy iterations  $\{\pi_t\}_{t\geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t\geq 0}$ : reward value  $V_{r}^{\pi_{t}}(\rho)$  (Left) and utility value  $V_{g}^{\pi_{t}}(\rho)$  (Right). In this experiment, for RPG-PD and OPG-PD, we use the stepsize  $\eta = 0.1$  and the regularization parameter  $\tau = 0.08$  for RPG-PD, and the initial distribution  $\rho$  is uniform. For PMD-PD, AR-CPO, and Accelerated Dual, we use the stepsize  $\eta = 0.1$  for the dual update, the regularized NPG stepsize  $\alpha = 1$ , and the regularization parameter  $\tau = 0.08$ , and the uniform initial distribution  $\rho$ .
We report our comparison for a primal method in Figure 5. In Figure 5, RPG-PD  $(- - )$  and OPG-PD  $(-)$  outperform CRPO [77]  $(--)$ . We notice that although CRPO [77] works in a single-time-scale fashion as RPG-PD and OPG-PD, the policy has to be updated by alternatively using the gradient directions of reward/utility value functions. To ensure constraint satisfaction, CRPO often switches between the gradient directions of reward/utility value functions depending on the amount of constraint violation. As a result, we see that CRPO reaches a slightly lower reward value than OPG-PD's, and the constraint satisfaction is relatively conservative and has mild oscillation behavior. In contrast, OPG-PD achieves the maximum reward value 8.16 and RPG-PD converges to a slightly smaller reward value because of regularization, while both methods enjoy the utility constraint satisfaction in the last-iterate fashion. Last but not least, we have supported RPG-PD and OPG-PD with a policy last-iterate convergence theory, while such theory is unknown for CRPO as far as we know.
Figure 5: Convergence performance of RPG-PD, OPG-PD, and primal methods. Learning curves of our RPG-PD  $(\dashv)$  and OPG-PD  $(\dashv)$ , and CRPO [77]  $(\dashv)$  methods. The horizontal axes represent the policy iterations  $\{\pi_t\}_{t \geq 0}$  that are generated by each method and the vertical axes mean the value functions of the policy iterates  $\{\pi_t\}_{t \geq 0}$ : reward value  $V_r^{\pi_t}(\rho)$  (Left) and utility value  $V_g^{\pi_t}(\rho)$  (Right). In this experiment, for RPG-PD and OPG-PD, we use the stepsize  $\eta = 0.1$  and the regularization parameter  $\tau = 0.08$  for RPG-PD, and the initial distribution  $\rho$  is uniform. For CRPO, we update the policy via the NPG step with stepsize  $\eta = 0.1$  and the uniform initial distribution  $\rho$ .
E.2 Last-iterate zero constraint violation comparison