SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the first inequality is due to triangle inequality, we use the fact:  $(1 - \gamma)q^{\pi_t^\star}(s) = d_\rho^{\pi_t^\star}(s)$ ,  $d_{\rho}^{\pi_t^\star}(s) \leq 1$ , and  $\| q_t(s,\cdot)\| \leq \frac{\sqrt{|A|}}{1 - \gamma}$  in the second inequality, the third inequality is due to that  $q^{\pi_t^\star}(s)$ ,  $q_t(s) \geq \rho_{\mathrm{min}}$ , and  $\rho_{\mathrm{min}} > 0$ , we apply Cauchy-Schwarz inequality in the fourth inequality, and finally we combine two square root terms by relaxing the first one in the last inequality.
First, we have
$$
\begin{array}{l} V _ {r} ^ {\pi_ {t} ^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \left(\pi_ {t} ^ {\star} (a \mid s) - \pi_ {t} (a \mid s)\right) Q _ {r} ^ {\pi_ {t}} (s, a) \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \| \pi_ {t} ^ {\star} (\cdot | s) - \pi_ {t} (\cdot | s) \| _ {1} \\ \leq \frac {\sqrt {| A |}}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \| \pi_ {t} ^ {\star} (\cdot | s) - \pi_ {t} (\cdot | s) \| \\ \end{array}
$$
where the equality is due to performance difference lemma in Lemma 28, we use Cauchy-Schwarz inequality in the first inequality, and the second inequality is due to  $\| x\| _1\leq \sqrt{d}\| x\| _2$  for  $x\in \mathbb{R}^d$  which shows  $V_{r}^{\pi^{*}_{t}}(\rho) - V_{r}^{\pi_{t}}(\rho)\leq O(\sqrt{\epsilon})$ . By the optimality of  $\pi_t^\star ,V_r^{\pi_t^\star}(\rho) = V_r^{\pi^\star}(\rho)$ . Therefore,  $V_{r}^{\pi^{*}}(\rho) - V_{r}^{\pi_{t}}(\rho)\leq O(\sqrt{\epsilon})$ .
$$
- V _ {g} ^ {\pi_ {t}} (\rho) = \underbrace {- V _ {g} ^ {\hat {\pi} _ {t} ^ {*}} (\rho)} _ {\mathrm {(i)}} + \underbrace {V _ {g} ^ {\hat {\pi} _ {t} ^ {*}} (\rho) - V _ {g} ^ {\pi_ {t}} (\rho)} _ {\mathrm {(i i)}}.
$$
Similar to bounding  $V_{r}^{\pi_{t}^{\star}}(\rho) - V_{r}^{\pi_{t}}(\rho)$ , we can show that (ii)  $\leq O(\sqrt{\epsilon})$ . By the optimality of  $\pi_t^\star$ , we have  $V_{g}^{\pi_{t}^{\star}}(\rho)\geq 0$ . Therefore,  $-V_{g}^{\pi_{t}}(\rho)\leq O(\sqrt{\epsilon})$ .
Finally, we replace the accuracy  $\sqrt{\epsilon}$  by  $\epsilon$  and combine all big O notation to conclude the proof.
Corollary 13 states that after an almost constant number of iterations a policy induced by the last occupancy measure iterate of OPD is an  $\epsilon$ -optimal constrained policy for Problem (1). We notice that recent last-iterate convergence result for convex-concave saddle-point problems [39] is also applicable to Problem (17), which provides the optimal rate without problem-dependent constants. It is worth mentioning that direct application of such last-iterate convergence results in convex minimax optimization to constrained MDPs with general utilities [125, 126] and convex MDPs [124, 47] in occupancy-measure space is also straightforward. We omit these exercises in this paper, and focus on the design and analysis of algorithms in policy space.
C Proofs in Section 3
In this section, we provide proofs of the claims in Section 3.
C.1 Existence and uniqueness of regularized saddle points
Lemma 14 (Existence and uniqueness). There exists a unique primal-dual pair  $(\bar{\pi},\bar{\lambda})\in \Pi \times \Lambda$  such that  $L_{\tau}(\bar{\pi},\lambda)\geq L_{\tau}(\bar{\pi},\bar{\lambda})\geq L_{\tau}(\pi ,\bar{\lambda})$  for any  $\pi \in \Pi$  and  $\lambda \in \Lambda$
Proof. We first re-write the regularized Lagrangian  $L_{\tau}(\pi, \lambda)$  in terms of occupancy measure  $q \in \mathcal{Q}$ ,
$$
L _ {\tau} (\pi , \lambda) = \langle r + \lambda g, q \rangle + \tau \left(\mathcal {H} (\pi) + \frac {1}{2} \lambda^ {2}\right)
$$
$$
\mathcal {H} (\pi) = - \sum_ {s, a} q (s, a) \log \frac {q (s , a)}{\sum_ {a ^ {\prime}} q (s , a ^ {\prime})} := \mathcal {H} (q).
$$
We next use notation  $L_{\pi}(q,\lambda)$  to represent  $L_{\tau}(\pi ,\lambda)$ . We first check that  $\mathcal{H}(q)$  is a concave function,
$$
\begin{array}{l} \mathcal {H} (\alpha q _ {1} + (1 - \alpha) q _ {2}) \\ = - \sum_ {s, a} (\alpha q _ {1} (s, a) + (1 - \alpha) q _ {2} (s, a)) \log \frac {\alpha q _ {1} (s , a) + (1 - \alpha) q _ {2} (s , a)}{\alpha \sum_ {a ^ {\prime}} q _ {1} (s , a ^ {\prime}) + (1 - \alpha) \sum_ {a ^ {\prime}} q _ {2} (s , a ^ {\prime})} \\ \geq - \sum_ {s, a} \alpha q _ {1} (s, a) \log \frac {\alpha q _ {1} (s , a)}{\alpha \sum_ {a ^ {\prime}} q _ {1} (s , a ^ {\prime})} - \sum_ {s, a} (1 - \alpha) q _ {2} (s, a) \log \frac {(1 - \alpha) q _ {2} (s , a)}{(1 - \alpha) \sum_ {a ^ {\prime}} q _ {2} (s , a ^ {\prime})} \\ = \alpha \mathcal {H} \left(q _ {1}\right) + (1 - \alpha) \mathcal {H} \left(q _ {2}\right) \\ \end{array}
$$
for any  $q_1, q_2 \in \mathcal{Q}$  and  $\alpha \in [0,1]$ , where the inequality is because of the log sum inequality  $\left(\sum_{i} a_i\right) \ln \frac{\sum_{i} a_i}{\sum_{i} b_i} \leq \sum_{i} a_i \ln \frac{a_i}{b_i}$  for non-negative  $a_i$  and  $b_i$ , and the equality holds if and only if
$$
\frac {q _ {1} (s , a)}{\sum_ {a ^ {\prime}} q _ {1} (s , a ^ {\prime})} = \frac {q _ {2} (s , a)}{\sum_ {a ^ {\prime}} q _ {2} (s , a ^ {\prime})} \text {f o r a l l} s, a.
$$
Therefore,  $L_{\tau}(q,\lambda)$  is concave in  $q \in \mathcal{Q}$  and strongly convex in  $\lambda \in \Lambda$ . We notice that  $\mathcal{Q}$  is a polytope and  $\Lambda$  is a bounded interval. By Sion's minimax theorem [127],  $L_{\tau}(q,\lambda)$  has a saddle point  $(\bar{q},\bar{\lambda}) \in \mathcal{Q} \times \Lambda$ . From the one-to-one correspondence between policy and occupancy measure,  $\bar{q}$  induces a policy  $\bar{\pi}$  and  $(\bar{\pi},\bar{\lambda})$  serves as a saddle point of  $L_{\tau}(\pi ,\lambda)$ , which proves the existence of saddle points.
To show the uniqueness of  $(\bar{\pi},\bar{\lambda})$  (or  $(\bar{q},\bar{\lambda}))$ , it is sufficient to show that  $L_{\tau}(q,\lambda)$  is strictly concave in  $q$  and strictly convex in  $\lambda$ . The second argument is straightforward from the strong convexity of  $L_{\tau}(q,\lambda)$  in  $\lambda$ . We next show the first argument that  $L_{\tau}(q,\lambda)$  is strictly concave in  $q$  for any  $\lambda \in \Lambda$ . Assume that there are two (different) policies  $\pi_1$  and  $\pi_2$  and the associated two occupancy measures are  $q_1$  and  $q_2$ . From the concavity of  $H(q)$ , there is another occupancy measure  $\alpha q_1 + (1 - \alpha)q_2$
$$
\begin{array}{l} L _ {\tau} \left(\alpha q _ {1} + (1 - \alpha) q _ {2}, \lambda\right) \\ = \langle r + \lambda g, \alpha q _ {1} + (1 - \alpha) q _ {2} \rangle + \tau \left(\mathcal {H} (\alpha q _ {1} + (1 - \alpha) q _ {2}) + \frac {1}{2} \lambda^ {2}\right) \\ \geq \alpha \left(\langle r + \lambda g, q _ {1} \rangle + \tau \mathcal {H} (q _ {1}) + \frac {\tau}{2} \lambda^ {2}\right) + (1 - \alpha) \left(\langle r + \lambda g, q _ {2} \rangle + \tau \mathcal {H} (q _ {2}) + \frac {\tau}{2} \lambda^ {2}\right) \\ = \alpha L _ {\tau} \left(q _ {1}, \lambda\right) + (1 - \alpha) L _ {\tau} \left(q _ {2}, \lambda\right) \\ \end{array}
$$
where the inequality  $\geq$  has to be strict since  $\pi_1$  and  $\pi_2$  differentiate by at least one state-action pair  $(\bar{s},\bar{a})$ ,
$$
\pi_ {1} (\bar {a} \mid \bar {s}) = \frac {q _ {1} (\bar {s} , \bar {a})}{\sum_ {a ^ {\prime}} q _ {1} (\bar {s} , a ^ {\prime})} \neq \frac {q _ {2} (\bar {s} , \bar {a})}{\sum_ {a ^ {\prime}} q _ {2} (\bar {s} , a ^ {\prime})} = \pi_ {2} (\bar {a} \mid \bar {s}).
$$
Hence, a new policy associated with  $\alpha q_{1} + (1 - \alpha)q_{2}$  can achieve a strictly higher value of the convex combination of two objective functions, unless  $q_{1} = q_{2}$  or  $\alpha$  equals either 0 or 1. Therefore,  $L_{\tau}(q,\lambda)$  is strictly concave in  $q\in \mathcal{Q}$  for any  $\lambda \in \Lambda$ .
Let a saddle point of  $L_{\tau}(\pi, \lambda)$  be  $\pi_{\tau}^{\star} = \operatorname{argmax}_{\pi \in \Pi} \min_{\lambda \in \Lambda} L_{\tau}(\pi, \lambda)$  and  $\lambda_{\tau}^{\star} = \operatorname{argmin}_{\lambda \in \Lambda} \max_{\pi \in \Pi} L_{\tau}(\pi, \lambda)$ . Existence of saddle points in Lemma 14 ensures that,
$$
L _ {\tau} \left(\pi , \lambda_ {\tau} ^ {\star}\right) \leq L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda_ {\tau} ^ {\star}\right) \leq L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda\right) \text {f o r a l l} (\pi , \lambda) \in \Pi \times \Lambda
$$
in which the first inequality implies that for any  $\pi \in \Pi$
$$
\begin{array}{l} V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \tau \left(\mathcal {H} \left(\pi_ {\tau} ^ {\star}\right) + \frac {1}{2} \left(\lambda_ {\tau} ^ {\star}\right) ^ {2}\right) \geq V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi} (\rho) + \tau \left(\mathcal {H} (\pi) + \frac {1}{2} \left(\lambda_ {\tau} ^ {\star}\right) ^ {2}\right) \\ \geq V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi} (\rho) + \tau \frac {1}{2} (\lambda_ {\tau} ^ {\star}) ^ {2} \\ \end{array}
$$
and the second inequality implies that for any  $\lambda \in \Lambda$
$$
\begin{array}{l} V _ {r + \lambda g} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \tau \left(\mathcal {H} \left(\pi_ {\tau} ^ {\star}\right) + \frac {1}{2} \lambda^ {2}\right) \geq V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \tau \left(\mathcal {H} \left(\pi_ {\tau} ^ {\star}\right) + \frac {1}{2} \left(\lambda_ {\tau} ^ {\star}\right) ^ {2}\right) \\ \geq V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \tau \mathcal {H} (\pi_ {\tau} ^ {\star}) \\ \end{array}
$$
Combination of two implied inequalities above leads to (5), i.e.,  $(\pi_{\tau}^{\star},\lambda_{\tau}^{\star})$  is a saddle point of the original Lagrangian  $V_{r + \lambda g}^{\pi}(\rho)$ , up to two  $\tau$ -terms.
Proof. We begin with the standard decomposition of the primal-dual gap,
$$
L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t}, \lambda_ {\tau} ^ {\star}\right) = \underbrace {L _ {\tau} \left(\pi_ {\tau} ^ {\star} , \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t} , \lambda_ {t}\right)} _ {\text {(i)}} + \underbrace {L _ {\tau} \left(\pi_ {t} , \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t} , \lambda_ {\tau} ^ {\star}\right)} _ {\text {(i i)}} \tag {19}
$$
and we next deal with (i) and (ii), separately. We notice that
$$
\mathcal {H} (\pi) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} - \gamma^ {t} \log \pi \left(a _ {t} \mid s _ {t}\right) \right] = - \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi} (s) \pi (a \mid s) \log \pi (a \mid s).
$$
We note that  $Q_{r + \lambda_t g + \tau \psi_t}^{\pi_t}(s, a)$  is a sum of a soft  $Q$  value function associated with a composite function  $r + \lambda_t g - \tau \log \pi_t$ , and  $-\tau \log \pi_t$ . Because of the boundedness of the soft  $Q$  value function and the  $\epsilon_0$ -restriction on the policy domain  $\hat{\Delta}(A)$ ,  $|Q_{r + \lambda_t g + \tau \psi_t}^{\pi_t}(s, a)| \leq \frac{1}{1 - \gamma} (1 + \frac{1}{(1 - \gamma)\xi} + \tau \log |A|) -$
$\tau \log \frac{\epsilon_0}{|A|} \coloneqq C_{\tau, \xi, \epsilon_0}$ . For the term (i),
$$
\begin{array}{l} L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t}, \lambda_ {t}\right) \\ = V _ {r + \lambda_ {t} g} ^ {\pi_ {\tau} ^ {*}} (\rho) - V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho) \\ - \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a | s) \log \pi_ {\tau} ^ {\star} (a | s) + \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {t}} (s) \pi_ {t} (a | s) \log \pi_ {t} (a | s) \\ = V _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {t}} (\rho) - \tau V _ {\psi_ {t}} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \tau V _ {\psi_ {t}} ^ {\pi_ {t}} (\rho) \\ - \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a | s) \log \pi_ {\tau} ^ {\star} (a | s) + \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {t}} (s) \pi_ {t} (a | s) \log \pi_ {t} (a | s) \\ = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\pi_ {\tau} ^ {\star} (a \mid s) - \pi_ {t} (a \mid s)\right) Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {t}} (s, a) \\ + \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {*}} (s) \pi_ {\tau} ^ {\star} (a | s) \log \pi_ {t} (a | s) - \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {t}} (s) \pi_ {t} (a | s) \log \pi_ {t} (a | s) \\ - \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a | s) \log \pi_ {\tau} ^ {\star} (a | s) + \frac {\tau}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {t}} (s) \pi_ {t} (a | s) \log \pi_ {t} (a | s) \\ = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\pi_ {\tau} ^ {\star} (a \mid s) - \pi_ {t} (a \mid s)\right) Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {t}} (s, a) - \frac {\tau}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \mathrm {K L} _ {t} (s) \\ \leq \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\frac {\mathrm {K L} _ {t} (s) - \mathrm {K L} _ {t + 1} (s)}{\eta (1 - \gamma)}\right) + \eta \left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2} - \frac {\tau}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \mathrm {K L} _ {t} (s) \\ = \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\frac {(1 - \eta \tau) \mathrm {K L} _ {t} (s) - \mathrm {K L} _ {t + 1} (s)}{\eta (1 - \gamma)}\right) + \eta \left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2} \\ = \frac {(1 - \eta \tau) \mathrm {K L} _ {t} (\rho) - \mathrm {K L} _ {t + 1} (\rho)}{\eta (1 - \gamma)} + \eta \left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}. \\ \end{array}
$$
where the first two equalities are because of the entropy regularization  $\mathcal{H}(\pi)$ , we apply the performance difference lemma in Lemma 28 and  $\psi_t(s, a) = -\log \pi_t(a|s)$  to the third equality, the first inequality is due to an application of Lemma 27 with  $x^{\star} = \pi_{\tau}^{\star}(\cdot |s)$ ,  $x = \pi_t(\cdot |s)$ ,  $g = -Q_{r + \lambda_t g + \tau \psi_t}^{\pi_t}(s, a) / (1 - \gamma)$ , and  $\eta \leq 1 / C_{\tau, \xi, \epsilon_0}$ .
Similarly, for the term (ii),
$$
\begin{array}{l} L _ {\tau} \left(\pi_ {t}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t}, \lambda_ {\tau} ^ {\star}\right) \\ = V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho) - V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {t}} (\rho) + \frac {1}{2} \tau (\lambda_ {t}) ^ {2} - \frac {1}{2} \tau (\lambda_ {\tau} ^ {\star}) ^ {2} \\ = \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) V _ {g} ^ {\pi_ {t}} (\rho) + \frac {1}{2} \tau \left(\lambda_ {t}\right) ^ {2} - \frac {1}{2} \tau \left(\lambda_ {\tau} ^ {\star}\right) ^ {2} \\ = \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) \left(V _ {g} ^ {\pi_ {t}} (\rho) + \tau \lambda_ {t}\right) - \frac {1}{2} \tau \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) ^ {2} \\ \leq \frac {\left(\lambda_ {\tau} ^ {\star} - \lambda_ {t}\right) ^ {2} - \left(\lambda_ {\tau} ^ {\star} - \lambda_ {t + 1}\right) ^ {2}}{2 \eta} + \frac {1}{2} \eta \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2} - \frac {1}{2} \tau \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) ^ {2} \\ = \frac {(1 - \eta \tau) (\lambda_ {\tau} ^ {\star} - \lambda_ {t}) ^ {2} - (\lambda_ {\tau} ^ {\star} - \lambda_ {t + 1}) ^ {2}}{2 \eta} + \frac {1}{2} \eta (C _ {\tau , \xi} ^ {\prime}) ^ {2} \\ \end{array}
$$
where the inequality is due to the standard descent lemma [128] and  $V_{g}^{\pi_{t}}(\rho) + \tau \lambda_{t} \leq \frac{1}{1 - \gamma} (1 + \frac{\tau}{\xi}) := C_{\tau, \xi}'$ .
Using the definition  $\Phi_t \coloneqq \mathrm{KL}_t(\rho) + \frac{1}{2} (\lambda_\tau^\star - \lambda_t)^2$ , we combine the two inequalities above to show that,
$$
\begin{array}{l} \Phi_ {t + 1} \leq (1 - \eta \tau) \Phi_ {t} - \eta \left(L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {t}, \lambda_ {\tau} ^ {\star}\right)\right) + \eta^ {2} \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \\ \leq (1 - \eta \tau) \Phi_ {t} + \eta^ {2} \max  \left((C _ {\tau , \xi , \epsilon_ {0}}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2}\right). \\ \end{array}
$$
where the second inequality is due to Lemma 14. If we expand the inequality above recursively, then,
$$
\begin{array}{l} \Phi_ {t + 1} \leq (1 - \eta \tau) \Phi_ {t} + \eta^ {2} \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \\ \leq \left(1 - \eta \tau\right) ^ {2} \Phi_ {t - 1} + \left(\eta^ {2} + \eta^ {2} (1 - \eta \tau)\right) \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \\ \leq \dots \\ \leq \left. (1 - \eta \tau) ^ {t} \Phi_ {1} + \left(\eta^ {2} \left(1 + (1 - \eta \tau) + (1 - \eta \tau) ^ {2} + \dots\right)\right) \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \right. \\ \leq (1 - \eta \tau) ^ {t} \Phi_ {1} + \frac {\eta}{\tau} \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \\ \leq \mathrm {e} ^ {- \eta \tau t} \Phi_ {1} + \frac {\eta}{\tau} \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \\ \end{array}
$$
C.3 Proof of Corollary 3
Proof. According to Theorem 2, if we take  $\tau = \Theta(\epsilon)$ ,  $\eta = \Theta(\epsilon^2)$ , and  $\epsilon_0 = \epsilon$ , then  $\Phi_{t+1} = O(\epsilon)$  for any  $t = \Omega\left(\frac{1}{\epsilon^3}\log \frac{1}{\epsilon}\right)$ , where  $\Omega(\cdot)$  hides some problem-dependent constant. We next consider a primal-dual iterate  $(\pi_t,\lambda_t)$  for some  $t = \Omega\left(\frac{1}{\epsilon^3}\log \frac{1}{\epsilon}\right)$ . It is straightforward to check that
$$
\mathrm {K L} _ {t} (\rho) = O (\epsilon) \text {a n d} \frac {1}{2} \left(\lambda_ {\tau} ^ {\star} - \lambda_ {t}\right) ^ {2} = O (\epsilon).
$$
$$
V _ {r} ^ {\pi^ {*}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) = \underbrace {V _ {r} ^ {\pi^ {*}} (\rho) - V _ {r} ^ {\pi_ {\tau} ^ {*}} (\rho)} _ {\text {(i)}} + \underbrace {V _ {r} ^ {\pi_ {\tau} ^ {*}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho)} _ {\text {(i i)}}. \tag {20}
$$
For the term (ii), because  $\mathrm{KL}_t(\rho) = O(\epsilon)$ , we have
$$
\begin{array}{l} \text {(i i)} = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\pi_ {\tau} ^ {\star} (a | s) - \pi_ {t} (a | s)\right) Q _ {r} ^ {\pi_ {t}} (s, a) \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \| \pi_ {\tau} ^ {\star} (\cdot | s) - \pi_ {t} (\cdot | s) \| _ {1} \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {\rho} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \sqrt {2 \mathrm {K L} _ {t} (s)} \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sqrt {\overline {{2 \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \mathrm {K L} _ {t} (s)}}} \\ = \frac {1}{(1 - \gamma) ^ {2}} \sqrt {2 \mathrm {K L} _ {t} (\rho)} \\ \end{array}
$$
where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to Pinsker's inequality, which shows  $V_{r}^{\pi_{\tau}^{\star}}(\rho) - V_{r}^{\pi_{t}}(\rho) \leq O(\sqrt{\epsilon})$ . For the term (i), if we take  $\pi = \pi^{\star}$  in (5), then,
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - \tau \mathcal {H} \left(\pi_ {\tau} ^ {\star}\right) \leq V _ {r} ^ {\pi_ {\tau} ^ {\star}} (\rho) + \lambda_ {\tau} ^ {\star} \left(V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {g} ^ {\pi^ {\star}} (\rho)\right).
$$
Meanwhile, if we take  $\lambda = 0$  in (5), then  $\lambda_{\tau}^{\star}V_{g}^{\pi_{\tau}^{\star}}(\rho)\leq 0$ . We notice that the feasibility of  $\pi^{\star}$  yields  $V_{g}^{\pi^{\star}}(\rho)\geq 0$ . Hence,
$$
\left(\mathrm {i}\right) = V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\tau} ^ {\star}} (\rho) \leq \tau \mathcal {H} \left(\pi_ {\tau} ^ {\star}\right).
$$
We now substitute the upper bounds of (i) and (ii) above into (20) to obtain  $V_{r}^{\pi^{*}}(\rho) - V_{r}^{\pi_{t}}(\rho) \leq O(\sqrt{\epsilon})$ , where we take  $\tau = \Theta(\epsilon)$ .
$$
- V _ {g} ^ {\pi_ {t}} (\rho) = \underbrace {- V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho)} _ {\text {(i i i)}} + \underbrace {V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {g} ^ {\pi_ {t}} (\rho)} _ {\text {(i v)}}. \tag {21}
$$
Similar to bounding  $V_{r}^{\pi_{\tau}^{\star}}(\rho) - V_{r}^{\pi_{t}}(\rho)$ , we can show that (iv)  $\leq O(\sqrt{\epsilon})$ . Let  $\lambda_{\max} \coloneqq \frac{1}{(1 - \gamma)\xi}$ . For the term (iii), if we take  $\lambda = \lambda_{\max}$  in (5), then,
$$
- \left(\lambda_ {\max } - \lambda_ {\tau} ^ {\star}\right) V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho) \leq \frac {\tau}{2} \left(\lambda_ {\max }\right) ^ {2}.
$$
By the definition  $\lambda_{\tau}^{\star} := \operatorname*{argmin}_{\lambda \in \Lambda} \left\{ \lambda V_{g}^{\pi_{\tau}^{\star}}(\rho) + \frac{\tau}{2} \lambda^{2} \right\}$ , there are three values for  $\lambda_{\tau}^{\star}$  to take:  $-\frac{V_{g}^{\pi_{\tau}^{\star}}(\rho)}{\tau}$ , or 0, or  $\lambda_{\max}$  as follows
(1) When  $0 < -\frac{V_g^{\pi_\tau^\star}(\rho)}{\tau} < \lambda_{\max}$ ,  $\lambda_\tau^\star = -\frac{V_g^{\pi_\tau^\star}(\rho)}{\tau}$  which shows that  $\lambda_{\max} - \lambda_\tau^\star > 0$ ;
(2) When  $-\frac{V_g^{\pi_\tau^\star}(\rho)}{\tau}\leq 0,\lambda_\tau^\star = 0$  which shows that  $\lambda_{\mathrm{max}} - \lambda_{\tau}^{\star} = \lambda_{\mathrm{max}} > 0$
(3) When  $-\frac{V_g^{\pi_\tau^\star}(\rho)}{\tau}\geq \lambda_{\max},\lambda_\tau^\star = \lambda_{\max}$ . In this case, using (5) with  $\pi = \pi^{\star}$  leads to
$$
\lambda_ {\max } \left(V _ {g} ^ {\pi^ {\star}} (\rho) - V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho)\right) \leq V _ {r} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {r} ^ {\pi^ {\star}} (\rho) + \tau \mathcal {H} \left(\pi_ {\tau} ^ {\star}\right). \tag {22}
$$
Meanwhile, for any saddle point  $(\pi^{\star},\lambda^{\star})\in \Pi^{\star}\times \Lambda^{\star}$ ,  $V_{r}^{\pi_{\tau}^{\star}}(\rho) - V_{r}^{\pi^{\star}}(\rho)\leq \lambda^{\star}(V_{g}^{\pi^{\star}}(\rho) - V_{g}^{\pi_{\tau}^{\star}}(\rho))$ , which in conjunction with (22) and  $V_{g}^{\pi^{\star}}(\rho)\geq 0$  yields,
$$
- (\lambda_ {\max } - \lambda^ {\star}) V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho) \leq \tau \mathcal {H} (\pi_ {\tau} ^ {\star}).
$$
It is easy to see that we can always take  $\lambda^{\star} < \lambda_{\mathrm{max}}$ . In fact, except for  $\lambda^{\star} = 0$ , we know that  $\lambda^{\star} V_{g}^{\pi^{\star}}(\rho) \leq 0$  leads to  $V_{g}^{\pi^{\star}}(\rho) = 0$ . By the definition  $\lambda^{\star} \in \operatorname*{argmin}_{\lambda \in \Lambda} V_{r + \lambda g}^{\pi^{\star}}(\rho)$ , any  $\lambda^{\star} < \lambda_{\mathrm{max}}$  is a min-max point. Therefore,
$$
\text {(i i i)} \leq \frac {\tau}{\left(\lambda_ {\max } - \lambda^ {\star}\right)} \mathcal {H} \left(\pi_ {\tau} ^ {\star}\right).
$$
By combining three cases above, we conclude that (iii)  $\leq O(\tau) = O(\epsilon)$
We now substitute the upper bounds of (iii) and (iv) above into (21) to obtain  $-V_{g}^{\pi_{t}}(\rho)\leq O(\sqrt{\epsilon})$
C.4 Zero constraint violation of RPG-PD (6)
Corollary 15 (Zero constraint violation). Let Assumption 1 hold. For small  $\epsilon$ , there exists  $\delta > 0$  such that if we instead use the conservative constraint  $V_{g'}^\pi(\rho) \geq 0$  for  $g' = g - (1 - \gamma)\delta$ , and take  $\eta = \Theta(\epsilon^4)$ ,  $\tau = \Theta(\epsilon^2)$ , and  $\epsilon_0 = \epsilon$ , then the policy iterates of RPG-PD (6) satisfy
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) \leq \epsilon \text {a n d} - V _ {g} ^ {\pi_ {t}} (\rho) \leq 0 \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$