SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to bounding  $\mathbb{E}\left[V_r^{\pi_\tau^\star}(\rho) - V_r^{\pi_{\theta_t}}(\rho)\right]$ , we can show that  $\mathbb{E}\left[(\mathrm{iv})\right] \leq O(\sqrt{\epsilon})$ . For the term (iii), we can show that (iii)  $\leq O(\tau) = O(\epsilon)$  in a similar way as dealing with (iii) in (21).
We now substitute the upper bounds of (iii) and (iv) above into (25) to obtain  $\mathbb{E}\left[-V_g^{\pi_{\theta_t}}(\rho)\right] \leq O(\sqrt{\epsilon})$ .
C.8 Zero constraint violation of inexact RPG-PD
Corollary 17 (Zero constraint violation). Let Assumptions 1-2 hold and  $\epsilon_{\mathrm{stat}}$ ,  $\epsilon_{\mathrm{bias}} = O(\epsilon^8)$  for small  $\epsilon$ ,  $\epsilon_0 > 0$ . For small  $\epsilon$ , there exists  $\delta > 0$  such that if we instead use the conservative constraint  $V_{g'}^\pi (\rho) \geq 0$  for  $g' = g - (1 - \gamma)\delta$ , and take the stepsize  $\eta = \Theta (\epsilon^4)$  and  $\tau = \Theta (\epsilon^2)$ , then the policy iterates of inexact RPG-PD satisfy
$$
\mathbb {E} \left[ V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon a n d \mathbb {E} \left[ - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq 0 f o r a n y t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
Proof. We apply the conservatism to the translated constraint  $V_{g}^{\pi}(\rho) \geq 0$  in Problem (1). Specifically, for any  $\delta < \min (\xi ,1)$ , we let  $g^{\prime} \coloneqq g - (1 - \gamma)\delta$  and define a conservative constraint,
It is straightforward to see that Assumption 1 ensures that  $V_{g'}^{\pi_t}(\rho) \geq 0$  is strictly feasible for a new slack variable  $\xi' \coloneqq \xi - \delta$ . We now can apply inexact RPG-PD (8) to a new regularized Lagrangian,
and Corollary 5 holds if we replace  $g$  in inexact RPG-PD by  $g'$  and  $\epsilon_{\mathrm{stat}}$ ,  $\epsilon_{\mathrm{bias}} = O(\epsilon^8)$  for small  $\epsilon > 0$ . Thus,
$$
\mathbb {E} \left[ V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon \text {a n d} \mathbb {E} \left[ - V _ {g ^ {\prime}} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\Omega (\cdot)$  has some problem-dependent constant, and  $\pi_{\delta}^{\star}$  is an optimal policy to the  $\delta$ -perturbed constrained policy optimization problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r} ^ {\pi} (\rho) \quad \text {s u b j e c t t o} V _ {g} ^ {\pi} (\rho) - \delta \geq 0. \tag {26}
$$
We notice that the above  $\Omega(\cdot)$  has  $\xi'$ -dependence and we denote it by  $\Xi(\xi')$ , where  $\Xi: \mathbb{R}_+ \to \mathbb{R}_+$  is a positive function. Thus, we select  $\delta$  such that  $\delta \geq \epsilon \Xi(\xi')$ , which is always possible for small enough  $\epsilon$ , for instance,  $\delta = \frac{\xi}{2}$  and  $\xi' = \frac{\xi}{2}$ . Hence, if we take  $\delta = \frac{\xi}{2}$  and such small  $\epsilon$ , then
$$
\mathbb {E} \left[ - V _ {g ^ {\prime}} ^ {\pi_ {\theta_ {t}}} (\rho) \right] = \mathbb {E} \left[ - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right] + \delta \leq \epsilon \Xi (\xi^ {\prime}) \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
which shows that  $\mathbb{E}\left[V_g^{\pi_{\theta_t}}(\rho)\right] \geq 0$  for some large  $t$ .
The rest is to show that  $\mathbb{E}\left[V_r^{\pi^\star}(\rho) - V_r^{\pi_\theta_t}(\rho)\right] \leq O(\epsilon)$ . We notice that  $\pi^\star$  is an optimal policy to Problem (26) when  $\delta = 0$ . Let  $q^\star$  and  $q_\delta^\star$  be associated occupancy measures of policies  $\pi^\star$  and  $\pi_\delta^\star$ . In the occupancy measure space, Problem (26) becomes a linear program and it has a solution  $q_\delta^\star$ . Thus, we can view  $q_\delta^\star$  as a  $\delta$ -perturbed solution of a convex optimization problem in which all functions are continuous differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution  $q_\delta^\star$  is continuous in  $\delta$ , which implies that for any  $\epsilon > 0$ , there exists  $\delta'$  such that  $|\langle r, q^\star \rangle - \langle r, q_\delta^\star \rangle| \leq O(\epsilon)$  for any  $\delta < \delta'$ . Thus,  $|V_r^{\pi^\star}(\rho) - V_r^{\pi_\delta^\star}(\rho)| \leq O(\epsilon)$  for small enough  $\epsilon$ . Therefore,
$$
\mathbb {E} \left[ V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \mathbb {E} \left[ V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] + | V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) | \leq O (\epsilon)
$$
for some large  $t$ . Collecting all conditions on  $\delta$  leads to our final choice of  $\delta = \min (\frac{\xi}{2},1,\delta^{\prime})$ . Finally, we combine all big O notation to complete the proof.
C.9 Sample-based inexact RPG-PD algorithm
We generalize the inexact RPG-PD to be a sample-based algorithm that only takes sample-based estimates. We propose a sample-based RPG-PD with linear function approximation as follows,
$$
\pi_ {\theta_ {t + 1}} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \hat {\Delta} (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) \phi_ {s, a} ^ {\top} \hat {w} _ {t} - \frac {1}{\eta} \mathrm {K L} \left(\pi (\cdot \mid s), \pi_ {\theta_ {t}} (\cdot \mid s)\right) \right\} \tag {27a}
$$
$$
\lambda_ {t + 1} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda \left(\widehat {V} _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) + \tau \lambda_ {t}\right) + \frac {1}{2 \eta} \left(\lambda - \lambda_ {t}\right) ^ {2} \right\}, \tag {27b}
$$
where  $\hat{w}_t$  and  $\hat{V}_g^{\pi_t}(\rho)$  are the sample-based estimates of NPG directions and value functions. It is standard to assume that there exists a policy simulator that generates policy rollouts for any given policies [49]. At time  $t$ , we can estimate  $\hat{w}_t$  by solving the regression problem  $\mathcal{E}_Q(w, \theta_t, d_{t,\nu}) = \mathbb{E}_{(s,a) \sim d_{t,\nu}}[(\phi_{s,a}^\top w - Q^{\pi_{\theta_t}}(s,a))^2]$  with  $Q^{\pi_{\theta_t}}(s,a) = Q_{r + \lambda_t g + \tau \psi_t}^{\pi_{\theta_t}}(s,a)$  via a projected stochastic gradient descent (SGD) method,
$$
w _ {t} ^ {k + 1} = \mathcal {P} _ {\| w \| \leq W} \left(w _ {t} ^ {k} - \alpha G _ {t} ^ {k}\right)
$$
where  $k \geq 0$  counts the number of SGD iterations, and  $G_{t}^{k}$  is a sample-based estimate of the population gradient  $\nabla_{w}\mathcal{E}_{Q}(w,\theta_{t},d_{t,\nu})$ ,
$$
G _ {t} ^ {k} = 2 \left(\phi_ {s, a} ^ {\top} w _ {t} ^ {k} - \hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) \phi_ {s, a}.
$$
From the projected SGD result [130], we use a weighted average  $\frac{2}{K(K + 1)}\sum_{k = 0}^{K - 1}(k + 1)w_t^k$  as our  $\hat{w}_t$ . We note that  $Q_{r + \lambda_tg + \tau \psi_t}^{\pi_{\theta_t}}(s,a)$  is a sum of a soft- $Q$  value function associated with a composite function  $r + \lambda_t g - \tau \log \pi_{\theta_t}$ , and  $-\tau \log \pi_{\theta_t}$ . Thus, we can estimate the value function  $\hat{Q}_{r + \lambda_t g + \tau \psi_t}^{\pi_{\theta_t}}(s,a)$  using policy rollouts in Algorithm 2, which provides an unbiased estimate and it has bounded variance [131],
$$
\mathbb {E} \left[ \hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a) \mid s, a \right] = Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a) \text {a n d} \mathbb {E} \left[ G _ {t} ^ {k} \right] = \nabla_ {w} \mathcal {E} _ {Q} \left(w _ {t} ^ {k}, \theta_ {t}, d _ {t, \nu}\right)
$$
where the expectation  $\mathbb{E}$  is taken over the randomness of drawing  $(s,a)\sim d_{t,\nu}$ . Another value function  $\hat{V}_g^{\pi_{\theta_t}}(\rho)$  can be estimated using policy rollouts in Algorithm 3,  $\mathbb{E}\left[\hat{V}_g^{\pi_{\theta_t}}(\rho)\right] = V_g^{\pi_{\theta_t}}(\rho)$ , which is also unbiased and has bounded variance [23]. Hence, simply replacing all population quantities in inexact RPG-PD by their sample-based estimates leads to a sample-based inexact RPG-PD that is detailed in Algorithm 1.
We are now ready to establish the sample complexity of Algorithm 1 by exploiting the projected SGD result [130].
Corollary 18 (Sample complexity of inexact RPG-PD). Let Assumptions 1-2 hold. Assume  $\Sigma_{\nu} = \mathbb{E}_{(s,a)}\sim_{\nu}\left[\phi_{s,a}\phi_{s,a}^{\top}\right]\geq \kappa_0I$  for any state-action distribution  $\nu$  and some  $\kappa_0 > 0$ . If we take the
Algorithm 1 Sample-based inexact RPG-PD algorithm with log-linear policy parametrization
Algorithm 2 Unbiased estimate  $Q$
Algorithm 3 Unbiased estimate  $V$
1: Input: Learning rate  $\eta$ , number of SGD iterations  $K$ , SGD learning rate  $\alpha$ .  
2: Initialize  $\theta_0 = 0$ ,  $\lambda_0 = 0$ ,  
3: for  $t = 0, \dots, T - 1$  do  
4: Initialize  $w_t^0 = 0$ .  
5: for  $k = 0, 1, \dots, K - 1$  do  
6: Estimate  $\hat{Q}_{r + \lambda_t g + \tau \psi_t}^{\pi_{\theta_t}}(s, a)$  for some  $(s, a) \sim d_{t,\nu}$ , using Algorithm 2 with policy  $\pi_{\theta_t}$ .  
7: Perform projected SGD step with  $\alpha_k = \frac{2}{\kappa_0(k + 2)}$ ,
$$
w _ {t} ^ {k + 1} = \mathcal {P} _ {\| w \| \leq W} \left(w _ {t} ^ {k} - 2 \alpha_ {k} \left(\phi_ {s, a} ^ {\top} w _ {t} ^ {k} - \hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) \phi_ {s, a}\right).
$$
8: end for
9: Set  $\hat{w}_t = \frac{2}{K(K + 1)}\sum_{k = 0}^{K - 1}(k + 1)w_t^k$
10: Estimate  $\hat{V}_g^{\pi_{\theta_t}}(\rho)$  using Algorithm 3 with policy  $\pi_{\theta_t}$ .
11: Perform inexact RPG-PD update,
$$
\begin{array}{rcl}\pi_{\theta_{t + 1}}(\cdot \mid s) & = & \operatorname *{argmax}_{\pi (\cdot \mid s)\in \hat{\Delta} (A)}\left\{\sum_{a}\pi (a\mid s)\phi_{s,a}^{\top}\hat{w}_{t} - \frac{1}{\eta}\operatorname {KL}(\pi (\cdot \mid s),\pi_{\theta_{t}}(\cdot \mid s))\right\} \\ \lambda_{t + 1} & = & \mathcal{P}_{\Lambda}\left((1 - \eta \tau)\lambda_{t} - \eta \hat{V}_{g}^{\pi_{\theta_{t}}}(\rho)\right). \end{array}
$$
1: Input: Initial state-action distribution  $\nu$ , policy  $\pi$ , dual variable  $\lambda$ , regularization parameter  $\tau$ , discount factor  $\gamma$ .
2: Sample  $(s_0, a_0) \sim \nu$ , execute the policy  $\pi$  with probability  $\gamma$  at each step  $h$ ; otherwise, accept  $(s_h, a_h)$  as the sample.
3: Start with  $(s_h, a_h)$ , execute the policy  $\pi$  with the termination probability  $1 - \sqrt{\gamma}$ . Once terminated, add all composite values  $\gamma^{(k-h)/2}(r + \lambda g + \tau \psi)$  from step  $k = h + 1$  onwards and  $-\tau \log \pi(a_h | s_h)$  as  $\hat{Q}_{r + \lambda g + \tau \psi}^{\pi}(s_h, a_h)$ .
4: Output:  $(s_h, a_h)$  and  $\hat{Q}_{r + \lambda g + \tau \psi}^{\pi}(s_h, a_h)$ .
1: Input: Initial state distribution  $\rho$ , policy  $\pi$ , discount factor  $\gamma$ .
2: Sample  $s_0 \sim \rho$ , execute the policy  $\pi$  with the termination probability  $1 - \gamma$ . Once terminated, add all utilities up as  $\hat{V}_g^\pi(\rho)$ .
3: Output:  $\hat{V}_g^\pi (\rho)$
stepsize  $\eta \leq 1 / C_W$ , then the primal-dual iterates of sample-based inexact RPG-PD in Algorithm 1 satisfy
$$
\mathbb {E} \big [ \Phi_ {t + 1} \big ] \leq \mathrm {e} ^ {- \eta \tau t} \mathbb {E} \big [ \Phi_ {1} \big ] + \frac {\eta}{\tau} \max \big ((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2} \big) + \frac {C _ {W , \xi , \tau , \epsilon_ {0}}}{\tau} \sqrt {\frac {| A | \kappa_ {\nu}}{K + 1}} + \frac {2}{\tau} \sqrt {| A | \epsilon_ {\mathrm {b i a s}}}
$$
where  $C_W \coloneqq 2W / (1 - \gamma)$ ,  $C_{W,\xi,\tau,\epsilon_0} \coloneqq 8(W + 2 / (\xi(1 - \sqrt{\gamma})^2) + \tau(2\log |A| + |\log \epsilon_0|) / ((1 - \sqrt{\gamma})^2\xi)) / \sqrt{\kappa_0}$ , and  $C_{\tau,\xi}' \coloneqq (1 + \tau / \xi) / (1 - \gamma)$ .
Proof. The proof is based on the proof of Theorem 4. Additionally, we have to take the randomness of sample-based estimates into account and bound the statistical error  $\epsilon_{\mathrm{stat}}$  using the projected SGD result [130]. We first check all conditions of the projected SGD [130] are indeed satisfied by the SGD step in Algorithm 1: (i) The domain  $\| w\| \leq W$  is convex and bounded; (ii) The gradient  $G_{t}^{k}$  is an unbiased estimate of the population gradient; (iii) The minimizer of  $\mathcal{E}_Q(w,\theta_t,d_{t,\nu})$  is unique, since  $\Sigma_{d_{t,\nu}}\geq \kappa_0I$  for some  $\kappa_0 > 0$ ; (iv) The squared norm of the estimated gradient  $G_{t}^{k}$  is bounded or the gradient has bounded variance. To show (iv), it is sufficient to check that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \phi_ {s, a} ^ {\top} \phi_ {s, a} w _ {t} ^ {k} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left\| \phi_ {s, a} ^ {\top} \phi_ {s, a} \right\| ^ {2} \left\| w _ {t} ^ {k} \right\| ^ {2} \right] \leq W ^ {2} \\ \mathbb {E} \left[ \left\| \hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a) \phi_ {s, a} \right\| ^ {2} \right] \leq \mathbb {E} \left[ \left(\hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) ^ {2} \| \phi_ {s, a} \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \left(\hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) ^ {2} \right] \\ \end{array}
$$
where we use the boundedness of  $\left\| w_t^k\right\| \leq W$  and  $\| \phi_{s,a}\| \leq 1$ . We notice that  $|r + \lambda_t g|\leq \frac{2}{(1 - \gamma)\xi}$ . By [131, Lemma 3.5], we know that
$$
\mathbb {E} \left[ \left(\hat {Q} _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) ^ {2} \right] \leq 4 \left(\frac {2 / ((1 - \gamma) \xi)}{1 - \sqrt {\gamma}}\right) ^ {2} + 4 \left(\frac {\tau \log | A |}{1 - \sqrt {\gamma}}\right) ^ {2} + 2 \left(\tau \log \frac {\epsilon_ {0}}{| A |}\right) ^ {2}
$$
where the first two terms in the upper bound is due to the variance of soft- $Q$  value function, and the last term is due to  $|\log \pi_{\theta_t}|\leq |\log \frac{\epsilon_0}{|A|} |$  for  $\pi_{\theta_t}\in \hat{\Delta} (A)$ . Hence, the estimated gradient  $G_{t}^{k}$  has bounded second-order moment (or variance), which verifies (iv). From the projected SGD result [130], if the SGD stepsize  $\alpha_{k} = \frac{2}{k + 2}$ , then
$$
\mathbb {E} \big [ \mathcal {E} _ {Q} (\hat {w} _ {t}, \theta_ {t}, d _ {t, \nu}) - \mathcal {E} _ {Q} (w _ {t} ^ {\star}, \theta_ {t}, d _ {t, \nu}) \big ] \leq \frac {1 6 \left(W + \frac {2}{\xi (1 - \sqrt {\gamma}) ^ {2}} + \frac {\tau (2 \log | A | + | \log \epsilon_ {0} |)}{(1 - \sqrt {\gamma}) ^ {2} \xi}\right) ^ {2}}{\kappa_ {0} (K + 1)}
$$
which leads to  $\sqrt{\epsilon_{\mathrm{stat}}} \leq \frac{4(W + 2 / (\xi(1 - \sqrt{\gamma})^2) + \tau(2\log|A| + |\log\epsilon_0|) / ((1 - \sqrt{\gamma})^2\xi))}{\sqrt{\kappa_0(K + 1)}}$ . Substituting the bound of  $\sqrt{\epsilon_{\mathrm{stat}}}$  into the proof of Theorem 4 yields our desired result.
Corollary 18 states a similar result as Theorem 4. The effect of using sample-based estimates to update inexact RPG-PD appears as the number  $K$  of SGD steps at each time  $t$ . Thus, we can interpret the iteration complexity in Corollary 5 in terms of the number of SGD steps by taking  $\epsilon_{\mathrm{stat}} = O(\epsilon^8)$ , i.e.,  $K = \Omega \left(\frac{1}{\epsilon^8}\right)$ . Thus, whenever  $\epsilon_{\mathrm{bias}} = O(\epsilon^8)$  for small  $\epsilon > 0$ , if we take  $(\eta, \tau)$  in Corollary 5 and  $K = \Omega \left(\frac{1}{\epsilon^8}\right)$  for Algorithm 1, then,
$$
\mathbb {E} \left[ V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon \text {a n d} \mathbb {E} \left[ - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\Omega (\cdot)$  only has some problem-dependent constant. In other words, the total number of policy rollouts or sampled trajectories  $tK = \Omega \left(\frac{1}{\epsilon^{14}}\right)$  is required for Algorithm 1 to output an  $\epsilon$ -optimal constrained policy. Furthermore, the zero constraint violation in Corollary 17 can be interpreted similarly. When  $\epsilon$  is small enough, we can design a conservative constraint such that the policy iterates of Algorithm 1 satisfy  $V_{r}^{\pi^{*}}(\rho) - V_{r}^{\pi_{t}}(\rho) \leq \epsilon$  and  $V_{g}^{\pi_{\theta_{t}}}(\rho) \geq 0$  for some  $t$ ,  $K$  that satisfy  $tK = \Omega \left(\frac{1}{\epsilon^{14}}\right)$ . This appears to be the first sample-based zero constraint violation result for constrained MDPs in the function approximation setting. We leave achieving the optimal sample complexity as future work.
D Proofs in Section 4
In this section, we provide proofs for the claims in Section 4.
D.1 Preliminary last-iterate analysis of OPG-PD (9)
To measure the proximity of the primal-dual iterates of OPG-PD (9) to the optimal pair  $(\pi^{\star},\lambda^{\star})$ , we introduce the following two distance metrics  $\Theta_t$  and  $\zeta_t$  at time  $t\geq 1$ ,
$$
\begin{array}{l} \Theta_ {t} := \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left\| \hat {\pi} _ {t} (\cdot | s) - \mathcal {P} _ {\Pi^ {*}} (\hat {\pi} _ {t} (\cdot | s)) \right\| ^ {2} + \frac {1}{2} (\hat {\lambda} _ {t} - \mathcal {P} _ {\Pi^ {*}} (\hat {\lambda} _ {t})) ^ {2} \\ + \frac {1}{1 6 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \hat {\pi} _ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| ^ {2} + \frac {1}{1 6} (\hat {\lambda} _ {t} - \lambda_ {t - 1}) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \zeta_ {t} := \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \right\| ^ {2} + \frac {1}{2} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \\ + \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \frac {1}{2} (\lambda_ {t} - \hat {\lambda} _ {t}) ^ {2} \\ \end{array}
$$
and a problem-dependent constant  $\iota$
$$
\iota := \max  \left(\frac {2 \kappa_ {\rho} ^ {2} | A |}{(1 - \gamma) ^ {6}}, \frac {8 \gamma^ {2} \sqrt {| A |} \kappa_ {\rho}}{(1 - \gamma) ^ {6}} \left(1 + \frac {1}{(1 - \gamma) ^ {2} \xi^ {2}}\right), \frac {4}{(1 - \gamma) ^ {3}}\right). \tag {28}
$$
Lemma 19. Let the optimal state visitation distribution be unique, i.e.,  $d_{\rho}^{\pi} = d_{\rho}^{\pi^{\star}}$  for any  $\pi \in \Pi^{\star}$ . If we set the stepsize  $\eta \leq 1 / (4\sqrt{\iota})$ , then the primal-dual iterates of OPG-PD (9) satisfy
$$
\Theta_ {t + 1} \leq \Theta_ {t} - \frac {7}{1 6} \zeta_ {t} \text {f o r a l l} t \geq 1. \tag {29}
$$
Proof. We begin with the standard decomposition of the primal-dual gap at time  $t \geq 1$ ,
$$
V _ {r + \lambda_ {t} g} ^ {\pi^ {*}} (\rho) - V _ {r + \lambda^ {*} g} ^ {\pi_ {t}} (\rho) = \underbrace {V _ {r + \lambda_ {t} g} ^ {\pi^ {*}} (\rho) - V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho)} _ {\text {(i)}} + \underbrace {V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho) - V _ {r + \lambda^ {*} g} ^ {\pi_ {t}} (\rho)} _ {\text {(i i)}} \tag {30}
$$
and we next deal with (i) and (ii), separately.
For the first term (i),
$$
\begin{array}{l} V _ {r + \lambda_ {t} g} ^ {\pi^ {\star}} (\rho) - V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho) \\ = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi^ {\star}} (s) (\pi^ {\star} (a \mid s) - \pi_ {t} (a \mid s)) Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) \\ = \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi^ {*}} (s) (\pi^ {*} (a \mid s) - \hat {\pi} _ {t + 1} (a \mid s)) Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) \\ + \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi^ {*}} (s) (\hat {\pi} _ {t + 1} (a | s) - \pi_ {t} (a | s)) Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a) \\ + \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi^ {\star}} (s) (\hat {\pi} _ {t + 1} (a | s) - \pi_ {t} (a | s)) \left(Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a)\right) \\ \leq \frac {1}{2 \eta (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ + \frac {1}{2 \eta (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} - \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ + \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\hat {\pi} _ {t + 1} (a | s) - \pi_ {t} (a | s)\right) \left(Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a)\right) \tag {31} \\ \end{array}
$$
where the first equality is due to performance difference lemma in Lemma 28, the second equality is because of some re-arrangement, the inequality is from an application of Lemma 24 to  $\hat{\pi}_{t + 1}(\cdot |s)$  and  $\pi_t(\cdot |s)$ :
$$
\begin{array}{l} \sum_ {a} \left(\pi^ {\star} (a \mid s) - \hat {\pi} _ {t + 1} (a \mid s)\right) Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) \\ \leq \frac {1}{2 \eta} \left(\| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {a} (\hat {\pi} _ {t + 1} (a \mid s) - \pi_ {t} (a \mid s)) Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a) \\ \leq \frac {1}{2 \eta} \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} - \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right). \\ \end{array}
$$
Similarly, we deal with the second term (ii) with some re-arrangement, and apply Lemma 24 to  $\hat{\lambda}_{t+1}$  and  $\lambda_t$ ,
$$
\begin{array}{l} V _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (\rho) - V _ {r + \lambda^ {*} g} ^ {\pi_ {t}} (\rho) = (\lambda_ {t} - \lambda^ {*}) V _ {g} ^ {\pi_ {t}} (\rho) \\ = \left(\lambda_ {t} - \hat {\lambda} _ {t + 1}\right) V _ {g} ^ {\pi_ {t - 1}} (\rho) + \left(\lambda_ {t} - \hat {\lambda} _ {t + 1}\right) \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t - 1}} (\rho)\right) \\ + \left(\hat {\lambda} _ {t + 1} - \lambda^ {\star}\right) V _ {g} ^ {\pi_ {t}} (\rho) \\ \leq \frac {1}{2 \eta} \left(\left(\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}\right) ^ {2} - \left(\hat {\lambda} _ {t + 1} - \lambda_ {t}\right) ^ {2} - \left(\lambda_ {t} - \hat {\lambda} _ {t}\right) ^ {2}\right) \\ + \left(\lambda_ {t} - \hat {\lambda} _ {t + 1}\right) \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t - 1}} (\rho)\right) \\ + \frac {1}{2 \eta} \left(\left(\lambda^ {\star} - \hat {\lambda} _ {t}\right) ^ {2} - \left(\lambda^ {\star} - \hat {\lambda} _ {t + 1}\right) ^ {2} - \left(\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}\right) ^ {2}\right). \tag {32} \\ \end{array}
$$
On the other hand, from Lemma 25, we have
$$
\left\| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \right\| _ {1} \leq \eta \left\| Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, \cdot) \right\| _ {\infty} \tag {33}
$$
$$
\left| \lambda_ {t} - \hat {\lambda} _ {t + 1} \right| \leq \eta \left| V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t - 1}} (\rho) \right|. \tag {34}
$$