SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Theorem 2, we show that the primal-dual iterates of RPG-PD converge in the last iterate; see Appendix C.2 for proof. We characterize the convergence via a distance metric  $\Phi_t = \mathrm{KL}_t(\rho) + \frac{1}{2} (\lambda_\tau^\star -\lambda_t)^2$ , where  $\mathrm{KL}_t(\rho)\coloneqq (1 / (1 - \gamma))\sum_s d_\rho^{\pi_\tau^\star}(s)\mathrm{KL}_t(s)$  and  $\mathrm{KL}_t(s)\coloneqq \mathrm{KL}(\pi_\tau^\star (\cdot |s),\pi_t(\cdot |s))$ . Theorem 2 (Linear convergence of RPG-PD). Let Assumption 1 hold. If we set the stepsize  $\eta \leq 1 / C_{\tau ,\xi ,\epsilon_0}$ , then the primal-dual iterates of RPG-PD (6) satisfy
$$
\Phi_ {t + 1} \leq \mathrm {e} ^ {- \eta \tau t} \Phi_ {1} + \frac {\eta}{\tau} \max  \left(\left(C _ {\tau , \xi , \epsilon_ {0}}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right)
$$
where  $C_{\tau, \xi, \epsilon_0} := (1 + 1 / ((1 - \gamma) \xi) + \tau \log |A|) / (1 - \gamma) - \tau \log (\epsilon_0 / |A|), C_{\tau, \xi}' := (1 + \tau / \xi) / (1 - \gamma)$ .
Theorem 2 states that the primal-dual iterates of RPG-PD converge to a neighborhood of  $(\pi_{\tau}^{\star}, \lambda_{\tau}^{\star})$  in a linear rate. The size of neighborhood scales with  $\eta / \tau + \eta \tau (1 + \log^2 \epsilon_0)$  and the convergence rate is  $\eta \tau$ . Even if  $\epsilon_0$  is very small, the log  $\epsilon_0$ -term is almost a constant. If we take  $\eta = \min(\epsilon \tau, 1 / C_{\tau, \xi, \epsilon_0})$  and  $\epsilon_0 = \epsilon$ , then after  $O(1 / \epsilon)$  iterations the RPG-PD's primal-dual iterate  $(\pi_t, \lambda_t)$  is  $\epsilon$ -close to  $(\pi_{\tau}^{\star}, \lambda_{\tau}^{\star})$ , i.e.,  $\Phi_t = O(\epsilon)$  for any  $t \geq (1 / (\epsilon \tau^2)) \log (1 / \epsilon)$ . For small  $\tau$ , we can translate the policy convergence for the value functions in Corollary 3; see Appendix C.3 for proof.
Corollary 3 (Nearly-optimal constrained policy). Let Assumption 1 hold. For small  $\epsilon >0$ , if we take  $\eta = \Theta (\epsilon^4)$ ,  $\tau = \Theta (\epsilon^2)$ , and  $\epsilon_0 = \epsilon$ , then the policy iterates of RPG-PD (6) satisfy
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) \leq \epsilon \text {a n d} - V _ {g} ^ {\pi_ {t}} (\rho) \leq \epsilon \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\Omega (\cdot)$  only has some problem-dependent constant.
Corollary 3 states that the last policy iterate of RPG-PD is an  $\epsilon$ -optimal policy for Problem (1) after  $\Omega(1/\epsilon^6)$  iterations. Compared with the single-time-scale methods [23, 54], RPG-PD improves the convergence from average-value (or regret-type) to last policy iterate. Not just being theoretically stronger, the last-iterate convergence is more appealing since it captures the stability of trajectories of an algorithm [29, 40]. Compared with the two-time-scale methods [28, 24, 34, 35], RPG-PD is free of nested loops, and uniform ergodicity and exploratory initial state distribution. We notice that the dual methods [28, 24] yield history-average policies and the dual methods [34, 35] return policies from a subroutine. In contrast, RPG-PD outputs a nearly-optimal policy in the last iterate, the first-of-its-kind in the constrained MDP literature, albeit the rate is worse than the average ones [23, 54].
To get zero constraint violation, i.e.,  $V_{g}^{\pi_{t}}(\rho) \geq 0$  at some  $t$ , it is straightforward to employ a conservative constraint  $V_{g'}^{\pi}(\rho) \geq 0$  with  $g' := g - (1 - \gamma)\delta$  for some  $\delta > 0$ . When  $\epsilon$  is small enough, there always exists some  $\delta$  such that the policy iterates of RPG-PD (6) satisfy  $V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\pi_{t}}(\rho) \leq \epsilon$  and  $V_{g}^{\pi_{t}}(\rho) \geq 0$  for large  $t$ ; see Appendix C.4 for proof. Our zero constraint violation ensures the last policy iterate of RPG-PD to satisfy the constraint, which is not the zero average constraint violation in the episodic setting [55, 56]. Compared with the zero constraint violation of a policy induced by an average of past occupancy measures [42], RPG-PD's zero constraint violation directly settles the policy iterates down, which appears to be the first policy-based zero constraint violation.
Last but not least, the iteration complexity of RPG-PD is nearly-free of the MDP dimension, except for an  $\log |A|$ -term, which inherits the dimension-free property of the NPG methods [49, 57, 23]. Hence, it is ready to view RPG-PD as a variant of NPG methods and generalize RPG-PD to constrained MDPs with large state spaces in the function approximation setting.
3.3 Linear function approximation case
To deal with large state spaces, we use a parametrized policy  $\pi_{\theta}$  with  $\theta \in \mathbb{R}^d$  for RPG-PD (6) without restricting  $\Delta(A)$ , where  $d$  is much smaller than the size of state/action spaces. To introduce function approximation, we begin with a tabular softmax policy  $\pi_{\theta}(a|s) = \frac{\exp(\theta_{s,a})}{\sum_{a'} \exp(\theta_{s,a'})}$  for all  $(s,a) \in S \times A$  and  $\theta \in \mathbb{R}^{|S||A|}$ . Connecting NPG to mirror descent [49, 58], we express RPG-PD (6) as a NPG method with the following update; see Appendix C.5 for proof,
$$
\theta_ {t + 1} = \theta_ {t} + \eta (1 - \gamma) F _ {\rho} \left(\theta_ {t}\right) ^ {\dagger} \cdot \nabla_ {\theta} L _ {\tau} \left(\pi_ {\theta_ {t}}, \lambda_ {t}\right) \tag {7a}
$$
$$
\lambda_ {t + 1} = \mathcal {P} _ {\Lambda} \left((1 - \eta \tau) \lambda_ {t} - \eta V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho)\right) \tag {7b}
$$
where  $F_{\rho}(\theta_t)^\dagger \cdot \nabla_\theta L_\tau(\pi_{\theta_t}, \lambda_t)$  is a NPG direction, and  $F_{\rho}(\theta)$  is the Fisher information matrix for a policy  $\pi_\theta$ , i.e.,  $F_{\rho}(\theta) := \mathbb{E}_{s \sim d_{\rho}^{\pi_\theta}} \mathbb{E}_{a \sim \pi_\theta(\cdot|s)}[\nabla_\theta \log \pi_\theta(a|s)(\nabla_\theta \log \pi_\theta(a|s))^{\top}]$ . A useful property of (7a) is that NPG can be related to a linear regression. For any policy  $\pi_\theta$  and a state-action value function  $Q^{\pi_\theta}$ , the associated compatible function approximation error is  $E_Q(w, \theta, \nu) := \mathbb{E}_{(s,a) \sim \nu}[(w^\top \nabla \log \pi_\theta(a|s) - Q^{\pi_\theta}(s,a))^2]$ , where  $\nu(s,a) = d_{\rho}^{\pi_\theta}(s)\pi_\theta(a|s)$  is a state-action distribution. It is known that (7a) is equivalent to  $\theta_{t+1} = \theta_t + \eta w_t^\star$ , where  $w_t^\star \in \operatorname{argmin}_{w \in \mathbb{R}^d} E_Q(w, \theta_t, \nu_t)$  in which  $Q^{\pi_\theta_t}(s,a) = Q_{r + \lambda_t g + \tau \psi_t}^{\pi_\theta_t}(s,a)$  and  $\nu_t(s,a) = d_{\rho}^{\pi_\theta_t}(s)\pi_\theta_t(a|s)$  (e.g., [59, Lemma 1]). In practice, only an approximate minimizer  $w_t^\star$  is available if a sample-based algorithm is used, e.g.,  $w_t \approx \operatorname{argmin}_{||w|| \leq W} E_Q(w, \theta_t, \nu_t)$ , where  $W > 0$ .
A useful generalization of the softmax policy to large state spaces is the log-linear policy based on linear function approximation. Let  $\phi_{s,a} \in \mathbb{R}^d$  be a feature map with  $\| \phi_{s,a} \| \leq 1$  for each state-action pair  $(s, a)$ . A log-linear policy  $\pi_\theta$ :  $S \to \Delta(A)$  is parametrized by a parameter  $\theta \in \mathbb{R}^d$ ,
$$
\pi_ {\theta} (a \mid s) = \frac {\exp \left(\phi_ {s , a} ^ {\top} \theta\right)}{\sum_ {a ^ {\prime}} \exp \left(\phi_ {s , a ^ {\prime}} ^ {\top} \theta\right)} \text {f o r a l l} (s, a) \in S \times A
$$
which takes the tabular softmax policy as a special case, i.e.,  $\phi_{s,a}$  is an indicator function. We notice that  $\nabla_{\theta} \log \pi_{\theta}(a|s) = \phi_{s,a} - \mathbb{E}_{a' \sim \pi_{\theta}(\cdot|s)}[\phi_{s,a'}]$ . Since the log-linear policy is invariant to any action-independent term, it is convenient to replace  $\nabla_{\theta} \log \pi_{\theta}(a|s)$  by  $\phi_{s,a}$  and we introduce a simplified compatible function approximation error,  $\mathcal{E}_Q(w,\theta,\nu) := \mathbb{E}_{(s,a) \sim \nu}[(\phi_{s,a}^\top w - Q^{\pi_{\theta}}(s,a))^2]$ . Thus, we can take  $w_t^\star \in \mathrm{argmin}_{w \in \mathbb{R}^d} \mathcal{E}_Q(w,\theta_t,\nu_t)$  in which  $Q^{\pi_{\theta_t}}(s,a) = Q_{r + \lambda_t g + \tau \psi_t}^{\pi_{\theta_t}}(s,a)$  and  $\nu_t(s,a) = d_\rho^{\pi_{\theta_t}}(s)\pi_{\theta_t}(a|s)$  to update  $\theta_{t+1} = \theta_t + \eta w_t^\star$ . Using the log-linear policy class, we replace the primal gradient direction of RPG-PD (6) by its linear function approximation  $\phi_{s,a}^\top w_t^\star$ ,
$$
\pi_ {\theta_ {t + 1}} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \hat {\Delta} (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) \phi_ {s, a} ^ {\top} w _ {t} ^ {\star} - \frac {1}{\eta} \mathrm {K L} \left(\pi (\cdot \mid s), \pi_ {\theta_ {t}} (\cdot \mid s)\right) \right\} \tag {8}
$$
which, together with Dual update (6b), leads to a general version of RPG-PD. The set  $\hat{\Delta}(A)$  ensures bounded true gradient direction  $Q_{r + \lambda_t g + \tau \psi_t}^{\pi \theta_t}(s, a)$ . When there is no function approximation error, (8) reduces to Primal update (6a). In practice, we can only compute  $w_t^\star$  approximately via
$$
w_{t}\approx \operatorname *{argmin}_{\| w\| \leq W}\mathcal{E}_{Q}(w,\theta_{t},d_{t,\nu})
$$
which leads to an inexact RPG-PD: Primal update (8) in which  $w_{t}^{\star}$  is replaced by  $w_{t}$  and Dual update (6b), where  $d_{t,\nu} \coloneqq (1 - \gamma)\mathbb{E}_{(s_0,a_0)\sim \nu}\sum_{t = 0}^{\infty}\gamma^{t}\mathrm{Pr}(s_{t} = s,a_{t} = a|\pi_{\theta_{t}},s_{0},a_{0})$  is a state-action distribution starting from any distribution  $\nu$ . Noticeably,  $d_{t,\nu}$  is more general than  $\nu_{t}$ . To control the function approximation error, we divide  $\mathcal{E}_Q(w_t,\theta_t,d_{t,\nu})$  into a statistical error  $\mathcal{E}_Q(w_t,\theta_t,d_{t,\nu}) - \mathcal{E}_Q(w_t^\star ,\theta_t,d_{t,\nu})$  that is similar to the excess risk in supervised learning, and an approximation error  $\mathcal{E}_Q(w_t^\star ,\theta_t,d_{t,\nu})$  that captures how well a linear function  $(w_{t}^{\star})^{\top}\phi_{s,a}$  approximates the true value function under  $d_{t,\nu}$ . If the on-policy distribution  $d_{t,\nu}$  in  $\mathcal{E}_Q(w_t^\star ,\theta_t,d_{t,\nu})$  is replaced by  $\nu^{\star}(s,a) = d_{\rho}^{\pi_{\tau}^{\star}}(s)$  Unif  $A(a)$ , we define a transfer error  $\mathcal{E}_Q(w_t^\star ,\theta_t,\nu^\star)$ . Let the covariance matrix of  $\phi_{s,a}$  in any state-action distribution  $\nu$  be  $\Sigma_{\nu} \coloneqq \mathbb{E}_{(s,a)}\sim_{\nu}[\phi_{s,a}\phi_{s,a}^{\top}]$ , and the relative condition number between  $\Sigma_{\nu}$  and  $\Sigma_{\nu^{\star}}$  be  $\kappa_{\nu} \coloneqq \max_w\in \mathbb{R}^d\frac{w^\top\Sigma_\nu\star w}{w^\top\Sigma_\nu w}$ .
We make an assumption on the statistical error, the transfer error, and the relative condition number.
Assumption 2. (i) There exist  $\epsilon_{\mathrm{stat}}$ ,  $\epsilon_{\mathrm{bias}} > 0$  such that  $\mathbb{E}[\mathcal{E}_Q(w_t,\theta_t,d_{t,\nu}) - \mathcal{E}_Q(w_t^\star ,\theta_t,d_{t,\nu})]\leq \epsilon_{\mathrm{stat}}$  and  $\mathbb{E}[\mathcal{E}_Q(w_t^\star ,\theta_t,\nu^\star)]\leq \epsilon_{\mathrm{bias}}$ ; (ii) The relative condition number is finite, i.e.,  $\kappa_{\nu} < \infty$ .
We assess the convergence of inexact RPG-PD via the distance metric  $\mathbb{E}[\Phi_t] \coloneqq \mathbb{E}[\mathrm{KL}_t(\rho)] + \frac{1}{2}\mathbb{E}\left[(\lambda_\tau^\star -\lambda_t)^2\right]$ , where the expectation  $\mathbb{E}$  is over the randomness of computing  $w_{t}$  via a sample-based algorithm. We state the convergence in Theorem 4 and delay its proof to Appendix C.6.
Theorem 4 (Linear convergence of inexact RPG-PD). Let Assumptions 1-2 hold. If we take the stepsize  $\eta \leq 1 / C_W$ , then the primal-dual iterates of inexact RPG-PD satisfy
$$
\mathbb {E} [ \Phi_ {t + 1} ] \leq \mathrm {e} ^ {- \eta \tau t} \mathbb {E} [ \Phi_ {1} ] + \frac {\eta}{\tau} \max \big ((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2} \big) + \frac {2}{\tau} \left(\sqrt {| A | \epsilon_ {\mathrm {b i a s}}} + \sqrt {| A | \kappa_ {\nu} \epsilon_ {\mathrm {s t a t}}}\right)
$$
where  $C_W\coloneqq 2W / (1 - \gamma)$  and  $C_{\tau ,\xi}^{\prime}\coloneqq (1 + \tau /\xi) / (1 - \gamma)$
Theorem 4 states that the primal-dual iterates of inexact RPG-PD converge to a neighborhood of  $(\pi_{\tau}^{\star},\lambda_{\tau}^{\star})$  in a linear rate. The convergence rate is  $\eta \tau$  and the size of neighborhood scales with a sum of an  $\eta /\tau$ -term and an  $1 / \tau$ -term that amplifies the effect of function approximation ( $\epsilon_{\mathrm{stat}},\epsilon_{\mathrm{bias}}$ ). We note that, Theorem 4 does not require the strong duality in the parametrized policy class, and the function approximation error includes the duality gap caused by the inexpensiveness of function class and the policy representation error caused by the restricted policy set  $\hat{\Delta} (A)$ . When there is no function approximation error, Theorem 4 has a similar result as Theorem 2. It is important to control
$(\epsilon_{\mathrm{stat}}, \epsilon_{\mathrm{bias}})$  to be small: (i) Application of stochastic gradient methods to the linear regression leads to  $\epsilon_{\mathrm{stat}} = O(1 / \sqrt{K})$  or  $O(1 / K)$ , where  $K$  is the number of gradient steps, and thus, it is easy to control  $\epsilon_{\mathrm{stat}}$ ; (ii) When  $\epsilon_0$  is very small, the parametrized policy iterate can be contained in  $\hat{\Delta}(A)$  and thus  $\epsilon_{\mathrm{bias}}$  becomes zero in some cases, e.g., tabular softmax case [49] or low-rank MDPs [60, 61] with  $d \geq |A|$ ; it can be made very small if the function class is rich, e.g., wide neural networks [62]. When the errors are small, it is ready to establish Corollary 5; see Appendix C.7 for proof.
Corollary 5 (Nearly-optimal constrained policy). Let Assumptions 1-2 hold and  $\epsilon_{\mathrm{stat}}$ ,  $\epsilon_{\mathrm{bias}} = O(\epsilon^8)$  for small  $\epsilon$ ,  $\epsilon_0 > 0$ . If we take the stepsize  $\eta = \Theta (\epsilon^4)$  and  $\tau = \Theta (\epsilon^2)$ , then the policy iterates of inexact RPG-PD satisfy
$$
\mathbb {E} \left[ V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon a n d \mathbb {E} \left[ - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right] \leq \epsilon f o r a n y t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
Corollary 5 states that the iteration complexity in Corollary 3 holds in the function approximation case. When  $\epsilon$  is small enough, we can design a conservative constraint such that the policy iterates of inexact RPG-PD satisfy  $V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\pi_{\theta_{t}}}(\rho)\leq \epsilon$  and  $V_{g}^{\pi_{\theta_{t}}}\left(\rho\right)\geq 0$  for large  $t$ ; see Appendix C.8 for proof. Compared with the zero average constraint violation [63], this appears to be the first policy-based zero constraint violation result in the function approximation setting. Moreover, we extend inexact RPG-PD to be a sample-based algorithm and provide its sample complexity in Appendix C.9.
4 Policy Last-Iterate Convergence: Optimistic Method
Having established sublinear policy last-iterate convergence via regularization, we turn to the optimistic gradient method [37] for a faster rate. We propose an optimistic method - Optimistic Policy Gradient Primal-Dual (OPG-PD) - that converges an optimal constrained policy at a linear rate.
4.1 Optimistic policy gradient primal-dual method
We propose an optimistic policy gradient primal-dual (OPG-PD) method by maintaining two sequences for policy and dual variables each:  $\{\pi_t\}_{t\geq 1}$  and  $\{\hat{\pi}_t\}_{t\geq 1}$  for the policy-player, and  $\{\lambda_t\}_{t\geq 1}$  and  $\{\hat{\lambda}_t\}_{t\geq 1}$  for the dual-player,
$$
\begin{array}{l} \pi_ {t} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \Delta (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a) - \frac {1}{2 \eta} \| \pi (\cdot \mid s) - \hat {\pi} _ {t} (\cdot \mid s) \| ^ {2} \right\} (9a) \\ \hat {\pi} _ {t + 1} (\cdot \mid s) = \underset {\pi (\cdot \mid s) \in \Delta (A)} {\operatorname {a r g m a x}} \left\{\sum_ {a} \pi (a \mid s) Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) - \frac {1}{2 \eta} \| \pi (\cdot \mid s) - \hat {\pi} _ {t} (\cdot \mid s) \| ^ {2} \right\}, (9a) \\ \lambda_ {t} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda V _ {g} ^ {\pi_ {t - 1}} (\rho) + \frac {1}{2 \eta} \left(\lambda - \hat {\lambda} _ {t}\right) ^ {2} \right\} (9b) \\ \hat {\lambda} _ {t + 1} = \underset {\lambda \in \Lambda} {\operatorname {a r g m i n}} \left\{\lambda V _ {g} ^ {\pi_ {t}} (\rho) + \frac {1}{2 \eta} (\lambda - \hat {\lambda} _ {t}) ^ {2} \right\} \\ \end{array}
$$
where  $\eta$  is the stepsize and  $(\hat{\pi}_0,\hat{\lambda}_0) = (\pi_0,\lambda_0)\in \Pi \times \Lambda$  is the initial point. OPG-PD concurrently works with two primal iterates and two dual iterates, and each two are updated consecutively to stabilize the algorithm dynamics. The "optimistic" in optimization, e.g., [64] views  $(\hat{\pi}_{t + 1},\hat{\lambda}_{t + 1})$  -update as a real policy gradient step and  $(\pi_t,\lambda_t)$  -update as a prediction step that generates an intermediate iterate  $(\pi_{t},\lambda_{t})$ . Not policy gradient at  $(\hat{\pi}_t,\hat{\lambda}_t)$ , the real step uses a policy gradient at  $(\pi_t,\lambda_t)$  from prediction, exhibiting the optimism towards the prediction. Specifically, Primal update (9a) works as the projected  $Q$  -ascent [58, 65], an application of the classical mirror descent with Euclidean distance [53], where the projection onto a probability simplex can be solved efficiently [66]. Dual update (9b) performs standard projected gradient descent. We note that OPG-PD is different from the one-step multiplicative weights update in the policy-based ReLOAD [40].
When there is no MDP transition dynamics, i.e., constrained bandit [40], last-iterate convergence of OPG-PD to a saddle point is known in the minimax optimization [67, 68, 38, 39], because Problem (3) reduces to a bilinear zero-sum game in this case. However, it is prohibitive to apply such
bilinear game results to the Lagrangian  $V_{r + \lambda g}^{\pi}(s)$  in every state  $s$ , as has been done for zero-sum Markov games [43, 44]. The main reason for this is that there may not exist an optimal constrained policy that is uniformly optimal across all states; see Appendix B.2.
4.2 Policy last-iterate convergence
We define the distribution mismatch coefficient over  $\rho$  as  $\kappa_{\rho} \coloneqq \sup_{\pi \in \Pi} \| d_{\rho}^{\pi} / \rho \|_{\infty}$ , which is the maximum distribution mismatch of policy  $\pi$  relative to  $\rho$ , where  $d_{\rho}^{\pi} / \rho$  is divided per state. Hence,  $\left\| d_{\rho}^{\pi} / d_{\rho}^{\pi^{*}} \right\|_{\infty} \leq \kappa_{\rho} / (1 - \gamma)$  for any policy  $\pi \in \Pi$  and  $\kappa_{\rho} \leq 1 / \rho_{\min}$  where  $\rho_{\min} \coloneqq \min_{s} \rho(s)$ . The projection operator  $\mathcal{P}_X$  on a closed convex set  $X$  defines  $\mathcal{P}_X(x) \coloneqq \operatorname{argmin}_{x' \in X} \| x' - x \|$ .
We state the policy last-iterate convergence of OPG-PD (9) in Theorem 6.
Theorem 6 (Linear convergence of OPG-PD). Let Assumption 1 hold. Assume the optimal state visitation distribution be unique, i.e.,  $d_{\rho}^{\pi} = d_{\rho}^{\pi^{\star}}$  for any  $\pi \in \Pi^{\star}$ , and  $\rho_{\mathrm{min}} > 0$ . If we set the stepsize  $\eta \leq \min \left(1 / (4\sqrt{\iota}), (1 - \gamma)^3 / (4|A|), (1 - \gamma)^3 / (2\kappa_{\rho})\right)$ , where  $\iota > 0$  is defined in Appendix D.1, then the primal-dual iterates of OPG-PD (9) satisfy
$$
\frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t} (\cdot | s)) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \frac {1}{2} (\mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t}) - \hat {\lambda} _ {t}) ^ {2} \leq \left(\frac {1}{1 + C}\right) ^ {t}
$$
where  $C \coloneqq \min(7(1 - \gamma)/8, 7\eta^2(1 - \gamma)^2(C_{\rho,\xi})^2\rho_{\min} / (6\kappa_{\rho,\gamma})^2)$  in which  $C_{\rho,\xi}$  and  $\kappa_{\rho,\gamma}$  are given by  $C_{\rho,\xi} \coloneqq c\rho_{\min} / (2\sqrt{|S||A|}) / (1 + 1 / ((1 - \gamma)\xi))$ ,  $\kappa_{\rho,\gamma} \coloneqq \max(\kappa_{\rho} / (1 - \gamma), 1)$ , and  $c > 0$  is a problem-dependent constant from Lemma 26.
Theorem 6 states that the primal-dual iterates of OPG-PD converge to  $\Pi^{\star} \times \Lambda^{\star}$  in a linear rate, or putting it differently, (9) is contracting to a set of optimal primal/dual variables. The rate is governed by a problem-dependent constant. Proof of Theorem 6 is provided in Appendix D. A key to our analysis is to bridge the per-state policy gradient update and the policy improvement for  $V_{r + \lambda g}^{\pi}(\rho)$  that is non-convex in policy  $\pi$ , which departs from the convex last-iterate analysis [38, 39]. In addition, we address two technical difficulties. First, the lack of uniformly optimal policies prevents learning an optimal policy from per-state bilinear games in zero-sum Markov games [43, 44]. Instead, we characterize the proximity of primal-dual iterates to a saddle point supported by an optimal state visitation distribution  $d_{\rho}^{\pi^*}$ . Second, Problem (3) is an asymmetric game since one plays a stochastic policy over a finite set of discrete actions and controls the transition dynamics, but the other selects an action in a continuous interval. Thus, our dual-player analysis handles the long-term effect of the policy-player, which did not appear in the symmetric game [43, 44].
A direct corollary of Theorem 6 is stated below; see Appendix D.3 for the proof.
Corollary 7 (Nearly-optimal constrained policy). Let Assumption 1 hold and the optimal state visitation distribution be unique, i.e.,  $d_{\rho}^{\pi} = d_{\rho}^{\pi^{\star}}$  for any  $\pi \in \Pi^{\star}$ , and  $\rho_{\mathrm{min}} > 0$ . If we use the stepsize  $\eta$  from Theorem 6, then the policy iterates of OPG-PD (9) satisfy
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\hat {\pi} _ {t}} (\rho) \leq \epsilon a n d - V _ {g} ^ {\hat {\pi} _ {t}} (\rho) \leq \epsilon f o r a n y t = \Omega \left(\log^ {2} \frac {1}{\epsilon}\right)
$$
Corollary 7 states that the last policy iterate of OPG-PD is an  $\epsilon$ -optimal policy for Problem (1) after an almost constant number of iterations, which improves the sublinear rate in Corollary 3. OPG-PD also improves the average-value convergence of the single-time-scale methods [23, 54] and the two-time-scale methods [28, 24, 34, 35], and matches the last-iterate convergence rate of the two-time-scale methods [34, 35]. We stress that our last-iterate convergence indicates the stability of whole primal-dual iterates, which is not the last policy iterate from a subroutine [34, 35]. Again, when  $\epsilon$  is small, we can design a conservative constraint such that the policy iterates of OPG-PD satisfy  $V_{r}^{\pi^{*}}(\rho) - V_{r}^{\hat{\pi}_{t}}(\rho) \leq \epsilon$  and  $V_{g}^{\hat{\pi}_{t}}(\rho) \geq 0$  for large  $t$ ; see Appendix D.4 for the proof.
5 Computational Experiment
We validate the effectiveness of RPG-PD (6) and OPG-PD (9) by comparing them with typical primal-dual methods in Figure 1. A few observations are in order. The initial oscillation of RPG-PD