SlowGuess/NeurIPS_2023_OCR huggingface dataset:
dual gradient method [110], Douglas-Rachford alternating direction method [111], inexact augmented Lagrangian method [112, 113], alternating proximal augmented Lagrangian algorithm [114], augmented Lagrangian-based decomposition method [115], faster Lagrangian method [116], and prediction-correction-based primal-dual method [117]. However, all these studies build on augmented Lagrangian methods to solve the classical convex optimization problems with linear constraints. In comparison, we have studied a class of non-convex constrained optimization problems that result from constrained MDPs using the standard Lagrange multiplier method. We notice that a Lagrangian-based two-player game has been used to study general non-convex constrained optimization problems with average performance analysis [118]. Our two policy-based primal-dual methods with sublinear and linear last-iterate convergence appear to be the first global non-asymptotic and last-iterate convergence result in non-convex constrained optimization.
B Proofs in Section 2
In this section, we make some helpful observations and provide proofs of the claims in Section 2.
B.1 Lack of uniformly optimal stationary policies in constrained MDPs
In unconstrained MDPs, there always exists an optimal policy that is optimal simultaneously for all states (e.g., see [119] and [120, Chapter 6]). In contrast, this is not true anymore for constrained MDPs. To see this, we adopt a counter-example from [46] and investigate it in the constrained MDP setting.
Figure 3: An example of a constrained MDP that has the objective function  $V_{r}^{\pi}(\rho)$  and the constraint set  $\{\pi \in \Pi | V_{g}^{\pi}(\rho) \geq 0\}$ . The pair  $(a, r, g)$  associated with a directed arrow represents (reward, utility) received when an action  $a$  at a certain state is taken.
We introduce a constrained MDP with two states: Left  $(L)$  and Right  $(R)$ , in Figure 3. In each state, there are two actions  $\{L,R\}$ . The MDP transition dynamics is deterministic. In state  $L$ , if the agent chooses action  $L$ , then the next state is  $L$  and the reward/utility  $(0,0)$  is received; otherwise, action  $R$  leads to next state  $R$  and reward/utility  $(1,-1)$ . In state  $R$ , no matter which action the agent takes, the next state is  $R$  and the reward/utility  $(1,1)$  is received.
Since the state  $R$  is trivial, a stationary policy  $\pi$  can be represented by the probability of taking action  $L$  in state  $L$  denoted by  $p$ . With a slight abuse of notation, we use notion  $\rho$  to represent the probability of starting off from state  $L$ . Thus, we can compute the value functions as follows via the Bellman equations, i.e.,  $V(s) = \sum_{a} \pi(a|s)(r(s,a) + \gamma \sum_{s'} P(s'|s,a)V(s))$  for all  $s$ .
$$
V _ {r} (L) = \underbrace {p \times (0 + \gamma (V _ {r} (L) \times 1 + V _ {r} (R) \times 0))} _ {\text {t a k e a c t i o n L}} + \underbrace {(1 - p) (1 + \gamma (V _ {r} (L) \times 0 + V _ {r} (R) \times 1))} _ {\text {t a k e a c t i o n R}}
$$
$$
V _ {r} (R) = \frac {1}{1 - \gamma}
$$
$$
V _ {g} (L) = \underbrace {p \times (0 + \gamma (V _ {g} (L) \times 1 + V _ {g} (R) \times 0))} _ {\text {t a k e a c t i o n L}} + \underbrace {(1 - p) (- 1 + \gamma (V _ {g} (L) \times 0 + V _ {g} (R) \times 1))} _ {\text {t a k e a c t i o n R}}
$$
$$
V _ {g} (R) = \frac {1}{1 - \gamma}
$$
or, equivalently,
$$
V _ {r} (L) = \frac {1 - p}{1 - \gamma p} \times \frac {1}{1 - \gamma}, V _ {r} (R) = \frac {1}{1 - \gamma}
$$
$$
\mathrm {a n d} V _ {g} (L) = \frac {1 - p}{1 - \gamma p} \times \frac {2 \gamma - 1}{1 - \gamma}, V _ {g} (R) = \frac {1}{1 - \gamma}.
$$
(i) It is easy to check a basic case:  $\gamma = 0$  [46]. We can compute the value functions as follows,
$$
V _ {r} ^ {p} (\rho) = (1 - p) \rho + (1 - \rho)
$$
$$
V _ {g} ^ {p} (\rho) = - (1 - p) \rho + (1 - \rho).
$$
Feasibility of the policy  $p$  requires that  $V_{g}^{p}(\rho)\geq 0$ , i.e.,
$$
p \geq \frac {2 \rho - 1}{\rho} \text {f o r a n y} \rho \in \left[ \frac {1}{2}, 1 \right]. \tag {10}
$$
Hence, the maximum  $V_{r}^{p}(\rho)$  within the feasible region can be reached at the optimal policy,
$$
p ^ {\star} = \frac {2 \rho - 1}{\rho}.
$$
Therefore, the optimal policy  $p^{\star}$  depends on the initial state distribution  $\rho$ . Moreover, except that  $\rho = 1$  or  $\frac{1}{2}$ , the optimal policy  $p^{\star}$  is a stochastic policy and is unique.
(ii) A slightly more general case is given by  $\gamma = \frac{1}{4}$ . Thus, we can compute the value functions as follows,
$$
V _ {r} ^ {p} (\rho) = \frac {4}{3} \times \frac {1 - p}{1 - p / 4} \times \rho + \frac {4}{3} \times (1 - \rho)
$$
$$
V _ {g} ^ {p} (\rho) = - \frac {2}{3} \times \frac {1 - p}{1 - p / 4} \times \rho + \frac {4}{3} \times (1 - \rho).
$$
Feasibility of the policy  $p$  requires that  $V_{g}^{p}(\rho) \geq 0$ , i.e.,
$$
p \geq 2 \times \frac {3 \rho - 2}{3 \rho - 1} \text {f o r a n y} \rho \in \left(\frac {1}{3}, 1 \right].
$$
In particular, if we take  $\rho = \frac{7}{9}$ , then  $p \geq \frac{1}{2}$ . In this case, the maximization of  $V_r^p(\rho)$  yields an optimal policy  $p^\star = \frac{1}{2}$ , which is a uniform policy and is unique.
B.2 Scalarization fallacy in constrained MDPs
In constrained RL, scalarization is often used to reduce a constrained MDP problem to a standard unconstrained one, which might permit many unconstrained RL algorithms [8, 9]. Unfortunately, as pointed out in the literature (e.g., [46, Part 4] and [30]), such a reduction does not necessarily provide a solution to the original constrained MDP problem. It is easy to see this from the previous examples in Figure 3. In the basic case:  $\gamma = 0$ , if we take  $\rho = \frac{3}{4}$ , then from (10) the optimal policy is given by  $p^{\star} = \frac{2}{3}$ , which is a stochastic policy; we see a uniform optimal policy when  $\gamma = \frac{1}{4}$ . By shaping a composite function  $r + \lambda g$  with some fixed  $\lambda \in [0,\infty]$ , the scalarization method aims to solve the following unconstrained MDP problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r + \lambda g} ^ {\pi} (\rho). \tag {11}
$$
However, by the optimality of dynamic programming [120, Chapter 6], an optimal policy is given in a deterministic form which has been widely used in theory and practice. Therefore, solving the above scalarized version of a constrained MDP problem does not necessarily provide an optimal solution for the original constrained MDP problem. We also notice that this phenomenon is reported in recent empirical studies [9, 30] and a more formal statement [47, Lemma 1]. Hence, it can be infeasible for dual descent methods [12, 30] to find an optimal constrained policy, because solving Problem (11) often yields a deterministic policy, which can be sub-optimal for a constrained MDP with a unique stochastic optimal policy, e.g., constrained MDP examples in Appendix B.1.
B.3 Properties of saddle points
First, we state the invariance property of saddle points [121] for our constrained saddle-point problem (3). By the invariance property of saddle points, we can restrict the problem domain without changing the saddle-point property when the original saddle points are contained in the restricted domain. Let the set of max-min points be  $\Pi^{\star} := \operatorname{argmax}_{\pi \in \Pi} \min_{\lambda \in [0,\infty]} V_{r + \lambda g}^{\pi}(\rho)$  and the set of min-max points be  $\Lambda^{\star} := \operatorname{argmin}_{\lambda \in [0,\infty]} \max_{\pi \in \Pi} V_{r + \lambda g}^{\pi}(\rho)$ .
Lemma 8 (Invariance of saddle points). Let  $(\pi^{\star},\lambda^{\star})\in \Pi^{\star}\times \Lambda^{\star}$  be a saddle point of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times [0, + \infty ]$  . For any subset  $\Lambda^{\prime}\subset [0, + \infty ]$  , if  $(\pi^{\star},\lambda^{\star})\in \Pi \times \Lambda^{\prime}$  , then  $(\pi^{\star},\lambda^{\star})$  is a saddle point of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times \Lambda^{\prime}$
Proof. From the saddle-point property of  $(\pi^{\star},\lambda^{\star})$ , we have
$$
\pi^{\star}\in \operatorname *{argmax}_{\pi \in \Pi}V_{r + \lambda^{*}g}^{\pi}(\rho)\text{and}\lambda^{\star}\in \operatorname *{argmin}_{\lambda \in [0, + \infty ]}V_{r + \lambda g}^{\pi^{\star}}(\rho)
$$
It is straightforward to see that
$$
V _ {r + \lambda^ {\star} g} ^ {\pi} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho) \text {f o r a n y} \pi \in \Pi . \tag {12}
$$
Since  $\Lambda' \subset [0, +\infty]$  and  $\lambda^{\star} \in \Lambda'$ ,  $V_{r + \lambda^{\star}g}^{\pi^{\star}}(\rho) = \min_{\lambda \in \Lambda'} V_{r + \lambda g}^{\pi^{\star}}(\rho)$ . Hence,
$$
V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \lambda g} ^ {\pi^ {\star}} (\rho) \text {f o r a n y} \lambda \in \Lambda^ {\prime}. \tag {13}
$$
Finally, combining (12) and (13) defines  $(\pi^{\star},\lambda^{\star})$  as a saddle point of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times \Lambda^{\prime}$ .
We next show the interchangeability of saddle points in two-player zero-sum games [122] for our non-convex game.
Lemma 9 (Interchangeability of saddle points). Let  $(\pi^{\star}, \lambda^{\star})$ ,  $(\bar{\pi}^{\star}, \bar{\lambda}^{\star}) \in \Pi^{\star} \times \Lambda^{\star}$  be two saddle points of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times [0, +\infty]$ . Then, both  $(\pi^{\star}, \bar{\lambda}^{\star})$  and  $(\bar{\pi}^{\star}, \lambda^{\star})$  are saddle points of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times [0, +\infty]$ .
Proof. By the definition of saddle points  $(\pi^{\star},\lambda^{\star})$  and  $(\bar{\pi}^{\star},\bar{\lambda}^{\star})$
$$
V _ {r + \lambda^ {\star} g} ^ {\pi} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \lambda g} ^ {\pi^ {\star}} (\rho) \text {f o r a l l} \pi \in \Pi \text {a n d} \lambda \in [ 0, \infty ]
$$
$$
V _ {r + \bar {\lambda} ^ {*} g} ^ {\pi} (\rho) \leq V _ {r + \bar {\lambda} ^ {*} g} ^ {\bar {\pi} ^ {\star}} (\rho) \leq V _ {r + \lambda g} ^ {\bar {\pi} ^ {\star}} (\rho) \text {f o r a l l} \pi \in \Pi \text {a n d} \lambda \in [ 0, \infty ].
$$
$$
V _ {r + \bar {\lambda} ^ {\star} g} ^ {\bar {\pi} ^ {\star}} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\bar {\pi} ^ {\star}} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho)
$$
$$
V _ {r + \bar {\lambda} ^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \bar {\lambda} ^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \bar {\lambda} ^ {\star} g} ^ {\bar {\pi} ^ {\star}} (\rho).
$$
$$
V _ {r + \bar {\lambda} ^ {\star} g} ^ {\pi} (\rho) \leq V _ {r + \bar {\lambda} ^ {\star} g} ^ {\bar {\pi} ^ {\star}} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \bar {\lambda} ^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \bar {\lambda} ^ {\star} g} ^ {\bar {\pi} ^ {\star}} (\rho) \leq V _ {r + \lambda^ {\star} g} ^ {\pi^ {\star}} (\rho) \leq V _ {r + \lambda g} ^ {\pi^ {\star}} (\rho)
$$
for all  $\pi \in \Pi$  and  $\lambda \in [0,\infty]$ , which shows that  $(\pi^{\star},\bar{\lambda}^{\star})$  is a saddle point of  $V_{r + \lambda g}^{\pi}(\rho)$  over  $\Pi \times [0, + \infty ]$ .
Similarly, we can prove it for  $(\bar{\pi}^{\star},\lambda^{\star})$
B.4 Constrained MDPs in occupancy-measure space
In the analytic approach [123, 5], the value functions in Problem (1) are bilinear in the occupancy measure  $V_{\diamond}^{\pi}(\rho) = \langle \diamond ,q\rangle$  for  $\diamond = r$  or  $u$ , where  $q^{\pi}\colon S\times A\to \mathbb{R}$  is an (un-normalized) occupancy measure over the state-action space,
$$
q ^ {\pi} (s, a) = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \Pr \left(s _ {t} = s, a _ {t} = a \mid \pi , s _ {0} \sim \rho\right) \tag {14}
$$
which adds up discounted probabilities of visiting  $(s, a)$  in the execution of  $\pi$ . Furthermore, we let the operator  $P^{\top} \colon \mathbb{R}^{|S||A|} \to \mathbb{R}^{|S|}$  be  $(P^{\top}q)(s) \coloneqq \sum_{s', a'} P(s|s', a')q(s', a')$ , and the operator  $E^{\top} \colon \mathbb{R}^{|S||A|} \to \mathbb{R}^{|S|}$  be  $(E^{\top}q)(s) \coloneqq \sum_{a} q(s, a)$  or simply  $q(s)$ . With a slight abuse of notation, we denote  $\rho = [\rho(s_1), \ldots, \rho(s_{|S|})]^{\top}$ . A valid occupancy measure  $q^{\pi} \in \mathbb{R}^{|S||A|}$  satisfies the Bellman flow equations,
$$
\mathcal {Q} := \left\{q \in \mathbb {R} ^ {| S | | A |} \mid E ^ {\top} q = \gamma P ^ {\top} q + \rho \text {a n d} q \geq 0 \right\}. \tag {15}
$$
It is known that the Bellman flow constraint is necessary and sufficient for any  $q \in \mathbb{R}^{|S||A|}$  to be a valid occupancy measure (e.g., [124, Lemma 1]).
Let the concatenation of  $r(s, a)$ ,  $u(s, a)$  for all  $(s, a)$  be  $r$ ,  $u \in \mathbb{R}^{|S||A|}$ , respectively. In the occupancy measure space, the goal of a constrained MDP is to find a solution  $q^\star$  of a linear program,
$$
\underset {q \in \mathcal {Q}} {\text {m a x i m i z e}} \langle r, q \rangle \quad \text {s u b j e c t} \quad \langle u, q \rangle \geq b. \tag {16}
$$
Denoting  $g \colon S \times A \to [-1,1]$  as  $g \coloneqq u - (1 - \gamma)b$ , we simplify the constraint  $\langle u,q \rangle \geq b$  as  $\langle g,q \rangle \geq 0$ . By the method of Lagrange multipliers, we dualize two constraints in (16) and introduce a standard Lagrangian,
$$
L (q, \lambda , \mu) := \langle r + \lambda g, q \rangle + \mu^ {\top} \left(\rho - (E - \gamma P) ^ {\top} q\right)
$$
where  $\pi \in \Pi$  is the primal variable,  $\lambda \in [0,\infty]$  is the dual variable for the constraint  $\langle g,q\rangle \geq 0$ , and  $\mu$  is the dual variable for the equality constraint in  $\mathcal{Q}$ . Since the strong duality holds for any feasible linear program, the boundedness of  $\lambda^{\star}$  in Lemma 1 holds for  $L(q,\lambda ,\mu)$ . Thus, any saddle point  $(q^{\star},\lambda^{\star},\mu^{\star})$  is also a max-min and min-max point, i.e.,  $q^{\star}$  is the occupancy measure associated with the optimal policy  $\pi^{\star}$ , and  $(\lambda^{\star},\mu^{\star})\in \mathrm{argmin}_{\lambda \geq 0,\mu}\max_{q\geq 0}L(q,\lambda ,\mu)$ . Boundedness of  $q^{\star}$  is straightforward from (14),
$$
q ^ {\star} \in Q := \left\{q \in \mathbb {R} ^ {| S | | A |} \mid 0 \leq q (s, a) \leq \frac {1}{1 - \gamma}, \forall (s, a) \in S \times A \right\}
$$
which allows us further restrict  $q \in Q \subset \mathcal{Q}$ . We next show boundedness of  $(\lambda^{\star}, \mu^{\star})$  in Lemma 10.
Lemma 10 (Boundedness). Let Assumption 1 hold. Then,  $\lambda^{\star} \in \Lambda$  and  $\mu^{\star} \in M$ , where  $\Lambda$  is stated below (3) and  $M := \{\mu \mid |\mu(s)| \leq \mu_{\max}, \forall s \in S\}$  where  $\mu_{\max} := \frac{1 - \gamma + 1 / \xi}{(1 - \gamma)^2}$ .
Proof. From the saddle-point property of  $(q^{\star},\lambda^{\star},\mu^{\star})$  , we have
$$
q^{\star}\quad \in \operatorname *{argmax}_{q\in Q}L(q,\lambda^{\star},\mu^{\star})
$$
$$
(\lambda^ {\star}, \mu^ {\star}) \quad \in \underset {\lambda \geq 0, \mu} {\operatorname {a r g m i n}} L (q ^ {\star}, \lambda , \mu)
$$
equivalently, for any  $q \in Q$ ,  $\partial_q L(q^\star, \lambda^\star, \mu^\star)^\top (q - q^\star) \leq 0$ , and for any  $\mu$  and  $\lambda \geq 0$ ,  $\partial_\mu L(q^\star, \lambda^\star, \mu^\star)^\top (\mu - \mu^\star) + \partial_\lambda L(q^\star, \lambda^\star, \mu^\star)(\lambda - \lambda^\star) \geq 0$ . Hence, for any  $q \in Q$ ,
$$
\langle r + (\gamma P - E) \mu^ {\star} + \lambda^ {\star} g, q - q ^ {\star} \rangle \leq 0.
$$
Arbitrary  $q \in Q$  demands the inequality  $r + (\gamma P - E)\mu^{\star} + \lambda^{\star}g \leq 0$ . By the definition of occupancy measure, we know that for any  $s \in S$  there exists an action  $a \in A$  such that  $q^{\star}(s,a) > 0$ . Thus, the equality  $r + (\gamma P - E)\mu^{\star} + \lambda^{\star}g = 0$  must hold at such state-action pairs in which we represent the associated reward and transition by  $(\bar{r},\bar{P})$ . Hence,
$$
\begin{array}{l} \left\| \bar {r} + \lambda^ {\star} g \right\| _ {\infty} = \left\| (\gamma \bar {P} - \bar {E}) \mu^ {\star} \right\| _ {\infty} \\ \geq (1 - \gamma) \| \mu^ {\star} \| _ {\infty}. \\ \end{array}
$$
Thus,  $(1 - \gamma)\| \mu^{\star}\|_{\infty}\leq 1 + \lambda^{\star}$  . However, by Assumption 1,  $L(\bar{q},\lambda^{\star},\mu^{\star})\leq L(q^{\star},\lambda^{\star},\mu^{\star})$  . Hence,
$$
\langle r, q ^ {\star} - \bar {q} \rangle + \lambda^ {\star} \langle g, q ^ {\star} - \bar {q} \rangle \geq 0
$$
which, together with the feasibility of  $\bar{q}$  and the optimality of  $q^{\star}$ , imply that  $0\leq \lambda^{\star}\xi \leq \langle r,q^{\star} - \bar{q}\rangle$  Hence,  $0\leq \lambda^{\star}\leq \frac{1}{(1 - \gamma)\xi}$ , which further yields a bound on  $\| \mu^{\star}\|_{\infty}$ .
We now obtain a constrained saddle-point problem in terms of the  $q$ -based Lagrangian,
$$
\underset {q \in Q} {\text {m a x i m i z e}} \underset {\lambda \in \Lambda , \mu \in M} {\text {m i n i m i z e}} L (q, \lambda , \mu) = \underset {\lambda \in \Lambda , \mu \in M} {\text {m i n i m i z e}} \underset {q \in Q} {\text {m a x i m i z e}} L (q, \lambda , \mu) \tag {17}
$$
where we take bounded polytopes  $Q$  and  $\Lambda \times M$  such that they contain  $q^{\star}$  and  $(\lambda^{\star},\mu^{\star})$ , respectively.
For notational brevity, we introduce  $z = (q,\lambda ,\mu)$ $Z\coloneqq Q\times \Lambda \times M$  , and  $Z^{\star}\coloneqq Q^{\star}\times \Lambda^{\star}\times M^{\star}$  which contains all saddle points  $z^{\star}\coloneqq (q^{\star},\lambda^{\star},\mu^{\star})$  . Let the gradient of  $L(q,\lambda ,\mu)$  be
$$
F (q, \lambda , \mu) := \left[ \begin{array}{c} - \nabla_ {q} L (q, \lambda , \mu) \\ \nabla_ {(\lambda , \mu)} L (q, \lambda , \mu) \end{array} \right].
$$
Due to the bilinearity of  $L(q, \lambda, \mu)$  over a compact domain,  $L(q, \lambda, \mu)$  has a gradient Lipschitz constant  $L_{f}$ . Let  $\mathcal{P}_X$  be the projection operator onto a set  $X$ , i.e.,  $\mathcal{P}_X(x) := \operatorname*{argmin}_{x' \in X} \| x' - x \|$ . Since the  $q$ -based Lagrangian  $L(q, \lambda, \mu)$  is bilinear and the domains are polytopes, Problem (17) satisfies the metric subregularity condition [38, Theorem 5].
Lemma 11 (Metric subregularity condition). Let Assumption 1 hold. Then, the gradient function  $F(z)$  satisfies that for any  $z \in Z / Z^{\star}$  with  $z^{\star} = \mathcal{P}_{Z^{\star}}(z)$ ,
$$
\sup _ {z ^ {\prime} \in Z} \frac {F (z) ^ {\top} (z - z ^ {\prime})}{\| z - z ^ {\prime} \|} \geq C \| z - z ^ {\star} \|
$$
where  $C > 0$  is a problem-dependent constant.
Hence, application of the optimistic gradient method [38] to Problem (17) yields an Optimistic Primal-Dual (OPD) algorithm that begins with two initial tuples of primal/dual variables  $(q_0,\lambda_0,\mu_0) = (\hat{q}_1,\hat{\lambda}_1,\hat{\mu}_1)\in Z$ , and performs two gradient steps for each primal/dual variable at time  $t\geq 1$
$$
z _ {t} = \mathcal {P} _ {Z} \left(\hat {z} _ {t} - \eta F \left(z _ {t - 1}\right)\right) \tag {18}
$$
$$
\hat {z} _ {t + 1} = \mathcal {P} _ {Z} (\hat {z} _ {t} - \eta F (z _ {t}))
$$
where  $\eta$  is the stepsize. Let the squared distance of a point  $z\in Z$  to the set  $Z^{\star}$  be  $\mathrm{dist}^2 (z,Z^\star)\coloneqq \| z - \mathcal{P}_{Z^\star}(z)\|^2$ . It is straightforward to employ [38, Theorem 8] to claim last-iterate convergence guarantee of OPD (18) below.
Theorem 12 (Linear convergence of OPD). Let Assumption 1 hold. If the stepsize  $\eta$  in OPD (18) satisfies  $\eta < \frac{1}{8L_f}$ , then the iterates  $\{z_t\}_{t \geq 0}$  converge to the set of saddle points  $Z^{\star}$  linearly,
$$
\operatorname {d i s t} ^ {2} \left(z _ {t}, Z ^ {\star}\right) \leq C _ {1} \left(\frac {1}{1 + C _ {2}}\right) ^ {t}
$$
where  $C_1 = 64\mathrm{dist}^2 (\hat{z}_1,Z^\star)$  and  $C_2 = \min \left(\frac{16\eta^2C^2}{81},\frac{1}{2}\right)$  for a problem-dependent constant  $C > 0$
Theorem 12 shows that the primal-dual iterates of OPD converge to the saddle point set  $Z^{\star}$  in linear rate. Compared with a contemporaneous work [40], OPD is free of projection to a occupancy measure set, and enjoys non-asymptotic and last-iterate linear convergence. If the underlying policy is recovered via  $\pi_t(a|s) = \frac{q_t(s,a)}{\sum_{a'} q_t(s,a')}$  for all  $(s,a)$ , these policy iterates  $\pi_t$  associated with the occupancy measure iterates  $q_t$  also converge to an optimal constrained policy  $\pi^{\star}$ .
Corollary 13 (Nearly-optimal constrained policy). Let Assumption 1 hold. Assume  $\rho_{\mathrm{min}} := \min_s \rho(s) > 0$  and re-define  $Q := \{q \in \mathbb{R}^{|S| |A|} \mid 0 \leq q(s, a) \leq 1 / (1 - \gamma), q(s) \geq \rho_{\mathrm{min}} / (1 - \gamma), \forall (s, a) \in S \times A\}$ . If we set the stepsize  $\eta$  in a similar way as in Theorem 12, then the recovered policy iterates  $\{\pi_t\}_{t \geq 0}$  of OPD (18) satisfy
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) \leq \epsilon a n d - V _ {g} ^ {\pi_ {t}} (\rho) \leq \epsilon f o r a n y t = \Omega \left(\log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\pi_t(a|s) = \frac{q_t(s,a)}{\sum_{a'} q_t(s,a')}$  for all  $(s,a)$ , and  $\Omega(\cdot)$  only has some problem-dependent constant.
Proof. The key to our proof is to connect the occupancy measure iterates  $q_{t}$  with associated policy iterates  $\pi_t$ . It is straightforward that Theorem 12 continues to hold, with a further restricted the domain  $Q$ . If we set the stepsize  $\eta$  in a similar way as in Theorem 12, then for any  $t = \Omega (\log \frac{1}{\epsilon})$ ,
$$
\max  \left\{\| \mathcal {P} _ {Q ^ {\star}} (q _ {t}) - q _ {t} \| ^ {2}, \| \mathcal {P} _ {\Lambda^ {\star}} (\lambda_ {t}) - \lambda_ {t} \| ^ {2}, \| \mathcal {P} _ {M ^ {\star}} (\mu_ {t}) - \mu_ {t} \| ^ {2} \right\} = O (\epsilon).
$$
Let  $\pi_t^\star$  be a policy that is associated with the occupancy measure  $q^{\pi_t^\star} = \mathcal{P}_{Q^\star}(q_t)$  and  $\pi_t$  be a policy that is associated with the occupancy measure iterate  $q_{t}$ , i.e.,  $\pi_t(a|s) = \frac{q_t(s,a)}{\sum_{a'} q_t(s,a')}$  for all  $(s,a)$ . We denote  $(\lambda_t^\star, \mu_t^\star) \coloneqq (\mathcal{P}_{\Lambda^\star}(\lambda_t), \mathcal{P}_{M^\star}(\mu_t))$ . Thus,
$$
\begin{array}{l} \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \left\| \pi_ {t} (\cdot | s) - \pi_ {t} ^ {\star} (\cdot | s) \right\| \\ = \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \left\| \frac {q _ {t} (s , \cdot)}{q _ {t} (s)} - \frac {q ^ {\pi_ {t} ^ {\star}} (s , \cdot)}{q ^ {\pi_ {t} ^ {\star}} (s)} \right\| \\ \leq \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \frac {\left\| q _ {t} (s , \cdot) - q _ {t} ^ {\star} (s , \cdot) \right\| q _ {t} (s)}{q _ {t} (s) q _ {t} ^ {\star} (s)} + \sum_ {s} d _ {\rho} ^ {\pi_ {t} ^ {\star}} (s) \frac {\left\| q _ {t} (s , \cdot) \right\| | q _ {t} (s) - q _ {t} ^ {\star} (s) |}{q _ {t} (s) q _ {t} ^ {\star} (s)} \\ \leq \sum_ {s} \frac {\left\| q _ {t} (s , \cdot) - q ^ {\pi_ {t} ^ {\star}} (s , \cdot) \right\|}{q ^ {\pi_ {t} ^ {\star}} (s)} + \sqrt {| A |} \sum_ {s} \frac {\left| q _ {t} (s) - q ^ {\pi_ {t} ^ {\star}} (s) \right|}{q _ {t} (s)} \\ \leq \frac {\sqrt {| A |}}{\rho_ {\min }} \sum \left(\left\| q _ {t} (s, \cdot) - q ^ {\pi_ {t} ^ {\star}} (s, \cdot) \right\| + | q _ {t} (s) - q ^ {\pi_ {t} ^ {\star}} (s) |\right) \\ \leq \frac {\sqrt {| A |}}{\rho_ {\min }} \left(\sqrt {| S |} \sqrt {\sum_ {s} \| q _ {t} (s , \cdot) - q ^ {\pi_ {t} ^ {\star}} (s , \cdot) \| ^ {2}} + \sqrt {| S | | A |} \sqrt {\sum_ {s , a} | q _ {t} (s , a) - q ^ {\pi_ {t} ^ {\star}} (s , a) | ^ {2}}\right) \\ \leq \frac {2 \sqrt {| S | | A |}}{\rho_ {\min}} \left\| q _ {t} - q ^ {\pi_ {t} ^ {\star}} \right\| \\ = \frac {2 \sqrt {| S |} | A |}{\rho_ {\min }} \| q _ {t} - \mathcal {P} _ {\mathcal {Q} ^ {*}} (q _ {t}) \| \\ \end{array}
$$