SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {s, a} d _ {\rho} ^ {\pi^ {*}} (s) \left(\hat {\pi} _ {t + 1} (a | s) - \pi_ {t} (a | s)\right) \left(Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, a) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, a)\right) \\ \leq \eta \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left\| Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, \cdot) \right\| _ {\infty} ^ {2} \\ \leq 2 \eta \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\left\| (\lambda_ {t} - \lambda_ {t - 1}) Q _ {g} ^ {\pi_ {t}} (s, \cdot) \right\| _ {\infty} ^ {2} + \left\| Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t - 1} g} ^ {\pi_ {t - 1}} (s, \cdot) \right\| _ {\infty} ^ {2}\right) \\ \leq \frac {2 \eta}{(1 - \gamma) ^ {2}} (\lambda_ {t} - \lambda_ {t - 1}) ^ {2} + \frac {4 \eta \gamma^ {2}}{(1 - \gamma) ^ {4}} \left(1 + \frac {1}{(1 - \gamma) ^ {2} \xi^ {2}}\right) \max  _ {s} \| \pi_ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| _ {1} ^ {2} \\ \leq \frac {2 \eta}{(1 - \gamma) ^ {2}} (\lambda_ {t} - \lambda_ {t - 1}) ^ {2} + \frac {4 \eta \gamma^ {2} \kappa_ {\rho}}{(1 - \gamma) ^ {5}} \left(1 + \frac {1}{(1 - \gamma) ^ {2} \xi^ {2}}\right) \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \pi_ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| _ {1} ^ {2} \\ \leq \frac {4 \eta}{(1 - \gamma) ^ {2}} \left(\left(\lambda_ {t} - \hat {\lambda} _ {t}\right) ^ {2} + \left(\hat {\lambda} _ {t} - \lambda_ {t - 1}\right) ^ {2}\right) \\ + \frac {8 \eta \gamma^ {2} \sqrt {| A |} \kappa_ {\rho}}{(1 - \gamma) ^ {5}} \left(1 + \frac {1}{(1 - \gamma) ^ {2} \xi^ {2}}\right) \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \| \hat {\pi} _ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| ^ {2}\right) \\ \end{array}
$$
where the first inequality is due to Cauchy-Schwarz inequality and (33), we subtract and add  $Q_{r + \lambda_{t - 1}g}^{\pi_t}$  and apply the inequality  $(x + y)^2 \leq 2x^2 + 2y^2$  in the second inequality, the third inequality
is due to Lemma 22 and the boundedness of value functions and dual variables, the fourth inequality comes from the property of  $\kappa_{\rho}$ , and the last inequality is due to the inequality  $(x + y)^{2} \leq 2x^{2} + 2y^{2}$  and the inequality  $\| p - p' \|_{1} \leq \sqrt{|A|} \| p - p' \|_{2}$  for two probability distributions  $p$  and  $p'$ . Meanwhile, using a similar reasoning, we can derive that
$$
\begin{array}{l} \left(\lambda_ {t} - \hat {\lambda} _ {t + 1}\right) \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t - 1}} (\rho)\right) \\ \leq \eta \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t - 1}} (\rho)\right) ^ {2} \\ \leq \frac {\eta \kappa_ {\rho} ^ {2}}{(1 - \gamma) ^ {6}} \left(\sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \pi_ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| _ {1}\right) ^ {2} \\ \leq \frac {\eta k _ {\rho} ^ {2}}{(1 - \gamma) ^ {6}} \sum_ {s} \left(\sqrt {d _ {\rho} ^ {\pi^ {\star}} (s)} \left\| \pi_ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \right\| _ {1}\right) ^ {2} \\ \leq \frac {2 \eta \kappa_ {\rho} ^ {2} | A |}{(1 - \gamma) ^ {6}} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \| \hat {\pi} _ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| ^ {2}\right) \\ \end{array}
$$
where the first inequality is due to Cauchy-Schwarz inequality and (34), the second inequality is due to Lemma 22, the third inequality is an application of Cauchy-Schwarz inequality, and the last inequality is due to the inequality  $(x + y)^{2} \leq 2x^{2} + 2y^{2}$  and the inequality  $\| p - p' \|_{1} \leq \sqrt{|A|} \| p - p' \|_{2}$  for two probability distributions  $p$  and  $p'$ .
We set notation,
$$
\iota := \max \left(\frac {2 \kappa_ {\rho} ^ {2} | A |}{(1 - \gamma) ^ {6}}, \frac {8 \gamma^ {2} \sqrt {| A |} \kappa_ {\rho}}{(1 - \gamma) ^ {6}} \left(1 + \frac {1}{(1 - \gamma) ^ {2} \xi^ {2}}\right), \frac {4}{(1 - \gamma) ^ {3}}\right).
$$
After applying the established inequalities above to (31) and (32), we combine them into (30) as,
$$
\begin{array}{l} V _ {r + \lambda_ {t} g} ^ {\pi^ {*}} (\rho) - V _ {r + \lambda^ {*} g} ^ {\pi_ {t}} (\rho) \\ \leq \frac {1}{2 \eta (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ + \frac {1}{2 \eta (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} - \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2}\right) \\ + \frac {1}{2 \eta} \left(\left(\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}\right) ^ {2} - \left(\hat {\lambda} _ {t + 1} - \lambda_ {t}\right) ^ {2} - \left(\lambda_ {t} - \hat {\lambda} _ {t}\right) ^ {2}\right) \\ + \frac {1}{2 \eta} \left(\left(\lambda^ {\star} - \hat {\lambda} _ {t}\right) ^ {2} - \left(\lambda^ {\star} - \hat {\lambda} _ {t + 1}\right) ^ {2} - \left(\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}\right) ^ {2}\right) \\ + \eta \iota \left(\lambda_ {t} - \hat {\lambda} _ {t}\right) ^ {2} + \eta \iota \left(\hat {\lambda} _ {t} - \lambda_ {t - 1}\right) ^ {2} \\ + \eta \iota \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \| \hat {\pi} _ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| ^ {2}\right). \\ \end{array}
$$
We notice that  $V_{r + \lambda_t g}^{\pi_*}(\rho) - V_{r + \lambda^* g}^{\pi_t}(\rho)\geq 0$  and non-expansiveness of projection operators  $\mathcal{P}_{\Pi^*}$ ,  $\mathcal{P}_{\Lambda^*}$ ,
$$
\begin{array}{l} \left\| \mathcal {P} _ {\Pi^ {\star}} \left(\hat {\pi} _ {t + 1} (\cdot | s)\right) - \hat {\pi} _ {t + 1} (\cdot | s) \right\| \leq \left\| \pi^ {\star} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \right\| \\ \left\| \mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t + 1}) - \hat {\lambda} _ {t + 1} \right\| \leq \left\| \lambda^ {\star} - \hat {\lambda} _ {t + 1} \right\| \\ \end{array}
$$
for any  $\pi^{\star} \in \Pi^{\star}$  and  $\lambda^{\star} \in \Lambda^{\star}$ . If we take  $\pi^{\star} = \mathcal{P}_{\Pi^{\star}}(\hat{\pi}_t(\cdot | s))$  and  $\lambda^{\star} = \mathcal{P}_{\Lambda^{\star}}(\hat{\lambda}_t)$ , then after some re-arrangement, we have
$$
\begin{array}{l} \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t + 1} (\cdot | s)) - \hat {\pi} _ {t + 1} (\cdot | s) \| ^ {2} + \frac {1}{2} (\mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t + 1}) - \hat {\lambda} _ {t + 1}) ^ {2} \\ \leq \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left\| \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t} (\cdot | s)) - \hat {\pi} _ {t + 1} (\cdot | s) \right\| ^ {2} + \frac {1}{2} (\mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t}) - \hat {\lambda} _ {t + 1}) ^ {2} \\ \leq \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \mathcal {P} _ {\Pi^ {*}} (\hat {\pi} _ {t} (\cdot | s)) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \frac {1}{2} (\mathcal {P} _ {\Lambda^ {*}} (\hat {\lambda} _ {t}) - \hat {\lambda} _ {t}) ^ {2} \\ - \frac {1}{2 (1 - \gamma)} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2} - \frac {1}{2} (\hat {\lambda} _ {t + 1} - \lambda_ {t}) ^ {2} \\ - \left(\frac {1}{2 (1 - \gamma)} - \eta^ {2} \iota\right) \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} - \left(\frac {1}{2} - \eta^ {2} \iota\right) (\lambda_ {t} - \hat {\lambda} _ {t}) ^ {2} \\ + \eta^ {2} \iota \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t} (\cdot | s) - \pi_ {t - 1} (\cdot | s) \| ^ {2} + \eta^ {2} \iota (\hat {\lambda} _ {t} - \lambda_ {t - 1}) ^ {2} \\ \end{array}
$$
where we also use the assumption  $d_{\rho}^{\pi^{\star}} = d_{\rho}^{\pi}$  for any  $\pi \in \Pi^{\star}$ . By taking  $\eta^2 \iota \leq \frac{1}{16}$  and using notation  $\Theta_t$  and  $\zeta_t$ , we have  $\Theta_{t+1} \leq \Theta_t - \frac{7}{16} \zeta_t$ .
To show the convergence, we next relate  $\zeta_t$  with  $\Theta_{t + 1}$ . An intermediate step is to show that  $\zeta_t$  has the following lower bound.
Lemma 20. Let  $\kappa_{\rho, \gamma} := \max \left(\frac{\kappa_{\rho}}{1 - \gamma}, 1\right)$ . If we set the stepsize  $\eta \leq \min \left(\frac{(1 - \gamma)^3}{4|A|}, \frac{(1 - \gamma)^3}{2\kappa_{\rho}}\right)$ , then the primal-dual iterates of OPG-PD (9) satisfy
$$
\begin{array}{l} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2}\right) + \left(| \lambda_ {t} - \hat {\lambda} _ {t} | ^ {2} + | \lambda_ {t} - \hat {\lambda} _ {t + 1} | ^ {2}\right) \\ \geq \frac {\eta^ {2}}{9 \kappa_ {\rho , \gamma} ^ {2}} \frac {\left[ V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho) \right] _ {+} ^ {2}}{\left(\max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| + | \lambda - \hat {\lambda} _ {t + 1} |\right) ^ {2}} f o r a l l (\pi , \lambda) \neq (\hat {\pi} _ {t + 1}, \hat {\lambda} _ {t + 1}). \\ \end{array}
$$
Proof. From the optimality of  $\hat{\pi}_{t + 1}(\cdot |s)$  in OPG-PD, we know that for any  $\pi \in \Pi$ ,
$$
\begin{array}{l} \left\langle \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \right\rangle \\ \geq \eta \langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \rangle \\ = \eta \langle Q _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \rangle + \eta \langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \rangle \\ + \eta \left\langle Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot) - Q _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \right\rangle . \tag {35a} \\ \end{array}
$$
Similarly, from the optimality of  $\pi_{t + 1}(\cdot |s)$  in OPG-PD, we know that for any  $\pi \in \Pi$
$$
\begin{array}{l} \left\langle \pi_ {t + 1} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s), \pi (\cdot | s) - \pi_ {t + 1} (\cdot | s) \right\rangle \\ \geq \eta \langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot), \pi (\cdot | s) - \pi_ {t + 1} (\cdot | s) \rangle \\ = \eta \langle Q _ {r + \lambda_ {t + 1} g} ^ {\pi_ {t + 1}} (s, \cdot), \pi (\cdot | s) - \pi_ {t + 1} (\cdot | s) \rangle + \eta \langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t} g} ^ {\pi_ {t + 1}} (s, \cdot), \pi (\cdot | s) - \pi_ {t + 1} (\cdot | s) \rangle \\ + \eta \left\langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t + 1}} (s, \cdot) - Q _ {r + \lambda_ {t + 1} g} ^ {\pi_ {t + 1}} (s, \cdot), \pi (\cdot | s) - \pi_ {t + 1} (\cdot | s) \right\rangle . \tag {35b} \\ \end{array}
$$
On the other hand, from the optimality of  $\hat{\lambda}_{t + 1}$  in OPG-PD, we know that for any  $\lambda \in \Lambda$
$$
\begin{array}{l} (\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}) (\lambda - \hat {\lambda} _ {t + 1}) \geq \eta V _ {g} ^ {\pi_ {t}} (\rho) (\hat {\lambda} _ {t + 1} - \lambda) \\ = \eta V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho) \left(\hat {\lambda} _ {t + 1} - \lambda\right) + \eta \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right) \left(\hat {\lambda} _ {t + 1} - \lambda\right) \tag {36a} \\ \end{array}
$$
and the optimality of  $\lambda_{t + 1}$  in OPG-PD yields,
$$
\begin{array}{l} \left(\lambda_ {t + 1} - \hat {\lambda} _ {t}\right) \left(\lambda - \lambda_ {t + 1}\right) \geq \eta V _ {g} ^ {\pi_ {t}} (\rho) \left(\lambda_ {t + 1} - \lambda\right) \\ = \eta V _ {g} ^ {\pi_ {t + 1}} (\rho) \left(\lambda_ {t + 1} - \lambda\right) + \eta \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\pi_ {t + 1}} (\rho)\right) \left(\lambda_ {t + 1} - \lambda\right). \tag {36b} \\ \end{array}
$$
First, we take the expectation of (35a) over some state distribution  $d_{\rho}^{\pi}$  on both sides and add it to (36a),
$$
\begin{array}{l} \frac {1}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \langle \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \rangle + (\hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t}) (\lambda - \hat {\lambda} _ {t + 1}) \\ \geq \frac {\eta}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \langle Q _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \rangle \\ + \frac {\eta}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \left\langle Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \right\rangle \\ + \frac {\eta}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \left\langle Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot) - Q _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot), \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \right\rangle \\ + \eta V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho) \left(\hat {\lambda} _ {t + 1} - \lambda\right) + \eta \left(V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right) \left(\hat {\lambda} _ {t + 1} - \lambda\right) \\ \end{array}
$$
$$
\begin{array}{l} \geq \eta \left(V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right) \\ - \frac {\eta}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \left\| Q _ {r + \lambda_ {t} g} ^ {\pi_ {t}} (s, \cdot) - Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot) \right\| _ {\infty} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| _ {1} \\ - \frac {\eta}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi} (s) \left\| Q _ {r + \lambda_ {t} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot) - Q _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (s, \cdot) \right\| \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \\ + \eta V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho) (\hat {\lambda} _ {t + 1} - \lambda) - \eta | V _ {g} ^ {\pi_ {t}} (\rho) - V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho) | | \hat {\lambda} _ {t + 1} - \lambda | \\ \end{array}
$$
$$
\begin{array}{l} \geq \eta \left(V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right) + \eta V _ {g} ^ {\hat {\pi} _ {t + 1}} (\rho) \left(\hat {\lambda} _ {t + 1} - \lambda\right) \\ - \frac {\eta \gamma | A |}{(1 - \gamma) ^ {3}} \max  _ {s} \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \sum_ {s} d _ {\rho} ^ {\pi} (s) \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \\ - \frac {\eta}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi} (s) | \lambda_ {t} - \hat {\lambda} _ {t + 1} | \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \\ - \frac {\eta \kappa_ {\rho} \sqrt {| A |}}{(1 - \gamma) ^ {3}} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \left| \hat {\lambda} _ {t + 1} - \lambda \right| \tag {37} \\ \end{array}
$$
where the second inequality is due to performance difference lemma in Lemma 28 and Cauchy-Schwarz inequality, and the last inequality results from Lemma 22 and the inequality  $\| p - p' \|_1 \leq \sqrt{|A|} \| p - p' \|_2$  for two probability distributions  $p$  and  $p'$ . By using the inequality  $ac + bd \leq$
$(a + b)(c + d)$  for  $a \geq 0, b \geq 0, c \geq 0$ , and  $d \geq 0$ , with some re-arrangement, we have
$$
\begin{array}{l} \left(\max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| + | \lambda - \hat {\lambda} _ {t + 1} |\right) \times \\ \left[ \frac {\kappa_ {\rho}}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \left(\frac {1}{1 - \gamma} \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {2 \eta | A |}{(1 - \gamma) ^ {3}} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) \right. \\ \left. + \left| \hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t} \right| + \frac {\eta \kappa_ {\rho}}{(1 - \gamma) ^ {3}} \left| \lambda_ {t} - \hat {\lambda} _ {t + 1} \right| \right] \\ \end{array}
$$
$$
\begin{array}{l} \geq \max _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \frac {\kappa_ {\rho}}{1 - \gamma} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \times \\ \left(\frac {1}{1 - \gamma} \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {\eta \gamma | A |}{(1 - \gamma) ^ {3}} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| + \frac {\eta}{(1 - \gamma) ^ {2}} | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right) \\ + \left| \lambda - \hat {\lambda} _ {t + 1} \right| \left(\left| \hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t} \right| + \frac {\eta \kappa_ {\rho} \sqrt {| A |}}{(1 - \gamma) ^ {3}} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) \\ \end{array}
$$
$$
\begin{array}{l} \geq \max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| \sum_ {s} d _ {\rho} ^ {\pi} (s) \times \\ \left(\frac {1}{1 - \gamma} \| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {\eta \gamma | A |}{(1 - \gamma) ^ {3}} \max  _ {s} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| + \frac {\eta}{(1 - \gamma) ^ {2}} | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right) \\ + \left| \lambda - \hat {\lambda} _ {t + 1} \right| \left(\left| \hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t} \right| + \frac {\eta \kappa_ {\rho} \sqrt {| A |}}{(1 - \gamma) ^ {3}} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) \\ \end{array}
$$
$$
\geq \eta \left(V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right)
$$
where the second inequality comes from the property of  $\kappa_{\rho}$  and the last inequality is straightforward from (37). We take  $\eta > 0$  such that  $\max \left(\frac{2\eta|A|}{(1 - \gamma)^3}, \frac{\eta\kappa_{\rho}}{(1 - \gamma)^3}\right) \leq \frac{1}{2}$  and denote  $\kappa_{\rho, \gamma} := \max \left(\frac{\kappa_{\rho}}{1 - \gamma}, 1\right)$ .
If we take the square of both sides of the inequality above, then the second product argument has the following upper bound,
$$
\begin{array}{l} \left(\sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {2 \eta | A |}{(1 - \gamma) ^ {3}} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) + | \hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t} | + \frac {\eta \kappa_ {\rho}}{(1 - \gamma) ^ {3}} | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right) ^ {2} \\ \leq \left(\sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \hat {\pi} _ {t + 1} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {1}{2} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) + | \hat {\lambda} _ {t + 1} - \hat {\lambda} _ {t} | + \frac {1}{2} | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right) ^ {2} \\ \leq \left(\sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \frac {3}{2} \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) + | \lambda_ {t} - \hat {\lambda} _ {t} | + \frac {3}{2} | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right) ^ {2} \\ \leq \left(\frac {3}{2} \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| + \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \|\right) + \frac {3}{2} \left(| \lambda_ {t} - \hat {\lambda} _ {t} | + | \lambda_ {t} - \hat {\lambda} _ {t + 1} |\right)\right) ^ {2} \\ \leq 9 \sum_ {s} d _ {\rho} ^ {\pi^ {*}} (s) \left(\| \pi_ {t} (\cdot | s) - \hat {\pi} _ {t} (\cdot | s) \| ^ {2} + \| \hat {\pi} _ {t + 1} (\cdot | s) - \pi_ {t} (\cdot | s) \| ^ {2}\right) + 9 \left(| \lambda_ {t} - \hat {\lambda} _ {t} | ^ {2} + | \lambda_ {t} - \hat {\lambda} _ {t + 1} | ^ {2}\right) \\ \end{array}
$$
where we use the inequality  $(x + y)^2\leq 2x^2 +2y^2$  and Jensen's inequality.
Recall the definition of  $\kappa_{\rho}$  and  $\kappa_{\rho} \leq 1 / \rho_{\min}$ , where  $\rho_{\min} \coloneqq \min_s \rho(s)$ .
Lemma 21. Assume  $\rho_{\mathrm{min}} > 0$ . For any  $t \geq 1$ , the primal-dual iterates of OPG-PD (9) satisfy
$$
\begin{array}{l} \sup  _ {\pi \in \Pi , \lambda \in \Lambda} \frac {V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho)}{\max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| + | \lambda - \hat {\lambda} _ {t + 1} |} \\ \geq C _ {\rho , \xi} \left(\sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \mathcal {P} _ {\Pi^ {\star}} (\hat {\pi} _ {t + 1} (\cdot | s)) \| + | \hat {\lambda} _ {t + 1} - \mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t + 1}) |\right) \\ \end{array}
$$
where  $C_{\rho, \xi} \coloneqq c\rho_{\min} / (2\sqrt{|S||A|}) / (1 + 1 / ((1 - \gamma)\xi))$  in which  $c > 0$  is described in Lemma 26.
Proof. Denote  $V^{\star} \coloneqq V_{r + \lambda^{*}g}^{\pi^{\star}}(\rho)$  and  $D_{\max} \coloneqq \max_{\pi, \pi' \in \Pi, \lambda, \lambda' \in \Lambda} (\max_{s} \| \pi(\cdot|s) - \pi'(\cdot|s) \| + |\lambda - \lambda'|)$ . We observe that if we can prove that there exist constants  $c_{1}, c_{2} > 0$  such that
$$
\max  _ {\pi \in \Pi} V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V ^ {\star} \geq c _ {1} | \hat {\lambda} _ {t + 1} - \mathcal {P} _ {\Lambda^ {\star}} (\hat {\lambda} _ {t + 1}) | \tag {38a}
$$
$$
V ^ {\star} - \min  _ {\lambda \in \Lambda} V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho) \geq c _ {2} \sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \mathcal {P} _ {\Pi^ {\star}} \left(\hat {\pi} _ {t + 1} (\cdot | s)\right) \|, \tag {38b}
$$
$$
\begin{array}{l} \sup  _ {\pi \in \Pi , \lambda \in \Lambda} \frac {V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho)}{\max  _ {s} \| \pi (\cdot | s) - \hat {\pi} _ {t + 1} (\cdot | s) \| + | \lambda - \hat {\lambda} _ {t + 1} |} \\ \geq \frac {1}{D _ {\max }} \sup  _ {\pi \in \Pi , \lambda \in \Lambda} V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho) \\ = \frac {1}{D _ {\max }} \left(\max  _ {\pi \in \Pi} V _ {r + \hat {\lambda} _ {t + 1} g} ^ {\pi} (\rho) - V ^ {\star}\right) + \frac {1}{D _ {\max }} \left(V ^ {\star} - \min  _ {\lambda \in \Lambda} V _ {r + \lambda g} ^ {\hat {\pi} _ {t + 1}} (\rho)\right) \\ \geq \frac {\min  \left(c _ {1} , c _ {2}\right)}{D _ {\max }} \left(\sum_ {s} d _ {\rho} ^ {\pi^ {\star}} (s) \| \hat {\pi} _ {t + 1} (\cdot | s) - \mathcal {P} _ {\Pi^ {\star}} \left(\hat {\pi} _ {t + 1} (\cdot | s)\right) \| + \left| \hat {\lambda} _ {t + 1} - \mathcal {P} _ {\Lambda^ {\star}} \left(\hat {\lambda} _ {t + 1}\right) \right|\right) \\ \end{array}
$$
which proves the lemma by taking  $C_{\rho, \xi} \coloneqq \frac{\min(c_1, c_2)}{D_{\max}}$ .
We next prove (38) using the bilinear game result in Lemma 26. By the linear program formulation of constrained MDP, we can express the value function in terms of the occupancy measure  $q^{\pi}$ , e.g.,  $V_{r + \lambda g}^{\pi}(\rho) = \langle q^{\pi}, r + \lambda g \rangle$ , where  $q^{\pi}$  is the occupancy measure that lives in a polytope  $\mathcal{Q}$  that is given by (15). Hence, the constrained saddle-point problem (3) reduces to,