SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We apply the conservatism to the translated constraint  $V_{g}^{\pi}(\rho) \geq 0$  in Problem (1). Specifically, for any  $\delta < \min(\xi, 1)$ , we let  $g' := g - (1 - \gamma)\delta$  and define a conservative constraint,
$$
V _ {g ^ {\prime}} ^ {\pi} := V _ {g} ^ {\pi} (\rho) - \delta \geq 0.
$$
It is straightforward to see that Assumption 1 ensures that  $V_{g'}^{\pi_t}(\rho) \geq 0$  is strictly feasible for a new slack variable  $\xi' \coloneqq \xi - \delta$ . We now can apply RPG-PD (6) to a new regularized Lagrangian,
$$
L _ {\tau} ^ {\prime} (\pi , \lambda) := V _ {r + \lambda g ^ {\prime}} ^ {\pi} (\rho) + \tau \left(\mathcal {H} (\pi) + \frac {1}{2} \lambda^ {2}\right)
$$
and Corollary 3 holds if we replace  $g$  in RPG-PD by  $g'$ . Thus,
$$
V _ {r} ^ {\pi_ {\delta} ^ {*}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) \leq \epsilon \text {a n d} - V _ {g ^ {\prime}} ^ {\pi_ {t}} (\rho) \leq \epsilon \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
where  $\Omega (\cdot)$  hides some problem-dependent constant, and  $\pi_{\delta}^{\star}$  is an optimal policy to the  $\delta$ -perturbed constrained policy optimization problem,
$$
\underset {\pi \in \Pi} {\text {m a x i m i z e}} V _ {r} ^ {\pi} (\rho) \quad \text {s u b j e c t t o} V _ {g} ^ {\pi} (\rho) - \delta \geq 0. \tag {23}
$$
We notice that the above  $\Omega(\cdot)$  has  $\xi'$ -dependence and we denote it by  $\Xi(\xi')$ , where  $\Xi: \mathbb{R}_+ \to \mathbb{R}_+$  is a positive function. Thus, we select  $\delta$  such that  $\delta \geq \epsilon \Xi(\xi')$ , which is always possible for small enough  $\epsilon$ , for instance,  $\delta = \frac{\xi}{2}$  and  $\xi' = \frac{\xi}{2}$ . Hence, if we take  $\delta = \frac{\xi}{2}$  and such small  $\epsilon$ , then,
$$
- V _ {g ^ {\prime}} ^ {\pi_ {t}} (\rho) = - V _ {g} ^ {\pi_ {t}} (\rho) + \delta \leq \epsilon \Xi (\xi^ {\prime}) \text {f o r a n y} t = \Omega \left(\frac {1}{\epsilon^ {6}} \log^ {2} \frac {1}{\epsilon}\right)
$$
which shows that  $V_{g}^{\pi_{t}}(\rho)\geq 0$  for some large  $t$ .
The rest is to show that  $V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\pi_{t}}(\rho) \leq O(\epsilon)$ . We notice that  $\pi^{\star}$  is an optimal policy to Problem (23) when  $\delta = 0$ . Let  $q^{\star}$  and  $q_{\delta}^{\star}$  be associated occupancy measures of policies  $\pi^{\star}$  and  $\pi_{\delta}^{\star}$ . In the occupancy measure space, Problem (23) becomes a linear program and it has a solution  $q_{\delta}^{\star}$ . Thus, we can view  $q_{\delta}^{\star}$  as a  $\delta$ -perturbed solution of a convex optimization problem in which all functions are continuously differentiable and the domain is convex and compact. It is known from [129, Theorem 3.1] that the optimal solution  $q_{\delta}^{\star}$  is continuous in  $\delta$ , which implies that for any  $\epsilon > 0$ , there exists  $\delta' > 0$  such that  $|\langle r, q^{\star} \rangle - \langle r, q_{\delta}' \rangle| \leq O(\epsilon)$  for any  $\delta < \delta'$ . Thus,  $|V_{r}^{\pi^{\star}}(\rho) - V_{r}^{\pi_{\delta}^{\star}}(\rho)| \leq O(\epsilon)$  for small enough  $\epsilon$ . Therefore,
$$
V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) \leq V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) - V _ {r} ^ {\pi_ {t}} (\rho) + | V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\delta} ^ {\star}} (\rho) | \leq O (\epsilon)
$$
for some large  $t$ . Collecting all conditions on  $\delta$  leads to our final choice of  $\delta = \min \left(\frac{\xi}{2}, 1, \delta'\right)$ . Finally, we combine all big O notation to complete the proof.
C.5 Reduction of RPG-PD (6) as a NPG variant (7)
We introduce some useful notation in the regularized MDP [52]. Let  $V_{\tau}^{\pi}(\rho) \coloneqq V_{r + \lambda g}^{\pi}(\rho) + \tau \mathcal{H}(\pi)$ . We introduce the soft- $Q$  value function  $Q_{\tau}^{\pi} \colon S \times A \to \mathbb{R}$  and  $V_{\tau}^{\pi} \colon S \to \mathbb{R}$  via Bellman equations,
$$
Q _ {\tau} ^ {\pi} (s, a) = r (s, a) + \lambda g (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V _ {\tau} ^ {\pi} \left(s ^ {\prime}\right) \right]
$$
$$
V _ {\tau} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ - \tau \log \pi (a | s) + Q _ {\tau} ^ {\pi} (s, a) \right].
$$
We also define  $A_{\tau}^{\pi}(s,a)\coloneqq Q_{\tau}^{\pi}(s,a) - \tau \log \pi (a|s) - V_{\tau}^{\pi}(s)$ . Hence,
$$
{Q _ {r + \lambda g + \tau \psi} ^ {\pi} (s, a)} = {Q _ {\tau} ^ {\pi} (s, a) - \tau \log \pi (a \mid s)}
$$
$$
{A _ {r + \lambda g + \tau \psi} ^ {\pi} (s, a)} = {Q _ {r + \lambda g + \tau \psi} ^ {\pi} (s, a) - V _ {r + \lambda g + \tau \psi} ^ {\pi} (s)}
$$
where  $\psi (s,a)\coloneqq -\log \pi (a\mid s)$  for all  $(s,a)\in S\times A$
Setting  $\epsilon_0 = 0$ , it is easy to show that RPG-PD (6) is a case of (7) in the tabular case by introducing the softmax policy that is widely used in policy optimization. A softmax policy  $\pi_{\theta}\colon S\to \Delta (A)$  is parametrized by a parameter  $\theta \in \mathbb{R}^{|S||A|}$  via a softmax function,
$$
\pi_ {\theta} (a \mid s) = \frac {\exp (\theta_ {s , a})}{\sum_ {a ^ {\prime}} \exp (\theta_ {s , a ^ {\prime}})} \text {f o r a l l} (s, a) \in S \times A.
$$
With a slight abuse of notation, we also use notation  $\pi_{\theta}$  as a vector in  $\mathbb{R}^{|S||A|}$ .
Lemma 16. Set  $\epsilon_0 = 0$ . Under the softmax policy parametrization, RPG-PD (6) is equivalent to (7).
Proof. Dual update (7b) is straightforward. We next show the equivalence for the primal update by applying the softmax function to both sides of Primal update (7a).
We first notice that  $F_{\rho}(\theta_t)^\dagger \cdot \nabla_\theta L_\tau (\pi_{\theta_t},\lambda_t) = F_{\rho}(\theta_t)^\dagger \cdot \nabla_\theta V_\tau^{\pi_\theta}(\rho)$ . Thus,
$$
\begin{array}{l} \exp (\theta_ {t + 1, s, a}) = \exp (\theta_ {t, s, a}) \exp \left(\eta (1 - \gamma) \left(F _ {\rho} (\theta_ {t}) ^ {\dagger} \cdot \nabla_ {\theta} L _ {\tau} \left(\pi_ {\theta_ {t}}, \lambda_ {t}\right)\right) _ {s, a}\right) \\ { = } { \exp ( \theta _ { t , s , a } ) \exp \left( \eta A _ { \tau } ^ { \pi _ { \theta _ { t } } } ( s , a ) + \eta c ( s ) \right) } \\ \end{array}
$$
where  $c(s)$  is an action-independent constant, and the second equality is the property of natural policy gradient (e.g., Lemma 29). Hence, after normalization over actions and some re-arrangement, we have
$$
\begin{array}{l} \pi_ {\theta_ {t + 1}} (a \mid s) = \pi_ {\theta_ {t}} (a \mid s) \frac {\exp \left(\eta A _ {\tau} ^ {\pi_ {\theta_ {t}}} (s , a) + \eta c (s)\right)}{\sum_ {a ^ {\prime}} \pi_ {\theta_ {t}} \left(a ^ {\prime} \mid s\right) \exp \left(\eta A _ {\tau} ^ {\pi_ {\theta_ {t}}} (s , a ^ {\prime}) + \eta c (s)\right)} \\ = \pi_ {\theta_ {t}} (a \mid s) \frac {\exp \left(\eta Q _ {\tau} ^ {\pi_ {\theta_ {t}}} (s , a) - \eta \tau \log \pi_ {\theta_ {t}} (a \mid s)\right)}{\sum_ {a ^ {\prime}} \pi_ {\theta_ {t}} \left(a ^ {\prime} \mid s\right) \exp \left(\eta Q _ {\tau} ^ {\pi_ {\theta_ {t}}} (s , a ^ {\prime}) - \eta \tau \log \pi_ {\theta_ {t}} \left(a ^ {\prime} \mid s\right)\right)} \\ = \pi_ {\theta_ {t}} (a \mid s) \frac {\exp \left(\eta Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s , a)\right)}{\sum_ {a ^ {\prime}} \pi_ {\theta_ {t}} (a ^ {\prime} \mid s) \exp \left(\eta Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s , a ^ {\prime})\right)} \\ \end{array}
$$
which is an explicit form of the policy update in (6). Since the above derivation holds in both directions, the proof is complete.
Proof. We utilize the decomposition of the primal-dual gap as in (19) and analyze the term (i) and the term (ii), separately.
For the term (i), we have
where the first two equalities are because of the entropy regularization  $\mathcal{H}(\pi)$ , we apply performance difference lemma in Lemma 28 and  $\psi_t(s, a) = -\log \pi_{\theta_t}(a|s)$  to the third equality, the inequality is due to an application of Lemma 27 with  $x^{\star} = \pi_{\tau}^{\star}(\cdot|s)$ ,  $x = \pi_{\theta_t}(\cdot|s)$ ,  $g = -\phi_{s,a}^{\top}w_t$ , and  $\eta \leq 1 / C_W$ , where  $|\phi_{s,a}^{\top}w_t / (1 - \gamma)| \leq 2W / (1 - \gamma) \coloneqq C_W$ . Moreover, the cross term has the following decomposition,
$$
\begin{array}{l} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\pi_ {\tau} ^ {\star} (a | s) - \pi_ {\theta_ {t}} (a | s)\right) \left(Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a) - \phi_ {s, a} ^ {\top} w _ {t}\right) \\ = \underbrace {\sum_ {s , a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s) \left(Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s , a) - \phi_ {s , a} ^ {\top} w _ {t} ^ {\star}\right)} _ {(\mathrm {a})} + \underbrace {\sum_ {s , a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s) \phi_ {s , a} ^ {\top} \left(w _ {t} - w _ {t} ^ {\star}\right)} _ {(\mathrm {b})} \\ + \underbrace {\sum_ {s , a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s) \phi_ {s , a} ^ {\top} \left(w _ {t} ^ {\star} - w _ {t}\right)} _ {\text {(c)}} + \underbrace {\sum_ {s , a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s) \left(\phi_ {s , a} ^ {\top} w _ {t} ^ {\star} - Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s , a)\right)} _ {\text {(d)}}. \\ \end{array}
$$
We now deal with the four terms (a), (b), (c), and (d), separately. For the term (a),
$$
\begin{array}{l} | (\mathbf {a}) | \leq \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s) \left| Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a) - \phi_ {s, a} ^ {\top} w _ {t} ^ {\star} \right| \\ \leq \sqrt {\sum_ {s , a} \frac {\left(d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s)\right) ^ {2}}{d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \operatorname {U n i f} _ {A} (a)} \sum_ {s , a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \operatorname {U n i f} _ {A} (a) \left(Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s , a) - \phi_ {s , a} ^ {\top} w _ {t} ^ {\star}\right) ^ {2}} \\ = \sqrt {\sum_ {s , a} \frac {\left(d _ {\rho^ {\tau}} ^ {\pi^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s)\right) ^ {2}}{d _ {\rho^ {\tau}} ^ {\pi^ {\star}} (s) \operatorname {U n i f} _ {A} (a)} \mathcal {E} _ {Q} \left(w _ {t} ^ {\star} , \theta_ {t} , \nu^ {\star}\right)} \\ \leq \sqrt {\sum_ {s , a} \frac {d _ {\rho^ {\tau}} ^ {\pi^ {\star}} (s) \pi_ {\tau} ^ {\star} (a \mid s)}{\operatorname {U n i f} _ {A} (a)} \mathcal {E} _ {Q} \left(w _ {t} ^ {\star} , \theta_ {t} , \nu^ {\star}\right)} \\ = \sqrt {| A | \mathcal {E} _ {Q} (w _ {t} ^ {\star} , \theta_ {t} , \nu^ {\star})}, \\ \end{array}
$$
where we recall the definition of  $\mathcal{E}_Q(w_t^\star, \theta_t, \nu^\star)$ . Similarly, we can bound the term (d) by  $|(d)| \leq \sqrt{|A| \mathcal{E}_Q(w_t^\star, \theta_t, \nu^\star)}$ . For the term (b),
$$
\begin{array}{l} | (\mathfrak {b}) | \leq \sum_ {s, a} d _ {\rho} ^ {\pi^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s) \left| \phi_ {s, a} ^ {\top} \left(w _ {t} - w _ {t} ^ {\star}\right) \right| \\ \leq \sqrt {\sum_ {s , a} \frac {\left(d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s)\right) ^ {2}}{d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \operatorname {U n i f} _ {A} (a)} \sum_ {s , a} d _ {\rho} ^ {\pi^ {\star}} (s) \operatorname {U n i f} _ {A} (a) \left(\phi_ {s , a} ^ {\top} \left(w _ {t} - w _ {t} ^ {\star}\right)\right) ^ {2}} \\ = \sqrt {\sum_ {s , a} \frac {\left(d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s)\right) ^ {2}}{d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \operatorname {U n i f} _ {A} (a)} \left\| w _ {t} - w _ {t} ^ {\star} \right\| _ {\Sigma_ {\nu^ {\star}}} ^ {2}} \\ \leq \sqrt {\sum_ {s , a} \frac {d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \pi_ {\theta_ {t}} (a \mid s)}{\operatorname {U n i f} _ {A} (a)} \| w _ {t} - w _ {t} ^ {\star} \| _ {\Sigma_ {\nu^ {\star}}} ^ {2}} \\ \leq \sqrt {| A | \| w _ {t} - w _ {t} ^ {\star} \| _ {\Sigma_ {\nu \star}} ^ {2}} \\ \leq \sqrt {| A | \kappa_ {\nu} \| w _ {t} - w _ {t} ^ {\star} \| _ {\Sigma_ {d _ {t , \nu}}} ^ {2}}, \\ \end{array}
$$
where we recall the definition of  $\kappa_{\nu}$  to obtain the last line. Similarly, we can bound the term (c) by  $|(\mathbf{c})|\leq \sqrt{|A||w_t - w_t^\star||_{\Sigma_{\nu^\star}}^2}$ . Moreover, the optimality of  $w_{t}^{\star}\in \mathrm{argmin}_{w\in \mathbb{R}^{d}}\mathcal{E}_{Q}(w,\theta_{t},d_{t,\nu})$  yields,
$$
\left(w - w _ {t} ^ {\star}\right) ^ {\top} \nabla_ {w} \mathcal {E} _ {Q} \left(w _ {t} ^ {\star}, \theta_ {t}, d _ {t, \nu}\right) \geq 0, \quad \text {f o r a n y} \| w \| \leq W
$$
which further implies that for any  $\| w\| \leq W$
$$
\begin{array}{l} \mathcal {E} _ {Q} \left(w, \theta_ {t}, d _ {t, \nu}\right) - \mathcal {E} _ {Q} \left(w _ {t} ^ {\star}, \theta_ {t}, d _ {t, \nu}\right) \\ = \mathbb {E} _ {(s, a) \sim d _ {t, \nu}} \left[ \left(\phi_ {s, a} ^ {\top} w - \phi_ {s, a} ^ {\top} w _ {t} ^ {\star} + \phi_ {s, a} ^ {\top} w _ {t} ^ {\star} - Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) ^ {2} \right] - \mathcal {E} _ {Q} \left(w _ {t} ^ {\star}, \theta_ {t}, d _ {t, \nu}\right) \\ = \mathbb {E} _ {(s, a) \sim d _ {t, \nu}} \left[ \left(\phi_ {s, a} ^ {\top} w - \phi_ {s, a} ^ {\top} w _ {t} ^ {\star}\right) ^ {2} \right] + 2 (w - w _ {t} ^ {\star}) ^ {\top} \mathbb {E} _ {(s, a) \sim d _ {t, \nu}} \left[ \phi_ {s, a} \left(\phi_ {s, a} ^ {\top} w _ {t} ^ {\star} - Q _ {r + \lambda_ {t} g + \tau \psi_ {t}} ^ {\pi_ {\theta_ {t}}} (s, a)\right) \right] \\ = \mathbb {E} _ {(s, a) \sim d _ {t, \nu}} \left[ \left(\phi_ {s, a} ^ {\top} w - \phi_ {s, a} ^ {\top} w _ {t} ^ {\star}\right) ^ {2} \right] + 2 (w - w _ {t} ^ {\star}) ^ {\top} \nabla_ {w} \mathcal {E} _ {Q} \left(w _ {t} ^ {\star}, \theta_ {t}, d _ {t, \nu}\right) \\ \geq \mathbb {E} _ {(s, a) \sim d _ {t, \nu}} \left[ \left(\phi_ {s, a} ^ {\top} w - \phi_ {s, a} ^ {\top} w _ {t} ^ {\star}\right) ^ {2} \right] \\ { = } { \| w - w _ { t } ^ { \star } \| _ { \Sigma _ { d _ { t , \nu } } } ^ { 2 } } \\ \end{array}
$$
$$
{| (\mathsf {b}) |} \leq \sqrt {| A | \kappa_ {\nu} \| w _ {t} - w _ {t} ^ {\star} \| _ {\Sigma_ {d _ {t , \nu}}} ^ {2}} \leq \sqrt {| A | \kappa_ {\nu} \left(\mathcal {E} _ {Q} (w _ {t} , \theta_ {t} , d _ {t , \nu}) - \mathcal {E} _ {Q} (w _ {t} ^ {\star} , \theta_ {t} , d _ {t , \nu})\right)}.
$$
With a similar reasoning, we can bound the term (c) by
$$
| (c) | \leq \sqrt {| A | \kappa_ {\nu} \left(\mathcal {E} _ {Q} \left(w _ {t} , \theta_ {t} , d _ {t , \nu}\right) - \mathcal {E} _ {Q} \left(w _ {t} ^ {\star} , \theta_ {t} , d _ {t , \nu}\right)\right)}.
$$
By applying the upper bounds above to the cross term, and then taking expectation over the randomness in  $w_{t}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ L _ {\tau} \left(\pi_ {\tau} ^ {\star}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {\theta_ {t}}, \lambda_ {t}\right) \right] \\ \leq \frac {(1 - \eta \tau) \mathbb {E} [ \mathrm {K L} _ {t} (\rho) ] - \mathbb {E} [ \mathrm {K L} _ {t + 1} (\rho) ]}{\eta} + \eta (C _ {W}) ^ {2} + 2 \mathbb {E} \left[ \sqrt {| A | \mathcal {E} _ {Q} (w _ {t} ^ {\star} , \theta_ {t} , \nu^ {\star})} \right] \\ + 2 \mathbb {E} \left[ \sqrt {\left| A \right| \kappa_ {\nu} \left(\mathcal {E} _ {Q} \left(w _ {t} , \theta_ {t} , d _ {t , \nu}\right) - \mathcal {E} _ {Q} \left(w _ {t} ^ {\star} , \theta_ {t} , d _ {t , \nu}\right)\right)} \right] \\ \leq \frac {(1 - \eta \tau) \mathbb {E} [ \mathrm {K L} _ {t} (\rho) ] - \mathbb {E} [ \mathrm {K L} _ {t + 1} (\rho) ]}{\eta} + \eta (C _ {W}) ^ {2} + 2 \sqrt {| A | \mathbb {E} [ \mathcal {E} _ {Q} (w _ {t} ^ {\star} , \theta_ {t} , \nu^ {\star}) ]} \\ + 2 \sqrt {\left| A \right| \kappa_ {\nu} \mathbb {E} \left[ \mathcal {E} _ {Q} \left(w _ {t} , \theta_ {t} , d _ {t , \nu}\right) - \mathcal {E} _ {Q} \left(w _ {t} ^ {\star} , \theta_ {t} , d _ {t , \nu}\right) \right]} \\ \leq \frac {(1 - \eta \tau) \mathbb {E} [ \mathrm {K L} _ {t} (\rho) ] - \mathbb {E} [ \mathrm {K L} _ {t + 1} (\rho) ]}{\eta} + \eta (C _ {W}) ^ {2} + 2 \sqrt {| A | \epsilon_ {\text {b i a s}}} + 2 \sqrt {| A | \kappa_ {\nu} \epsilon_ {\text {s t a t}}}. \\ \end{array}
$$
$$
\begin{array}{l} L _ {\tau} \left(\pi_ {\theta_ {t}}, \lambda_ {t}\right) - L _ {\tau} \left(\pi_ {\theta_ {t}}, \lambda_ {\tau} ^ {\star}\right) \\ = V _ {r + \lambda_ {t} g} ^ {\pi_ {\theta_ {t}}} (\rho) - V _ {r + \lambda_ {\tau} ^ {\star} g} ^ {\pi_ {\theta_ {t}}} (\rho) + \frac {1}{2} \tau (\lambda_ {t}) ^ {2} - \frac {1}{2} \tau (\lambda_ {\tau} ^ {\star}) ^ {2} \\ = \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) + \frac {1}{2} \tau \left(\lambda_ {t}\right) ^ {2} - \frac {1}{2} \tau \left(\lambda_ {\tau} ^ {\star}\right) ^ {2} \\ = \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) \left(V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) + \tau \lambda_ {t}\right) - \frac {1}{2} \tau \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) ^ {2} \\ \leq \frac {\left(\lambda_ {\tau} ^ {\star} - \lambda_ {t}\right) ^ {2} - \left(\lambda_ {\tau} ^ {\star} - \lambda_ {t + 1}\right) ^ {2}}{2 \eta} + \frac {1}{2} \eta \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2} - \frac {1}{2} \tau \left(\lambda_ {t} - \lambda_ {\tau} ^ {\star}\right) ^ {2} \\ = \frac {(1 - \eta \tau) (\lambda_ {\tau} ^ {\star} - \lambda_ {t}) ^ {2} - (\lambda_ {\tau} ^ {\star} - \lambda_ {t + 1}) ^ {2}}{2 \eta} + \frac {1}{2} \eta (C _ {\tau , \xi} ^ {\prime}) ^ {2} \\ \end{array}
$$
where the inequality is due to the standard descent lemma [128] and  $V_{g}^{\pi_{\theta_{t}}}(\rho) + \tau \lambda_{t} \leq \frac{1}{1 - \gamma} (1 + \frac{\tau}{\xi}) := C_{\tau, \xi}'$ . By taking expectation over the randomness in  $w_{t}$ ,
$$
\mathbb {E} \left[ L _ {\tau} (\pi_ {\theta_ {t}}, \lambda_ {t}) - L _ {\tau} (\pi_ {\theta_ {t}}, \lambda_ {\tau} ^ {\star}) \right] \leq \frac {(1 - \eta \tau) \mathbb {E} \left[ (\lambda_ {\tau} ^ {\star} - \lambda_ {t}) ^ {2} \right] - \mathbb {E} \left[ (\lambda_ {\tau} ^ {\star} - \lambda_ {t + 1}) ^ {2} \right]}{2 \eta} + \frac {1}{2} \eta (C _ {\tau , \xi} ^ {\prime}) ^ {2}.
$$
Using the definition  $\mathbb{E}\left[\Phi_t\right] := \mathbb{E}\left[\mathrm{KL}_t(\rho)\right] + \frac{1}{2}\mathbb{E}\left[(\lambda_\tau^\star -\lambda_t)^2\right]$ , we combine the two inequalities above to show that,
$$
\begin{array}{l} \mathbb {E} \left[ \Phi_ {t + 1} \right] \leq (1 - \eta \tau) \mathbb {E} \left[ \Phi_ {t} \right] - \eta \mathbb {E} \left[ L _ {\tau} (\pi_ {\tau} ^ {\star}, \lambda_ {t}) - L _ {\tau} (\pi_ {\theta_ {t}}, \lambda_ {\tau} ^ {\star}) \right] + \eta^ {2} \max \left((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2}\right) \\ + 2 \eta \sqrt {| A | \epsilon_ {\text {b i a s}}} + 2 \eta \sqrt {| A | \kappa_ {\nu} \epsilon_ {\text {s t a t}}} \\ \leq (1 - \eta \tau) \mathbb {E} \left[ \Phi_ {t} \right] + \eta^ {2} \max \left((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2}\right) + 2 \eta \sqrt {| A | \epsilon_ {\mathrm {b i a s}}} + 2 \eta \sqrt {| A | \kappa_ {\nu} \epsilon_ {\mathrm {s t a t}}}, \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \Phi_ {t + 1} \right] \leq (1 - \eta \tau) \mathbb {E} \left[ \Phi_ {t} \right] + \eta^ {2} \max \left((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2}\right) + 2 \eta \sqrt {| A | \epsilon_ {\mathrm {b i a s}}} + 2 \eta \sqrt {| A | \kappa_ {\nu} \epsilon_ {\mathrm {s t a t}}}. \\ \leq \left. (1 - \eta \tau) ^ {2} \mathbb {E} \left[ \Phi_ {t - 1} \right] + \left(\eta^ {2} + \eta^ {2} (1 - \eta \tau)\right) \max  \left((C _ {W}) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \right. \\ + 2 \eta (1 + (1 - \eta \tau)) (\sqrt {| A | \epsilon_ {\text {b i a s}}} + 2 \sqrt {| A | \kappa_ {\nu} \epsilon_ {\text {s t a t}}}) \\ <   \dots \\ \leq \left. (1 - \eta \tau) ^ {t} \mathbb {E} \left[ \Phi_ {1} \right] + \left(\eta^ {2} \left(1 + (1 - \eta \tau) + (1 - \eta \tau) ^ {2} + \dots\right)\right) \max  \left(\left(C _ {W}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) \right. \\ + 2 \eta (1 + (1 - \eta \tau) + (1 - \eta \tau) ^ {2} + \dots) (\sqrt {| A | \epsilon_ {\text {b i a s}}} + \sqrt {| A | \kappa_ {\nu} \epsilon_ {\text {s t a t}}}) \\ \leq \left(1 - \eta \tau\right) ^ {t} \mathbb {E} \left[ \Phi_ {1} \right] + \frac {\eta}{\tau} \max  \left(\left(C _ {W}\right) ^ {2}, \left(C _ {\tau , \xi} ^ {\prime}\right) ^ {2}\right) + \frac {2}{\tau} \left(\sqrt {| A | \epsilon_ {\mathrm {b i a s}}} + \sqrt {| A | \kappa_ {\nu} \epsilon_ {\mathrm {s t a t}}}\right) \\ \leq \mathrm {e} ^ {- \eta \tau t} \mathbb {E} \left[ \Phi_ {1} \right] + \frac {\eta}{\tau} \max  \left((C _ {W}) ^ {2}, (C _ {\tau , \xi} ^ {\prime}) ^ {2}\right) + \frac {2}{\tau} \left(\sqrt {| A | \epsilon_ {\text {b i a s}}} + \sqrt {| A | \kappa_ {\nu} \epsilon_ {\text {s t a t}}}\right) \\ \end{array}
$$
C.7 Proof of Corollary 5
Proof. The proof is similar to the proof of Corollary 3, except that we take the expectation over the randomness of computing  $w_{t}$  via a sample-based algorithm.
According to Theorem 4 and  $\epsilon_{\mathrm{stat}} = O(\epsilon^4)$ ,  $\epsilon_{\mathrm{stat}} = O(\epsilon^4)$ , if we take  $\tau = \Theta(\epsilon)$  and  $\eta = \Theta(\epsilon^2)$ , then  $\mathbb{E}[\Phi_{t+1}] = O(\epsilon)$  for any  $t = \Omega\left(\frac{1}{\epsilon^3}\log \frac{1}{\epsilon}\right)$ . We next consider a primal-dual iterate  $(\pi_{\theta_t},\lambda_t)$  for some  $t = \Omega\left(\frac{1}{\epsilon^3}\log \frac{1}{\epsilon}\right)$ . It is straightforward to check that
$$
\mathbb {E} \left[ \mathrm {K L} _ {t} (\rho) \right] = O (\epsilon) \text {a n d} \frac {1}{2} \mathbb {E} \left[ \left(\lambda_ {\tau} ^ {\star} - \lambda_ {t}\right) ^ {2} \right] = O (\epsilon).
$$
$$
\mathbb {E} \left[ V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right] = \underbrace {V _ {r} ^ {\pi^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\tau} ^ {\star}} (\rho)} _ {\text {(i)}} + \underbrace {\mathbb {E} \left[ V _ {r} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {r} ^ {\pi_ {\theta_ {t}}} (\rho) \right]} _ {\text {(i i)}}. \tag {24}
$$
For the term (ii), because of  $\mathrm{KL}_t(\rho) = O(\epsilon)$ , we have
$$
\begin{array}{l} \text {(i i)} = \mathbb {E} \left[ \frac {1}{1 - \gamma} \sum_ {s, a} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \left(\pi_ {\tau} ^ {\star} (a | s) - \pi_ {\theta_ {t}} (a | s)\right) Q _ {r} ^ {\pi_ {\theta_ {t}}} (s, a) \right] \tag {ii} \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \mathbb {E} \left[ \| \pi_ {\tau} ^ {\star} (\cdot | s) - \pi_ {\theta_ {t}} (\cdot | s) \| _ {1} \right] \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sum_ {\sigma} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \sqrt {2 \mathbb {E} [ \mathrm {K L} _ {t} (s) ]} \\ \leq \frac {1}{(1 - \gamma) ^ {2}} \sqrt {\sum_ {s} d _ {\rho} ^ {\pi_ {\tau} ^ {\star}} (s) \mathbb {E} [ \mathrm {K L} _ {t} (s) ]} \\ = \frac {1}{(1 - \gamma) ^ {2}} \sqrt {2 \mathbb {E} [ \mathrm {K L} _ {t} (\rho) ]} \\ \end{array}
$$
where we use Cauchy-Schwarz inequality in the first and third inequalities, and the second inequality is due to Pinsker's inequality and Jensen's inequality, which shows  $\mathbb{E}\left[V_r^{\pi_\tau^\star}(\rho) - V_r^{\pi_{\theta_t}}(\rho)\right] \leq O(\sqrt{\epsilon})$ . For the term (i), if we take  $\pi = \pi^{\star}$  in (5), then,
We now substitute the upper bounds of (i) and (ii) into (24) to obtain  $\mathbb{E}\left[V_r^{\pi^*}(\rho) - V_r^{\pi_{\theta_t}}(\rho)\right] \leq O(\sqrt{\epsilon})$ , where we take  $\tau = \Theta(\epsilon)$ .
$$
\mathbb {E} \left[ - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right] = \underbrace {- V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho)} _ {\text {(i i i)}} + \underbrace {\mathbb {E} \left[ V _ {g} ^ {\pi_ {\tau} ^ {\star}} (\rho) - V _ {g} ^ {\pi_ {\theta_ {t}}} (\rho) \right]} _ {\text {(i v)}}. \tag {25}
$$