SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| Q ^ {u _ {N}} - Q ^ {u _ {*} ^ {t}} \right\| _ {\nu^ {t} \circ \pi^ {t}} ^ {2} \right] = \mathbb {E} \left[ \left(u _ {*} ^ {t} - u _ {N}\right) ^ {\top} \left(\Phi^ {\top} \operatorname {d i a g} \left(\nu^ {t} \circ \pi^ {t}\right) \Phi\right) \left(u _ {*} ^ {t} - u _ {N}\right) \right] \\ \leq \lambda_ {m a x} (\Phi^ {\top} \operatorname {d i a g} (\nu^ {t} \circ \pi^ {t}) \Phi) \mathbb {E} [ \| u _ {*} ^ {t} - u _ {N} \| ^ {2} ] = \tilde {O} (1 / N). \\ \end{array}
$$
For the update at step  $t$ ,  $\theta^{t + 1} = \mathsf{RQNPG}(\theta^t,\eta^t,w^t,N)$  and  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}},K)$ , let  $\pi^t = \pi_{\theta^t}$  be the policy at step  $t$ . Let  $u^{t} = u_{N}$  ( $u_{N}$  in Algorithm 3) and Lemma 3 above shows that  $\mathbb{E}[\| Q^{u^t} - Q_{*}^{u_t}\|_{\nu^t\circ \pi^t}^2 ] = \tilde{O} (\frac{1}{N})$ . This does not necessarily imply that  $Q^{u^t}$  and  $Q^{\pi^t}$  are close since the property of the critic returned  $w^{t}$  is required. We measure the difference between  $Q^{u^t}$  and  $Q^{\pi^t}$  in the following lemma.
Let  $w_{*}^{t} = w^{\pi^{t}}$  and  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}}, K)$  is the output of RLTD at step  $t$ . Define  $\epsilon_{V,stat} := \max_{t=0,1,\dots,T-1}\{\sqrt{\mathbb{E}[||V_{w_{*}^{t}} - V_{w^{t}}||_{\nu^{t}}^{2}]} \}$ , and  $\epsilon_{V,stat} = \tilde{O}(1/\sqrt{K})$  by Theorem 3.
Lemma 4. Under the conditions in Theorem 5 and Assumption 3, there is some  $\epsilon = \epsilon_{stat} + \epsilon_{bias}$  that for any  $\pi, \pi'$  and any  $d = d_{\rho}^{\pi', \kappa_{\pi'}}$ ,  $d_{\rho}^{\pi^{*}, \kappa_{\pi^{t}}}$  or  $\nu^{\pi^{*}}$ ,
$$
\left| \mathbb {E} \left[ \mathbb {E} _ {(s, a) \sim d \circ \pi} \left[ Q ^ {u ^ {t}} (s, a) - Q ^ {\pi^ {t}} (s, a) \right] \right] \right| \leq \epsilon , \tag {16}
$$
where the outside expectation is taken w.r.t. the randomness of the data collected in  $w^t = R\mathsf{LTD}(\pi_{\theta^t},K)$  and  $\theta^{t + 1} = RQNPG(\theta^t,\eta^t,w^t,N)$ . Moreover,  $\epsilon_{stat} = \sqrt{|\mathcal{A}|}$  ( $\xi \beta \epsilon_{V,stat} + \xi \epsilon_{Q,stat}$ ) with  $\epsilon_{V,stat} = \tilde{O}\left(\frac{1}{\sqrt{K}}\right)$  and  $\epsilon_{Q,stat} = \tilde{O}\left(\frac{1}{\sqrt{N}}\right)$ ;  $\epsilon_{bias} = \sqrt{|\mathcal{A}|}\left(\frac{\xi\beta}{1 - \beta}\epsilon_{V,bias} + \epsilon_{Q,bias}\right)$  with  $\epsilon_{V,bias}$  and  $\epsilon_{Q,bias}$  in Definitions 3 and 5, respectively.
Proof of Lemma 4. For any  $d = d_{s}^{\pi^{t + 1},\kappa^{t + 1}}$ ,  $d_{\rho}^{\pi^{*},\kappa^{t}}$  or  $\nu^{\pi^{*}}$  and for any  $\pi$ , we know for any  $Q,Q'$
$$
\begin{array}{l} \left| \mathbb {E} \left[ \mathbb {E} _ {(s, a) \sim d \circ \pi} [ Q (s, a) - Q ^ {\prime} (s, a) ] \right] \right| \leq \mathbb {E} \left[ \sqrt {| \mathcal {A} | \mathbb {E} _ {(s , a) \sim d \circ \text {U n i f}} [ (Q (s , a) - Q ^ {\prime} (s , a)) ^ {2} ]} \right] \\ = \sqrt {| \mathcal {A} |} \mathbb {E} \left[ \| Q - Q ^ {\prime} \| _ {d \circ \text {U n i f}} \right]. \\ \end{array}
$$
To quantify the error between  $Q^{u^t} - Q^{\pi^t}$ , recall  $Q_{*}^{u^t} = \Pi_{\Phi}^{\pi^t}Q_{w^t}$  and we decompose it into
$$
Q ^ {u ^ {t}} - Q ^ {\pi^ {t}} = \left(Q ^ {u ^ {t}} - Q ^ {u _ {*} ^ {t}}\right) + \left(\Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}}\right) + \left(\Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q ^ {\pi^ {t}}\right) + \left(\Pi_ {\Phi} ^ {\pi^ {t}} Q ^ {\pi^ {t}} - Q ^ {\pi^ {t}}\right),
$$
and bound each term respectively. We can transfer the norm within the space spanned by  $\Phi$  via the assumption that  $\| \Phi u\|_{d\circ \mathrm{Unif}}\leq \xi \| \Phi u\|_{\nu^{\pi}\circ \pi}$ . Note that the first three terms all lie in the  $span(\Phi)$ , we have the first term bounded by
$$
\mathbb {E} \left[ \| Q ^ {u ^ {t}} - Q _ {*} ^ {u _ {*} ^ {t}} \| _ {d \circ \mathrm {U n i f}} \right] \leq \xi \mathbb {E} \left[ \| Q ^ {u ^ {t}} - Q _ {*} ^ {u _ {*} ^ {t}} \| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} \right] \leq \xi \sqrt {\mathbb {E} \left[ \| Q ^ {u ^ {t}} - Q _ {*} ^ {u _ {*} ^ {t}} \| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} ^ {2} \right]} \leq \xi \epsilon_ {Q, s t a t},
$$
for some  $\epsilon_{Q,stat} = \tilde{O}\left(\frac{1}{\sqrt{N}}\right)$  as in Lemma 3. The second term is bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}} \right\| _ {d \circ \mathrm {U n i f}} \right] \leq \xi \mathbb {E} \left[ \left\| \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}} \right\| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} \right] \\ \leq \xi \mathbb {E} \left[ \| Q _ {w ^ {t}} - Q _ {w _ {*} ^ {t}} \| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} \right] \leq \xi \beta \mathbb {E} [ \| V _ {w ^ {t}} - V _ {w _ {*} ^ {t}} \| _ {\nu^ {t}} ] \leq \xi \beta \sqrt {\mathbb {E} [ \| V _ {w ^ {t}} - V _ {w _ {*} ^ {t}} \| _ {\nu^ {t}} ^ {2} ]} \leq \xi \beta \epsilon_ {V, s t a t} \\ \end{array}
$$
for some  $\epsilon_{V,stat} = \tilde{O}\left(\frac{1}{\sqrt{K}}\right)$  as in Theorem 5. The third term is bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q ^ {\pi^ {t}} \right\| _ {d \circ \mathrm {U n i f}} \right] \leq \xi \mathbb {E} \left[ \left\| \Pi_ {\Phi} ^ {\pi^ {t}} Q _ {w _ {*} ^ {t}} - \Pi_ {\Phi} ^ {\pi^ {t}} Q ^ {\pi^ {t}} \right\| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} \right] \leq \xi \mathbb {E} \left[ \left\| Q _ {w _ {*} ^ {t}} - Q ^ {\pi^ {t}} \right\| _ {\nu^ {\pi^ {t}} \circ \pi^ {t}} \right] \\ \leq \xi \beta \mathbb {E} [ \| V _ {w _ {*} ^ {t}} - V ^ {t} \| _ {\nu^ {\pi}} ] \leq \xi \beta \frac {\mathbb {E} [ \| \Pi^ {t} V ^ {t} - V ^ {t} \| _ {\nu^ {\pi}} ]}{1 - \beta} \leq \frac {\xi \beta \epsilon_ {V , b i a s}}{1 - \beta}, \\ \end{array}
$$
where the last inequality is by Definition 3 and inequality (13). The last term is then bounded by Definition 5  $\mathbb{E}[\| \Pi_{\Phi}^{\pi^t}Q^{\pi^t} - Q^{\pi^t}\|_{d\circ \mathrm{Unif}}]\leq \epsilon_{Q,bias}$  . We thus have
$$
\left| \mathbb {E} \left[ \mathbb {E} _ {(s, a) \sim d \circ \pi} [ Q ^ {u ^ {t}} (s, a) - Q ^ {\pi^ {t}} (s, a) ] \right] \right| \leq \sqrt {| \mathcal {A} |} \left(\xi \epsilon_ {Q, s t a t} + \xi \beta \epsilon_ {V, s t a t} + \frac {\xi \beta}{1 - \beta} \epsilon_ {V, b i a s} + \epsilon_ {Q, b i a s}\right),
$$
D.2.2 RQNPG One-step Analysis - Mirror Ascent Update
Now we look at the policy improvement of the update  $\theta^{t + 1} = \mathbb{R}\mathbb{Q}\mathbb{N}\mathbb{P}\mathbb{G}(\theta^t,\eta^t,w^t,N)$  with  $\zeta_{n} = \Theta (1 / n)$  (Algorithm 3), where  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}},K)$ . Let  $u^{t} = u_{N}$  ( $u_{N}$  in Algorithm 3), and we know the RQNPG update is  $\theta^{t + 1} = \theta^t +\eta^t u^t$
This RQNPG update is equivalent to a certain mirror ascent update. Specifically, recall  $Q^{u^t}(s,a) = \phi (s,a)^\top u^t$  is the approximated robust Q function, the RQNPG update  $\theta^{t + 1} = \theta^t +\eta^t u^t$  is equivalent to [4]
$$
\pi_ {s} ^ {t + 1} \leftarrow \arg \max  _ {\pi \in \Delta_ {\mathcal {A}}} \left\{\eta^ {t} \left\langle Q _ {s} ^ {u ^ {t}}, \pi \right\rangle - \mathrm {K L} \left(\pi , \pi_ {s} ^ {t}\right) \right\} \quad \forall s \in \mathcal {S}, \tag {17}
$$
where we let  $\pi_s \coloneqq \pi(\cdot|s)$  and  $Q_s \coloneqq Q(s, \cdot)$  for simplicity. Note that this update can be viewed as a mirror descent step with KL-divergence as Bregman divergence. Given this mirror descent formulation of policy update in Eq (17), the pushback property indicates that for any policy  $\pi$  (Eq (2) in [68]),
$$
\eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t + 1} \rangle - \mathrm {K L} (\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}) \geq \eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} \rangle - \mathrm {K L} (\pi_ {s}, \pi_ {s} ^ {t}) + \mathrm {K L} (\pi_ {s}, \pi_ {s} ^ {t + 1}),
$$
which is equivalent to the following fundamental inequality
$$
\eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} - \pi_ {s} ^ {t} \rangle + \eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \leq \mathrm {K L} (\pi , \pi_ {s} ^ {t}) - \mathrm {K L} (\pi , \pi_ {s} ^ {t + 1}) - \mathrm {K L} (\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}). \tag {18}
$$
Restatement of Theorem 4 (Approximate policy improvement) For any  $t \geq 0$ , we know
$$
V ^ {\pi^ {t + 1}} (\rho) \geq V ^ {\pi^ {t}} (\rho) + \frac {\mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa_ {\pi^ {t + 1}}}} \left(\pi^ {t} , \pi^ {t + 1}\right) + \mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa_ {\pi^ {t + 1}}}} \left(\pi^ {t + 1} , \pi^ {t}\right)}{(1 - \gamma) \eta^ {t}} - \frac {\epsilon_ {t}}{1 - \gamma}, \tag {19}
$$
Proof of Theorem 4. Let  $\kappa^{t + 1} = \kappa_{\pi^{t + 1}}$  be the worst-case transition kernel w.r.t. policy  $\pi^{t + 1}$ , and let  $\epsilon_t = \mathbb{E}_{s\sim d_\rho^{\pi^{t + 1},\kappa^{t + 1}}}[\langle Q_s^{\pi^t} - Q_s^{u^t},\pi_s^{t + 1} - \pi_s^t\rangle ]$ . We have
$$
\begin{array}{l} V ^ {\pi^ {t + 1}} (\rho) - V ^ {\pi^ {t}} (\rho) \geq \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s} ^ {\pi^ {t}}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle ] \\ = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle ] + \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s} ^ {\pi^ {t}} - Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle ] \\ \end{array}
$$
$$
\geq \frac {\mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa^ {t + 1}} \left(\pi^ {t} , \pi^ {t + 1}\right) + \mathrm {K L} _ {d _ {\rho} ^ {\pi^ {t + 1} , \kappa^ {t + 1}} \left(\pi^ {t + 1} , \pi^ {t}\right)}}}{(1 - \gamma) \eta^ {t}} - \frac {\epsilon_ {t}}{1 - \gamma},
$$
where the first inequality is by Lemma 8, and the last inequality is by taking  $\pi = \pi^t$  in the fundamental inequality Eq (18), which implies
$$
\eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle \geq \mathrm {K L} (\pi_ {s} ^ {t}, \pi_ {s} ^ {t + 1}) + \mathrm {K L} (\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}).
$$
$\epsilon_{t}$  can then be bounded by Lemma 4 with  $\mathbb{E}[\epsilon_t] \leq 2\epsilon = 2\epsilon_{stat} + 2\epsilon_{bias}$ , where  $\epsilon_{stat}, \epsilon_{bias}$  are specified in Lemma 4 with  $\epsilon_{stat} = \tilde{O}\left(\frac{1}{\sqrt{N}} + \frac{1}{\sqrt{K}}\right)$ .
D.3 Extension of General Function Approximation of Policy
Setting: In this subsection, we consider a general policy class of form
$$
\left\{\pi_ {\theta} (a | s) = \frac {\exp \left(f _ {\theta} (s , a)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp \left(f _ {\theta} \left(s , a ^ {\prime}\right)\right)} \mid \theta \in \mathbb {R} ^ {d} \right\}, \tag {20}
$$
where  $f_{\theta}$  is a differentiable function. This general policy class contains the log-linear policy class as a special case by  $f_{\theta}(s,a) = \phi (s,a)^{\top}\theta$ .
D.3.1 Robust Natural Policy Gradient with General Function Approximation
For the general policy class in Eq (20), we propose a Robust NPG (RNPG) algorithm.
This algorithm can be applied to RNAC (Algorithm 1) for the robust natural actor update. Now we look at a specific update  $\theta^{t + 1} = \mathsf{RNPG}(\theta^t,\eta^t,w^t,N)$ , where  $w^{t}$  is the output of the robust critic at step  $t$ . Note that for the critic with general function approximation Eq. (14), we slightly abuse the notation by  $V_{w^t} = g^t$ , where  $g^{t}$  is the output of the FRVE (Algorithm 5) at step  $t$  of RNAC. We can view  $g\in \mathcal{G}$  is parameterized by some  $w$ , as indicated in the RNAC algorithm Algorithm 1.
Denote by  $\phi^{\theta}(s,a)\coloneqq \nabla_{\theta}\log \pi_{\theta}(a_n|s_n)$  and  $\Phi^{\theta}\in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times d}$  as the feature matrix stacking up feature vector  $\phi^{\theta}$ . For each  $t = 0,1,\ldots ,T - 1$ , we let  $\phi^t\coloneqq \phi^{\theta^t}$  and  $\Phi^t\coloneqq \Phi^{\theta^t}$  for simplicity. The RNPG update is  $\theta^{t + 1}\gets \theta^t +\eta^t u^t$ , where  $u^{t} = u_{N}$  as the output of the stochastic gradient descent in Algorithm 6,
$$
u _ {n + 1} = (1 - \lambda) u _ {n} + \zeta_ {n} \nabla_ {\theta} \log \pi_ {\theta^ {t}} (a _ {n} | s _ {n}) \left[ (\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi_ {\theta^ {t}}} V _ {w ^ {t}}) (s _ {n}, a _ {n}, y _ {n + 1}) - V _ {w ^ {t}} (s _ {n}) - \nabla_ {\theta} \log \pi_ {\theta} (a _ {n} | s _ {n}) ^ {\top} u _ {n} \right].
$$
RNPG approximates the value approximated advantage function  $A_{w^t}(s,a) \coloneqq r(s,a) + \gamma \inf_{p\in \mathcal{P}_{s,a}}p^\top V_{w^t} - V_{w^t}(s)$  by  $A_t^u \coloneqq \Phi^t u$ . Note that  $(\hat{T}_{\mathcal{P}}^{\pi^t}V_{w^t})(s_n,a_n,y_{n + 1}) - V_{w^t}(s_n)$  is an unbiased estimate of  $A_{w^t}(s_n,a_n)$ , i.e.,
$$
\mathbb {E} \left[ \left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi^ {t}} V _ {w ^ {t}}\right) \left(s _ {n}, a _ {n}, y _ {n + 1}\right) - V _ {w ^ {t}} \left(s _ {n}\right) \mid s _ {n}, a _ {n} \right] = A _ {w ^ {t}} \left(s _ {n}, a _ {n}\right).
$$
RNPG thus is iteratively solving the following optimization
$$
\frac {1}{2} \| A _ {w ^ {t}} - \Phi^ {t} u \| _ {\nu^ {t} \circ \pi^ {t}} ^ {2} + \frac {\lambda}{2} \| u \| ^ {2}, \tag {21}
$$
by stochastic approximation (stochastic gradient descent with Markovian data). Denote by  $u_{*}^{\pi^{t}}$  the optimal value of the optimization, which is also a solution of the equation
$$
\begin{array}{l} 0 = \left(\Phi^ {t}\right) ^ {\top} \left(A _ {w ^ {t}} - \Phi^ {t} u\right) - \lambda u \\ = \mathbb {E} _ {(s, a, y ^ {\prime}) \sim \nu^ {\pi} \circ \pi \circ p ^ {\circ}} \left[ \phi^ {t} (s, a) \left(\left(\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi^ {t}} V _ {w ^ {t}}\right) (s, a, y ^ {\prime}) - V _ {w ^ {t}} (s) - \phi^ {t} (s, a) ^ {\top} u\right) \right] - \lambda u. \\ \end{array}
$$
Theorem 7 (Convergence of compatible advantage function approximation (SGD with Markovian data)). Under assumption 1,  $RNPG(\theta^t,\eta^t,w^t,N)$  (Algorithm 6) with step sizes  $\zeta_{n} = \Theta (1 / n)$  guarantees  $\mathbb{E}[\| u_N - u_*^t\|^2 ] = \tilde{O} (\frac{1}{N})$  and  $\mathbb{E}[\| A_t^{u_N} - A_t^{u_*}\|_{\nu^t\circ \pi^t}^2 ] = \tilde{O} (\frac{1}{N})$
Proof of Theorem 7. The theorem can be proved in the same manner as that for Lemma 3, since the objective function in Eq (21) is strongly convex.  $\square$
With slight abuse of notation, denote by  $\Sigma_{\nu,\pi}^t \coloneqq \mathbb{E}_{(s,a) \sim \nu \circ \pi}[\phi^t(s,a)\phi^t(s,a)^\top]$ . The optimal value  $u_*^t = (\lambda I + (\Phi^t)^\top \mathrm{diag}(\nu^t \circ \pi^t)\Phi^t)^{-1}((\Phi^t)^\top \mathrm{diag}(\nu^t \circ \pi^t)A_{w^t})$  satisfies  $\| u_*^t \| \leq \frac{\| A_{w^t} \|_{\nu^{t_0\pi^t}} \sqrt{\sum_{i=1}^d \| \phi_i^t \|_{\nu^{t_0\pi^t}}^2}}{\lambda + \lambda_{\min}(\Sigma_{\nu^{t_0\pi^t}}^t)}$ . Let  $u^t = u_N$ , Theorem 7 gives  $\mathbb{E}[ \| u^t \|^2] \leq 2\mathbb{E}[ \| u_*^t \|^2] + \tilde{O}\left(\frac{1}{N}\right)$ . We then have  $\mathbb{E}[ \| u^t \|^2] \leq U$  under the following assumption.
Assumption 4 (Bounded feature). Assume  $\sup_{\theta} \sum_{i=1}^{d} \|\phi_i^\theta\|_{\nu^{\pi_\theta \circ \pi_\theta}}^2 < \infty$ . Since  $\mathcal{G}$  (14) is bounded,  $\|A_{w^t}\|_{v^t \circ \pi^t} < \infty$  and there exists  $0 < U < \infty$  that  $\max_t \mathbb{E}[\|u^t\|^2] \leq U$ .
D.3.2 Robust General Advantage Function Approximation
Denote by  $\Pi_{\Phi^t}^t$  the projection mapping to the space  $\{\Phi^t u:\| u\| \leq \| u_*^t\| \}$  under metric  $\| \cdot \|_{\nu^t\circ \pi^t}$ .
Definition 6.  $\epsilon_{A,bias} := \max_{t = 0,1,\dots,T - 1}\mathbb{E}\left[\| \Pi_{\Phi^t}^t A^t -A^t\|_{\nu^{\pi_*}\circ \pi^*}\right]$
Note that  $\epsilon_{A,bias}$  also implicitly depends on the choice of  $\lambda$  since  $u_{*}^{t}$  depends on  $\lambda$ . If the realizable case, i.e.,  $A^{\pi^t} \in \text{span}(\Phi^t)$ ,  $\epsilon_{A,bias} = 0$  if  $\lambda = 0$ .
Assumption 5. There is some  $0 < \xi' < \infty$  that for any  $\theta$ ,  $\sup_u \frac{\|\Phi^\theta u\|_{\nu^{\pi^*} \circ \pi^*}}{\|\Phi^\theta u\|_{\nu^{\pi_\theta} \circ \pi_\theta}} \leq \xi'$ .
Lemma 5. Under the conditions in Theorems 7 and 6 and Assumption 5, for any  $t$ ,
$$
\left| \mathbb {E} \left[ \mathbb {E} _ {(s, a) \sim \nu^ {\pi^ {*}} \circ \pi^ {*}} \left[ A ^ {u ^ {t}} (s, a) - A ^ {\pi^ {t}} (s, a) \right] \right] \right| \leq \epsilon^ {\prime} = \epsilon_ {s t a t} ^ {\prime} + \epsilon_ {b i a s} ^ {\prime}, \tag {22}
$$
where the outside expectation is taken w.r.t. the randomness of the data collected the robust natural actor and robust critic update. Moreover,  $\epsilon_{stat}^{\prime} = \tilde{O}\left(\frac{1}{\sqrt{K}} + \frac{1}{\sqrt{N}}\right)$ ;  $\epsilon_{bias}^{\prime} = O(\epsilon_{A,bias} + \epsilon_{g,bias})$  or  $O(\epsilon_{A,bias} + \epsilon_{V,bias})$  for linear robust critic.
Proof of Lemma 5. To quantify the error between  $A^{u^t} - A^t$ , we decompose it into
$$
A ^ {u ^ {t}} - A ^ {t} = (A ^ {u ^ {t}} - A ^ {u _ {*} ^ {t}}) + (\Pi_ {\Phi^ {t}} ^ {t} A _ {w ^ {t}} - \Pi_ {\Phi^ {t}} ^ {t} A _ {w _ {*} ^ {t}}) + (\Pi_ {\Phi^ {t}} ^ {t} A _ {w _ {*} ^ {t}} - \Pi_ {\Phi^ {t}} ^ {t} A ^ {t}) + (\Pi_ {\Phi^ {t}} ^ {t} A ^ {t} - A ^ {t}),
$$
and bound each term respectively. The proof follows similarly to that in Lemma 4 by applying Theorem 7 and Theorem 6.
E Robust Natural Actor-Critic Analysis
In this section, we first state and prove the formal versions of the main theorems Theorem 1 and Theorem 2 in Theorem 8 and Theorem 9, respectively. We then give the convergence of RNAC employing general function approximation in Theorem 10.
E.1 Linear Function Approximation
We introduce the detailed setup of the RNAC algorithm with linear function approximation, based on which the theorems are stated and proved.
RNAC-Linear Setting: We study the RNAC (Algorithm 1) with robust critic performing RLTD (Algorithm 2 with  $\xi_{n} = \Theta(1/k)$  as in Theorem 5) and robust natural actor performing RQNPG (Algorithm 3 with  $\zeta_{n} = \Theta(1/n)$  as in Lemma 3) under linear value function approximation as in Eq (12) and log-linear policy as in Eq (15), for DS or IPM uncertainty sets taking  $\delta$  suggested by Proposition 4 or Lemma 1, respectively. We assume Assumption 1 and Assumption 3 hold.
For each time  $t = 0,1,\ldots ,T - 1$ , the value approximation and policy in RNAC are updated as  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}},K)$  and  $\theta^{t + 1} = \mathsf{RQNPG}(\theta^t,\eta^t,w^t,N)$ , respectively. Denote by  $u^{t}$  as  $u_{N}$  in RQNPG  $(\theta^t,\eta^t,w^t,N)$  (Algorithm 3), which defines a robust Q-function approximation  $Q^{u^t}$ . Let  $\pi^t = \pi_{\theta^t},\kappa^t = \kappa_{\pi^t},V^t = V^{\pi^t}$  for simplicity. We have the following lemma.
Lemma 6. Under the RNAC-Linear Setting, for any  $t = 0,1,\dots ,T - 1$  we have  $\langle Q_s^{u^t},\pi_s^{t + 1} - \pi_s^t\rangle \geq 0,\forall s\in S$  and
$$
\mathbb {E} [ \mathbb {E} _ {s \sim \rho} [ \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle ] ] \geq \mathbb {E} [ V ^ {t} (\rho) ] - \mathbb {E} [ V ^ {t + 1} (\rho) ] - \frac {2 \epsilon}{1 - \gamma},
$$
where the expectation  $\mathbb{E}$  is taken w.r.t. the data sampled by RNAC, and  $\epsilon = \epsilon_{stat} + \epsilon_{bias}$  is the same as that in Lemma 4.
Proof of Lemma 6. Taking  $\pi = \pi^t$  in the fundamental inequality Eq (18) gives
$$
\eta^ {t} \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \leq - \mathrm {K L} (\pi_ {s} ^ {t}, \pi_ {s} ^ {t + 1}) - \mathrm {K L} (\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}) \leq 0,
$$
which implies  $\pi^{t + 1}$  is indeed improving  $\pi^t$  along  $Q^{u^t}$  direction. We then have
$$
\begin{array}{l} \left\langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \right\rangle \geq \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d _ {s} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} \left[ \left\langle Q _ {s ^ {\prime}} ^ {u ^ {t}}, \pi_ {s ^ {\prime}} ^ {t} - \pi_ {s ^ {\prime}} ^ {t + 1} \right\rangle \right] \\ = - \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d _ {s} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s ^ {\prime}} ^ {\pi^ {t}}, \pi_ {s ^ {\prime}} ^ {t + 1} - \pi_ {s ^ {\prime}} ^ {t} \rangle ] + \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d _ {s} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s ^ {\prime}} ^ {u ^ {t}} - Q _ {s ^ {\prime}} ^ {\pi^ {t}}, \pi_ {s ^ {\prime}} ^ {t} - \pi_ {s ^ {\prime}} ^ {t + 1} \rangle ] \\ = -\frac{1}{1 - \gamma}\mathbb{E}_{s^{\prime}\sim d_{s}^{\pi^{t + 1},\kappa^{t + 1}},a^{\prime}\sim \pi_{s^{\prime}}^{t + 1}}[A^{t}(s^{\prime},a^{\prime})]\bigg] + \frac{1}{1 - \gamma}\mathbb{E}_{s^{\prime}\sim d_{s}^{\pi^{t + 1},\kappa^{t + 1}}}[\langle Q_{s^{\prime}}^{u^{t}} - Q_{s^{\prime}}^{\pi^{t}},\pi_{s^{\prime}}^{t} - \pi_{s^{\prime}}^{t + 1}\rangle ] \\ \geq V ^ {t} (s) - V ^ {t + 1} (s) + \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d _ {s} ^ {\pi^ {t + 1}, \kappa^ {t + 1}}} [ \langle Q _ {s ^ {\prime}} ^ {u ^ {t}} - Q _ {s ^ {\prime}} ^ {\pi^ {t}}, \pi_ {s ^ {\prime}} ^ {t} - \pi_ {s ^ {\prime}} ^ {t + 1} \rangle ], \\ \end{array}
$$
where the second inequality is by Lemma 8. The proof of the lemma is then concluded by taking expectation  $\mathbb{E}[\mathbb{E}_{s\sim \rho}[\cdot ]]$  on both sides and applying Lemma 4.
Assumption 6. For initial state distribution  $\rho$ , there exists  $M$  that  $\sup_{\kappa \in \mathcal{P}}\| \frac{d^{\ast,\kappa}}{\rho}\|_{\infty} \leq M < \infty$ .
Theorem 8 (Formal statement of Theorem 1). Under RNAC-Linear Setting and Assumption 6, RNAC with geometrically increasing step sizes  $\eta^t \geq \frac{\frac{M}{1 - \gamma}}{1 - \frac{1 - \gamma}{M}}\eta^{t - 1}$  for each  $t = 1,2,\ldots,T$ , satisfies
$$
\begin{array}{l} V ^ {*} (\rho) - \mathbb {E} [ V ^ {T} (\rho) ] \leq \left(1 - \frac {1 - \gamma}{M}\right) ^ {T - 1} \left((1 - \frac {1 - \gamma}{M}) (V ^ {*} (\rho) - V ^ {0} (\rho)) + \log | \mathcal {A} |\right) \\ + \left(\frac {2}{1 - \gamma} + \frac {2 M}{(1 - \gamma) ^ {2}}\right) \epsilon , \\ \end{array}
$$
where  $\epsilon = \epsilon_{stat} + \epsilon_{bias}$  as in Lemma 4 with  $\epsilon_{stat} = \tilde{O} \left( \frac{1}{\sqrt{K}} + \frac{1}{\sqrt{N}} \right)$ . Omitting the approximation  $\epsilon_{bias}$ , the sample complexity for achieving  $\varepsilon$  robust optimal value (i.e.,  $V^{*}(\rho) - \mathbb{E}[V^{T}(\rho)] \leq \varepsilon$ ) is  $\tilde{O}(1 / \varepsilon^2)$  by taking  $N = K = \tilde{\Theta}(1 / \varepsilon^2)$  and  $T = \Theta(\log(1 / \varepsilon))$ .
Proof of Theorem 8. Taking  $\pi = \pi^{*}$  in inequality (18), we have
$$
\left\langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \right\rangle + \left\langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \right\rangle \leq \frac {1}{\eta^ {t}} \mathrm {K L} \left(\pi_ {s} ^ {*}, \pi_ {s} ^ {t}\right) - \frac {1}{\eta^ {t}} \mathrm {K L} \left(\pi_ {s} ^ {*}, \pi_ {s} ^ {t + 1}\right). \tag {23}
$$
We then take the expectation  $\mathbb{E}[\mathbb{E}_{s\sim d_p^{*,\kappa^t}}[\cdot ]] \text{ on both sides. Note that}$
$$
\begin{array}{l} \mathbb {E} [ \mathbb {E} _ {s \sim d _ {\rho} ^ {*}, \kappa^ {t}} [ \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle ] ] = \mathbb {E} [ \mathbb {E} _ {s \sim d _ {\rho} ^ {*}, \kappa^ {t}} [ \langle Q _ {s} ^ {\pi^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle ] ] + \mathbb {E} [ \mathbb {E} _ {s \sim d _ {\rho} ^ {*}, \kappa^ {t}} [ \langle Q _ {s} ^ {u ^ {t}} - Q _ {s} ^ {\pi^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle ] ] \\ \geq (1 - \gamma) (V ^ {*} (\rho) - \mathbb {E} [ V ^ {t} (\rho) ]) - 2 \epsilon , \\ \end{array}
$$
where the inequality is by Lemma 8 and Lemma 4. Then by Lemma 6 and Assumption 6 that  $M = \sup_{\kappa} \| \frac{d^{*,\kappa}}{\rho} \|_{\infty}$ , we have