SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\rho} ^ {*}, \kappa^ {t}} \left[ \left\langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \right\rangle \right] \right] \geq M \left(\mathbb {E} \left[ V ^ {t} (\rho) \right] - \mathbb {E} \left[ V ^ {t + 1} (\rho) \right] - \frac {2 \epsilon}{1 - \gamma}\right),
$$
since  $\langle Q_s^{u^t},\pi_s^t -\pi_s^{t + 1}\rangle \leq 0$ . The outside expectation  $\mathbb{E}$  is taken w.r.t. the data sampled by RNAC, and since all the following statements are under expectation  $\mathbb{E}$ , we omit  $\mathbb{E}$  in the proof for simplicity and bring it back at the end. Combining the inequality above and Eq (23), we have
$$
\begin{array}{l} (1 - \gamma) (V ^ {*} (\rho) - V ^ {t} (\rho)) - 2 \epsilon + M \left(V ^ {t} (\rho) - V ^ {t + 1} (\rho) - \frac {2 \epsilon}{1 - \gamma}\right) \\ \leq \frac {1}{\eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} \left(\pi^ {*}, \pi^ {t}\right) - \frac {1}{\eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} \left(\pi^ {*}, \pi^ {t + 1}\right). \\ \end{array}
$$
$$
\begin{array}{l} V ^ {*} (\rho) - V ^ {t + 1} (\rho) \leq \left(1 - \frac {1 - \gamma}{M}\right) (V ^ {*} (\rho) - V ^ {t} (\rho)) + \frac {1}{M \eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} (\pi^ {*}, \pi^ {t}) \\ - \frac {1}{M \eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} (\pi^ {*}, \pi^ {t + 1}) + 2 \left(\frac {\epsilon}{M} + \frac {\epsilon}{1 - \gamma}\right), \\ \end{array}
$$
$$
\begin{array}{l} V ^ {*} (\rho) - V ^ {T} (\rho) \leq \left(1 - \frac {1 - \gamma}{M}\right) ^ {T} (V ^ {*} (\rho) - V ^ {0} (\rho)) + \left(1 - \frac {1 - \gamma}{M}\right) ^ {T - 1} \frac {\mathrm {K L} _ {d _ {\rho} ^ {* *}} ^ {*} (\pi^ {*} , \pi^ {0})}{M \eta^ {0}} \\ + \sum_ {t = 1} ^ {T - 1} \left(1 - \frac {1 - \gamma}{M}\right) ^ {T - 1 - t} \left(\frac {1}{M \eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} \left(\pi^ {*}, \pi^ {t}\right) - \frac {1 - \frac {1 - \gamma}{M}}{M \eta^ {t - 1}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t - 1}} \left(\pi^ {*}, \pi^ {t}\right)\right) \\ + \frac {2 M}{1 - \gamma} \left(\frac {\epsilon}{M} + \frac {\epsilon}{1 - \gamma}\right). \\ \end{array}
$$
Take geometrically increasing step sizes  $\eta^t \geq \frac{\frac{M}{1 - \gamma}}{1 - \frac{1 - \gamma}{M}}\eta^{t-1}$  for each  $t$ . Since  $M = \sup_{\kappa} \left\| \frac{d_{\rho}^{*,\kappa}}{\rho} \right\|_{\infty} \geq (1 - \gamma) \sup_{\kappa, \kappa'} \left\| \frac{d_{\rho}^{*,\kappa}}{d_{\rho}^{*,\kappa'}} \right\|_{\infty}$ , we have
$$
\begin{array}{l} \frac {1}{M \eta^ {t}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} \left(\pi^ {*}, \pi^ {t}\right) - \frac {1 - \frac {1 - \gamma}{M}}{M \eta^ {t - 1}} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t - 1}} \left(\pi^ {*}, \pi^ {t}\right) \\ \leq \frac {1 - \frac {1 - \gamma}{M}}{M \eta^ {t - 1}} \left(\frac {1 - \gamma}{M} \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t}} \left(\pi^ {*}, \pi^ {t}\right) - \mathrm {K L} _ {d _ {\rho} ^ {*}, \kappa^ {t - 1}} \left(\pi^ {*}, \pi^ {t}\right)\right) \leq 0. \\ \end{array}
$$
We bring back the expectation  $\mathbb{E}$  over the data and finally arrive at
$$
\begin{array}{l} V ^ {*} (\rho) - \mathbb {E} [ V ^ {T} (\rho) ] \leq \left(1 - \frac {1 - \gamma}{M}\right) ^ {T} (V ^ {*} (\rho) - V ^ {0} (\rho)) + \left(1 - \frac {1 - \gamma}{M}\right) ^ {T - 1} \frac {\mathrm {K L} _ {d _ {\rho} ^ {* k ^ {0}}} (\pi^ {*} , \pi^ {0})}{M \eta^ {0}} \\ + \frac {2 M}{1 - \gamma} \left(\frac {\epsilon}{M} + \frac {\epsilon}{1 - \gamma}\right), \\ \end{array}
$$
where  $\mathrm{KL}_{d_{\rho}^{*\kappa^0}}(\pi^*,\pi^0)\leq \log |\mathcal{A}|$  when  $\pi^0$  is a uniform policy.
Theorem 9 (Formal statement of Theorem 2). Under RNAC-Linear Setting and Assumption 6 when IPM uncertainty set is considered, take  $\rho = \nu^{\pi^{*}}$  and  $\eta^t = \eta = (1 - \gamma)\log (|\mathcal{A}|)$ , RNAC satisfies
$$
V ^ {*} (\rho) - \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} [ V ^ {t} (\rho) ] \leq \frac {2}{(1 - \gamma) (1 - \beta) T} + \frac {2}{1 - \beta} \frac {2 - \gamma}{1 - \gamma} \epsilon ,
$$
where  $\beta$  is in Assumption 6 (guaranteed by Proposition 4 for DS uncertainty set and assumed for IPM uncertainty set), and  $\epsilon = \epsilon_{stat} + \epsilon_{bias}$  as in Lemma 4 with  $\epsilon_{stat} = \tilde{O}\left(\frac{1}{\sqrt{K}} + \frac{1}{\sqrt{N}}\right)$ . Omitting the approximation  $\epsilon_{bias}$ , the sample complexity for achieving  $\varepsilon$  robust optimal value on average (i.e.,  $V^{*}(\rho) - \frac{1}{T}\sum_{t=1}^{T}\mathbb{E}[V^{t}(\rho)] \leq \varepsilon$ ) is  $\tilde{O}(1/\varepsilon^{3})$  by  $N = K = \tilde{\Theta}(1/\varepsilon^{2})$  and  $T = \Theta(1/\varepsilon)$ .
Proof of Theorem 9. Taking  $\pi = \pi^{*}$  in inequality (18), we have
$$
\langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle + \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \leq \frac {1}{\eta} \mathrm {K L} (\pi_ {s} ^ {*}, \pi_ {s} ^ {t}) - \frac {1}{\eta} \mathrm {K L} (\pi_ {s} ^ {*}, \pi_ {s} ^ {t + 1}).
$$
Since all the following statements are under expectation  $\mathbb{E}$  over the data, we omit  $\mathbb{E}$  in the proof for simplicity and bring it back at the end. According to Lemma 9 and Lemma 4, we have
$$
\begin{array}{l} \mathbb {E} _ {s \sim \rho} [ \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle ] = \mathbb {E} _ {s \sim \rho} [ \langle Q _ {s} ^ {\pi^ {t}}, \pi_ {s} ^ {*} - \pi_ {s} ^ {t} \rangle ] + \mathbb {E} _ {s \sim \rho} [ \langle Q _ {s} ^ {u ^ {t}} - Q _ {s} ^ {\pi^ {t}}, \pi_ {s} ^ {*} - \pi_ {s /} ^ {t} \rangle ] \\ \geq (1 - \beta) \left(V ^ {*} (\rho) - V ^ {t} (\rho)\right) - 2 \epsilon . \\ \end{array}
$$
Based on Lemma 6, we have
$$
\mathbb {E} _ {s \sim \rho} [ \langle Q _ {s} ^ {u ^ {t}}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle ] \geq V ^ {t} (\rho) - V ^ {t + 1} (\rho) - \frac {2 \epsilon}{1 - \gamma}.
$$
$$
(1 - \beta) (V ^ {*} (\rho) - V ^ {t + 1} (\rho)) \leq V ^ {t + 1} (\rho) - V ^ {t} (\rho) + \frac {\mathrm {K L} _ {\rho} (\pi^ {*} , \pi^ {t}) - \mathrm {K L} _ {\rho} (\pi^ {*} , \pi^ {t + 1})}{\eta} + \frac {2 - \gamma}{1 - \gamma} 2 \epsilon .
$$
Taking summation from  $t = 0$  to  $T - 1$  from both sides we have
$$
V ^ {*} (\rho) - \frac {1}{T} \sum_ {t = 1} ^ {T} V ^ {t} (\rho) \leq \frac {V ^ {T} (\rho)}{(1 - \beta) T} + \frac {\mathrm {K L} _ {\rho} (\pi^ {*} , \pi^ {0})}{\eta (1 - \beta) T} + \frac {2 \epsilon}{1 - \beta} \frac {2 - \gamma}{1 - \gamma}.
$$
We then conclude the theorem by choosing  $\eta = (1 - \gamma)\log (|\mathcal{A}|)$  and bringing back the expectation  $\mathbb{E}$  over data,
$$
V ^ {*} (\rho) - \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} [ V ^ {t} (\rho) ] \leq \frac {2}{(1 - \gamma) (1 - \beta) T} + \frac {2}{1 - \beta} \frac {2 - \gamma}{1 - \gamma} \epsilon .
$$
Discussion: The linear convergence for NPG with linear function approximation has been previously studied in canonical MDP [4]. We present the linear convergence of RNAC (Algorithm 1) in Theorem 8. Compared to Theorem 8, Theorem 9 utilizing constant step sizes and leads to sublinear convergence, and it is proved only for initial state distribution  $\rho = \nu^{\pi^{*}}$ . Moreover, Theorem 9 does not require Assumption 6 though may need Assumption 2 for IPM uncertainty set.
E.2 General Function Approximation
RNAC-General Setting. We study the RNAC (Algorithm 1) with robust critic performing FRVE (Algorithm 5) and robust natural actor performing RNPG (Algorithm 6 with  $\zeta_{n} = \Theta(1/n)$  as in Theorem 7) under general value function approximation as in Eq (14) and general policy class as in Eq (20), for DS uncertainty set taking  $\delta$  suggested by Proposition 4. We assume Assumption 1, Assumption 5 and Assumption 4 ( $\|u_{*}^{t}\| \leq U$ ), hold.
Theorem 10. Under RNAC-General Setting, suppose  $\log \pi_{\theta}(a|s)$  is an  $L$ -smooth function of  $\theta$ . Take  $\rho = \nu^{\pi^{*}}$  and  $\eta^t = \eta = \sqrt{\frac{2KL_{\rho}(\pi^{*},\pi^0)}{LU^2T}}$  for each  $t$ , RNAC satisfies
$$
V ^ {*} (\rho) - \mathbb {E} \left[ \frac {1}{T} \sum_ {t = 0} ^ {T - 1} V ^ {t} (\rho) \right] \leq \frac {\sqrt {2 L U ^ {2} \mathrm {K L} _ {\rho} (\pi^ {*} , \pi^ {0})}}{(1 - \beta) \sqrt {T}} + \frac {\epsilon^ {\prime}}{1 - \beta},
$$
where  $\beta$  is in Assumption 6 (guaranteed by Proposition 4 for DS uncertainty set), and  $\epsilon' = \epsilon_{stat}' + \epsilon_{bias}'$  as in Lemma 5 with  $\epsilon_{stat}' = \tilde{O}\left(\frac{1}{\sqrt{K}} + \frac{1}{\sqrt{N}}\right)$ . Omitting the approximation  $\epsilon_{bias}'$ , the sample complexity for achieving  $\varepsilon$  robust optimal value is  $\tilde{O}(1/\varepsilon^4)$  by  $N = K = \tilde{\Theta}(1/\varepsilon^2)$  and  $T = \Theta(1/\varepsilon^2)$ .
$$
\begin{array}{l} (1 - \beta) (V ^ {*} (\rho) - \mathbb {E} [ V ^ {t} (\rho) ]) \leq \mathbb {E} [ \mathbb {E} _ {(s, a) \sim \rho \circ \pi^ {*}} [ A ^ {t} (s, a) ] ] \\ = \mathbb {E} [ \mathbb {E} _ {(s, a) \sim \rho \circ \pi^ {*}} [ \nabla_ {\theta} \log \pi^ {t} (a | s) ^ {\top} u ^ {t} ] + \mathbb {E} [ \mathbb {E} _ {(s, a) \sim \rho \circ \pi^ {*}} [ A ^ {t} (s, a) - A ^ {u ^ {t}} (s, a) ] ] \\ \leq \mathbb {E} \left[ \mathbb {E} _ {(s, a) \sim \rho \circ \pi^ {*}} \left[ \frac {1}{\eta} \log \frac {\pi^ {t + 1} (a | s)}{\pi^ {t} (a | s)} \right] \right] + \frac {\eta L \mathbb {E} [ \| u ^ {t} \| ^ {2} ]}{2} + \epsilon^ {\prime} \\ \leq \frac {1}{\eta} \mathbb {E} \left[ \mathrm {K L} _ {\rho} \left(\pi^ {*}, \pi^ {t}\right) - \mathrm {K L} _ {\rho} \left(\pi^ {*}, \pi^ {t + 1}\right) \right] + \frac {\eta L U ^ {2}}{2} + \epsilon^ {\prime}, \\ \end{array}
$$
where the first inequality is by Lemma 9, the first equality is by the definition of  $A^{u^t}(s,a) = \nabla \log \pi^t (a|s)^\top u^t$ , and the second inequality is by the  $L$ -smoothness of  $\log \pi_{\theta}(a|s)$  and Lemma 5.
We then conclude the proof by summing from  $t = 0$  to  $T - 1$  from both sides and taking  $\eta = \sqrt{\frac{2\mathrm{KL}_{\rho}(\pi^{*},\pi^{0})}{LU^{2}T}}$ .
Note that in RNAC-General Setting, the critic is employing general function approximation, where the IPM uncertainty set is not defined (c.f. Section C.2 for more discussion). It is not hard to show a similar result as in Theorem 10 for the RNAC with robust critic performing RLTD (Algorithm 2) and robust natural actor performing RNPG (Algorithm 6) for both DS and IPM uncertainty sets.
We introduce some supporting lemmas for proving the results of this paper.
F.1 Performance Difference Lemmas
A key supporting lemma in the proof of convergence of policy gradient-based methods is the performance difference lemma. In the canonical RL with a fixed transition kernel  $\kappa$ , we have the following performance lemma.
Lemma 7 (Performance difference [23]). For any policy  $\pi, \pi'$ , and transition  $\kappa$ , we have
$$
V _ {\kappa} ^ {\pi^ {\prime}} (\rho) - V _ {\kappa} ^ {\pi} (\rho) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {\prime}, \kappa}} \mathbb {E} _ {a \sim \pi^ {\prime} (\cdot | s)} [ A ^ {\pi} (s, a) ].
$$
However, due to the non-singleton uncertainty set  $\mathcal{P}$ , the worst-case transition kernel is a function of policy  $\pi$ , and we have the following performance difference inequality lemma.
Lemma 8 (Robust performance difference). For any policy  $\pi, \pi'$ , denote  $\kappa' = \kappa_{\pi'}$  and  $\kappa = \kappa_{\pi}$  be their worst-case transition kernels, respectively. Then
$$
\frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {\prime}, \kappa^ {\prime}}} \mathbb {E} _ {a \sim \pi^ {\prime} (\cdot | s)} [ A ^ {\pi} (s, a) ] \leq V ^ {\pi^ {\prime}} (\rho) - V ^ {\pi} (\rho) \leq \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {\prime}, \kappa}} \mathbb {E} _ {a \sim \pi^ {\prime} (\cdot | s)} [ A ^ {\pi} (s, a) ].
$$
Proof of Lemma 8. The LHS is by
$$
\begin{array}{l} V _ {\kappa^ {\prime}} ^ {\pi^ {\prime}} (\rho) - V ^ {\pi} (\rho) = \mathbb {E} _ {\kappa^ {\prime}, \pi^ {\prime}} \left[ \sum_ {t \geq 0} \gamma^ {t} (r (s _ {t}, a _ {t}) + V ^ {\pi} (s _ {t}) - V ^ {\pi} (s _ {t})) \right] - V ^ {\pi} (\rho) \\ = \mathbb {E} _ {\kappa^ {\prime}, \pi^ {\prime}} \left[ \sum_ {t \geq 0} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}\right) + \gamma V ^ {\pi} \left(s _ {t + 1}\right) - V ^ {\pi} \left(s _ {t}\right)\right) \right] \\ \geq \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {\prime}, \kappa^ {\prime}}} \mathbb {E} _ {a \sim \pi^ {\prime} (\cdot | s)} [ A ^ {\pi} (s, a) ]. \\ \end{array}
$$
The RHS is by
$$
V _ {\kappa^ {\prime}} ^ {\pi^ {\prime}} (\rho) - V _ {\kappa} ^ {\pi} (\rho) \leq V _ {\kappa} ^ {\pi^ {\prime}} (\rho) - V _ {\kappa} ^ {\pi} (\rho) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {\prime}, \kappa}} \mathbb {E} _ {a \sim \pi^ {\prime} (· | s)} [ A ^ {\pi} (s, a) ].
$$
Moreover, the optimality gap between the optimal policy  $\pi^{*}$  and any policy  $\pi$  is upper bounded as in the lemma below.
Lemma 9 (Robust optimality gap lemma). Under Assumption 2, for any  $\pi$  and any  $\rho$ ,
$$
V ^ {*} (\rho) - V ^ {\pi} (\rho) \leq \frac {1}{1 - \beta} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {*}, p ^ {\circ}}, a \sim \pi_ {s} ^ {*}} [ A ^ {\pi} (s, a) ].
$$
Proof of Lemma 9.
$$
\begin{array}{l} V ^ {*} (\rho) - V ^ {\pi} (\rho) = \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi_ {s} ^ {*}} [ r (s, a) + \gamma \sum_ {s ^ {\prime}} \kappa_ {\pi^ {*}} (s ^ {\prime} | s, a) V ^ {*} (s ^ {\prime}) ] - V ^ {\pi} (\rho) \\ = \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} [ r (s, a) + \gamma \sum_ {s ^ {\prime}} \kappa_ {\pi} \left(s ^ {\prime} \mid s, a\right) V ^ {\pi} \left(s ^ {\prime}\right) - V ^ {\pi} (\rho) ] \\ + \gamma \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} \sum_ {s ^ {\prime}} \left(\kappa_ {\pi^ {*}} \left(s ^ {\prime} | s, a\right) V ^ {*} \left(s ^ {\prime}\right) - \kappa_ {\pi^ {t}} \left(s ^ {\prime} | s, a\right) V ^ {\pi} \left(s ^ {\prime}\right)\right) \\ \leq \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} [ A ^ {\pi} (s, a) ] + \gamma \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} \sum_ {s ^ {\prime}} \kappa_ {\pi^ {t}} (s ^ {\prime} | s, a) (V ^ {*} (s ^ {\prime}) - V ^ {\pi} (s ^ {\prime})) \\ \leq \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} [ A ^ {\pi} (s, a) ] + \beta \mathbb {E} _ {s \sim \rho} \mathbb {E} _ {a \sim \pi^ {*}} \sum_ {s ^ {\prime}} p _ {s, a} ^ {\circ} (s ^ {\prime}) (V ^ {*} (s ^ {\prime}) - V ^ {\pi} (s ^ {\prime})) \\ \leq \dots \\ \leq \frac {1}{1 - \beta} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi^ {*}, p ^ {\circ}}, a \sim \pi_ {s} ^ {*}} [ A ^ {\pi} (s, a) ]. \\ \end{array}
$$
F.2 Concentration Lemmas with Markov samples
Assumption 7 (Uniformly geometric mixing, quantitative restatement of Assumption 1). There exists some  $0 < \lambda < 1$  and  $C > 0$  such that for any policy  $\pi$ , the Markov chain  $\{s_k\}$  induced by applying  $\pi$  in the nominal model  $p^\circ$  is geometrically ergodic with unique stationary distribution  $\nu^\pi$ , i.e.,  $\max_s\| \mathbb{P}(s_k\in \cdot |s_0 = s) - \nu^\pi \|_{TV}\leq C\lambda^k$ ,  $\forall k$ . The uniform convergence to unique stationary distribution implies that for any policy  $\pi$  the spectral gap (i.e., the difference between the largest and the second largest eigenvalues) of the state transition kernel has a strictly positive lower bound.
Lemma 10 (Bernstein's inequality for stationary Markovian data, Theorem 2 in [21]). Suppose  $Z_{1}, Z_{2}, \ldots, Z_{n}$  follow some stationary Markovian data in space  $\mathcal{Z}$  with stationary distribution  $\nu \in \Delta_{\mathcal{Z}}$  and spectral gap of the Markov chain strictly greater than 0. For any function  $f: \mathcal{Z} \to [-c, c]$  that,  $\mathbb{E}_{Z \sim \nu}[f(Z)] = 0$  and  $\mathbb{E}_{Z \sim \nu}[f(Z)^{2}] = \sigma^{2}$ , with probability at least  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} f (Z _ {i}) \leq C _ {1} \sqrt {\frac {\sigma^ {2} \ln (1 / \delta)}{n}} + C _ {2} \frac {c \ln (1 / \delta)}{n},
$$
for some spectral gap-dependent multiplicative factors  $C_1, C_2$ .
Since we assume Assumption 7 throughout the paper during the analysis, the spectral gap is uniformly lower bounded by some constant great than zero, and we view  $C_1, C_2$  as some constant that can be omitted in the big-O notation. We refer to [21] for further details of Bernstein's inequality for stationary Markovian data.
Lemma 11 (Lemma A.11 in [2] with Markovian data). Consider a stationary Markov chain on domain  $\mathcal{X} \times \mathcal{Z}$  that is geometrically mixing with transition  $p(x', z'|x, z) = p_{X'|Z}(x'|z)p_{Z|X}(z'|x)$  and stationary distribution  $\nu \circ p_{Z|X}$ . Let  $\mathcal{D} = \{(X_1, Z_1), (X_2, Z_2), \ldots, (X_n, Z_n)\}$  be stationary Markovian data sampled with  $X_1 \sim \nu$ . Given a function class  $\mathcal{G} \subset \mathbb{R}^{\mathcal{X}}$  and a function  $G: \mathcal{X} \times \mathcal{Z} \to \mathbb{R}$  that  $\mathbb{E}_{Z \sim p_{Z|X}(\cdot|x)}[G(x, Z)] = g^*(x)$  and  $|G(x, z)| \leq G_{\max}$ . Let  $\hat{g} = \arg \min_{g \in \mathcal{G}} \frac{1}{2} \sum_{(x, z) \in \mathcal{D}} (G(x, z) - g(x))^2$  be the MSE estimate of  $g^*$  within  $\mathcal{G}$ , then with probability at least  $1 - \delta$ ,
$$
\sqrt {\mathbb {E} _ {x \sim \nu} \left[ (\hat {g} (x) - g ^ {*} (x)) ^ {2} \right]} \leq O \left(G _ {\max } \sqrt {\frac {\log (| \mathcal {G} | / \delta)}{n}}\right) + O \left(\epsilon_ {\text {b i a s}}\right), \tag {24}
$$
where  $\epsilon_{bias} = \min_{g\in \mathcal{G}}\| g - g^{*}\|_{\nu}$
Proof. The proof follows the same as [2]. The only difference is replacing the canonical Bernstein's inequality with i.i.d. data by Bernstein's inequality for stationary Markovian data in Lemma 10.  $\square$
F.3 Convergence of Stochastic Approximation
$F: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}^d$  is a stochastic operator. The stochastic approximation algorithm updates the parameter by  $w_{k+1} = w_k + \alpha_k F(w, Z_k)$ , where  $(Z_0, Z_1, \ldots)$  is a Markov chain, that is geometrically ergodic for some  $C > 0$ ,  $1 > \lambda > 0$  with stationary distribution  $\nu$ , i.e.,  $\max_z \| \mathbb{P}(Z_k \in \cdot | Z_0 = z) - \nu \|_{TV} \leq C \lambda^k$ ,  $\forall k$ . (c.f. Assumption 1). The stochastic approximation algorithm solves the following equation
$$
0 = \bar {F} (w) := \mathbb {E} _ {Z \sim \nu} [ F (w, Z) ],
$$
under the following assumption and appropriate step sizes  $(\alpha_{1},\alpha_{2},\ldots)$
Assumption 8. 1.  $(Z_0,Z_1,\ldots)$  are geometrically mixed to the stationary distribution.
2.  $\exists L_{1} > 0$  that  $\| F(w,z) - F(w^{\prime},z)\| \leq L_{1}\| w - w^{\prime}\|$ ,  $\| F(0,z)\| \leq L_{1}$  for any  $w,w^{\prime}\in \mathbb{R}^{d},z\in \mathcal{Z}$ .
3.  $\bar{F}(w) = 0$  has unique solution  $w^*$  and there exists  $c_0 > 0$  that  $(w - w^*)^\top \bar{F}(w) \leq -c_0 \| w - w^* \|^2$  for any  $w \in \mathbb{R}^d$ .
The second condition of Assumption 8 is a Lipschitz condition of the operator  $F$ , and the third condition is the "strongly-concavity" structure of the averaged operator  $\bar{F}$ .
Lemma 12 (Big-O version of Corollary 2.1.2 in [10]). Under Assumption 8, taking appropriate  $\alpha_{k} = \Theta (1 / k)$  gives  $\mathbb{E}[\| w_K - w^*\| ^2 ] = \tilde{O} (1 / K)$ .
To apply the lemma above, we need to show that the proposed operator  $F$  satisfies Assumption 8.
The framework of the robust Markov decision process (RMDP) is proposed by [19, 40] to learn the optimal robust policy that achieves the optimal worst-case performance over all possible models in the uncertainty set. For the planning problem of RMDP, Iyengar [19] and Nilim et al. [40] show that value iteration achieves linear convergence to the optimal robust values.
When the transition model is unknown, several model-based and value-based approaches have been proposed and studied for robust RL in both the tabular setting and the function approximation setting. Under the tabular setting, Xu et al. [64] design the model-based robust phase value learning algorithm and demonstrates an  $O(1 / \epsilon^2)$  sample complexity bound. Under the function approximation setting, Panaganti et al. [43] develop a robust fitted Q-iteration with the total variation uncertainty set and show an  $O(1 / \epsilon^2)$  sample complexity to achieve approximate optimality and [6] develop a model-based method with the same sample complexity under certain coverage assumption. However, these methods are not scalable to continuous control problems. Instead, policy-based approaches directly parameterize policy and have more representation power in modeling stochastic policies, more flexibility for policy manipulation, and a better ability to solve robotics control problems with large action space.
The existing policy-based methods with convergence guarantees mainly focus on the tabular setting. Wang et al. [61] show an  $O(1 / \epsilon^7)$  sample complexity for robust actor-critic with  $R$ -contamination uncertainty sets. Li et al. [29] prove an  $\tilde{O}(1 / \epsilon^2)$  sample complexity for robust natural actor-critic assuming the existence of an oracle to solve the inner optimization problem. Kumar et al. [27] develop a robust policy gradient method based on  $\ell_{\mathrm{p}}$  norm uncertainty sets and extends its formulation under the  $s$ -rectangular assumption. All works mentioned above are limited to the tabular case with critic Q and adopt computationally infeasible uncertainty sets in the case of the large state space. Kuang et al. [26] illustrate a state disturbance view of Wasserstein metric-based RMDP, which can be generalized to the large state space. However, they only provide a policy iteration guarantee in the tabular setting (contraction under  $\| \cdot \|_{\infty}$ ). Instead, we focus on large-scale robust RL, propose
two computationally efficient uncertainty sets, and demonstrate an  $\tilde{O}(1/\epsilon^2)$  (resp.,  $\tilde{O}(1/\epsilon^4)$ ) sample complexity under linear (resp., general) function approximation. Additionally, we regard  $V$  as a critic instead of  $Q$ , which is closer to the actual implementation of on-policy natural actor-critic algorithms.