SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] \\ \leq 2 \rho \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} \} | \mathcal {F} _ {t} ], \tag {24} \\ \end{array}
$$
which follows from Cauchy-Schwarz inequality, triangle inequality and the fact that  $\max \{\|\Theta(t)\|_2, \|\Theta^*\|_2\} \leq \rho$  due to projection. Using Hölder's inequality on the RHS of (24), we obtain:
$$
\mathbb {E} \big [ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} \} | \mathcal {F} _ {t} \big ] \leq 2 \rho u ^ {\frac {1}{1 + p}} \big [ \mathbb {P} (\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} | \mathcal {F} _ {t}) \big ] ^ {\frac {p}{1 + p}}.
$$
Using Markov's inequality, we bound the bias due to using the clipped stochastic gradient as:
$$
\mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] \leq 2 \rho u b _ {t} ^ {- p}. \tag {25}
$$
Substituting (25) into (19), and taking expectation over the trajectory  $\mathcal{F}_t$ , we obtain:
$$
\mathbb {E} [ \mathcal {L} (\Theta (t + 1)) - \mathcal {L} (\Theta (t)) ] \leq - 2 \eta (1 - \gamma) \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2} + 4 \eta \rho u b _ {t} ^ {- p} + \eta^ {2} u b _ {t} ^ {1 - p}.
$$
Telescoping sum over  $t = 1,2,\ldots ,T$  yields:
$$
\begin{array}{l} \mathbb {E} \mathcal {L} (\Theta (T + 1)) - \mathcal {L} (\Theta (1)) \leq - 2 \eta (1 - \gamma) \sum_ {t = 1} ^ {T} \left(\mathbb {E} \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2}\right) \\ + 4 \eta \rho u \int_ {0} ^ {T} b _ {s} ^ {- p} d s + \eta^ {2} u \int_ {0} ^ {T} b _ {s} ^ {1 - p} d s. \tag {26} \\ \end{array}
$$
Rearranging the terms, using Jensen's inequality and  $\mathcal{L}(\Theta(1)) \leq 4\rho^2$ , and substituting the step-size  $\eta$  yields the result.
(b) For the full-rank case, note that
$$
\begin{array}{l} \| f _ {\Theta} - \mathcal {V} \| _ {\mu} ^ {2} = (\Theta - \Theta^ {\star}) ^ {\top} \left(\sum_ {x \in \mathbb {X}} \mu (x) \Phi (x) \Phi^ {\top} (x)\right) (\Theta - \Theta^ {\star}), \\ \geq \lambda_ {\min } \| \Theta - \Theta^ {*} \| _ {2} ^ {2}, \\ \end{array}
$$
which implies (together with (23)) that:
$$
\mathbb {E} \| \Theta (t + 1) - \Theta^ {\star} \| _ {2} ^ {2} \leq (1 - \eta_ {t} \lambda (1 - \gamma)) \| \Theta (t) - \Theta^ {\star} \| _ {2} ^ {2} - \eta_ {t} (1 - \gamma) \mathbb {E} \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2} + 4 \eta_ {t} \rho u b _ {t} ^ {- p} + \eta_ {t} ^ {2} b _ {t} ^ {1 - p} u.
$$
With the step-size choice  $\eta_t = \frac{1}{(1 - \gamma)\lambda t}$ , we obtain by induction:
$$
\mathbb {E} \| \Theta (t + 1) - \Theta^ {\star} \| _ {2} ^ {2} \leq - \frac {1}{\lambda t} \sum_ {k = 1} ^ {t} \mathbb {E} \| f _ {\Theta (k)} - \mathcal {V} \| _ {\mu} ^ {2} + \frac {4 \rho u}{\lambda_ {\min } t} \sum_ {k = 1} ^ {t} b _ {k} ^ {- p} + \frac {u}{\lambda_ {\min } ^ {2} t} \sum_ {k = 1} ^ {t} \frac {b _ {k} ^ {1 - p}}{k}.
$$
By rearranging the terms and using the integral bound for the summations above, and using the Jensen's inequality for the  $\mu$ -norm, we obtain the result.
Proof of Theorem 2. Let  $\mathcal{F}_t^{+ + } = \sigma (\Theta (1),\dots ,\Theta (t),X_t,X_{t + 1})$  and  $\mathbb{E}_{t}^{+ + }[\cdot ] = \mathbb{E}[\cdot |\mathcal{F}_{t}^{+ + }]$  . Also, let
$$
\begin{array}{l} \hat {g} (\Theta) = \mathbb {E} _ {t} ^ {+ +} g _ {t} (\Theta), \\ \bar {g} (\Theta) = \sum_ {x, x ^ {\prime} \in \mathbb {X}} \mu (x) \mathcal {P} (x, x ^ {\prime}) (r (x) + \gamma f _ {\Theta} (x ^ {\prime}) - f _ {\Theta} (x)) \Phi (x). \\ \end{array}
$$
The bias due to Markovian sampling is:
$$
Z _ {t} (\Theta) = \left(\hat {g} _ {t} (\Theta) - \bar {g} (\Theta)\right) ^ {\top} (\Theta - \Theta^ {\star}).
$$
With the above definitions, the Lyapunov drift at time  $t \geq 1$  can be bounded as follows:
$$
\begin{array}{l} \mathbb {E} _ {t} ^ {+ +} \| \Theta (t + 1) - \Theta^ {\star} \| _ {2} ^ {2} \leq \| \Theta (t) - \Theta^ {\star} \| _ {2} ^ {2} - 2 \eta (1 - \gamma) \| \mathcal {V} - f _ {\Theta (t)} \| _ {\mu} ^ {2} + \eta^ {2} \mathbb {E} _ {t} ^ {+ +} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \chi_ {t} ] \\ + 2 \eta \hat {g} ^ {\top} (\Theta (t) - \Theta^ {\star}) \bar {\chi} _ {t} + 2 \eta Z _ {t} (\Theta (t)), \\ \end{array}
$$
where  $\chi_t = 1 - \bar{\chi}_t = \mathbb{1}\{\| g_t(\Theta(t))\|_2 \leq b_t\}$ . Compared to the case of iid sampling in Theorem 1, the difference is  $Z_t(\Theta(t))$ . In the following, we bound  $\mathbb{E}Z_t(\Theta(t))$  by using the mixing time analysis in [4]. First, we provide two essential properties of  $Z_t(\Theta)$  to verify the conditions in Lemma 10 in [4].
Lemma 3. Under Assumption 1, we have:
$$
\left| Z _ {t} (\Theta) \right| \leq (1 + 2 \rho) ^ {2}, \Theta \in B _ {2} (0, \rho), \tag {27}
$$
$$
\left| Z _ {t} \left(\Theta\right) - Z _ {t} \left(\Theta^ {\prime}\right) \right| \leq 6 \left(1 + 2 \rho\right) ^ {2} \| \Theta - \Theta^ {\prime} \| _ {2} ^ {2}, \Theta , \Theta^ {\prime} \in B _ {2} (0, \rho). \tag {28}
$$
$$
\mathbb {E} Z _ {t} (\Theta (t)) \leq \mathbb {E} \left[ Z _ {t} (\Theta (t - \tau)) \right] + 6 (1 + 2 \rho) ^ {2} \mathbb {E} \| \Theta (t) - \Theta (t - \tau) \| _ {2}. \tag {29}
$$
We have the following inequality:
$$
\| \Theta (t) - \Theta (t - \tau) \| _ {2} \leq \sum_ {k = t - \tau} ^ {t - 1} \| \Theta (k + 1) - \Theta (k) \| _ {2} \leq \eta \sum_ {k = t - \tau} ^ {t - 1} \| g _ {t} (\Theta (t)) \| _ {2} \chi_ {t}.
$$
Taking the expectation above, and using Hölder's inequality:
$$
\mathbb {E} \| \Theta (t) - \Theta (t - \tau) \| _ {2} \leq \sum_ {k = t - \tau} ^ {t - 1} \left(\mathbb {E} [ \| g _ {k} (\Theta (k)) \| _ {2} ^ {1 + p} ]\right) ^ {\frac {1}{1 + p}} \leq \eta \tau u ^ {\frac {1}{1 + p}}.
$$
By using the information theoretic bound in Lemma 9 in [4], we obtain
$$
\mathbb {E} Z _ {t} (\Theta (t - \tau)) \leq 2 (1 + 2 \rho) ^ {2} \eta ,
$$
under the uniform ergodicity assumption in Assumption 1. Using the last two inequalities in (29), we obtain:
$$
\mathbb {E} Z _ {t} (\Theta (t)) \leq 2 (1 + 2 \rho) ^ {2} \left(1 + 6 \tau u ^ {\frac {1}{1 + p}}\right) \eta . \tag {30}
$$
By using the above result, we obtain the ultimate inequality for the Lyapunov drift as follows:
$$
\begin{array}{l} \mathbb {E} \| \Theta (t + 1) - \Theta^ {\star} \| _ {2} ^ {2} \leq \mathbb {E} \| \Theta (t) - \Theta^ {\star} \| _ {2} ^ {2} - 2 \eta (1 - \gamma) \mathbb {E} \| \mathcal {V} - f _ {\Theta (t)} \| _ {\mu} ^ {2} + \eta^ {2} \mathbb {E} [ \| g _ {t} \| _ {2} ^ {2} \chi_ {t} ] + 4 \eta \rho \mathbb {E} [ \| g _ {t} \| _ {2} \chi_ {t} ] \\ + 4 \eta^ {2} (1 + 2 \rho) ^ {2} \Big (1 + 6 \tau u ^ {\frac {1}{1 + p}} \Big). \\ \end{array}
$$
The proof follows from identical steps as Theorem 1.
Proof of Theorem 3. The main idea in the proof is to establish a centering argument for both the bias (due to using clipped stochastic gradients) and the variability (controlled by  $b_{t}$ ), and to use martingale concentration arguments based on Freedman's inequality and Azuma-Hoeffding inequality to bound the sample mean for the bias and variability, respectively. This strategy extends the approach in [6] for robust mean estimation to reinforcement learning, which has a dynamic behavior unlike the mean estimation problem. Namely, for any  $t \in \{1,2,\dots,T\}$ , we have:
$$
\begin{array}{l} \mathcal {L} (\Theta (t + 1)) \leq \mathcal {L} (\Theta (t)) - 2 \eta (1 - \gamma) \| f _ {\Theta (t)} - \mathcal {V} \| _ {\mu} ^ {2} \\ + \eta^ {2} \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\} | \mathcal {F} _ {t} ] \\ + 2 \eta B (t) + \eta^ {2} V (t), \\ \end{array}
$$
where the first line follows from Lemma 2, and
$$
B (t) = - \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) | \mathcal {F} _ {t} \right] + g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\left\| g _ {t} (\Theta (t)) \right\| _ {2} \leq b _ {t} \right\}, \tag {31}
$$
is the bias term, and
$$
Z (t) = \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \} - \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \} | \mathcal {F} _ {t} ]
$$
is the variability. By telescoping sum over  $t = 1,2,\ldots ,T$  and some algebraic manipulations, we have:
$$
\begin{array}{l} \frac {\mathcal {L} (\Theta (T + 1))}{T} - \frac {\mathcal {L} (\Theta (1))}{T} \leq - 2 \eta \left(\frac {1}{T} \sum_ {t = 1} ^ {T} f (\Theta (t)) - f \left(\Theta^ {*}\right)\right) \tag {32} \\ + \eta^ {2} \frac {u}{T} \sum_ {t = 1} ^ {T} b _ {t} ^ {1 - p} + \frac {2 \eta}{T} \sum_ {t = 1} ^ {T} B (t) + \frac {\eta^ {2}}{T} \sum_ {t = 1} ^ {T} Z (t), \\ \end{array}
$$
where we used (20) in the second line. In the following, we will bound the empirical processes  $\frac{1}{T}\sum_{t=1}Z(t)$  and  $\frac{1}{T}\sum_{t=1}B(t)$ .
Note that  $\{Z(t):t\in \mathbb{N}\}$  is a martingale difference sequence (MDS) adapted to the filtration  $\{\mathcal{F}_t:t\in \mathbb{N}\}$ . Furthermore, note that
$$
| Z (t) | \leq 2 b _ {t} ^ {2} \leq 2 b _ {T} ^ {2},
$$
almost surely for any  $t \leq T$ . Thus,  $\sum_{t=1}^{n} V(t) \mathbb{1}\{n \leq T\}$  forms a martingale with bounded differences, and by using Azuma-Hoeffding inequality [55], we have:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} Z (t) \leq b _ {T} \sqrt {\frac {L _ {\delta}}{T}} = \frac {u ^ {\frac {1}{1 + p}} T ^ {\frac {1 - p}{2 (1 + p)}}}{L _ {\delta} ^ {\frac {1 - p}{2 (1 + p)}}} \leq \frac {u ^ {\frac {2}{1 + p}} T ^ {\frac {1 - p}{1 + p}}}{L _ {\delta} ^ {\frac {1 - p}{1 + p}}}, \tag {33}
$$
with probability at least  $1 - \delta /2$  where the last inequality holds since  $T > L_{\delta}u^{-\frac{2}{1 - p}}$
We decompose  $B(t)$  into predictable and non-predictable components as follows:
$$
B (t) = \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\left\| g _ {t} (\Theta (t)) \right\| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] + B _ {0} (t), \tag {34}
$$
where the martingale difference sequence  $B_0(t)$  is defined as follows:
$$
\begin{array}{l} B _ {0} (t) = - \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\} \mid \mathcal {F} _ {t} \right] \\ + g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \right\}. \tag {35} \\ \end{array}
$$
By (25) (with  $b_{t}$  replaced by  $b_{t}$ ), the first term on the RHS of (34) is bounded by  $2\rho u c_{t,\delta}^{-p}$ . Thus, we have:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} B (t) \leq \frac {2 \rho u ^ {\frac {1}{1 + p}} L _ {\delta} ^ {\frac {p}{1 + p}}}{T ^ {\frac {p}{1 + p}}} + \frac {1}{T} \sum_ {t = 1} ^ {T} B _ {0} (t). \tag {36}
$$
In order to upper bound  $\frac{1}{T}\sum_{t = 1}^{T}B_{0}(t)$ , we use Freedman's inequality for martingales [12]. To use Freedman's inequality, we verify the following conditions.
1. For any  $t \leq T$ , we have:
$$
\left| g _ {t} \left(\Theta (t)\right) ^ {\top} \left(\Theta (t) - \Theta^ {*}\right) \mathbb {I} \left\{\left\| g _ {t} \left(\Theta (t)\right) \right\| _ {2} \leq b _ {t} \right\} \right| \leq 2 \rho b _ {t} \leq 2 \rho b _ {t},
$$
2. The normalized quadratic variation process satisfies:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} | B _ {0} (t) | ^ {2} \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left| g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \right| ^ {2} \mathbb {1} \left\{\left| \left| g _ {t} (\Theta (t)) \right| \right| _ {2} \leq b _ {t} \right\} \mid \mathcal {F} _ {t} \right], \\ \leq \frac {4 \rho^ {2}}{T} \sum_ {t = 1} ^ {T} \mathbb {E} [ \| g _ {t} (\Theta (t)) \| _ {2} ^ {2} \mathbb {1} \{\| g _ {t} (\Theta (t)) \| _ {2} \leq b _ {t} \} | \mathcal {F} _ {t} ], \\ \leq \frac {4 \rho^ {2}}{T} \sum_ {t = 1} ^ {T} u b _ {t} ^ {1 - p} \leq 4 \rho^ {2} \frac {u ^ {\frac {2}{1 + p}} T ^ {\frac {1 - p}{1 + p}}}{L _ {\delta} ^ {\frac {1 - p}{1 + p}}}, \\ \end{array}
$$
where the first inequality is due to  $Var(Z) \leq \mathbb{E}[Z^2]$  for any random variable  $Z$  with a finite variance, the second inequality follows from Cauchy-Schwarz inequality and triangle inequality with  $\max \{\| \Theta(t) \|_2, \| \Theta^* \|_2\} \leq \rho$  due to projection, the third inequality follows from (20) with  $b_t$  replaced by  $b_t$ .
Thus, by Freedman's inequality, we have:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} B _ {0} (t) \leq \frac {2 \sqrt {2} \rho u ^ {\frac {1}{1 + p}} L _ {\delta} ^ {\frac {p}{1 + p}}}{T ^ {\frac {p}{1 + p}}} + \frac {4 \rho L _ {\delta} b _ {t}}{3 T}, \\ \leq (2 \sqrt {2} + 4 / 3) \rho \frac {u ^ {\frac {1}{1 + p}} L _ {\delta} ^ {\frac {p}{1 + p}}}{T ^ {\frac {p}{1 + p}}}, \\ \end{array}
$$
with probability at least  $1 - \delta / 2$ . Therefore, from (36) and the above inequality, with probability at least  $1 - \delta / 2$ , we have:
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} B (t) \leq \frac {7 \rho u ^ {\frac {1}{1 + p}} L _ {\delta} ^ {\frac {p}{1 + p}}}{T ^ {\frac {p}{1 + p}}} \tag {37}
$$
Hence, by substituting (33) and (37) into (32) with union bound, and using the specified step-size together with the facts that  $\mathcal{L}(\Theta(1)) \leq 4\rho^2$  and  $\mathcal{L}(\Theta(T + 1)) \geq 0$ , we conclude the proof.
A.1 Finite-Time Analysis of Robust TD Learning without Realizability
In the following, we release the realizability assumption, and show that an additional function approximation error appears in the bounds for the general case.
Theorem 5 (Performance of Robust TD learning - without realizability). Let
$$
\epsilon_ {\mathrm {a p p}} = \inf  _ {\Theta \in B _ {2} (0, \rho)} \sqrt {\mathbb {E} | \mathcal {V} (x) - \langle \Theta , \Phi (x) \rangle | ^ {2}}.
$$
Then, under Assumptions 1 and 2a we have the following bounds for Robust TD learning:
For  $b_{t} = (ut)^{\frac{1}{1 + p}}$  for any  $t \in \mathbb{Z}_{+}$  and  $\eta_t = \eta = \frac{2\rho(1 - \gamma)}{(uT)^{\frac{1}{1 + p}}}$ , we have:
$$
\underset {\Theta (1), \underset {x} {\Theta (2)}, \dots , \Theta (T)} {\mathbb {E}} \left[ \left(\mathcal {V} (x) - \langle \bar {\Theta} (T), \Phi (x) \rangle\right) ^ {2} \right] \leq \frac {6 \rho u ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}} + (\rho + \frac {1}{1 - \gamma}) \frac {2 \epsilon_ {\mathrm {a p p}}}{1 - \gamma}, \tag {38}
$$
for any  $T > 1$ .
Proof of Theorem 5. Note that we define
$$
\Theta^{\star}\in \arg \min_{\Theta \in B_{2}(0,\rho)}\mathbb{E}[|\mathcal{V}(x) - \langle \Theta ,\Phi (x)\rangle |^{2}].
$$
By (19), we have:
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} (\Theta (t + 1)) \mid \mathcal {F} _ {t} \right] \leq \mathcal {L} (\Theta (t)) + 2 \eta \mathbb {E} _ {t} \left[ g _ {t} ^ {\top} (\Theta (t)) (\Theta (t) - \Theta^ {\star}) \right] \\ - 2 \eta \mathbb {E} \left[ g _ {t} (\Theta (t)) ^ {\top} (\Theta (t) - \Theta^ {*}) \mathbb {1} \left\{\left\| g _ {t} (\Theta (t)) \right\| _ {2} > b _ {t} \right\} \mid \mathcal {F} _ {t} \right] + \eta^ {2} u b _ {t} ^ {1 - p}, \tag {39} \\ \end{array}
$$
To eliminate the realizability assumption (Assumption 3), we make the following decomposition:
$$
\begin{array}{l} \mathbb {E} _ {t} [ g _ {t} ^ {\top} (\Theta (t)) (\Theta (t) - \Theta^ {\star}) ] = \mathbb {E} _ {t} [ (R _ {t} + \gamma f _ {\Theta (t)} (X _ {t} ^ {\prime}) - f _ {\Theta (t)} (X _ {t})) (f _ {\Theta (t)} (X _ {t}) - \langle \Theta^ {\star}, \Phi (X _ {t}) \rangle) ], \\ = \mathbb {E} _ {t} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(f _ {\Theta (t)} \left(X _ {t}\right) - \langle \Theta^ {\star}, \Phi \left(X _ {t}\right) \rangle\right) \right], \\ = \mathbb {E} _ {t} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(f _ {\Theta (t)} \left(X _ {t}\right) - \mathcal {V} \left(X _ {t}\right)\right) \right] \\ + \mathbb {E} _ {t} \Big [ \Big ((\mathcal {T} f _ {\Theta (t)}) (X _ {t}) - f _ {\Theta (t)} (X _ {t}) \Big) \Big (\mathcal {V} (X _ {t}) - \langle \Theta^ {\star}, \Phi (X _ {t}) \rangle \Big) \Big ]. \\ \end{array}
$$
Note that the last term corresponds to the approximation error, and we can bound its expectation as:
$$
\mathbb {E} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(\mathcal {V} \left(X _ {t}\right) - \langle \Theta^ {\star}, \Phi (X _ {t}) \rangle\right) \right] \leq (1 - \gamma + 1) \| \mathcal {V} - f _ {\Theta (t)} \| _ {\mu} \cdot \epsilon_ {\mathrm {a p p}}.
$$
Since  $|\mathcal{V}(x)| \leq \frac{1}{1 - \gamma}$  and  $|f_{\Theta(t)}(x)| \leq \rho$  for all  $x$ , and  $\gamma \in (0, 1)$ , we have:
$$
\mathbb {E} \left[ \left(\left(\mathcal {T} f _ {\Theta (t)}\right) \left(X _ {t}\right) - f _ {\Theta (t)} \left(X _ {t}\right)\right) \left(\mathcal {V} \left(X _ {t}\right) - \langle \Theta^ {\star}, \Phi \left(X _ {t}\right) \rangle\right) \right] \leq 2 (\rho + \frac {1}{1 - \gamma}) \epsilon_ {\mathrm {a p p}},
$$
for any  $t$ . Using this result, and following identical steps as Theorem 1 with the same step-size yields the result.
B Proofs for Robust Natural Actor-Critic
Proof of Theorem 4. We use the following Lyapunov function for the analysis [1, 57, 34]:
$$
\mathcal {L} (\pi) = \sum_ {s \in \mathbb {S}} \mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \sum_ {a \in \mathbb {A}} \pi^ {\star} (a | s) \log \frac {\pi^ {\star} (a | s)}{\pi (a | s)}. \tag {40}
$$
For the Lyapunov drift, at any iteration  $k$ , we have:
$$
\mathcal {L} \left(\pi_ {k + 1}\right) - \mathcal {L} \left(\pi_ {k}\right) = \sum_ {s, a} \mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \pi^ {\star} (a | s) \log \frac {\pi_ {k} (a | s)}{\pi_ {k + 1} (a | s)}. \tag {41}
$$
Since  $\sup_{s,a}\| \Phi (s,a)\| _2\leq 1$ $\nabla_W\log \pi_W(a|s)$  is 1-Lipschitz continuous [1]. Thus, we have:
$$
\left| \log \pi_ {k + 1} (a | s) - \log \pi_ {k} (a | s) - \nabla^ {\top} \log \pi_ {k} (a | s) (W (k + 1) - W (k)) \right| \leq \frac {1}{2} \| W (k + 1) - W (k) \| _ {2} ^ {2}. \tag {42}
$$
Since  $W(k + 1) = W(k) + \alpha \bar{\Theta}_k(T)$ , we have:
$$
\mathcal {L} \left(\pi_ {k + 1}\right) - \mathcal {L} \left(\pi_ {k}\right) \leq \frac {\eta^ {2}}{2} \| \bar {\Theta} _ {k} (T) \| _ {2} ^ {2} - \eta \cdot \mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \pi^ {\star} (a | s) \nabla^ {\top} \log \pi_ {k} (a | s) \bar {\Theta} _ {k} (T). \tag {43}
$$
By performance difference lemma [26], we have:
$$
\mathcal {V} ^ {\pi} (s) - \mathcal {V} ^ {\pi^ {\prime}} (s) = \frac {1}{1 - \gamma} \underset { \begin{array}{c} s \sim \mathrm {d} _ {\lambda} ^ {\pi} \\ a \sim \pi (\cdot | s) \end{array} } {\mathbb {E}} [ \mathcal {A} ^ {\pi^ {\prime}} (s, a) ]. \tag {44}
$$
Using the last two inequalities, we have the drift inequality:
$$
\begin{array}{l} \mathcal {L} \left(\pi_ {k + 1}\right) - \mathcal {L} \left(\pi_ {k}\right) \leq \frac {\eta^ {2}}{2} \| \bar {\Theta} _ {k} (T) \| _ {2} ^ {2} - \eta \sum_ {s, a} \mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \pi^ {\star} (a | s) \left(\nabla^ {\top} \log \pi_ {k} (a | s) \bar {\Theta} _ {k} (T) - \mathcal {A} ^ {\pi_ {k}} (s, a)\right) \\ - \eta \left(\mathcal {V} ^ {\pi^ {*}} (\lambda) - \mathcal {V} ^ {\pi_ {k}} (\lambda)\right). \\ \end{array}
$$
For the log-linear policy parameterization, we have
$$
\nabla \log \pi_ {W} (a | s) = \Phi (s, a) - \sum_ {a ^ {\prime} \in \mathbb {A}} \pi_ {W} (a ^ {\prime} | s) \Phi (s, a ^ {\prime}).
$$
Also, from the definition of  $\mathcal{A}^{\pi}(s,a) = \mathcal{Q}^{\pi}(s,a) - \sum_{a^{\prime}\in \mathbb{A}}\pi (a^{\prime}|s)\mathcal{Q}^{\pi}(s,a^{\prime})$
$$
\begin{array}{l} \mathbb {E} \left[ \left(\nabla^ {\top} \log \pi_ {k} (a | s) \bar {\Theta} _ {k} (T) - \mathcal {A} ^ {\pi_ {k}} (s, a)\right) ^ {2} \right] \leq \mathbb {E} \left[ \left(\langle \Phi (s, a), \bar {\Theta} _ {k} (T) \rangle - \mathcal {Q} ^ {\pi_ {k}} (s, a)\right) ^ {2} \right], \\ = \sum_ {s, a} \mathrm {d} _ {\lambda} ^ {\pi^ {\star}} (s) \pi^ {\star} (a | s) \left(\langle \Phi (s, a), \bar {\Theta} _ {k} (T) \rangle - \mathcal {Q} ^ {\pi_ {k}} (s, a)\right) ^ {2}, \\ \leq C _ {\text {c o n c}} \sum_ {s, a} \mu^ {\pi_ {k}} (s, a) \left(\langle \Phi (s, a), \bar {\Theta} _ {k} (T) \rangle - \mathcal {Q} ^ {\pi_ {k}} (s, a)\right) ^ {2}, \\ \end{array}
$$
where the first line follows from the fact that  $Var(X) \leq \mathbb{E}[X^2]$  for any random variable  $X$  with finite second-order moments, and the last line follows from a change of measure argument. Then, by Theorem 3, we have:
$$
\sum_ {s, a} \mu^ {\pi_ {k}} (s, a) \Big (\langle \Phi (s, a), \bar {\Theta} _ {k} (T) \rangle - \mathcal {Q} ^ {\pi_ {k}} (s, a) \Big) ^ {2} \leq \frac {\rho u _ {k} ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}} \Big (3 L _ {\delta} ^ {- \frac {1 - p}{2 (1 + p)}} + 7 L _ {\delta} ^ {\frac {p}{1 + p}} \Big),
$$
with probability at least  $1 - \delta /K$ . Furthermore, we have:
$$
\left\| \bar {\Theta} _ {k} (T) \right\| _ {2} ^ {2} \leq \rho^ {2},
$$
for any  $k, T$  by the projection. As such, we can bound the drift inequality as follows:
$$
\begin{array}{l} \mathcal {L} (\pi_ {k + 1}) - \mathcal {L} (\pi_ {k}) \leq \frac {\eta^ {2}}{2} \rho^ {2} + \eta \sqrt {C _ {\text {c o n c}} \frac {\rho u _ {k} ^ {\frac {1}{1 + p}}}{(1 - \gamma) T ^ {\frac {p}{1 + p}}} \left(3 L _ {\delta} ^ {- \frac {1 - p}{2 (1 + p)}} + 7 L _ {\delta} ^ {\frac {p}{1 + p}}\right)} \\ - \eta (1 - \gamma) \left(\mathcal {V} ^ {\pi^ {\star}} (\lambda) - \mathcal {V} ^ {\pi_ {k}} (\lambda)\right), \tag {45} \\ \end{array}
$$
with probability at least  $1 - \delta / K$ . By telescoping sum of the above inequality, using union bound, and noting that  $\pi_0(a|s) = \frac{1}{|\mathbb{A}|}$  for any  $s, a$ , which leads to  $\mathcal{L}(\pi_1) = \log |\mathbb{A}|$ , we conclude the proof.