SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The following theorem shows that choosing a small  $\delta$  is not a panacea for the well-known  $f$ -divergence uncertainty set. Note that many well-known metrics, such as KL-divergence, total variation, and  $\chi^2$ -divergence are special cases of  $f$ -divergence.
Proposition 5 (Restatement of Proposition 3). For any  $f$ -divergence and radius  $\delta > 0$ , there exists a geometrically mixing nominal model such that the  $f$ -divergence defined uncertainty set violates Assumption 2.
Proof of Proposition 3. Given a continuous strict convex function  $f: \mathbb{R}_+ \to \mathbb{R}$  with  $f(1) = 0$ , the  $f$ -divergence is defined as  $\mathrm{d}_f(q, p) = \sum_s p(s) f\left(\frac{q(s)}{p(s)}\right)$ .
Given a nominal model  $p^\circ$  and a radius  $\delta > 0$ , the  $f$ -divergence-based uncertainty set is  $\mathcal{P}$  with  $\mathcal{P}_{s,a} = \{q : \mathrm{d}_f(q, p_{s,a}^\circ) \leq \delta\}$ . For any  $f$  and  $\delta > 0$ , consider a nominal model  $p^\circ$ , which has a uniform transition probability, i.e., transits to a uniformly and randomly selected next state at any state-action pair, except for  $(s,a)$ . The transition probability at  $(s,a)$  is  $p_{s,a}^\circ = (\alpha, \frac{1 - \alpha}{|S| - 1}, \frac{1 - \alpha}{|S| - 1}, \dots, \frac{1 - \alpha}{|S| - 1})$ , where  $\alpha$  is some parameter to be determined later. It is clear that this nominal model  $p^\circ$  is well mixed for any  $\alpha \in [0,1]$ . Consider another model  $q$  which coincides with  $p^\circ$  except for states  $(s,a)$  with  $q_{s,a} = (\alpha / \gamma, \frac{1 - \alpha / \gamma}{|S| - 1}, \frac{1 - \alpha / \gamma}{|S| - 1}, \dots, \frac{1 - \alpha / \gamma}{|S| - 1})$ . Note that  $\frac{q_{s,a}(1)}{p_{s,a}^\circ(1)} = \frac{1}{\gamma}$ , thus if  $q \in \mathcal{P}$ , the Assumption 2 of  $\beta < 1$  is violated. Since as  $\alpha \to 0$ , by the continuity of  $f$  and  $f(1) = 0$ , we have
$$
\begin{array}{l} \mathrm {d} _ {f} (q _ {s, a}, p _ {s, a} ^ {\circ}) = \sum_ {s ^ {\prime}} p _ {s, a} ^ {\circ} (s ^ {\prime}) f (\frac {q _ {s , a} (s ^ {\prime})}{p _ {s , a} ^ {\circ} (s ^ {\prime})}) = \alpha f (1 / \gamma) + (1 - \alpha) f (\frac {1 - \alpha / \gamma}{1 - \alpha}) \\ = \alpha f (1 / \gamma) + (1 - \alpha) f (1 - \frac {1 / \gamma - 1}{1 - \alpha} \alpha) \rightarrow 0 <   \delta . \\ \end{array}
$$
Therefore, with a sufficiently small  $\alpha$ , we have  $q_{s,a} \in \mathcal{P}_{s,a}$  and clearly  $q \in \mathcal{P}$ . Thus there does not exist a universal choice of  $\delta$  for  $f$ -divergence-based uncertainty set to guarantee Assumption 2.
We next prove Lemma 1, which shows the contraction of robust Bellman operator for IPM-based RMDP without Assumption 2.
Proof of Lemma 1. Let  $P^{\pi}(s'|s) \coloneqq \sum_{a} p_{s, \pi(a|s)}^{\circ}(s')$  be the state transition kernel of executing policy  $\pi$  on the nominal model  $p^{\circ}$ . We can view  $P^{\pi} \in \mathbb{R}^{S \times S}$  as a matrix. Recall  $D^{\pi} = \mathrm{diag}(\nu^{\pi})$ , we have
$$
\begin{array}{l} \left\| \mathcal {T} _ {\mathcal {P}} ^ {\pi} \Psi w - \mathcal {T} _ {\mathcal {P}} ^ {\pi} \Psi w ^ {\prime} \right\| _ {\nu^ {\pi}} = \left\| \gamma P ^ {\pi} \Psi w - \gamma \delta \| w _ {2: d} \| \mathbf {1} - \gamma P ^ {\pi} \Psi w ^ {\prime} + \gamma \delta \| w _ {2: d} ^ {\prime} \| \mathbf {1} \right\| _ {\nu^ {\pi}} \\ \leq \gamma \| \Psi w - \Psi w ^ {\prime} \| _ {\nu^ {\pi}} + \gamma \delta \big | \| w \| - \| w ^ {\prime} \| \big | \leq \gamma \| \Psi w - \Psi w ^ {\prime} \| _ {\nu^ {\pi}} + \frac {\gamma \delta}{\lambda_ {\min} (\Psi^ {\top} D ^ {\pi} \Psi)} \| \Psi w - \Psi w ^ {\prime} \| _ {\nu^ {\pi}}. \\ \end{array}
$$
Since  $\Pi^{\pi}$  is a non-expansion mapping w.r.t.  $\| \cdot \|_{\nu^{\pi}}$ ,  $\Pi^{\pi}T_{\mathcal{P}}^{\pi}$  is a contraction mapping if  $\gamma (1 + \frac{\delta}{\lambda_{\min}(\Psi^\top D^\pi\Psi)}) < 1$ , which is equivalent to the condition  $\delta < \lambda_{\min}(\Psi^\top D^\pi \Psi)\frac{1 - \gamma}{\gamma}$ .
C.1.2 Convergence of Robust Linear TD
As discussed in the paragraph under Assumption 1, robust linear TD (RLTD) is a stochastic approximation (c.f. Section F.3 for a brief overview of stochastic approximation) since empirical operator  $\hat{T}_{\mathcal{P}}^{\pi}$  ((3) or (6)) is unbiased with
$$
\mathbb {E} _ {(s, a, y ^ {\prime}) \sim \nu^ {\pi} \circ \pi \circ p ^ {\circ}} \left[ \psi (s) \left[ (\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w}) (s, a, y ^ {\prime}) - \psi (s) ^ {\top} w \right] \right] = \Psi^ {\top} D ^ {\pi} (\mathcal {T} _ {\mathcal {P}} ^ {\pi} V _ {w} - V _ {w}).
$$
We can then let  $F(w,x) = \psi(s)\left[(\hat{\mathcal{T}}_{\mathcal{P}}^{\pi}V_w)(s,a,y') - \psi(s)^\top w\right]$  with  $x = (s,a,y')$  and RLTD is solving the following equation
$$
\bar {F} (w) := \Psi^ {\top} D (\mathcal {T} _ {\mathcal {P}} ^ {\pi} \Psi w - \Psi w) = 0,
$$
through stochastic approximation:
$$
w _ {k + 1} = w _ {k} + \alpha_ {k} F (w _ {k}, x _ {k}), \quad x _ {k} = \left(s _ {k}, a _ {k}, y _ {k + 1}\right).
$$
It is clear that  $\{x_{k}\}$  is also a Markov chain with domain  $\mathcal{X} = \mathcal{S} \times \mathcal{A} \times \mathcal{Y}$ ,  $\mathcal{Y} = \mathcal{S}^{m}$  for the double-sampling uncertainty set with  $m$  samples and  $\mathcal{Y} = \mathcal{S}$  for the IPM uncertainty set.
Theorem 5 (Formal statement of Theorem 3). Take  $\delta$  as in Lemma 1 and in Proposition 4, for the DS uncertain set or the IPM uncertainty set, respectively. Under Assumption 1, RLTD with step sizes  $\alpha_{k} = \Theta(1/k)$  guarantees  $\mathbb{E}[\|w_{K} - w^{\pi}\|^{2}] = \tilde{O}\left(\frac{1}{K}\right)$  and  $\mathbb{E}[\|V_{w_{K}} - V_{w^{\pi}}\|_{\nu^{\pi}}^{2}] = \tilde{O}\left(\frac{1}{K}\right)$ .
Proof of Theorem 5. By Proposition 4 and Lemma 1, DS or IPM with small  $\delta$  can guarantee  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$ -contraction w.r.t.  $\| \cdot \|_{\nu^{\pi}}$  for some  $\beta < 1$ . The theorem can be proved by applying Lemma 12. For this purpose, we only need to show that Assumption 8 is satisfied for RLTD. We check the three conditions in Assumption 8 as follows.
1. The geometric mixing property of  $\{x_{k} = (s_{k},a_{k},y_{k + 1})\}_{k\geq 0}$  is straightforward given geometrically mixed  $\{s_k\}_{k\geq 0}$ .
2. Since  $r \in [0,1]$  and  $\| \psi(s) \|$  is bounded  $\forall s \in S$ , we have  $\| F(0,x) \| = \| \psi(s)r(s,a) \| \leq \| \psi(s) \|$  is also bounded for any  $x = (s,a,y')$ .
For double-sampling (3) and any  $x = (s, a, s_{1:m}^{\prime})$ ,
$$
F (w, x) = \psi (s) \left(r (s, a) + \gamma \inf  _ {\alpha \in \Delta_ {[ m ]}: \mathrm {d} (\alpha , \operatorname {U n i f} (m)) \leq \delta} \sum_ {i = 1} ^ {m} \alpha_ {i} V _ {w} (s _ {i} ^ {\prime}) - \psi (s) ^ {\top} w\right).
$$
Let  $\Psi_{s_{1:m}^{\prime}}\in \mathbb{R}^{m\times d}$  be the matrix by stacking up  $\psi (s_1^{\prime})^\top ,\ldots ,\psi (s_m^{\prime})^\top$  , we have
$$
\begin{array}{l} \| F (w _ {1}, x) - F (w _ {2}, x) \| \\ \leq \gamma \| \psi (s) \| \left| \inf  _ {\alpha} \alpha^ {\top} \Psi_ {s _ {1: m} ^ {\prime}} w _ {1} - \inf  _ {\alpha} \alpha^ {\top} \Psi_ {s _ {1: m} ^ {\prime}} w _ {2} \right| + \gamma \| \psi (s) \psi (s) ^ {\top} (w _ {1} - w _ {2}) \| \\ \leq \gamma \| \psi (s) \| | \sup  _ {\alpha} \alpha^ {\top} \Psi_ {s _ {1: m} ^ {\prime}} \left(w _ {1} - w _ {2}\right) | + \gamma \| \psi (s) \| ^ {2} \| w _ {1} - w _ {2} \| \\ \leq \max _ {s ^ {\prime}} 2 \gamma \| \psi (s ^ {\prime}) \| ^ {2} \| w _ {1} - w _ {2} \|, \\ \end{array}
$$
where  $\sup_{\alpha}, \inf_{\alpha}$  are taking in  $\{\alpha \in \Delta_{[m]} : \mathrm{d}(\alpha, \mathrm{Unif}([m])) \leq \delta\}$ . Thus there exists  $L_1 = \max (\max_s \| \psi(s) \|, 2\gamma \max_s \| \psi(s) \|^2)$  that guarantee  $\| F(0,x) \| \leq L_1$  and  $\| F(w,x) - F(w',x) \| \leq L_1 \| w - w' \|$  for the double-sampling RMDP.
For IPM (6) and any  $x = (s,a,s')$ ,
$$
F (w, x) = \psi (s) \left(r (s, a) + \gamma \psi (s ^ {\prime}) ^ {\top} w - \gamma \delta \| w _ {2: d} \| - \psi (s) ^ {\top} w\right).
$$
$$
\| F (w, x) - F (w ^ {\prime}, x) \| \leq \max _ {s} \left((\gamma + 1) \| \psi (s) \| ^ {2} + \gamma \delta \| \psi (s) \|\right) \| w - w ^ {\prime} \|,
$$
which also satisfies the second item of Assumption 8 by setting  $L_{1} = \max(\max_{s} \| \psi(s) \|, \max_{s} \left( (\gamma + 1) \| \psi(s) \|^2 + \gamma \delta \| \psi(s) \| \right)$ .
3. Note that for small  $\delta$ ,  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$ -contraction w.r.t.  $\| \cdot \|_{\nu^{\pi}}$ . Since there exists  $w^{\pi}$  with  $\bar{F}(w^{\pi}) = 0$ , let  $D := D^{\pi} = \mathrm{diag}(\nu^{\pi})$ , we have
$$
\begin{array}{l} \langle w - w ^ {\pi}, \bar {F} (w) \rangle = \langle w - w ^ {\pi}, \bar {F} (w) - \bar {F} (w ^ {\pi}) \rangle \\ = \left\langle w - w ^ {\pi}, \Psi^ {\top} D \left(\mathcal {T} ^ {\pi} \Psi w - \mathcal {T} ^ {\pi} \Psi w ^ {\pi}\right) \right\rangle - \| \Psi (w - w ^ {\pi}) \| _ {\nu^ {\pi}} ^ {2} \\ = \left\langle \Psi^ {\top} D \Psi \left(w - w ^ {\pi}\right), \left(\Psi^ {\top} D \Psi\right) ^ {- 1} \Psi^ {\top} D \left(\mathcal {T} ^ {\pi} \Psi w - \mathcal {T} ^ {\pi} \Psi w ^ {\pi}\right) \right\rangle - \| \Psi \left(w - w ^ {\pi}\right) \| _ {\nu^ {\pi}} ^ {2} \\ = \left\langle D ^ {1 / 2} \Psi \left(w - w ^ {\pi}\right), D ^ {1 / 2} \Psi \left(\Psi^ {\top} D \Psi\right) ^ {- 1} \Psi^ {\top} D \left(\mathcal {T} ^ {\pi} \Psi w - \mathcal {T} ^ {\pi} \Psi w ^ {\pi}\right) \right\rangle - \| \Psi \left(w - w ^ {\pi}\right) \| _ {\nu^ {\pi}} ^ {2} \\ \leq \| \Psi (w - w ^ {\pi}) \| _ {\nu^ {\pi}} \| \Pi^ {\pi} \mathcal {T} _ {\mathcal {P}} ^ {\pi} \Psi w - \Pi^ {\pi} \mathcal {T} _ {\mathcal {P}} ^ {\pi} \Psi w ^ {\pi} \| _ {\nu^ {\pi}} - \| \Psi (w - w ^ {\pi}) \| _ {\nu^ {\pi}} ^ {2} \\ \leq - (1 - \beta) \| \Psi (w - w ^ {\pi}) \| _ {\nu^ {\pi}} ^ {2} \leq - (1 - \beta) \lambda_ {\min } (\Psi^ {\top} D \Psi) \| w - w ^ {\pi} \| ^ {2}, \\ \end{array}
$$
where the first inequality is due to the Cauchy-Schwarz inequality and the second inequality is due to the  $\beta$ -contraction property of  $\Pi^{\pi}\mathcal{T}_{\mathcal{P}}^{\pi}$ .
$$
\begin{array}{l} \mathbb {E} \left[ \| V _ {w _ {K}} - V _ {w ^ {\pi}} \| _ {\nu^ {\pi}} ^ {2} \right] = \mathbb {E} \left[ \left(w _ {K} - w ^ {\pi}\right) ^ {\top} \left(\Psi^ {\top} D ^ {\pi^ {\pi}} \Psi\right) \left(w _ {K} - w ^ {\pi}\right) \right] \\ \leq \lambda_ {m a x} (\Psi^ {\top} D ^ {\pi} \Psi) \mathbb {E} [ \| w _ {K} - w ^ {\pi} \| ^ {2} ] = \tilde {O} (1 / K). \\ \end{array}
$$
C.2 Extension of General Value Function Approximation
Setting: In the general function approximation setting, we consider a known finite and bounded function class  $\mathcal{G} \subset \mathbb{R}^S$  to fit the robust value function  $V^{\pi}$ , i.e.,
$$
\mathcal {G} \subset \mathbb {R} ^ {\mathcal {S}}, \quad | \mathcal {G} | <   \infty , \quad | g (s) | <   \infty , \forall g \in \mathcal {G}, \forall s \in \mathcal {S}. \tag {14}
$$
Note that IPM-based RMDP with empirical robust Bellman operator (6) only applies to linear function approximation. We implement it similarly by adding a negative regularization with neural network approximating the robust values, which also induces robust behavior of the learned policy as illustrated in the experiments (Section 7). However, it may not directly match any specific uncertainty set. We thus only consider the DS-based RMDP in this general value function approximation setting.
We define the robust Bellman error as follows.
Definition 4 (Robust Bellman Error).  $\epsilon_{g,bias} := \max_{\pi} \max_{g \in \mathcal{G}} \min_{g' \in \mathcal{G}} \| \mathcal{T}_{\mathcal{P}}^{\pi} g - g' \|_{\nu^{\pi}}$ .
If  $\epsilon_{g,bias} = 0$ , then the robust value is realizable within the function class, i.e.,  $V^{\pi} \in \mathcal{G}$  for any  $\pi \in \mathcal{G}$ .
C.2.1 Fitted Robust Value Evaluation Algorithm
We propose the fitted robust value evaluation (FRVE) in Algorithm 5, a robust version of the fitted value evaluation commonly used in offline RL. This algorithm first samples a batch of data from the nominal model, then select the last half as training data for analytical purpose without losing the order while the Markovian data are close to the stationary distribution due to geometric mixing. Assumption 1, and iteratively solve for a better robust value approximation based on the current approximation  $g_{k}$  and its robust value estimate  $\hat{T}_{\mathcal{P}}^{\pi}(g_k)$ .
Theorem 6 (Convergence of FRVE). For DS RMDP with  $\delta$  suggested by Proposition 4,  $\mathcal{T}_{\mathcal{P}}^{\pi}$  is a  $\beta$ -contraction mapping for some  $\beta < 1$ . Under Assumption 7, the return of FRVE Algorithm 5 has
$$
\mathbb {E} [ \| V ^ {\pi} - g _ {K} \| _ {\nu^ {\pi}} ] \leq \beta^ {K} (G _ {\max } + \frac {1}{1 - \gamma}) + \frac {\epsilon_ {g , s t a t}}{1 - \beta} + \frac {\epsilon_ {g , b i a s}}{1 - \beta},
$$
where  $G_{\mathrm{max}} \coloneqq \max_{g \in \mathcal{G}}((\hat{\mathcal{T}}_{\mathcal{P}}^{\pi}g)(s,a,y'), \| \mathcal{T}_{\mathcal{P}}^{\pi}g\|_{\nu^{\pi}}, \| g\|_{\nu^{\pi}})$  and  $\epsilon_{g,stat} = \tilde{O}(1 / \sqrt{K})$ . Thus  $\mathbb{E}[\| V^{\pi} - g_K\|_{\nu^{\pi}}] = \tilde{O}(1 / \sqrt{K}) + O(\epsilon_{g,bias})$ .
Proof. By the contraction mapping of  $\mathcal{T}_{\mathcal{P}}^{\pi}$
$$
\begin{array}{l} \left\| V ^ {\pi} - g _ {K} \right\| _ {\nu^ {\pi}} \leq \left\| \mathcal {T} _ {\mathcal {P}} ^ {\pi} V ^ {\pi} - \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {K - 1} \right\| _ {\nu^ {\pi}} + \left\| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {K - 1} - g _ {K} \right\| _ {\nu^ {\pi}} \\ \leq \beta \| V ^ {\pi} - g _ {K - 1} \| _ {\nu^ {\pi}} + \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {K - 1} - g _ {K} \| _ {\nu^ {\pi}} \\ \leq \sum_ {k = 1} ^ {K} \beta^ {K - k} \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} + \beta^ {K} \| V ^ {\pi} - g _ {0} \| _ {\nu^ {\pi}} \\ \leq \sum_ {k = 1} ^ {K} \beta^ {K - k} \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} + \beta^ {K} (\| g _ {0} \| _ {\nu^ {\pi}} + \frac {1}{1 - \gamma}). \\ \end{array}
$$
Note that  $\mathcal{T}_{\mathcal{P}}^{\pi}g_{k - 1}$  is a target that  $g_{k}$  is an approximation through MSE with Markovian data. If the data is stationary, i.e.,  $s_{K / 2}\sim \nu^{\pi}$ , applying Lemma 11 and the union bound over  $\mathcal{G}$  (taking  $\mathcal{T}_{\mathcal{P}}^{\pi}g$  as a target for each  $g\in \mathcal{G}$ ), we know with probability at least  $1 - \delta$ ,
$$
\| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} = O \left(G _ {m a x} \sqrt {\frac {\log (| \mathcal {G} |) + \log (1 / \delta)}{K}}\right) + O (\epsilon_ {g, b i a s}), \quad \forall k = 1, 2, \dots , K.
$$
Let  $P_{\mu}$  be a distribution on  $\mathcal{D}$  with  $x_{K / 2} \sim \mu$ , and let  $P_{\nu^{\pi}}$  be a distribution on  $\mathcal{D}$  with  $x_{K / 2} \sim \nu^{\pi}$ . Take  $\mu$  as the true distribution of  $x_{K / 2}$  according to the data sampling process as in Algorithm 5. We know  $\| \mu - \nu^{\pi} \|_{TV} = O(e^{-K})$  by Assumption 1, and thus
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} \sim P _ {\mu}} \left[ \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} \right] \leq 2 G _ {\max } \| P _ {\mu} - P _ {\nu^ {\pi}} \| _ {T V} + \mathbb {E} _ {\mathcal {D} \sim P _ {\nu^ {\pi}}} \left[ \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} \right] \\ = 2 G _ {\max } \| \mu - \nu^ {\pi} \| _ {T V} + \mathbb {E} _ {\mathcal {D} \sim P _ {\nu^ {\pi}}} \left[ \| \mathcal {T} _ {\mathcal {P}} ^ {\pi} g _ {k - 1} - g _ {k} \| _ {\nu^ {\pi}} \right] \\ = O \left(G _ {\max } e ^ {- K}\right) + O \left(G _ {\max } \sqrt {\frac {\log (| \mathcal {G} |) + \log (K)}{K}}\right) + O \left(\epsilon_ {g, b i a s}\right). \\ \end{array}
$$
The proof of the theorem is thus concluded.
D Robust Natural Actor Analysis
We analyze the robust natural component in the RNAC algorithm (Algorithm 1).
We first discuss the Fréchet supergradient of the robust value function in the first subsection. The second subsection focuses on the linear function approximation and proves the main theorems as stated in Section 5 of the main paper. The last subsection focuses on the general function approximation.
D.1 Policy Gradient and Performance Difference
The robust value function  $J(\theta) = V^{\pi_{\theta}}(\rho)$  is in general not differentiable. But since it is Lipschitz (w.r.t. to Lipschitz policy parameterization), it is differentiable almost everywhere according to Rademacher's theorem [14]. At the place not differentiable, Fréchet supergradient  $\nabla J$  of  $J$  is then defined as
$$
\limsup_{\theta^{\prime}\to 0}\frac{J(\theta^{\prime}) - J(\theta) - \langle\nabla J(\theta),\theta^{\prime} - \theta\rangle}{\| \theta^{\prime} - \theta \|}\leq 0.
$$
When function  $J$  is differentiable,  $J$  at any point  $\theta$  has a unique Fréchet supergradient, which is the gradient of  $J$  at  $\theta$ .
Proof of Lemma 2. The Fréchet supergradient exists for tabular RMDP [29], which is
$$
[ \nabla_ {\pi} V ^ {\pi} (\rho) ] _ {s, a} = \frac {1}{1 - \gamma} d _ {\rho} ^ {\pi , \kappa_ {\pi}} (s) Q ^ {\pi} (s, a), \quad \forall s, a,
$$
where  $[\nabla_{\pi}V^{\pi}(\rho)]_{s,a}$  indicates the  $(s,a)$  coordinate of vector  $\nabla_{\pi}V^{\pi}(\rho)$ .  $\pi = \pi_{\theta}$  is a policy parameterized by  $\theta$ .
$$
\nabla_ {\theta} V ^ {\pi_ {\theta}} (\rho) = \sum_ {s, a} \left[ \nabla_ {\pi} V ^ {\pi} (\rho) \right] _ {s, a} \nabla_ {\theta} \pi_ {\theta} (a | s)
$$
$$
\begin{array}{l} = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi , \kappa_ {\pi}}} \mathbb {E} _ {a \sim \pi_ {s}} [ Q ^ {\pi} (s, a) \nabla_ {\theta} \log \pi_ {\theta} (a | s) ] \\ = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi , \kappa_ {\pi}}} \mathbb {E} _ {a \sim \pi_ {s}} \left[ A ^ {\pi} (s, a) \nabla_ {\theta} \log \pi_ {\theta} (a | s) \right], \\ \end{array}
$$
where the first relation is by the chain rule of supergradient.
D.2 Linear Function Policy Approximation
Setting: This subsection considers the log-linear policy with
$$
\pi_ {\theta} (a | s) = \frac {\exp (\phi (s , a) ^ {\top} \theta)}{\sum_ {a ^ {\prime}} \exp (\phi (s , a ^ {\prime}) ^ {\top} \theta)}, \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \tag {15}
$$
where  $\phi (s,a)\in \mathbb{R}^d$  is some known feature vector and  $\theta \in \mathbb{R}^d$  is the policy parameter. Let  $\Phi \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times d}$  be the feature matrix by stacking up the feature vectors  $\phi (s,a)^{\top}$ .
Recall the discussion of the proposed RQNPG in Section 6. We approximate the robust Q function  $Q^{\pi}$  via  $Q_{w}(s,a) = r(s,a) + \inf_{p\in \mathcal{P}_{s,a}}p^{\top}V_{w}$  given a value function approximation  $V_{w}$  from the robust critic, and then approximate  $Q_{w}$  by a policy-compatible [52] robust Q-approximation  $Q^{u} = \Phi u$ . In other words, we project  $Q_{w}$  onto  $\text{span}(\Phi)$ . Denote by  $\Pi_{\Phi}^{\pi} \coloneqq \Phi (\Phi^{\top}\mathrm{diag}(\nu^{\pi}\circ \pi)\Phi)^{-1}\Phi^{\top}\mathrm{diag}(\nu^{\pi}\circ \pi)\in \mathbb{R}^{|S||\mathcal{A}|\times |S||\mathcal{A}|}$  the projection matrix onto space  $\text{span}(\Phi)$  w.r.t. norm  $\| \cdot \|_{\nu^{\pi}\circ \pi}$ . We then define the approximation error  $\epsilon_{Q,bias}$  below. When realizable, i.e.,  $Q^{\pi}\in \text{span}(\Phi)$ , the approximation error  $\epsilon_{Q,bias} = 0$ .
Definition 5.  $\epsilon_{Q,bias} := \max_{\pi,\pi'} \max_{d=d_{\rho}^{\pi',\kappa_{\pi'}}}, d_{\rho}^{\pi*}, \kappa_{\pi} \text{ or } \nu^{\pi*}} \left\| \Pi_{\Phi}^{\pi} Q^{\pi} - Q^{\pi} \right\|_{d \circ Unif}$ .
We assume a finite relative condition number (Assumption 3) (similar to that in [3]). The relative condition number is not necessarily related to the size of the state space (details are shown in Remark 6.3 of [3]).
Assumption 3.  $\max_{\pi ,\pi^{\prime}}\max_{d = d_{\rho}^{\pi^{\prime},\kappa_{\pi^{\prime}}},d_{\rho}^{\pi^{*},\kappa_{\pi}}}\sup_{u}\frac{\|\Phi u\|_{d\circ Unif}}{\|\Phi u\|_{\nu^{\pi}\circ\pi}}\leq \xi <  \infty$  for some  $\xi$
Now we look at a specific update  $\theta^{t + 1} = \mathsf{RQNPG}(\theta^t,\eta^t,w^t,N)$ , where  $w^{t} = \mathrm{RLTD}(\pi_{\theta^{t}},K)$ .
D.2.1 RQNPG One-Step Analysis - Robust Q Function Approximation
In this update  $\theta^{t + 1} = \mathsf{RQNPG}(\theta^t,\eta^t,w^t,N)$ , where  $w^{t} = \mathsf{RLTD}(\pi_{\theta^{t}},K)$ . RQNPG first approximates  $Q_{w^t}(s,a) = r(s,a) + \gamma \inf_{p\in \mathcal{P}_{s,a}}p^\top V_{w^t} = \mathbb{E}[(\hat{\mathcal{T}}_\mathcal{P}^\pi V_{w^t})(s,a,y')|s,a]$  by  $Q^{u}(s,a) = \phi (s,a)^{\top}u$ , as the calculation of  $u_{N}$  in Algorithm 3. Let
$$
u _ {*} ^ {t} := \arg \min _ {u} \mathbb {E} _ {(s, a) \sim \nu^ {t} \circ \pi^ {t}} [ (Q _ {w ^ {t}} (s, a) - Q ^ {u} (s, a)) ^ {2} ].
$$
be the optimal approximation for the target  $Q_{w^t}$ .  $u_{*}^{t}$  is approximated by stochastic approximation (c.f. Section F.3 for a brief overview of stochastic approximation) with a mean squared error loss
$$
L (u; V _ {w ^ {t}}, \pi) = \mathbb {E} _ {(s, a, y ^ {\prime}) \sim \nu^ {\pi} \circ \pi \circ p ^ {\circ}} \left[ \left((\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w ^ {t}}) (s, a, y ^ {\prime}) - \phi (s, a) ^ {\top} u\right) ^ {2} \right].
$$
We know  $u_{*}^{t}$  is the unique solution of
$$
0 = - \nabla_ {u} L (u; V _ {w ^ {t}}, \pi) = \mathbb {E} _ {(s, a, y ^ {\prime}) \sim \nu^ {\pi} \circ \pi \circ p ^ {\circ}} \left[ \phi (s, a) \left((\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w ^ {t}}) (s, a, y ^ {\prime}) - \phi (s, a) ^ {\top} u\right) \right].
$$
Let  $F(u, x)$  be the function inside the expectation with  $x = (s, a, y')$ , and  $\bar{F}(u)$  be the negative gradient  $-\nabla_u L(u; V, \pi)$ . We then solve this stochastic zero point problem by stochastic approximation as in
$$
u _ {n + 1} = u _ {n} + \zeta_ {n} \phi (s _ {n}, a _ {n}) \left[ (\hat {\mathcal {T}} _ {\mathcal {P}} ^ {\pi} V _ {w}) (s _ {n}, a _ {n}, y _ {n + 1}) - \phi (s _ {n}, a _ {n}) ^ {\top} u _ {n} \right] = u _ {n} + \zeta_ {n} F (u _ {n}, x _ {n}),
$$
where  $x_{n} = (s_{n},a_{n},y_{n + 1})$
Lemma 3 (Convergence of compatible Q function approximation (SGD with Markovian data)). Under Assumption 1, RQNPG (Algorithm 3) with step sizes  $\zeta_{n} = \Theta(1/n)$  guarantees  $\mathbb{E}[\|u_{N}-u_{*}^{t}\|^{2}] = \tilde{O}\left(\frac{1}{N}\right)$  and  $\mathbb{E}[\|Q^{u_{N}}-Q^{u_{*}^{t}}\|_{\nu^{t} \circ \pi^{t}}^{2}] = \tilde{O}\left(\frac{1}{N}\right)$ .
Proof of Lemma 3. The lemma is implied by Lemma 12. To see this, we only need to show that Assumption 8 is satisfied. We check the three conditions in Assumption 8 as follows.
1. The geometric mixing property of  $\{x_{k}\}_{k\geq 0}$  is straightforward given geometrically mixed  $\{s_k\}_{k\geq 0}$ .
2. Since  $\| \phi (s,a)\|$  is bounded  $\forall (s,a)\in S\in \mathcal{A}$ , similar proof follows as in the proof of 5.
3. Since there exists  $u_{\pi}$  with  $\bar{F} (u_{\pi}) = 0$  and  $L(u;V,\pi)$  is  $\lambda_{min}(\mathbb{E}_{s,a}[\phi_{s,a}\phi_{s,a}^{\top}])$ -strongly convex, we have
$$
\begin{array}{l} \langle u - u _ {\pi}, \bar {F} (u) \rangle = \langle u - u _ {\pi}, \bar {F} (u) - \bar {F} (u _ {\pi}) \rangle \\ = - \left\langle u - u _ {\pi}, \nabla L (u; V, \pi) - \nabla L \left(u ^ {\pi}; V, \pi\right) \right\rangle \leq - \lambda_ {m i n} \left(\Sigma_ {\nu^ {\pi} \circ \pi}\right) \| u - u _ {\pi} \| ^ {2}, \\ \end{array}
$$
where  $\Sigma_{\nu^{\pi}\circ \pi}:= \mathbb{E}_{s\sim \nu^{\pi},a\sim \pi}[\phi_{s,a}\phi_{s,a}^{\top}]$
Denote by  $\Pi_{\Phi}^{\pi} \coloneqq \Phi (\Phi^{\top}\mathrm{diag}(\nu^{\pi}\circ \pi)\Phi)^{-1}\Phi^{\top}\mathrm{diag}(\nu^{\pi}\circ \pi) \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times |\mathcal{S}||\mathcal{A}|}$  the projection matrix of function  $Q \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$  onto matrix  $\Phi \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times d}$  under norm  $\| \cdot \|_{\nu^{\pi}\circ \pi}$ . We know  $Q^{u_{*}^{t}} = \Pi_{\Phi}^{\pi^{t}}Q_{w^{t}}$ . We have