SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition D.4. For all  $Q \in \mathbb{R}^{\mathcal{X}}$ , denote  $v(s) \coloneqq \langle \pi_s, Q(s, \cdot) \rangle_{\mathcal{A}}, \forall s \in S$ . Then, for any policy  $\pi \in \Pi$ , the  $Q$ -value iteration defined according to  $Q_{n+1} = \mathcal{L}_{\mathcal{U}}^{\pi} Q_n$  induces
$$
v _ {n + 1} := \mathcal {T} _ {\mathcal {U}} ^ {\pi} v _ {n}.
$$
Proof. By construction, for all  $s \in S$  we have
$$
\begin{array}{l} v _ {n + 1} (s) = \left\langle \pi_ {s}, Q _ {n + 1} (s, \cdot) \right\rangle_ {\mathcal {A}} \\ = \left\langle \pi_ {s}, \left(\mathcal {L} _ {\mathcal {U}} ^ {\pi} Q _ {n}\right) (s, \cdot) \right\rangle_ {\mathcal {A}} \\ = \sum_ {a \in \mathcal {A}} \pi_ {s} (a) \left[ R _ {\mathcal {U}, v _ {n}} ^ {\pi} (s, a) + \gamma \sum_ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {X}} P _ {\mathcal {U}, v _ {n}} ^ {\pi} \left(s ^ {\prime} \mid s, a\right) \pi_ {s ^ {\prime}} \left(a ^ {\prime}\right) Q _ {n} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \tag {ByEq.9} \\ = \sum_ {a \in \mathcal {A}} \pi_ {s} (a) \left[ R _ {\mathcal {U}, v _ {n}} ^ {\pi} (s, a) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} P _ {\mathcal {U}, v _ {n}} ^ {\pi} \left(s ^ {\prime} | s, a\right) v _ {n} \left(s ^ {\prime}\right) \right] \quad (\text {B y}) \\ = \left(\mathcal {T} _ {\mathcal {U}} ^ {\pi} v _ {n}\right) (s), \\ \end{array}
$$
where the last equality holds because  $(P_{\mathcal{U},v_n}^\pi, R_{\mathcal{U},v_n}^\pi) \in \arg \min_{(P,R) \in \mathcal{U}} \mathcal{T}_{(P,R)}^\pi v_n$ .
As a result of the above proposition, the value iteration induced by our Q-value iteration rule converges linearly to the robust value function, i.e.,  $\| v_{n} - v_{\mathcal{U}}^{\pi}\|_{\infty}\leq \gamma^{n}\| v_{0}\|_{\infty}$ . Therefore, Q-value iterates converge to a fixed point. Precisely,  $v_{n}\to_{n}v_{\mathcal{U}}^{\pi}$  implies that  $(P_{\mathcal{U},v_n}^{\pi},R_{\mathcal{U},v_n}^{\pi})\to_n(P_\mathcal{U}^\pi ,R_\mathcal{U}^\pi)$ , which in turn implies that  $Q_{n}\rightarrow_{n}Q_{\mathcal{U}}^{\pi}$ . The result below further characterizes the convergence rate.
Proposition D.5. For any policy  $\pi \in \Pi$ , the recursion  $Q_{n+1} \coloneqq \mathcal{L}_{\mathcal{U}}^{\pi} Q_n$ ,  $n \in \mathbb{N}$  converges linearly to  $Q_{\mathcal{U}}^{\pi}$ .
Proof. Thanks to Prop. D.4, we have:
$$
\begin{array}{l} \left\| Q _ {n + 1} - Q _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty} = \left\| R _ {\mathcal {U}, v _ {n}} ^ {\pi} + \gamma P _ {\mathcal {U}, v _ {n}} ^ {\pi} v _ {n} - R _ {\mathcal {U}} ^ {\pi} + \gamma P _ {\mathcal {U}} ^ {\pi} v _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty} \\ = \gamma \| P _ {\mathcal {U}, v _ {n}} ^ {\pi} v _ {n} - P _ {\mathcal {U}} ^ {\pi} v _ {\mathcal {U}} ^ {\pi} \| _ {\infty} \quad (R _ {\mathcal {U}, v} ^ {\pi} = R _ {\mathcal {U}} ^ {\pi}, \quad \forall v), \\ = \gamma \| \left(P _ {0} - B ^ {\pi} u _ {n}\right) v _ {n} - \left(P _ {0} - B ^ {\pi} u _ {\mathcal {U}} ^ {\pi}\right) v _ {\mathcal {U}} ^ {\pi} \| _ {\infty}, \\ \end{array}
$$
where by the worst kernel characterization,  $B^{\pi}(s,a) \coloneqq \beta_{s,a}$  for  $\mathcal{U} = \mathcal{U}_p^{\mathrm{sa}}$  and  $B^{\pi}(s,a) \coloneqq \beta_s\left(\frac{\pi_s(a)}{\|\pi_s\|_q}\right)^{q - 1}$  for  $\mathcal{U} = \mathcal{U}_p^{\mathrm{sa}}$ . This implies
$$
\begin{array}{l} \left\| Q _ {n + 1} - Q _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty} \leq \gamma \left\| P _ {0} \left(v _ {n} - v _ {\mathcal {U}} ^ {\pi}\right) \right\| _ {\infty} + \gamma \left\| B ^ {\pi} \left(u _ {n}\right) ^ {\top} v _ {n} - B ^ {\pi} \left(u _ {\mathcal {U}} ^ {\pi}\right) ^ {\top} v _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty} \\ \leq \gamma^ {n + 1} \| v _ {0} - v _ {\mathcal {U}} ^ {\pi} \| _ {\infty} + \gamma \| (u _ {n}) ^ {\top} v _ {n} - (u _ {\mathcal {U}} ^ {\pi}) ^ {\top} v _ {\mathcal {U}} ^ {\pi} \| \quad (B ^ {\pi} (s, a) \leq 1), \\ \leq \gamma^ {n + 1} \| v _ {0} - v _ {\mathcal {U}} ^ {\pi} \| _ {\infty} + \gamma \| \left(u _ {n}\right) ^ {\top} \left(v _ {n} - v _ {\mathcal {U}} ^ {\pi}\right) \| + \gamma \| \left(u _ {n} - u _ {\mathcal {U}} ^ {\pi}\right) ^ {\top} v _ {\mathcal {U}} ^ {\pi} \| \\ \leq \gamma^ {n + 1} \| v _ {0} - v _ {\mathcal {U}} ^ {\pi} \| _ {\infty} + \gamma^ {n + 1} S \| v _ {0} - v _ {\mathcal {U}} ^ {\pi} \| _ {\infty} + \gamma \frac {\| \mathcal {R} \| _ {\infty}}{1 - \gamma} \| u _ {n} - u _ {\mathcal {U}} ^ {\pi} \| _ {\infty}. \\ \end{array}
$$
Here,  $u_{n}, u_{\mathcal{U}}^{\pi}$  is the balanced-normalized vector associated with vector  $v_{n}$  and  $v_{\mathcal{U}}^{\pi}$ , respectively. Recall that the  $p$ -balanced normalized vector  $u$  associated with vector  $v$  is given by
$$
u (s) := \frac {\operatorname {s i g n} (v (s) - \omega_ {q} (v) \| v (s) - \omega_ {q} (v) \| ^ {q - 1}}{\kappa_ {q} (v) ^ {q - 1}}, \tag {10}
$$
where  $\kappa_{p}(v) = \min_{w}\| v - w\mathbf{1}\|_{p}$  and  $\omega_p(v) = _w\| v - w\mathbf{1}\| _p$  . It is easy to see that  $\omega_{p},\kappa$  are Lipschitz in  $v$  . Hence,  $\| u_n - u_\mathcal{U}^\pi \|_\infty \leq C\cdot \mathrm{Pol}(\| v_n - v_\mathcal{U}^\pi \|_\infty)\cdot \psi (\kappa (v_\mathcal{U}^\pi),S,A)$  , where Pol is a polynomial and  $\psi$  a real function. This implies that
$$
\left\| Q _ {n + 1} - Q _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty} \leq \gamma^ {n + 1} \psi^ {\prime} \left(\kappa \left(v _ {\mathcal {U}} ^ {\pi}\right), \left\| v _ {0} - v _ {\mathcal {U}} ^ {\pi} \right\| _ {\infty}, S, A\right),
$$
E Robust Policy Gradient
Theorem (RPG). For any rectangular  $\ell_p$ -ball-constrained uncertainty, the robust policy gradient is given by:
$$
\partial_ {\pi} \rho_ {\mathcal {U}} ^ {\pi} = \sum_ {(s, a) \in \mathcal {X}} \left(d _ {P _ {0}, \mu} ^ {\pi} (s) - c ^ {\pi} (s)\right) Q _ {\mathcal {U}} ^ {\pi} (s, a) \nabla \pi_ {s} (a), \tag {11}
$$
Proof. The proof directly follows from plugging the robust occupation measure of Thm. 4.6 and the robust Q-value into standard policy-gradient theorem [26].  $\square$
In this section, we aim to derive the complexity of one RPG iteration for different uncertainty sets. We first focus on non-robust MDPs, to then see how the complexity increases with the uncertainty structure.
Non-robust MDPs. Computing non-robust Q-values and occupation measure takes  $O(S^2 A \log(\epsilon^{-1}))$  each, which represent the most expensive computations in PG. The product of  $d^\pi$ ,  $Q^\pi$  and  $\nabla \pi$  in PG requires  $O(SA)$  operations, which is insignificant. Hence, the total cost of PG corresponds to the computational cost of Q-values. More precisely, approximate the Q-value by  $Q$  and the occupation measure by  $d$  with  $\frac{\epsilon}{SA}$  tolerance, that is,  $\| Q - Q^\pi \|_\infty \leq \frac{\epsilon}{SA}$  and  $\| d - d^\pi \|_\infty \leq \frac{\epsilon}{SA}$ . This involves  $O(S^2 A \log(SA\epsilon^{-1}))$  operations for each. Then, we have
$$
\sum_ {(s, a) \in \mathcal {X}} d ^ {\prime} (s) Q ^ {\prime} (s, a) \nabla \pi_ {s} (a) = \sum_ {(s, a) \in \mathcal {X}} \left(Q ^ {\pi} (s, a) + \epsilon_ {1} (s, a)\right) \left(d ^ {\pi} (s, a) + \epsilon_ {2} (s, a)\right) \nabla \pi_ {s} (a)
$$
where  $\epsilon_1(s,a) \coloneqq Q(s,a) - Q^\pi(s,a)$  and  $\epsilon_2(s,a) \coloneqq d(s,a) - d^\pi(s,a)$ . Let  $B$  be an upper bound of  $\|Q^\pi\|_\infty$  and  $\|d^\pi\|_\infty$ . Then
$$
\begin{array}{l} \sum_ {(s, a) \in \mathcal {X}} d ^ {\prime} (s) Q ^ {\prime} (s, a) \nabla \pi_ {s} (a) = \sum_ {(s, a) \in \mathcal {X}} \left(Q ^ {\pi} (s, a) + \epsilon_ {1} (s, a)\right) \left(d ^ {\pi} (s, a) + \epsilon_ {2} (s, a)\right) \nabla \pi_ {s} (a) \\ = \sum_ {(s, a) \in \mathcal {X}} Q ^ {\pi} (s, a) d ^ {\pi} (s, a) \nabla \pi_ {s} (a) + O (\epsilon), \\ \end{array}
$$
so the exact complexity of policy gradient for non-robust MDP is  $O(S^2 A \log (SA\epsilon^{-1})$ .
Convex non-rectangular uncertainty set. Robust policy improvement is strongly NP-Hard for non-rectangular uncertainty sets, even if convex [33]. The PG method finds global optimal given oracle access to policy gradient in polynomial time [30]. This implies that the computation of RPG must be NP-Hard.
F.1 Helper results for Robust MDPs
Variance and mean functions. Computing  $\kappa_{p}(v)$  (resp.  $\omega_{p}(v)$ ) with  $\epsilon$ -tolerance requires  $O(S\log (S\epsilon^{-1}))$  (resp.  $O(S\log (\epsilon^{-1}))$ ) computations if we use binary search [14].
Occupation measure. Let  $k\in \mathbb{R}^S$  be any vector. By definition,
$$
d _ {P, k} ^ {\pi} = \sum_ {n = 0} ^ {\infty} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n}
$$
$$
\left\| d _ {P, k} ^ {\pi} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} \right\| = \left\| \sum_ {n = N} ^ {\infty} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} \right\| \leq \| k \| \sum_ {n = N} ^ {\infty} \| \gamma P ^ {\pi} \| ^ {n}.
$$
Since  $P^{\pi}$  is a stochastic matrix,  $\| P^{\pi}\| \leq 1$  and
$$
\left\| d _ {P, k} ^ {\pi} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} \right\| \leq \frac {\| k \| \gamma^ {N} \| P ^ {\pi} \| ^ {N}}{1 - \gamma \| P ^ {\pi} \|} \leq \frac {\| k \| \gamma^ {N}}{1 - \gamma}.
$$
This implies that  $\sum_{n=0}^{N-1} \gamma^n k^\top (P^\pi)^n$  is an  $O(\gamma^N)$  approximation of  $d_{P,k}^\pi$ . Now, take  $u_0 = k$  and
$$
u _ {n + 1} := \gamma (u _ {n}) ^ {\top} P,
$$
then  $\sum_{n=0}^{N-1} \gamma^n k^\top (P^\pi)^n = \sum_{n=0}^{N-1} u_n$ . Each iteration takes  $O(S^2)$  computations, leading to total cost  $O(S^2 N)$  for  $N$  iterations. Computing  $P^\pi$  from  $P$  is  $O(S^2 A)$ . We conclude that computing the occupation measure has a complexity of  $O(S^2 A + S^2 \log (\epsilon^{-1}))$ .
Lemma F.1. We can approximate  $d_{P,k}^{\pi}$  by  $\sum_{n=0}^{N-1} \gamma^n (k')^\top (P^\pi)^n$  with complexity  $O(S^2 A + S^2 \log (\epsilon^{-1}))$  and tolerance
$$
\left\| d _ {P, k} ^ {\pi} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\prime \top} (P ^ {\pi}) ^ {n} \right\| \leq O \left(\frac {\| k \| \gamma^ {N} + \| k - k ^ {\prime} \|}{1 - \gamma}\right).
$$
$$
\begin{array}{l} \left\| d _ {P, k} ^ {\pi} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\prime \top} (P ^ {\pi}) ^ {n} \right\| \leq \left\| d _ {P, k} ^ {\pi} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} \right\| + \left\| \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\prime \top} (P ^ {\pi}) ^ {n} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} \right\| \\ \leq O \left(\frac {\| k \| \gamma^ {N}}{1 - \gamma}\right) + \left\| \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\top} (P ^ {\pi}) ^ {n} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} k ^ {\prime \top} (P ^ {\pi}) ^ {n} \right\| \\ \leq O \left(\frac {\| k \| \gamma^ {N}}{1 - \gamma}\right) + \| k - k ^ {\prime} \| \left\| \sum_ {n = 0} ^ {N - 1} \gamma^ {n} (P ^ {\pi}) ^ {n} - \sum_ {n = 0} ^ {N - 1} \gamma^ {n} (P ^ {\pi}) ^ {n} \right\| \\ \leq O \left(\frac {\| k \| \gamma^ {N}}{1 - \gamma}\right) + O \left(\frac {\| k - k ^ {\prime} \|}{1 - \gamma}\right). \\ \end{array}
$$
Computing Q-value given value function. Let  $v$  be an  $\epsilon_{1}$  approximation of robust value function  $v_{\mathcal{U}}^{\pi}$ , that is  $\| v - v_{\mathcal{U}}^{\pi}\|_{\infty} \leq \epsilon_{1}$ . We want to compute the Q-value using the relation:
$$
\begin{array}{l} Q _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {\mathcal {U}} ^ {\pi} (s, a) + \sum_ {s, a} \pi_ {s} (a) P _ {\mathcal {U}} ^ {\pi} \left(s ^ {\prime} \mid s, a\right) v _ {\mathcal {U}} ^ {\pi} \left(s ^ {\prime}\right) \\ = R _ {0} (s, a) + \gamma \sum_ {s ^ {\prime}} P _ {0} (s ^ {\prime} | s, a) v _ {\mathcal {U}} ^ {\pi} (s ^ {\prime}) - \Omega_ {\mathcal {U}} (v _ {\mathcal {U}} ^ {\pi}, \pi). \\ \end{array}
$$
where  $\Omega_{\mathcal{U}_p^{\mathrm{s}}}(v_{\mathcal{U}_p^{\mathrm{s}}},\pi) = \frac{\pi_s(a)^{q - 1}}{\|\pi_s\|_q^{q - 1}}\left(\alpha_s + \gamma \beta_s\kappa_q(v_{\mathcal{U}_p^{\mathrm{s}}})\right)$  and  $\Omega_{\mathcal{U}_p^{\mathrm{sa}}}(\nu_{\mathcal{U}_p^{\mathrm{sa}}},\pi) = \alpha_{sa} + \gamma \beta_{sa}\kappa_q(\nu_{\mathcal{U}_p^{\mathrm{sa}}})$  Let  $Q$  be approximated from  $v$  by
$$
Q (s, a) = R _ {0} (s, a) + \gamma \sum_ {s ^ {\prime}} P _ {0} (s ^ {\prime} | s, a) v (s ^ {\prime}) - \Omega_ {\mathcal {U}} (v, \pi).
$$
$$
\begin{array}{l} \| Q _ {\mathcal {U}} ^ {\pi} (s, a) - Q (s, a) \| _ {\infty} = \gamma \left\| \sum_ {s ^ {\prime} \in \mathcal {S}} P _ {0} \left(s ^ {\prime} \mid s, a\right) \left(v \left(s ^ {\prime}\right) - v _ {\mathcal {U}} ^ {\pi}\right) \right\| + \| \Omega_ {\mathcal {U}} (v, \pi) - \Omega_ {\mathcal {U}} \left(v _ {\mathcal {U}} ^ {\pi}, \pi\right) \| \\ \leq \gamma \epsilon_ {1} + \| \Omega_ {\mathcal {U}} (v, \pi) - \Omega_ {\mathcal {U}} (v _ {\mathcal {U}} ^ {\pi}, \pi) \| \\ \leq \gamma \epsilon_ {1} + \| \beta \| _ {\infty} \| \kappa_ {q} (v) - \kappa_ {q} (v _ {\mathcal {U}} ^ {\pi}) \| \\ \leq \gamma \epsilon_ {1} + \| \beta \| _ {\infty} S ^ {\frac {1}{q}} \epsilon_ {1} \tag {By Lemma F.2} \\ = O \left(S ^ {\frac {1}{q}} \epsilon_ {1}\right), \\ \end{array}
$$
so that  $\| Q - Q_{\mathcal{U}}^{\pi}\|_{\infty}\leq O(S^{\frac{1}{q}}\epsilon_1)$
Lemma F.2. The variance function  $\kappa_{p}$  is Lipschitz. More precisely,
$$
\left\| \kappa_ {p} (v _ {1}) - \kappa_ {p} (v _ {2}) \right\| \leq S ^ {\frac {1}{p}} \| v _ {1} - v _ {2} \| _ {\infty} \leq S ^ {\frac {1}{p}} \| v _ {1} - v _ {2} \| _ {\infty}, \quad \forall v _ {1}, v _ {2} \in \mathbb {R} ^ {\mathcal {S}}.
$$
Proof. Let  $v_{i} \in \mathbb{R}^{S}$  and  $w_{i} \in \arg \min_{w \in \mathbb{R}} \| v_{i} - w\mathbf{1} \|_{p}$  for  $i = 1,2$ . Without loss of generality, further assume that  $\kappa_{p}(v_{1}) \geq \kappa_{p}(v_{2})$ . Then,
$$
\begin{array}{l} \left\| \kappa_ {p} (v _ {1}) - \kappa_ {p} (v _ {2}) \right\| = \kappa_ {p} (v _ {1}) - \kappa_ {p} (v _ {2}) \\ = \min  _ {w \in \mathbb {R}} \| v _ {1} - w \mathbf {1} \| _ {p} - \min  _ {w \in \mathbb {R}} \| v _ {2} - w \mathbf {1} \| _ {p} \quad (\text {B y}) \\ \leq \left\| v _ {1} - w _ {2} \mathbf {1} \right\| _ {p} - \left\| v _ {2} - w _ {2} \mathbf {1} \right\| _ {p} \quad (\text {B y}) \\ \leq \left\| \left(v _ {1} - w _ {2} \mathbf {1}\right) - \left(v _ {2} - w _ {2} \mathbf {1}\right) \right\| _ {p} \quad (\text {R e v e r s e t r i a n g l e i n e q u a l i t y}) \\ = \left\| v _ {1} - v _ {2} \right\| _ {p} \\ = \left(\sum_ {s \in \mathcal {S}} (v _ {1} (s) - v _ {2} (s)) ^ {p}\right) ^ {\frac {1}{p}} \leq S ^ {\frac {1}{p}} \| v _ {1} - v _ {2} \| _ {\infty}. \\ \end{array}
$$
Lemma F.3.  $Q_{\mathcal{U}_p^{sa}}^{\pi}$  can be approximated to  $\epsilon$  tolerance with the same complexity as computing  $v_{\mathcal{U}_p^{sa}}^{\pi}$  to  $S^{-\frac{1}{q}}\epsilon$ .
Proof. We can compute the value function with  $S^{-\frac{1}{q}} \epsilon$  tolerance. The rest of the operations are insignificant. The result follows from above.
F.2 RPG Complexity
Let  $O_{p}^{\mathrm{sa}}(\epsilon)$  be the complexity of computing  $v_{\mathcal{U}_{p}^{\mathrm{sa}}}^{\pi}$  with  $\epsilon$ -tolerance (see [14] for more details). Calculating Q-value up to  $\epsilon_{1}$  requires  $O_{p}^{\mathrm{sa}}(S^{-\frac{1}{q}}\epsilon_{1})$  computations according to Lemma F.3. Letting  $d_{1}$  and  $d_{2}$  be  $\epsilon_{2}$ -approximations of  $d_{P_0,\mu}^{\pi}$  and  $d_{P_0,k}^{\pi}$  respectively, their complexity is insignificant compared to  $O_{p}^{\mathrm{sa}}(S^{-\frac{1}{q}}\epsilon_{2})$ . Now, approximating the gradient using  $d_{1}, d_{2}, Q, \nabla \pi$  as in Thm. 5.1 has a complexity
of  $O(SA)$ . Since the uncertainty set  $\mathcal{U}$  is compact, all quantities are bounded. There are  $O(SA)$  operations in Thm. 5.1, so taking  $\epsilon_1, \epsilon_2 = O\left(\frac{\epsilon}{SA}\right)$ , we get  $O(\epsilon)$  for the gradient. Hence, the total complexity is  $O_p^{\mathrm{sa}}(S^{-\frac{1}{q} - 1}A^{-1}\epsilon)$  which is  $\tilde{O}(S^2 A \log (\epsilon^{-1}))$  by hiding log factors. A similar analysis follows for the  $s$ -rectangular case.
G Generalization to arbitrary norms
In this section, we analyze how to generalize our result to general norms that are not  $\ell_p$ .
G.1  $(s,a)$ -rectangular robust MDPs
Consider an  $(s,a)$ -rectangular uncertainty set  $\mathcal{U} = \mathcal{U}_{\|,||}^{\mathrm{sa}}$  constrained by:
$$
\mathcal {U} _ {\| \cdot \|} ^ {\mathrm {s a}} = (P _ {0} + \mathcal {P}) \times (R _ {0} + \mathcal {R}), \qquad \text {w h e r e} \qquad (\mathcal {P}, \mathcal {R}) = (\times_ {s, a} \mathcal {P} _ {s a}, \times_ {s, a} \mathcal {P} _ {s a}),
$$
$$
\mathcal {R} _ {(s, a)} = \left\{r \in \mathbb {R} \mid \| r \| \leq \alpha_ {s, a} \right\}, \quad \text {a n d} \quad \mathcal {P} _ {(s, a)} = \left\{p \in \mathbb {R} ^ {\mathcal {S}} \mid \langle p, \mathbf {1} \rangle_ {\mathcal {S}} = 0, \| p \| \leq \beta_ {s, a} \right\}.
$$
The robust Bellman operator  $\mathcal{T}_{\mathcal{U}}^{\pi}$  can be evaluated as
$$
(\mathcal {T} _ {\mathcal {U}} ^ {\pi} v) (s) = \sum_ {a} \pi_ {s} (a) \left[ R (s, a) - \gamma \beta_ {s, a} \kappa_ {\| \cdot \|} (v) + \gamma \sum_ {s ^ {\prime}} P (s ^ {\prime} | s, a) v (s ^ {\prime}) \right],
$$
where the variance function is defined as
$$
\kappa_{\| \cdot \|}(v):= \min_{\langle u,\mathbf{1}\rangle_{\mathcal{S}} = 0,\| u\| \leq 1}\langle u,v_{\mathcal{U}}^{\pi}\rangle .
$$
This can be used to compute the robust value function. Then the worst values can found using robust Bellman operator  $\mathcal{T}_{\mathcal{U}}^{\pi}$  and robust value function  $v_{\mathcal{U}}^{\pi}$  as
$$
\left(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi}\right) \in \underset {(P, R) \in \mathcal {U}} {\arg \min } \mathcal {T} _ {(P, R)} ^ {\pi} v _ {\mathcal {U}} ^ {\pi}, \quad [ 3 3 ].
$$
It is easy to see that the worst values are given as
$$
R _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {0} (s, a) - \alpha_ {s, a} \quad \mathrm {a n d} \quad P _ {\mathcal {U}} ^ {\pi} (\cdot | s, a) = P _ {0} ^ {\pi} (\cdot | s, a) - \beta_ {s, a} u _ {\mathcal {U}} ^ {\pi},
$$
where normalized-balanced value function  $u_{\mathcal{U}}^{\pi}$  is a solution to
$$
\min_{\substack{\langle u,1\rangle_{S} = 0,\| u\| \leq 1}}\langle u,v_{\mathcal{U}}^{\pi}\rangle .
$$
Observe that the worst kernel is still a rank-one perturbation of the nominal kernel. Hence, the robust occupation measure can be obtained using Lemma 4.4 as
$$
d _ {\mathcal {U}, \mu} ^ {\pi} = d _ {P _ {0}, \mu} ^ {\pi} - \gamma \frac {\langle d _ {P _ {0} , \mu} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}}{1 + \gamma \langle d _ {P _ {0} , u _ {\mathcal {U}} ^ {\pi}} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}} d _ {P _ {0}, u _ {\mathcal {U}} ^ {\pi}} ^ {\pi}, \tag {12}
$$
where  $\beta^{\pi}(s) = \sum_{a}\pi_{s}(a)\beta_{s,a}$ . The last ingredient to compute RPG is the robust Q-value which can be computed using robust value function and worst values. However, it can be computed directly using the following iterates:
$$
\begin{array}{l} Q _ {n + 1} (s) = \min  _ {(P, R) \in \mathcal {U}} \left[ R (s, a) + \gamma \sum_ {s ^ {\prime}} P \left(s ^ {\prime} \mid s, a\right) v \left(s ^ {\prime}\right) \right] \\ = R (s, a) - \alpha_ {s, a} - \gamma \beta_ {s, a} \kappa_ {\| \cdot \|} (v) + \gamma \sum_ {s ^ {\prime}} P (s ^ {\prime} | s, a) v (s ^ {\prime}), \\ \end{array}
$$
as  $Q_{n}$  converges to robust Q-value  $Q_{\mathcal{U}}^{\pi}$  linearly.
The proofs of the above claims are very similar to the  $\ell_p$  case so they are omitted. Finally, computing the variance function  $\kappa_{\| \cdot \|}$  and the normalized-value function  $u_{\mathcal{U}}^{\pi}$  can be done using numerical convex optimization methods for general norms. For the  $\ell_p$  case, these can be obtained in concrete form, so we choose to focus on  $\ell_p$  in our main study.
G.2  $s$ -rectangular case
Generalization to  $s$ -rectangular balls of a general norm is not straightforward and may not be possible for all types of norms. The crucial property of the  $\ell_p$ -norm exploited in our rank-one perturbation proof is 'decoupling', that is, for all  $x \in \mathbf{R}^{A \times S}$ , there must exist  $k, l, m \in \mathbb{R}_+$  such that
$$
\left\| x \right\| _ {p} ^ {k} = \sum_ {a \in \mathcal {A}} \left\| x _ {a} \right\| _ {m} ^ {l}.
$$
This holds in the  $\ell_p$  case with  $k = l = m = p$ . Further analysis of this setting is left for future work.
G.3 Generalization to metrics and divergences
Generalizing RPG to distance or divergence-based uncertainty sets is also not obvious. Our RPG method, in particular the robust occupation measure, crucially relies on the rank-one perturbation characterization of the worst kernel, which might not apply for example with KL-ball constraints. We leave this analysis for future work.
Parameters. All the nominal transition kernels and reward functions are generated randomly. The number of states and the number of actions are varied. Discount factor  $\gamma = 0.9$ , reward noise radius  $\alpha_{s,a}$ ,  $\alpha_s = 0.1$ , transition noise kernel  $\beta_{s,a}, \beta_s = \frac{0.01}{SA}$ .
Hardware Experiments are done on the machine with the following configuration: Intel(R) Core(TM) i7-6700 CPU @3.40GHz, size:3598MHz, capacity 4GHz, width 64 bits, memory size 64 GiB.
Software and codes All the experiments were done in Python using numpy, matplotlib. All codes and results are available at https://github.com/navtech/rpg.
Procedure and Results. All the experiments were repeated 100 times, except for Linear Programming (LP) cases as LP methods were very time-consuming. In LP methods, experiments were repeated 5 times except for the case  $(S = 500, A = 100)$  which was done only once. As this case was prohibitively expensive. Standard deviation in all cases was less than  $10\%$ , and typically  $1 - 2\%$ . This conveniently illustrates the superiority of our methods over LP methods.
- Scalability of our methods. Note that our methods scale very well with large state-action space. It takes a (small) constant times the time required by non-robust MDPs. On the other hand, LP methods explode. Both observations confirm the theoretical time complexity.
- sa-case vs s case in LP methods. We see s-case outperforms sa-case for small state-action spaces via LP methods. This is opposite to the theoretical time complexity of s-case which expensive than sa-case. We believe this is due to the internal implementation issues. Note that computing the robust value function is the most expensive step which requires evaluation of the robust Bellman operator. In sa case, one evaluation requires solving SA LP programs with  $S$  variables each, while for s-case, it is  $S$  LP programs with  $SA$  variables each. To solve LP, scipy.linprog is used, we believe it does some parallelization for large LPs. Hence, we observe less cost for sa-case. However, we observe that the cost of s-case increases much faster than s-case, and eventually under-performing than sa-case.
H.1 RPG by LP
We compute RPG using LP in the following steps:
1. (Robust Value Iteration) Approximately compute the robust value function  $v_{\mathcal{U}}^{\pi}$  using the iterates  $v_{n+1} := \mathcal{T}_{\mathcal{U}}^{\pi}v_{n}$ . Evaluation of robust Bellman operator  $\mathcal{T}_{\mathcal{U}}^{\pi}$  is done via LPs as described below. This is the most expensive step as it requires evaluating robust Bellman operators  $O(\log(\epsilon^{-1}))$  times, and each evaluation requires many LPs.
2. (Adversarial Values) Compute the worst values  $(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})$  using the robust value function from the following relation:
$$
(P_{\mathcal{U}}^{\pi},R_{\mathcal{U}}^{\pi})\in \operatorname *{arg  min}_{(P,R)\in \mathcal{U}}\mathcal{T}_{\mathcal{U}}^{\pi}v_{\mathcal{U}}^{\pi}.
$$
This is also solved by LP.
3. (Policy Gradient Theorem) We now compute the RPG using policy gradient Theorem [26] w.r.t. the adversarial values computed above, as
$$
\partial \rho_ {\mathcal {U}} ^ {\pi} = \sum_ {s, a} d _ {P _ {\mathcal {U}} ^ {\pi}} ^ {\pi} (s) Q _ {P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi}} ^ {\pi} (s, a) \nabla \pi_ {s} (a).
$$