SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| c ^ {*} \right\| _ {p} = \left(\sum_ {s \in \mathcal {S}} \left| c ^ {*} (s) \right| ^ {p}\right) ^ {\frac {1}{p}} \\ = \left( \right.\sum_ {s \in \mathcal {S}} \left. \right| - \epsilon \operatorname {S I G N} (v (s) - \omega_ {q} (v)) \left(\frac {| v (s) - \omega_ {q} (v) |}{\kappa_ {q} (v)}\right) ^ {q - 1} \Bigg | ^ {p}\left. \right) ^ {\frac {1}{p}} \\ = \left(\left(\frac {\epsilon}{\kappa_ {q} (v) ^ {q - 1}}\right) \sum_ {s \in \mathcal {S}} \left| \left(\frac {| v (s) - \omega_ {q} (v) |}{\kappa_ {q} (v)}\right) ^ {q - 1} \right| ^ {p}\right) ^ {\frac {1}{p}} \\ = \frac {\epsilon}{\kappa_ {q} (v) ^ {q - 1}} \left(\sum_ {s \in \mathcal {S}} | v (s) - \omega_ {q} (v) | ^ {(q - 1) p}\right) ^ {\frac {1}{p}} \\ = \frac {\epsilon}{\kappa_ {q} (v) ^ {q - 1}} \left(\sum_ {s \in \mathcal {S}} | v (s) - \omega_ {q} (v) | ^ {q}\right) ^ {\frac {1}{p}} \\ = \frac {\epsilon}{\kappa_ {q} (v) ^ {q - 1}} \kappa_ {q} (v) ^ {\frac {q}{p}} \\ = \epsilon , \quad \left(\frac {q}{p} - (q - 1) = \frac {q - p q + p}{p} = 0\right) \\ \end{array}
$$
so the norm constraint is satisfied. We check the noise constraint by computing:
$$
\begin{array}{l} \sum_ {s \in \mathcal {S}} c ^ {*} (s) = \sum_ {s \in \mathcal {S}} - \epsilon \mathrm {S I G N} \big (v (s) - \omega_ {q} (v) \big) \left(\frac {| v (s) - \omega_ {q} (v) |}{\kappa_ {q} (v)}\right) ^ {q - 1} \\ = \frac {- \epsilon}{\kappa_ {q} (v) ^ {q - 1}} \sum_ {s \in \mathcal {S}} \operatorname {S I G N} (v (s) - \omega_ {q} (v)) | v (s) - \omega_ {q} (v) | ^ {q - 1}. \\ \end{array}
$$
Now, considering the real function  $\varphi : w \to \|v - w\mathbf{1}\|_q$  and taking its derivative, we remark the proportional relation:
$$
\sum_ {s \in \mathcal {S}} c ^ {*} (s) = C \cdot \varphi^ {\prime} (\omega_ {q} (v)),
$$
where  $C \in \mathbb{R}$  is the proportionality coefficient. By construction,  $\omega_q(v)$  is a minimizer of  $\varphi$ , so we must have  $\varphi'(\omega_q(v)) = 0$  and  $c^*$  satisfies the noise constraint.
We finally show that  $c^*$  reaches the optimal value:
$$
\begin{array}{l} - \frac {1}{\epsilon} \langle c ^ {*}, v \rangle = - \frac {1}{\epsilon} \langle c ^ {*}, v - \omega_ {q} (v) \mathbf {1} \rangle \quad (\langle c ^ {*}, \mathbf {1} \rangle_ {\mathcal {S}} = 0) \\ = \sum_ {s \in \mathcal {S}} \frac {\left| v (s) - \omega_ {q} (v) \right| ^ {q}}{\kappa_ {q} (v) ^ {q - 1}} \quad \text {(P u t t i n g t h e v a l u e o f} c ^ {*} \text {)} \\ = \frac {\kappa_ {q} (v) ^ {q}}{\kappa_ {q} (v) ^ {q - 1}} \quad (\kappa_ {q} (v) = \| v - \omega_ {q} \mathbf {1} \| _ {q}) \\ = \kappa_ {q} (v). \\ \end{array}
$$
Proposition. For any policy  $\pi \in \Pi$  and  $\ell_p$ -ball rectangular uncertainty set, the following holds:
$$
u _ {\mathcal {U}} ^ {\pi} = \nabla_ {v} \kappa_ {q} (v) \big | _ {v = v _ {\mathcal {U}} ^ {\pi}},
$$
$$
\left\langle u _ {\mathcal {U}} ^ {\pi}, v _ {\mathcal {U}} ^ {\pi} \right\rangle = \kappa_ {q} (v _ {\mathcal {U}} ^ {\pi}).
$$
Proof. The second claim directly follows from Lemma A.1 applied to  $v \coloneqq v_{\mathcal{U}}^{\pi}$ , so that by optimality,  $\kappa_q(v_{\mathcal{U}}^\pi) = \langle u_{\mathcal{U}}^\pi, v_{\mathcal{U}}^\pi \rangle$ . For the first claim, we take the gradient of  $\kappa_p(v) \coloneqq \min_{w \in \mathbb{R}} \| v - w\mathbf{1} \|_p$  w.r.t.  $v$  using the envelope theorem [19]. Then, the  $p$ -balanced-normalized vector  $u_p(v)$  is a sub-gradient of  $\kappa_p(v)$ , that is,
$$
u _ {p} (v) = \nabla \kappa_ {q} (v),
$$
which we apply to  $v\coloneqq v_{\mathcal{U}}^{\pi}$
We have the additional properties below:
- The variance function  $\kappa_{q}$  is translation-invariant in all-ones directions, i.e., for all  $\omega \in \mathbb{R}, \kappa_{q}(v) = \kappa_{q}(v + \omega \mathbf{1})$ . As a result,  $\langle \nabla \kappa_{q}(v), \mathbf{1} \rangle_{S} = 0$ .
- The balanced-normalized vector  $u_{p}(v)$  has unit norm, i.e.,  $\| u_{p}(v)\|_{p} = 1$  by Lemma A.1.
B Worst Kernel and Reward
Here we present the proofs for the worst/adversarial kernel and reward function characterization.
B.1 Proof of Theorem 4.2
Theorem  $((s,a)$ -rectangular case). Given uncertainty set  $\mathcal{U} = \mathcal{U}_p^{sa}$  and any policy  $\pi \in \Pi$ , the worst model is related to the nominal one through:
$$
(P_{\mathcal{U}_{p}^{\mathrm{sa}}}^{\pi},R_{\mathcal{U}_{p}^{\mathrm{sa}}}^{\pi})\in \operatorname *{arg  min}_{(P,R)\in \mathcal{U}_{p}^{\mathrm{sa}}}T_{(P,R)}^{\pi}v_{\mathcal{U}_{p}^{\mathrm{sa}}}^{\pi}.
$$
Additionally, since  $\mathcal{U}_p^{\mathrm{sa}} = (R_0 + \mathcal{R})\times (P_0 + \mathcal{P})$  , it results that:
$$
\left(R _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}}, P _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}\right) = \left(P _ {0} + P ^ {*}, R _ {0} + R ^ {*}\right)
$$
$$
(P ^ {*}, R ^ {*}) \in \underset {(P, R) \in \mathcal {P} \times \mathcal {R}} {\arg \min } T _ {(P, R)} ^ {\pi} v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}.
$$
By the  $(s,a)$ -rectangularity assumption, we get that for all  $(s,a) \in \mathcal{X}$ ,
$$
\left(P ^ {*} (\cdot | s, a), R ^ {*} (s, a)\right) \in \operatorname * {a r g m i n} _ {\left(p _ {s, a}, r _ {s, a}\right) \in \mathcal {P} _ {s, a} \times \mathcal {R} _ {s, a}} \left\{r _ {s, a} + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {s, a} \left(s ^ {\prime}\right) v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right) \right\}
$$
It is clear from the above that the worst reward is independent of policy  $\pi$ . Thus, by the ball constraint, it is given by
$$
R ^ {*} (s, a) = - \alpha_ {s, a}, \quad \forall (s, a) \in \mathcal {X}.
$$
Differently, the worst kernel depends on the value function which itself depends on the policy. It is given by
$$
P ^ {*} (\cdot | s, a) = \operatorname * {a r g   m i n} _ {p _ {s, a} \in \mathcal {P} _ {s a}} \left\{\sum_ {s ^ {\prime} \in \mathcal {S}} p _ {s, a} (s ^ {\prime}) v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}) \right\}, \quad \forall (s, a) \in \mathcal {X}.
$$
The optimization is of the form
$$
\operatorname *{arg  min}_{\| c\|_{p}\leq \beta ,\langle c,\mathbf{1}\rangle = 0}\langle c,v\rangle ,
$$
so by Lemma A.1,
$$
P ^ {*} (s ^ {\prime} | s, a) = - \beta_ {s, a} \mathrm {S I G N} \left(v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}})\right) \frac {\left| v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}) \right| ^ {q - 1}}{\kappa_ {q} (v) ^ {q - 1}}.
$$
As a result, we proved that for all  $(s,a)\in \mathcal{X}$ $R_{\mathcal{U}_p^{\mathrm{sa}}}^\pi (s,a) = R_0(s,a) - \alpha_{s,a}$  and
$$
P _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime} | s, a) = P _ {0} (s ^ {\prime} | s, a) - \beta_ {s, a} \mathrm {S I G N} \left(v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi})\right) \frac {\left| v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}) \right| ^ {q - 1}}{\kappa_ {q} (v) ^ {q - 1}}.
$$
Theorem (s-rectangular case). Given uncertainty set  $\mathcal{U} = \mathcal{U}_p^s$  and any policy  $\pi \in \Pi$ , the worst model is related to the nominal one through:
$$
(P _ {\mathcal {U} _ {p} ^ {\mathtt {s}}} ^ {\pi}, R _ {\mathcal {U} _ {p} ^ {\mathtt {s}}} ^ {\pi}) \in \operatorname * {a r g   m i n} _ {(P, R) \in \mathcal {U} _ {p} ^ {\mathtt {s}}} T _ {(P, R)} ^ {\pi} v _ {\mathcal {U} _ {p} ^ {\mathtt {s}}} ^ {\pi},
$$
and since  $\mathcal{U}_p^{\mathsf{s}} = (R_0 + \mathcal{R})\times (P_0 + \mathcal{P})$  , we have
$$
\left(P _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi}, R _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi}\right) = \left(P _ {0} + P ^ {*}, R _ {0} + R ^ {*}\right)
$$
By the  $s$ -rectangularity assumption, we get that for all  $s \in S$
$$
\left(P ^ {*} (\cdot | s, \cdot), R ^ {*} (s, \cdot)\right) = \underset {\left(p _ {s}, r _ {s}\right) \in \mathcal {P} _ {s} \times \mathcal {R} _ {s}} {\arg \min } \sum_ {a \in \mathcal {A}} \pi_ {s} (a) \left\{r _ {s, a} + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {s, a} \left(s ^ {\prime}\right) v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right) \right\}.
$$
Here, the worst reward does depend on policy  $\pi$  and is given by
$$
R ^ {*} (s, a) = - \alpha_ {s} \frac {\pi_ {s} (a) ^ {q - 1}}{\sum_ {a} \pi_ {s} (a) ^ {q - 1}}, \quad \forall (s, a) \in \mathcal {X}.
$$
As for the worst kernel, it depends both on the value function and the policy. It is given by
$$
P ^ {*} (\cdot | s, \cdot) = \underset {p _ {s} \in \mathcal {P} _ {s}} {\arg \min } \left\{\sum_ {a \in \mathcal {A}} \pi_ {s} (a) \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {s, a} (s ^ {\prime}) v _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi} (s ^ {\prime}) \right\}.
$$
The optimization of interest is of the form
$$
\min  _ {\| c _ {a} \| _ {p} \leq \beta_ {s}, \langle c _ {a}, \mathbf {1} \rangle = 0, a \in \mathcal {A}} \left\{\sum_ {a ^ {\prime} \in \mathcal {A}} \pi_ {s} (a ^ {\prime}) \langle c _ {a ^ {\prime}}, v \rangle \right\},
$$
which is equivalent to the following two-fold minimization:
$$
\min  _ {\sum_ {a \in \mathcal {A}} (\beta_ {s, a}) ^ {p} \leq (\beta_ {s}) ^ {p}} \quad \min  _ {\| c _ {a} \| _ {p} \leq \beta_ {s}, \langle c _ {a}, \mathbf {1} \rangle = 0, a \in \mathcal {A}} \left\{\sum_ {a ^ {\prime} \in \mathcal {A}} \pi_ {s} (a ^ {\prime}) \langle c _ {a ^ {\prime}}, v \rangle \right\}.
$$
Thus, rewriting the problem in our context,
$$
\begin{array}{l} \min  _ {\sum_ {a} (\beta_ {s, a}) ^ {p} \leq (\beta_ {s}) ^ {p}} \quad \min  _ {\| p _ {s a} \| _ {p} \leq \beta_ {s, a}, \sum_ {s ^ {\prime}} p _ {s a} (s ^ {\prime}) = 0} \quad \sum_ {a} \pi_ {s} (a) \langle p _ {s, a}, v \rangle \\ = \min  _ {\sum_ {a} (\beta_ {s, a}) ^ {p} \leq (\beta_ {s}) ^ {p}} \sum_ {a} \pi_ {s} (a) \quad \min  _ {\| p _ {s a} \| _ {p} \leq \beta_ {s, a}, \sum_ {s ^ {\prime}} p _ {s a} (s ^ {\prime}) = 0} \quad \langle p _ {s, a}, v \rangle \\ = \min  _ {\sum_ {a} \left(\beta_ {s a}\right) ^ {p} \leq \left(\beta_ {s}\right) ^ {p}} \sum_ {a} \pi_ {s} (a) (- \beta_ {s a} \kappa_ {q} (v)) \tag {By Lemma A.1} \\ = - \kappa_ {q} (v) \max _ {\sum_ {a} (\beta_ {s a}) ^ {p} \leq (\beta_ {s}) ^ {p}} \sum_ {a} \pi_ {s} (a) \beta_ {s a}. \\ \end{array}
$$
Computing the optimal  $\beta$  above, the optimization is now the same as in the  $(s,a)$ -rectangular case. Hence, we have
$$
P ^ {*} (s ^ {\prime} | s, a) = - \beta_ {s} \frac {\pi_ {s} (a) ^ {q - 1}}{\| \pi_ {s} \| _ {q} ^ {q - 1}} \mathrm {S I G N} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s}}})) \frac {\left| v _ {\mathcal {U} _ {p} ^ {\mathrm {s}}} ^ {\pi} (s ^ {\prime}) - \omega_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s}}} ^ {\pi}) \right| ^ {q - 1}}{\kappa_ {q} (v) ^ {q - 1}},
$$
which ends the proof by definition of the balanced value function  $u_{\mathcal{U}}^{\pi}$ .
C Occupation Matrix
Lemma. Let  $b, k \in \mathbb{R}^S$  and  $P_0, P_1 \in (\Delta_S)^S$  two transition matrices. If  $P_1 = P_0 - bk^\top$ , i.e.,  $P_1$  is a rank-one perturbation of  $P_0$ , then their occupation matrices  $D_i := (I - \gamma P_i)^{-1}, i = 0, 1$  are related through:
Proof. By definition,  $D_{1} = (I_{S} - \gamma P_{1})^{-1}$  so it follows that:
$$
\begin{array}{l} (I _ {S} - \gamma P _ {1}) D _ {1} = I _ {S} \\ \Longleftrightarrow I _ {S} + \gamma P _ {1} D _ {1} = D _ {1} \\ \Longleftrightarrow I _ {S} + \gamma \left(P _ {0} - b k ^ {\top}\right) D _ {1} = D _ {1} \quad (\text {B y a s s u m p t i o n}, P _ {1} = P _ {0} - b k ^ {\top}) \\ \Longleftrightarrow I _ {S} - \gamma b k ^ {\top} D _ {1} = \left(I _ {S} - \gamma P _ {0}\right) D _ {1} \\ \Longleftrightarrow \left(I _ {\mathcal {S}} - \gamma P _ {0}\right) ^ {- 1} \left(I _ {\mathcal {S}} - \gamma b k ^ {\top} D _ {1}\right) = D _ {1} \quad (\text {M u l t i p l y i n g b o t h s i d e s b y} \left(I _ {\mathcal {S}} - \gamma P _ {0}\right) ^ {- 1}) \\ \Longleftrightarrow D _ {0} \left(I _ {S} - \gamma b k ^ {\top} D _ {1}\right) = D _ {1} \quad (\text {B y d e f i n i t i o n}, D _ {0} = \left(I _ {S} - \gamma P _ {0}\right) ^ {- 1}) \\ \Longleftrightarrow D _ {0} - \gamma D _ {0} b k ^ {\top} D _ {1} = D _ {1}. \tag {7} \\ \end{array}
$$
Now, multiplying both sides by  $k$  and noticing that  $k^{\top}D_{0}b$  is a scalar we get
$$
\begin{array}{l} k ^ {\top} D _ {0} - \gamma k ^ {\top} D _ {0} b k ^ {\top} D _ {1} = k ^ {\top} D _ {1} \\ \Longleftrightarrow k ^ {\top} D _ {0} = (1 + \gamma k ^ {\top} D _ {0} b) k ^ {\top} D _ {1} \\ \Longleftrightarrow k ^ {\top} D _ {1} = \frac {k ^ {\top} D _ {0}}{\left(1 + \gamma k ^ {\top} D _ {0} b\right)}. \tag {8} \\ \end{array}
$$
Combining Eqs. (7) and (8) thus yields:
$$
D _ {1} = D _ {0} - \gamma \frac {D _ {0} b k ^ {\top} D _ {0}}{(1 + \gamma k ^ {\top} D _ {0} b)},
$$
Theorem. For any rectangular  $\ell_p$ -ball-constrained uncertainty and  $\pi \in \Pi$ , it holds that:
$$
d _ {\mathcal {U}, \mu} ^ {\pi} = d _ {P _ {0}, \mu} ^ {\pi} - \gamma \frac {\langle d _ {P _ {0} , \mu} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}}{1 + \gamma \langle d _ {P _ {0} , u _ {\mathcal {U}} ^ {\pi}} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}} d _ {P _ {0}, u _ {\mathcal {U}} ^ {\pi}} ^ {\pi}.
$$
Proof. From Thms. 4.2 and 4.3, it holds that:
$$
P _ {\mathcal {U}} ^ {\pi} \left(s ^ {\prime} \mid s\right) = P _ {0} ^ {\pi} \left(s ^ {\prime} \mid s\right) - \beta_ {s} ^ {\pi} u _ {\mathcal {U}} ^ {\pi} \left(s ^ {\prime}\right), \quad \forall s, s ^ {\prime} \in \mathcal {S}.
$$
Therefore, setting  $P_{1} \coloneqq P_{\mathcal{U}}^{\pi}$ ,  $P_{0} \coloneqq P_{0}^{\pi}$ ,  $b \coloneqq \beta^{\pi}$  and  $k \coloneqq u_{\mathcal{U}}^{\pi}$ , we can apply Lemma 4.4 and relate the corresponding occupation matrices. Additionally multiplying both sides of the relation by  $\mu^{\top} \in \mathbb{R}^{1 \times S}$  yields the desired result.
D Robust Q-value
D.1 Basic Properties
In the literature, robust Q-values are defined in various ways that turn out to be conflicting for  $s$  but non- $(s, a)$  rectangular uncertainty sets. In this section, we propose to define the robust Q-value solely based on the worst model. Define the robust Q-value, the robust value function, and the robust occupation respectively as:
$$
Q _ {\mathcal {U}} ^ {\pi} := Q _ {(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi})} ^ {\pi}, \quad d _ {\mathcal {U}} ^ {\pi} := d _ {(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi})} ^ {\pi}, \quad v _ {\mathcal {U}} ^ {\pi} := v _ {(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi})} ^ {\pi}.
$$
For  $s$ -rectangular uncertainty sets (in particular, for  $(s,a)$ -rectangular), the above definition of robust value function coincides with the common one, i.e.,  $v_{(P_{\mathcal{U}}^{\pi},R_{\mathcal{U}}^{\pi})}^{\pi} = \min_{(P,R)\in \mathcal{U}}v_{(P,R)}^{\pi}$  [33]. If the uncertainty set is additionally  $(s,a)$ -rectangular (as in [31] or [3, 14]), the above definition of robust Q-value also coincides with the common one because then,
$$
Q _ {\mathcal {U} ^ {\mathrm {s a}}} ^ {\pi} (s, a) = \min  _ {(P, R) \in \mathcal {U} ^ {\mathrm {s a}}} \left(R (s, a) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} P \left(s ^ {\prime} | s, a\right) v _ {\mathcal {U} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right)\right), \quad \forall (s, a) \in \mathcal {X}.
$$
Getting back to our own definition, robust Q-value and value functions are related through:
$$
v _ {\mathcal {U}} ^ {\pi} (s) = \left\langle \pi_ {s}, Q _ {\mathcal {U}} ^ {\pi} (s, \cdot) \right\rangle_ {\mathcal {A}}
$$
$$
Q _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {\mathcal {U}} ^ {\pi} (s, a) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} \pi_ {s} (a) P _ {\mathcal {U}} ^ {\pi} (s ^ {\prime} | s, a) v _ {\mathcal {U}} ^ {\pi} (s ^ {\prime}),
$$
as both quantities are defined based on worst kernel and reward, i.e.,  $Q_{\mathcal{U}}^{\pi} \coloneqq Q_{(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})}^{\pi}$  and  $v_{\mathcal{U}}^{\pi} \coloneqq v_{(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})}^{\pi}$ .
Given an optimal robust policy  $\pi_{\mathcal{U}}^{*}$ , we further use  $P_{\mathcal{U}}^{*}, R_{\mathcal{U}}^{*}, v_{\mathcal{U}}^{*}, Q_{\mathcal{U}}^{*}, d_{\mathcal{U}}^{*}$  as a shorthand for  $P_{\mathcal{U}}^{\pi_{\mathcal{U}}^{*}}, R_{\mathcal{U}}^{\pi_{\mathcal{U}}^{*}}, v_{\mathcal{U}}^{\pi_{\mathcal{U}}^{*}}, Q_{\mathcal{U}}^{\pi_{\mathcal{U}}^{*}}, d_{\mathcal{U}}^{\pi_{\mathcal{U}}^{*}}$  respectively. For  $(s, a)$ -rectangular uncertainty set  $\mathcal{U}^{\mathrm{sa}}$ , the optimal value function is the best optimal Q-value, that is
$$
v _ {\mathcal {U} ^ {\mathrm {s a}}} ^ {*} (s) = \max  _ {a \in \mathcal {A}} Q _ {\mathcal {U} ^ {\mathrm {s a}}} ^ {*} (s, a), \quad \forall s \in \mathcal {S}.
$$
because an optimal policy deterministically takes the action with the highest Q-value [20, 12]. This does no longer hold for  $s$ -rectangular or coupled uncertainty sets, as there, an optimal policy may be stochastic [33]. Still, based on Thms. 4.2 and 4.3, we get the Bellman recursion below.
Proposition D.1. Let an  $\ell_p$ -ball constrained uncertainty set. Then, for all  $(s,a)\in \mathcal{X}$  and  $\pi \in \Pi$ , the robust  $Q$ -value satisfies the following recursion in the  $(s,a)$  and  $s$ -rectangular case respectively:
$$
Q _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} (s, a) = T _ {(P _ {0}, R _ {0})} ^ {\pi} Q _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} (s, a) - \alpha_ {s a} - \gamma \beta_ {s a} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi}),
$$
$$
Q _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi} (s, a) = T _ {(P _ {0}, R _ {0})} ^ {\pi} Q _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi} (s, a) - \left(\frac {\pi_ {s} (a)}{\| \pi_ {s} \| _ {q}}\right) ^ {q - 1} \left(\alpha_ {s} + \gamma \beta_ {s} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi})\right).
$$
Proof. We give proof for the  $(s, a)$ -rectangular case only. The  $s$ -rectangular case follows the exact same lines except that it uses Thm. 4.3 instead of Thm. 4.2. We have:
$$
\begin{array}{l} Q _ {\mathcal {U}} ^ {\pi} (s, a) = Q _ {(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi})} ^ {\pi} (s, a) \quad (\text {B y d e f i n i t i o n}) \\ = R _ {\mathcal {U}} ^ {\pi} (s, a) + \sum_ {s ^ {\prime} \in \mathcal {S}} P _ {\mathcal {U}} ^ {\pi} \left(s ^ {\prime} | s, a\right) v _ {\mathcal {U} _ {p} ^ {\mathrm {a s}}} ^ {\pi} \left(s ^ {\prime}\right) \\ = R _ {0} (s, a) - \alpha_ {s a} + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} \left(P _ {0} \left(s ^ {\prime} \mid s, a\right) - \beta_ {s a} u _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right)\right) v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right) (By Thm. 4.2) \\ = R _ {0} (s, a) - \alpha_ {s a} + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} P _ {0} \left(s ^ {\prime} \mid s, a\right) v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} \left(s ^ {\prime}\right) - \gamma \beta_ {s a} \kappa_ {q} \left(v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}\right) (2d statement of Prop. 4.1) \\ = R _ {0} (s, a) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P _ {0} (s ^ {\prime} | s, a) \pi_ {s ^ {\prime}} (a ^ {\prime}) Q _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) - \alpha_ {s a} - \gamma \beta_ {s a} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}) \\ = T _ {(P _ {0}, R _ {0})} ^ {\pi} Q _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} (s, a) - \alpha_ {s a} - \gamma \beta_ {s a} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi}). \\ \end{array}
$$
The above recursion applies the standard Bellman operator on robust Q-values. We can similarly apply it on the robust value function (itself can be computed efficiently based on [3, 14]).
Corollary D.2. Let an  $\ell_p$ -ball constrained uncertainty set. Then, for all  $(s,a)\in \mathcal{X}$  and  $\pi \in \Pi$ , the robust  $Q$ -value satisfies the following recursion in the  $(s,a)$  and  $s$ -rectangular case respectively:
$$
Q _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} (s, a) = R _ {0} (s, a) + \gamma \sum_ {s ^ {\prime}} P _ {0} (s ^ {\prime} | s, a) v _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} (s ^ {\prime}) - \alpha_ {s a} - \gamma \beta_ {s a} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi}),
$$
$$
Q _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi} (s, a) = R _ {0} (s, a) + \gamma \sum_ {s ^ {\prime}} P _ {0} (s ^ {\prime} | s, a) v _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} (s ^ {\prime}) - \left(\frac {\pi_ {s} (a)}{\| \pi_ {s} \| _ {q}}\right) ^ {q - 1} \Big (\alpha_ {s} + \gamma \beta_ {s} \kappa_ {q} (v _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi}) \Big).
$$
Based on the Bellman recursion above, we now derive robust Q-learning equations to learn a robust Q-value. Precisely, we investigate if the linear operator below is contracting and can be evaluated efficiently:
$$
\left(\mathcal {L} _ {\mathcal {U}} ^ {\pi} Q\right) (s, a) := R _ {\mathcal {U}, v} ^ {\pi} (s, a) + \gamma \sum_ {\left(s ^ {\prime}, a ^ {\prime}\right) \in \mathcal {X}} P _ {\mathcal {U}, v} ^ {\pi} \left(s ^ {\prime} \mid s, a\right) \pi_ {s ^ {\prime}} \left(a ^ {\prime}\right) Q \left(s ^ {\prime}, a ^ {\prime}\right), \quad \forall Q \in \mathbb {R} ^ {\mathcal {X}}, \tag {9}
$$
where  $(P_{\mathcal{U},v}^{\pi},R_{\mathcal{U},v}^{\pi})\in \arg \min_{(P,R)\in \mathcal{U}}T_{(P,R)}^{\pi}v$  and  $v(s) = \langle \pi_s,Q(s,\cdot)\rangle_{\mathcal{A}},\quad \forall s\in S.$
Proposition D.3. Consider an  $\ell_p$ -ball constrained uncertainty set. Then, for all  $Q \in \mathbb{R}^{\mathcal{X}}$  and  $\pi \in \Pi$ , the operator  $\mathcal{L}^\pi$  can be evaluated as:
$$
\left(\mathcal {L} _ {\mathcal {U} _ {p} ^ {s a}} ^ {\pi} Q\right) (s, a) = T _ {(P _ {0}, R _ {0})} ^ {\pi} Q (s, a) - \alpha_ {s a} - \gamma \beta_ {s a} \kappa_ {q} (v),
$$
$$
(\mathcal {L} _ {\mathcal {U} _ {p} ^ {s}} ^ {\pi} Q) (s, a) = T _ {(P _ {0}, R _ {0})} ^ {\pi} Q (s, a) - \left(\frac {\pi_ {s} (a)}{\| \pi_ {s} \| _ {q}}\right) ^ {q - 1} (\alpha_ {s} + \gamma \beta_ {s} \kappa_ {q} (v)),
$$
where for all  $Q \in \mathbb{R}^{\mathcal{X}}$ , its corresponding value is  $v(s) \coloneqq \langle \pi_s, Q(s, \cdot) \rangle$ ,  $\forall s \in S$ .
Proof. We give proof for the  $(s, a)$ -rectangular case only. The  $s$ -rectangular case follows the exact same lines except that we take the worst model for  $s$ -rectangular balls. By definition,
$$
\begin{array}{l} \left(\mathcal {L} _ {\mathcal {U} _ {p} ^ {\mathrm {s a}}} ^ {\pi} Q\right) (s, a) = \min  _ {(P, R) \in \mathcal {U} _ {p} ^ {\mathrm {s a}}} \left\{R (s, a) + \gamma \sum_ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {X}} P (s ^ {\prime} | s, a) \pi_ {s ^ {\prime}} (a ^ {\prime}) Q (s ^ {\prime}, a ^ {\prime}) \right\} \\ = \min  _ {R \in \mathcal {R} _ {p} ^ {\mathrm {s a}}} R (s, a) + \gamma \min  _ {P \in \mathcal {P} _ {p} ^ {\mathrm {s a}}} \left\{\sum_ {s ^ {\prime} \in \mathcal {S}} P \left(s ^ {\prime} | s, a\right) v \left(s ^ {\prime}\right) \right\} \\ = R _ {0} (s, a) - \alpha_ {s, a} + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} P _ {0} \left(s ^ {\prime} \mid s, a\right) v \left(s ^ {\prime}\right) - \beta_ {s, a} \kappa_ {q} (v) \tag {By[14]} \\ = R _ {0} (s, a) - \alpha_ {s, a} + \gamma \sum_ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {X}} P _ {0} (s ^ {\prime} | s, a) \pi_ {s ^ {\prime}} (a ^ {\prime}) Q (s ^ {\prime}, a ^ {\prime}) - \beta_ {s, a} \kappa_ {q} (v) \\ = T _ {\left(P _ {0}, R _ {0}\right)} ^ {\pi} Q (s, a) - \alpha_ {s, a} - \beta_ {s, a} \kappa_ {q} (v). \\ \end{array}
$$
D.3 Convergence
In the remainder of this section, we focus on  $\ell_p$ -ball constrained uncertainty sets of the form  $\mathcal{U}_p^{\mathrm{sa}}$  or  $\mathcal{U}_p^{\mathrm{s}}$ . Let our Q-value iteration  $Q_{n + 1}\coloneqq \mathcal{L}_{\mathcal{U}}^{\pi}Q_{n}$ , and denote  $v_{n}(s) = \langle \pi_{s},Q_{n}(s,\cdot)\rangle_{\mathcal{A}},\forall s\in S,n\in \mathbb{N}$ .