SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By construction, it has zero mean and unit norm, i.e.,  $\langle u_{\mathcal{U}}^{\pi}, \mathbf{1} \rangle_{\mathcal{S}} = 0$  and  $\| u_{\mathcal{U}}^{\pi} \|_p = 1$ . In fact, as stated in the result below,  $u_{\mathcal{U}}^{\pi}$  is the gradient of the  $q$ -variance function, and correlates with the (unnormized, unbalanced) robust value function according to the same  $q$ -variance.
Proposition 4.1. For any policy  $\pi \in \Pi$  and  $\ell_p$ -ball rectangular uncertainty set, the following holds:
$$
u _ {\mathcal {U}} ^ {\pi} = \nabla_ {v} \kappa_ {q} (v) \Big | _ {v = v _ {\mathcal {U}} ^ {\pi}},
$$
$$
\left\langle u _ {\mathcal {U}} ^ {\pi}, v _ {\mathcal {U}} ^ {\pi} \right\rangle = \kappa_ {q} \left(v _ {\mathcal {U}} ^ {\pi}\right).
$$
4.1 Worst Kernel and Reward
In the following results, we explicit the relationship between the nominal and the worst-case model for  $(s,a)$  and  $s$ -rectangular  $\ell_p$ -balls. We will then leverage this relationship to compute the robust Q-values and the robust occupation measure, both necessary for RPG.
Theorem 4.2 ((s, a)-rectangular case). Given uncertainty set  $\mathcal{U} = \mathcal{U}_p^{s\alpha}$  and any policy  $\pi \in \Pi$ , the worst model is related to the nominal one through:
$$
R _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {0} (s, a) - \alpha_ {s, a} \quad \text {a n d} \quad P _ {\mathcal {U}} ^ {\pi} (\cdot | s, a) = P _ {0} (\cdot | s, a) - \beta_ {s, a} u _ {\mathcal {U}} ^ {\pi}.
$$
Based on Thm. 4.2, it follows that in the  $(s,a)$ -rectangular case, the worst reward function is independent of the employed policy. As we establish in Thm. 4.3 below, this no longer applies under  $s$ -rectangularity. In either case, the worst kernel is policy-dependent, discouraging the system to move toward high-rewarding states and directing it to low-rewarding ones instead. Surprisingly, the vector penalty  $u_{\mathcal{U}}^{\pi} \in \mathbb{R}^{S}$  additionally illustrates that the worst kernel is a rank-one perturbation of the nominal. Indeed, considering the stochastic matrix induced by any policy  $\pi \in \Pi$ , we have
$$
[ P _ {\mathcal {U}} ^ {\pi} - P _ {0} ^ {\pi} ] (s ^ {\prime} | s) = - \left(\sum_ {a \in \mathcal {A}} \beta_ {s, a} \pi_ {s} (a)\right) u _ {\mathcal {U}} ^ {\pi} (s ^ {\prime}), \quad \forall s \in \mathcal {S},
$$
so that the perturbation matrix  $P_{\mathcal{U}}^{\pi} - P_{0}^{\pi}$  is of rank one. In the sequel, we will leverage this finding to compute the robust visitation frequency.
Theorem 4.3 (s-rectangular case). Given uncertainty set  $\mathcal{U} = \mathcal{U}_p^s$  and any policy  $\pi \in \Pi$ , the worst model is related to the nominal one through:
$$
R _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {0} (s, a) - \alpha_ {s} \left(\frac {\pi_ {s} (a)}{\| \pi_ {s} \| _ {q}}\right) ^ {q - 1} \quad a n d \quad P _ {\mathcal {U}} ^ {\pi} (\cdot | s, a) = P _ {0} (\cdot | s, a) - \beta_ {s} u _ {\mathcal {U}} ^ {\pi} \left(\frac {\pi_ {s} (a)}{\| \pi_ {s} \| _ {q}}\right) ^ {q - 1}.
$$
Similarly to the  $(s, a)$ -case, the adversarial kernel is a rank-one perturbation of the nominal. Yet, an extra dependence on the policy through the coefficient  $\left(\frac{\pi_s(a)}{\|\pi_s\|_q}\right)^{q-1}$  appears in the  $s$ -case, affecting both the worst reward and the worst kernel. Intuitively, it means that the worst model cannot be chosen independently for each action, but must instead depend on the agent's policy. This further explains why optimal policies can all be stochastic in  $s$ -rectangular robust MDPs [33].
Thms. 4.2 and 4.3 enable us to derive the worst MDP model in closed form with time complexity  $O(S^2 A \log \epsilon^{-1})$ , up to logarithmic factors (please see the appendix for a detailed discussion). It thus holds the same complexity as non-robust value iteration, since we additionally need to compute the value function to derive its corresponding regularizer [3, 14]. On the other hand, if we employ convex optimization using value methods instead, obtaining the worst model requires a time complexity of  $O(S^4 A \log \epsilon^{-1})$  in the  $(s, a)$ -rectangular case, and  $O(S^4 A^3 \log \epsilon^{-1})$  in the  $s$ -rectangular case [30][Sec. 4.1].
4.2 Robust Occupation Measure
We finally derive the robust occupation measure using nominal values, which will lead to an explicit RPG. Although intractable in general, we show that focusing on ball-constrained uncertainty enables deriving the robust occupation matrix efficiently from the (nominal) occupation measure. We first establish the lemma below, which leverages the fact that the worst transition function is a rank-one perturbation of the nominal and represents our core contribution.
Lemma 4.4. Let  $b, k \in \mathbb{R}^S$  and  $P_0, P_1 \in (\Delta_S)^S$  two transition matrices. If  $P_1 = P_0 - bk^\top$ , i.e.,  $P_1$  is a rank-one perturbation of  $P_0$ , then their occupation matrices  $D_i := (I - \gamma P_i)^{-1}, i = 0, 1$  are related through:
$$
D _ {1} = D _ {0} - \gamma \frac {D _ {0} b k ^ {\top} D _ {0}}{(1 + \gamma k ^ {\top} D _ {0} b)}.
$$
Combining Thms. 4.2 and 4.3 with the above lemma, we obtain the robust occupation in terms of the nominal, as stated in Thm. 4.6 below. Prior to this, we introduce the notion of expected transition uncertainty below.
Definition 4.5. Let  $\mathcal{U}$  a rectangular  $\ell_p$ -ball-constrained uncertainty set of transition radius  $\beta$ . For any policy  $\pi \in \Pi$ , the expected transition uncertainty at any state  $s \in S$  is given by  $\beta_s^\pi := \sum_{a \in \mathcal{A}} \pi_s(a) \beta_{s,a}$  if  $\mathcal{U} = \mathcal{U}_p^{sa}$ , and  $\beta_s^\pi := \beta_s \| \pi_s \|_q$  if  $\mathcal{U} = \mathcal{U}_p^s$ .
Theorem 4.6. For any rectangular  $\ell_p$ -ball-constrained uncertainty and  $\pi \in \Pi$ , it holds that:
$$
d _ {\mathcal {U}, \mu} ^ {\pi} = d _ {P _ {0}, \mu} ^ {\pi} - \gamma \frac {\langle d _ {P _ {0} , \mu} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}}{1 + \gamma \langle d _ {P _ {0} , u _ {\mathcal {U}} ^ {\pi}} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}} d _ {P _ {0}, u _ {\mathcal {U}} ^ {\pi}} ^ {\pi}. \tag {3}
$$
Thm. 4.6 explicitly highlights the relationship between the robust visitation frequency and the nominal one. Thus, according to Eq. (3), the standard non-robust occupation measure in the first term needs to be penalized by another one,  $d_{P_0,u_\mathcal{U}}^\pi = (u_\mathcal{U}^\pi)^\top (I_S - \gamma P_0^\pi)^{-1}$ , to obtain the robust occupation measure. Recall that  $u_{\mathcal{U}}^{\pi}$  is the balanced-scaled value function determined by  $\pi \in \Pi$  and uncertainty set  $\mathcal{U}$ . Thus, the penalty term  $d_{P_0,u_\mathcal{U}}^\pi$  tends to zero if all coordinates of the robust value function vector converge to the same value.
Nonetheless, our expression (3) does present some challenges. First, the visitation frequency appearing in the correction term indicates that instead of taking a fixed initial state distribution, we should start from a varying and signed measure represented by the balanced value function. Although it suggests putting more weight on worst-performing states, obtaining a non-biased estimator for this occupancy measure remains unclear in model-free learning. One may use importance sampling, but as any off-policy approach, both variance and bias would need to be controlled then. Such statistical analysis goes beyond the scope of this work.
4.3 Robust Q-values
In this section, we focus on the last element needed for RPG and aim to estimate the robust Q-value denoted previously by  $Q_{\mathcal{U}}^{\pi} \coloneqq Q_{(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})}^{\pi}$ . Define its associated value function as  $v_{\mathcal{U}}^{\pi}(s) = \langle \pi_s, Q_{\mathcal{U}}^{\pi}(s, \cdot) \rangle$ ,  $\forall s \in S, \pi \in \Pi$ . Based on standard Bellman recursion, it thus holds that:
$$
Q _ {\mathcal {U}} ^ {\pi} (s, a) = R _ {\mathcal {U}} ^ {\pi} (s, a) + \gamma \left\langle P _ {\mathcal {U}} ^ {\pi} (\cdot | s, a), v _ {\mathcal {U}} ^ {\pi} \right\rangle_ {\mathcal {S}}, \quad \forall (s, a) \in \mathcal {X}, \pi \in \Pi ,
$$
while  $Q_{\mathcal{U}}^{\pi}$  is the unique fixed point of the  $\gamma$ -contracting operator
$$
\left(\mathcal {L} _ {\mathcal {U}} ^ {\pi} Q\right) (s, a) := T _ {\left(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi}\right)} ^ {\pi} Q (s, a), \quad \forall Q \in \mathbb {R} ^ {\mathcal {X}}. \tag {4}
$$
The relations above hold for general uncertainty sets, provided that we have access to the worst model. The  $s$ -rectangularity assumption additionally enables us to retrieve the robust value function using the Bellman operator above [33]. Concretely, we have:  $v_{\mathcal{U}}^{\pi} = \min_{(P,R)\in \mathcal{U}}v_{(P,R)}^{\pi} = v_{(P_{\mathcal{U}}^{\pi},R_{\mathcal{U}}^{\pi})}^{\pi}$ .
The following result derives a regularized operator equivalent to  $\mathcal{L}_{\mathcal{U}}^{\pi}$ , which results in an efficient iteration method to compute the robust Q-value.
Proposition 4.7. The Bellman operator  $\mathcal{L}_{\mathcal{U}}^{\pi}$  defined in Eq. (4) is equivalent to:
$$
\left(\mathcal {L} _ {\mathcal {U}} ^ {\pi} Q\right) (s, a) = T _ {\left(P _ {0}, R _ {0}\right)} ^ {\pi} Q (s, a) + \Omega_ {q} ^ {\prime} \left(\alpha_ {s, a}, \beta_ {s, a}, v\right),
$$
where  $v(s)\coloneqq \langle \pi_s,Q(s,\cdot)\rangle_{\mathcal{A}},\Omega_q'(\alpha ,\beta ,v)\coloneqq -(a_{s,a} + \gamma \beta_{s,a}\kappa_q(v))$  for  $(s,a)$  -rectangular uncer
tainty  $\mathcal{U}_p^{sa}$ , and  $\Omega_q^\prime (\alpha ,\beta ,v)\coloneqq -\left(\frac{\pi_s(a)}{\|\pi_s\|_q}\right)^{q - 1}(\alpha_s + \gamma \beta_s\kappa_q(v))$  for s-rectangular  $\mathcal{U}_p^s$
5 Robust Policy Gradient
We are now able to derive an RPG by combining our previous results. Notably, unlike previous works that need to sample next-state transitions based on all models from the uncertainty set [21, 17, 4], here, we only need the nominal kernel to get the occupation measures.
Theorem 5.1 (RPG). For any rectangular  $\ell_p$ -ball-constrained uncertainty, the robust policy gradient is given by:
$$
\partial_ {\pi} \rho_ {\mathcal {U}} ^ {\pi} = \sum_ {(s, a) \in \mathcal {X}} \left(d _ {P _ {0}, \mu} ^ {\pi} (s) - c ^ {\pi} (s)\right) Q _ {\mathcal {U}} ^ {\pi} (s, a) \nabla \pi_ {s} (a), \tag {5}
$$
$$
c ^ {\pi} (s) := \frac {\gamma \langle d _ {P _ {0} , \mu} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}}{1 + \gamma \langle d _ {P _ {0} , u _ {\mathcal {U}} ^ {\pi}} ^ {\pi} , \beta^ {\pi} \rangle_ {\mathcal {S}}} d _ {P _ {0}, u _ {\mathcal {U}} ^ {\pi}} ^ {\pi} (s), \quad \forall s \in \mathcal {S}.
$$
The implementation of RPG directly follows and can be found in Alg. 1. Thm. 5.1 is a straightforward application of non-robust PG, as its proof simply consists in plugging Eq. (3) into the standard PG expression  $\partial_{\pi}\rho_{\mathcal{U}}^{\pi} = \sum_{(s,a)\in \mathcal{X}}d_{\mathcal{U},\mu}^{\pi}(s)Q_{\mathcal{U}}^{\pi}(s,a)\nabla \pi_{s}(a)$ . We obtain a regular PG in the first term, with the robust Q-value instead of the non-robust one, plus a correction term  $c^{\pi}$  resulting from taking the visitation frequency of the worst kernel instead of the nominal. Unlike previous work that uses policy regularization to achieve empirical robustness in PG methods [2, 11], Thm. 5.1 establishes an RPG that accounts for transition uncertainty and targets a robust optimal policy. It crucially relies on the rank-one-perturbation structure of the worst transition kernel (see Lem. 4.4). As established in Thm. 4.6,  $\ell_p$ -ball uncertainty implies such property, but the converse as to whether any convex set leads to the worst transition kernel being a rank-one perturbation of the nominal remains an open question. For example, it would be interesting to investigate the structural properties needed on the uncertainty set for the rank-one perturbation to hold.
Algorithm 1 RPG
Input:  $\mu, \eta$
Initialize:  $v_{k},\pi_{k}$
1: for  $k = 1,2,\ldots$  do
2:  $\partial_{\pi}\rho_{k} = \sum_{(s,a)\in \mathcal{X}}\left(d_{P_{0},\mu}^{\pi}(s) - c^{\pi}(s)\right)Q_{\mathcal{U}}^{\pi}(s,a)\nabla \pi_{s}(a)$
3:  $\pi_{k}\gets \mathbf{proj}_{\Pi}(\pi_{k} + \eta \partial_{\pi}\rho_{k})$
4: end for
$\triangleright$  Compute policy gradient
> Update policy
5.1 Complexity Analysis
A major concern in solving robust MDPs is time complexity [33]. Similarly, it is of major importance to assess the additional time required for computing an RPG update, compared to its non-robust variant. Although previous work has analyzed the convergence rate of RPG to a global optimum [30], it assumes access to an oracle gradient, thus occulting the computational concerns raised from gradient estimation. In fact, the NP-hardness of non-rectangular and/or non-convex robust MDPs [33] already indicates that their resulting RPG can be intractable.
To compute RPG in Thm. 5.1, we first need to evaluate the robust Q-value. Based on Lemma 4.7 and the Bellman operators introduced there, our evaluation method involves an additional estimation of the variance function  $\kappa_{p}$ . According to [14], this takes logarithmic time at most, using binary search. As to the compensation term  $c^{\pi}$  in Eq. (11), it requires computing occupancy measures with respect to two different initial vectors, namely the balanced value function and the initial distribution. Thus, the computational cost for estimating  $c^{\pi}$  is the same as estimating a non-robust occupancy measure. Tab. 1 summarizes the complexity of different approaches while a detailed discussion can be found in the appendix. We refer to [30][Sec. 4.1] for the complexity of RPG based on convex optimization.
Generalization to arbitrary norms. Until now, we have focused on  $\ell_p$ -norm for concreteness. However, the above results apply to any norm  $\| \cdot \|$ , at least if the uncertainty set is  $(s,a)$ -rectangular, in which case the variance function changes to  $\kappa (v)\coloneqq \min_{\| c\| \leq 1,\mathbf{1}^{\top}c = 0}\langle c,v\rangle$  and the balanced value to  $\arg \min_{\| c\| \leq 1,\mathbf{1}^{\top}c = 0}\langle c,v\rangle$ . The rank-one perturbation structure of the worst kernel is preserved, so the robust occupation measure can be obtained similarly using Lemma 4.4. The  $s$ -rectangular is more involved. We defer its discussion to the appendix and leave its complete derivation for future work.
In order to test the effectiveness of our RPG update, we evaluate its increased time complexity relative to non-robust PG. In the following experiments, we randomly generate nominal models for arbitrary state-action space sizes. Each experiment was averaged over 100 runs. We refer the reader to the appendix for more details on the radius levels and other implementation choices.
We first focus on  $\ell_1$ -robust MDPs to compare our RPG with a convex optimization approach. Specifically, we consider a robust PG with an optimization solver, which we designate by LP-RPG. Indeed, recall that  $\ell_1$ -ball-constraints induce a linear program (LP) rather than a more general convex optimization problem. Therefore, to compute the robust value function for a given policy, we iteratively evaluate the robust Bellman operator using LP [30, Section 4.1]. Using this approximated value function, we can compute the worst value parameters to apply PG theorem by [26] and deduce an LP-based robust PG update. Differently, our RPG method relies on the regularized formulation of robust value iteration proposed in [3, 14], from which we deduce the normalized-balanced value function as in Eq. (2). We finally apply Thm. 4.6 to compute the robust occupation measure, and Prop. 4.7 to obtain the robust Q-value.
Tab. 3 displays the results obtained for the two alternative methods described above. In all experiments, the standard deviation was typically  $2 - 10\%$  so we omitted it for brevity. As can be seen in Tab. 3, LP-RPG does not scale well compared to RPG, whereas RPG has similar time complexity as PG. Notably, the running time of  $s$ -rectangular LP-RPG scales much better with the space size than its  $(s,a)$ -rectangular equivalent, which confirms the theoretical complexities from Tab. 1. Yet, since these methods were time-consuming, we repeated these for a few runs only. In fact, LP-RPG is more expensive than RPG by 1-3 orders of magnitude, which illustrates its inefficiency. We emphasize that here, we only focused on  $\ell_1$ -robust MDPs to leverage LP solvers in robust policy evaluation. We expect the computational cost of LP-RPG to scale even more poorly for other  $\ell_p$ -robust MDPs that involve polynomial time-consuming convex programs.
Table 3: Comparison of the relative running time between RPG and the convex optimization approach (here, LP). Our method is faster than LP-based updates by 1 to 3 orders of magnitude.
{ (P0, R0) }	Usa	Us
S	A	PG	RPG	LP-RPG	RPG	LP-RPG
10	10	1	1.4	326	1.4	77
30	10	1	1.4	351	1.4	109
50	10	1	1.4	408	1.4	159
100	20	1	1.5	469	1.3	268
500	50	1	1.3	925	1.3	5343
We further compare our RPG to non-robust PG on different  $\ell_p$ -balls. Tab. 4 confirms the comparable time complexity of RPG to non-robust PG, thus demonstrating the effectiveness of our method. We note that for  $p \in \{1,2,\infty\}$ , the corresponding regularization quantities can be computed in closed form, whereas they involve a binary search for other values [14]. We thus get a slight running-time increase for  $p \in \{5,10\}$ .
Table 4: Relative running time for computing RPG under different types of uncertainty sets.
S	A	{ (P0,R0) }	U2sa	U2s	U5sa	U5s	U10sa	U10s	Usa∞	Usa∞
10	10	1	1.5	1.5	4.9	4.7	4.7	4.9	1.5	1.6
30	10	1	1.4	1.5	4.2	4.3	4.2	4.0	1.4	1.4
50	10	1	1.5	1.4	4.5	4.1	4.0	4.0	1.4	1.4
100	20	1	1.4	1.3	2.6	2.5	2.5	2.4	1.3	1.2
500	50	1	1.2	1.2	1.7	1.7	1.7	1.7	1.2	1.3
This paper introduced an explicit expression of RPG for rectangular robust MDPs. Our approach involved auxiliary results such as deriving the worst model in closed form and showing that it is a rank-one perturbation of the nominal kernel. The resulting RPG extends vanilla PG with additional correction terms that can be derived in closed form as well. Thus, the computational time of RPG is similar to its non-robust variant.
A key assumption that would be interesting to relax is the normed-ball structure of the uncertainty sets considered in this study. Indeed, since the proofs of our technical results rely on norm properties, it is still unclear if and how RPG can generalize to metric-based or  $f$ -divergence uncertainty sets. The latter type of uncertainty can be particularly useful for data-driven settings, as the radius can be chosen according to cross-validation or statistical bounds [10]. Another compelling direction would be to explore other variants of RPG using mirror descent or natural policy gradient and examine their compatibility with deep architectures, which would further demonstrate the practical efficiency of our RPG method.
[1] Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for  $\ell_{\infty}$ -constrained s-rectangular robust MDPs. Advances in Neural Information Processing Systems, 34:25982-25992, 2021.
[2] Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Gregoire Delétang, Markus Kunesch, Shane Legg, and Pedro Ortega. Your policy regularizer is secretly an adversary. Transactions on Machine Learning Research (TMLR), 2022.
[3] Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized MDPs and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34:22274-22287, 2021.
[4] Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. AUAI press for Association for Uncertainty in Artificial Intelligence, pages 208–218, 2018.
[5] Esther Derman, Yevgeniy Men, Matthieu Geist, and Shie Mannor. Twice regularized Markov decision processes: The equivalence between robustness and regularization. arXiv preprint arXiv:2303.06654, 2023.
[6] Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. International Conference on Learning Representations, 2022.
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[8] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast Bellman updates for robust MDPs. In International Conference on Machine Learning, pages 1979-1988. PMLR, 2018.
[9] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for  $l_{1}$ -robust Markov decision processes. J. Mach. Learn. Res., 22:275-1, 2021.
[10] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Robust  $\phi$ -divergence MDPs. arXiv preprint arXiv:2205.14202, 2022.
[11] Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics, pages 64-72. PMLR, 2021.
[12] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[13] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning. CiteSeer, 2002.
A Balanced and Normed Vectors 13
A.1 Proof of Proposition 4.1 14
B Worst Kernel and Reward 14
B.1 Proof of Theorem 4.2 14
B.2 Proof of Theorem 4.3 15
C Occupation Matrix 16
C.1 Proof of Lemma 4.4 16
C.2 Proof of Theorem 4.6 17
D Robust Q-value 17
D.1 Basic Properties 17
D.2 Evaluation 18
D.3 Convergence 19
E Robust Policy Gradient 20
F Complexity Analysis 20
F.1 Helper results for Robust MDPs 21
F.2 RPG Complexity 22
G Generalization to arbitrary norms 23
G.1  $(s,a)$  -rectangular robust MDPs 23
G.2  $s$  -rectangular case 24
G.3 Generalization to metrics and divergences 24
H Experiments 24
H.1 RPG by LP 24
A Balanced and Normed Vectors
In this section, we lay down some basic properties of  $p$ -normalized-balanced vectors.
First recall the  $p$ -variance and the  $p$ -mean defined as:
$$
\kappa_ {p} (v) = \underset {\omega \in \mathbb {R}} {\min } \| v - \omega \mathbf {1} \| _ {p}, \qquad \omega_ {p} (v) = \underset {\omega \in \mathbb {R}} {\arg \min } \| v - \omega \mathbf {1} \| _ {p}.
$$
Given any  $v\in \mathbb{R}^S$  , let also the  $p$  -balanced-normalized function:
$$
u _ {p} (v) (s) := \operatorname {S I G N} (v (s) - \omega_ {p} (v)) \left(\frac {| v (s) - \omega_ {p} (v) |}{\kappa_ {p} (v)}\right) ^ {p - 1}, \quad \forall v \in \mathbb {R} ^ {\mathcal {S}}, s \in \mathcal {S}.
$$
According to [14][Sec. 16.1, Lemma 1], the following holds:
$$
\kappa_ {q} (v) = - \frac {1}{\epsilon} \left[ \min  _ {\| c \| _ {p} \leq \epsilon , \langle c, \mathbf {1} \rangle = 0} \langle c, v \rangle \right], \tag {6}
$$
namely, the  $p$ -variance function is the optimal value of a linear optimization under kernel noise constraint. The result below further characterizes the solution to the above problem.
Lemma A.1. The vector defined as  $c^* \coloneqq -\epsilon u_q(v)$  is an optimal solution to the optimization problem
$$
\min_{\| c\|_{p}\leq \epsilon ,\langle c,\mathbf{1}\rangle_{\mathcal{S}} = 0}\langle c,v\rangle .
$$
Proof. It suffices to show that  $c^*$  satisfies both constraints  $\| c^* \|_p \leq \epsilon$  and  $\langle c^*, \mathbf{1} \rangle_S = 0$ , and that it reaches optimal value, i.e.,  $-\frac{1}{\epsilon} \langle c^*, v \rangle = \kappa_q(v)$ . We thus compute: