SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Navdeep Kumar*
Esther Derman
MILA, Université de Montréal
Matthieu Geist
Goodle Deepmind
Kfir Levy
Technion, NVIDIA Research
Policy gradient methods have become a standard for training reinforcement learning agents in a scalable and efficient manner. However, they do not account for transition uncertainty, whereas learning robust policies can be computationally expensive. In this paper, we introduce robust policy gradient (RPG), a policy-based method that efficiently solves rectangular robust Markov decision processes (MDPs). We provide a closed-form expression for the worst occupation measure. Incidentally, we find that the worst kernel is a rank-one perturbation of the nominal. Combining the worst occupation measure with a robust Q-value estimation yields an explicit form of the robust gradient. Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent. Hence, it relieves the computational burden of convex optimization problems required for training robust policies by current policy gradient approaches.
Markov decision processes (MDPs) provide an analytical framework to solve sequential decision-making problems and seek the best performance in a fixed environment. Since the resulting policy can be highly sensitive to parameter values [18], the robust MDP setting alternatively maximizes return under the worst scenario, thus yielding robustness to uncertain environments [20, 12]. In practice, the robust MDP paradigm quantifies the level of uncertainty through a set  $\mathcal{U}$  determining the possible range of model perturbations. Then, a policy is said to be robust-optimal if it reaches maximal performance under the most adversarial model within the uncertainty set. Developing efficient solvers for robust MDPs is of great interest, as it can lead to behavior policies with generalization guarantees [34].
If not computationally expensive, robust MDPs can be strongly NP-hard [33]. Thus, to preserve tractability, we commonly assume that  $\mathcal{U}$  is convex and  $s$ -rectangular, i.e.,  $\mathcal{U} = \times_{s \in S} \mathcal{U}_s$  [20, 12, 33]. Well established in the robust reinforcement learning (RL) literature [8, 3, 9, 10, 28], the latter assumption means that the overall uncertainty should be designed independently for each state. Further simplification may consider  $(s, a)$ -rectangular uncertainty sets of the form  $\mathcal{U} = \times_{(s, a) \in \mathcal{X}} \mathcal{U}_{(s, a)}$ , albeit this naturally leads to more conservative strategies. In any case, planning in robust MDPs can be computationally costly, as it involves successive max-min problems [9, 1, 33]. To address this issue, the works [3, 14] have established an equivalence between robustness and regularization in RL in order to derive efficient robust planning methods for  $s$  and  $(s, a)$ -rectangular robust MDPs. Indeed, it appears that resorting to proper regularization instead of solving a minimization problem can yield robust behavior without requiring the polynomial time complexity of convex optimization problems [3].
*Contact author navdeepkumar@campus.technion.ac.il
Alternatively to planning, policy gradient algorithms (PG) directly learn an optimal policy by applying gradient steps towards better performance [25]. Due to its scalability, ease of implementation, and adaptability to many different settings such as model-free and continuous state-action spaces [13, 24], PG has become the workhorse of RL. Although regularization techniques such as max-entropy [7] or Tsallis [15] have shown robust behavior without impairing computational cost, they only account for adversarial reward [2, 6, 3]. Differently, robust PG formulations (RPG) formulations aim to address uncertainty to reward and transition functions.
Despite their ability to propel robust behavior, RPG methods that target robust optimal policies are still rare in the RL literature. The global convergence of RPG established in [16, 30] already ensures global convergence of our proposed algorithm, but motivates a practical method for estimating the gradients. Indeed, [16, 30] occult the estimation part, as they assume full access to the policy gradient. Alternatively, the inner loop solution proposed in [30][Sec. 4.1] requires solving convex optimization problems to find the worst model, which represents a larger time complexity of  $O(S^4 A \log \epsilon^{-1})$  for  $(s, a)$ -rectangular, or  $O(S^4 A^3 \log \epsilon^{-1})$  for  $s$ -rectangular uncertainty sets. These worst kernel and reward models are needed to compute RPG using the policy gradient theorem [26]. Other approaches that elicit an expression for RPG rely on a specific type of uncertainty set such as reward uncertainty with known kernel [3, 5],  $r$ -contaminated kernel with known reward [32], or  $(s, a)$ -rectangular uncertainty [16], whereas we aim to tackle more general robust MDPs.
In this work, we introduce an RPG method for both  $s$  and  $(s,a)$ -rectangular ball-constrained uncertainty sets, with similar complexity as non-robust PG. Our approach provides a closed-form expression of RPG without relying on an oracle while applying to the most common robust MDPs. To this end, we derive the worst reward and transition functions, thus revealing the adversarial nature of the corresponding uncertainty set. Surprisingly, we also find that the worst kernel is a rank-one perturbation of the nominal kernel. Leveraging this rank-one perturbation enables us to derive a robust occupation measure. We concurrently propose an alternative definition of the robust Q-value together with an efficient way to estimate it. Combining these results enables us to obtain RPG in closed form. Our resulting RPG update requires  $O(S^2 A\log \epsilon^{-1})$  computations, thus showing similar time complexity as non-robust PG.
To summarize our contributions: (i) We establish the worst reward and transition models in closed-form; (ii) We show that the worst-case transition function is a rank-one perturbation of the nominal; (iii) We introduce alternative robust Q-values that can be evaluated through efficient Bellman recursion while retrieving the robust value function; (iv) We establish an expression of RPG that can be estimated with similar time complexity as non-robust PG. Experiments show that our RPG speeds up state-of-the-art robust PG updates by 2 orders of magnitude.
Although some previous works use gradient methods to learn robust policies, they seek empirical robustness to adversarial behavior rather than robust MDP solutions [21, 29, 4]. In that sense, our study differs from adversarial RL as we explicitly optimize the max-min objective to find a robust optimal policy. Accordingly, the risk-averse approach focuses on the internal uncertainty due to the stochasticity of the system, whereas robust RL addresses the external uncertainty of the system's dynamics. As a result, common risk-averse objectives can be reformulated as robust problems with specific uncertainty sets [27].
Previous studies that did aim to derive robust policy-based methods are [3, 32, 30]. These are summarized in Table 1, which also displays the complexity of existing approaches. [3] established RPG for  $s$ -rectangular reward-robust MDPs, i.e., robust MDPs with uncertain reward but given kernel. Although it applies to general norms, their result does not account for transition perturbation. Differently, in [32], the authors introduced RPG for  $r$ -contaminated MDPs, i.e., robust MDPs with uncertainty set  $\mathcal{U} \coloneqq \{R_0\} \times [(1 - r)P_0 + r\Delta_S^{S \times A}]$ . Although it has similar complexity as non-robust PG, by construction, their setting is limited to  $(s,a)$ -rectangularity with known reward and mixed transition. As such, the proof techniques in [32] are tailor-made to the  $r$ -contamination framework and do not apply to more general robust MDPs. In fact, we remark that the  $r$ -contamination setting is equivalent to the action robustness approach introduced in [29], which emphasizes its limitation to action perturbation. Differently, our RPG holds whenever the worst kernel is a rank-one perturbation of the nominal transition function (see Lemma 4.4).
The work [16] provides a convergence proof of robust policy mirror-descent in the  $(s,a)$ -rectangular case, whereas we study robust policy optimization for  $s$ -rectangular uncertainty sets. In fact, its restriction to the  $(s,a)$ -case prevents us from transposing the analysis to our setting. This is due to the fact that the standard robust Bellman operator on  $Q$ -functions can no longer be applied on  $s$ -rectangular sets. To address generic robust MDPs, [30] recently introduced RPG for general uncertainty sets. Their gradient update has a complexity of  $O(S^6 A^4 \epsilon^{-4})$ , which is more expensive than non-robust PG by a factor of  $S^4 A^3 \epsilon^{-4}$ . Both works [16, 30] additionally assume access to an oracle gradient of the robust return with respect to the transition model. Avoiding this oracle assumption naturally leads to even higher time complexity in [30] which is not scalable. At the same time, the two works [16, 30] guarantee global convergence of projected robust gradient iterates, thus establishing the potential promise of RPG. In fact, equipped with RPG convergence, the remaining challenge in making it practical is to efficiently estimate the gradient. This represents the main focus of our study: We aim to explicit an RPG method that generalizes existing results on specific uncertainty sets [3, 32] while holding for  $s$ -rectangular robust MDPs.
Table 1: Time complexity of RPG update according to the type of uncertainty set. For conciseness, the displayed complexity hides logarithmic factors in  $A$  and  $S$ . Our RPG method has the same complexity as non-robust PG while it generalizes other RPG methods with similar efficiency.
UNCERTAINTY SET U	TIME COMPLEXITY	REFERENCE
{R0} × {P0}	S2A log ε-1	[26]
{R0} × [(1-r)P0+rΔS×A](s,a)-rectangular ball Usa(s,a)-rectangular, convex Usa	S2A log ε-1S2A log ε-1S4A log ε-1	[32]This workConvex optimization
s-rectangular ball Us	S2A log ε-1	This work
s-rectangular ball (R0+Rs) × {P0}	S2A log ε-1	[3]
s-rectangular, convex Us	S4A3 log ε-1	Convex optimization
s-rectangular, convex Us	S6A4ε-4	[30]
s-rectangular, non-convex Us	NP-hard	[33]
Non-rectangular, convex U	NP-hard	[33]
Notation: We denote the cardinal of an arbitrary finite set  $\mathcal{Z}$  by  $|\mathcal{Z}|$ . Given two real functions  $\mathbf{a}, \mathbf{b}: \mathcal{Z} \to \mathbb{R}$ , their inner product is  $\langle \mathbf{a}, \mathbf{b} \rangle_{\mathcal{Z}} \coloneqq \sum_{z \in \mathcal{Z}} \mathbf{a}(z) \mathbf{b}(z)$ , which induces the  $\ell_2$ -norm  $\| \mathbf{a} \|_2 \coloneqq \sqrt{\langle \mathbf{a}, \mathbf{a} \rangle_{\mathcal{Z}}}$ . More generally, the  $\ell_p$ -norm of  $\mathbf{a}$  is denoted by  $\| \mathbf{a} \|_p$  whose conjugate norm is  $\| \mathbf{a} \|_q \coloneqq \max_{\| \mathbf{b} \|_p \leq 1} \langle \mathbf{a}, \mathbf{b} \rangle_{\mathcal{Z}}$  with  $q^{-1} = 1 - p^{-1}$ , by Hölder's inequality. The vector of all zeros (resp. all ones) with appropriate dimensions is denoted by  $\mathbf{0}$  (resp.  $\mathbf{1}$ ); the probability simplex over  $\mathcal{Z}$  by  $\Delta_{\mathcal{Z}} \coloneqq \{\mathbf{a}: \mathcal{Z} \to \mathbb{R}_+ | \langle \mathbf{a}, \mathbf{1} \rangle_{\mathcal{Z}} = 1\}$ , and  $I_{\mathcal{Z}}$  designates the identity matrix in  $\mathbb{R}^{\mathcal{Z} \times \mathcal{Z}}$ . Given  $v \in \mathbb{R}^{\mathcal{Z}}$ , we finally define the variance function as  $\kappa_q(v) = \min_{w \in \mathbb{R}} \| v - w\mathbf{1} \|_q$  and the mean function as  $\omega_q(v) \in \arg \min_{w \in \mathbb{R}} \| v - w\mathbf{1} \|_q$  (see Tab. 2 for their closed-form expression when  $q \in \{1, 2, \infty\}$ ).
A Markov decision process (MDP) is a tuple  $(S, \mathcal{A}, \gamma, \mu, P, R)$  such that  $S$  and  $\mathcal{A}$  are finite state and action spaces of cardinal  $S$  and  $A$  respectively,  $\gamma \in [0,1)$  is a discount factor and  $\mu \in \Delta_S$  the initial state distribution. Denoting  $\mathcal{X} := S \times \mathcal{A}$ , the couple  $(P, R)$  corresponds to the MDP model with  $P: \mathcal{X} \to \Delta_S$  being a transition kernel and  $R: \mathcal{X} \to \mathbb{R}$  a reward function. A policy  $\pi: S \to \Delta_A$  maps each state to a probability distribution over  $\mathcal{A}$ , and we denote by  $\Pi$  the set of such functions. For any policy  $\pi \in \Pi$ ,  $R^{\pi} \in \mathbb{R}^{S}$  is the expected immediate reward defined as  $R^{\pi}(s) := \langle \pi_s, R(s, \cdot) \rangle_{\mathcal{A}}$ ,  $\forall s \in S$ , where  $\pi_s$  is a shorthand for  $\pi(\cdot|s)$ . We similarly define the stochastic matrix induced by  $\pi$  as  $P^{\pi}(s'|s) := \langle \pi_s, P(s'|s, \cdot) \rangle_{\mathcal{A}}$ ,  $\forall s, s' \in S$ , and extend the occupation measure to an arbitrary initial vector  $\nu \in \mathbf{R}^{S}$  by defining
$$
d _ {P, \nu} ^ {\pi} := \nu^ {\top} \left(I _ {\mathcal {S}} - \gamma P ^ {\pi}\right) ^ {- 1}.
$$
Note that the initial vector here is not necessarily a probability measure: it can be the initial state distribution, but also the balanced value function introduced in Sec. 4[Eq. (2)]. The performance measure we aim to maximize is the value function  $v_{(P,R)}^{\pi} \coloneqq (I_S - \gamma P^{\pi})^{-1}R^{\pi}$ , or alternatively, the return  $\rho_{(P,R)}^{\pi} \coloneqq \langle \mu ,v_{(P,R)}^{\pi}\rangle_{\mathcal{S}}$ . We denote the optimal value function (resp. optimal return) by  $v_{(P,R)}^{*} = \max_{\pi \in \Pi}v_{(P,R)}^{\pi}$  (resp.  $\rho_{(P,R)}^{*} = \langle \mu ,v_{(P,R)}^{*}\rangle$ ). It can be obtained using Bellman operators, which are defined as  $T_{(P,R)}^{\pi}v \coloneqq R^{\pi} + \gamma P^{\pi}v$  and  $T_{(P,R)}^{*}v \coloneqq \max_{\pi \in \Pi}T_{(P,R)}^{\pi}v, \forall v\in \mathbb{R}^{\mathcal{S}}$ , respectively [22]. For any vector  $v\in \mathbb{R}^{\mathcal{S}}$ , we associate its Q-function  $Q\in \mathbb{R}^{\mathcal{X}}$  such that
$$
Q (s, a) = R (s, a) + \gamma \langle P (\cdot | s, a), v \rangle_ {\mathcal {S}}, \quad \forall (s, a) \in \mathcal {X}.
$$
Table 2: Expressions of the  $q$ -mean, the  $q$ -variance, and its gradient. We assume that the vector  $v$  is sorted, i.e.,  $v(s_{i}) \geq v(s_{i+1}), \forall i \in \{1,2,\dots,S\}$ , and denote  $n_{l} := \lfloor (S+1)/2 \rfloor$ ,  $n_{u} := \lceil (S+1)/2 \rceil$ .
ωq(v)	κq(v)	∇vκq(v)
q	arg minw∈R||v-w1||q	minω∈R||v-ω1||q	∂κq(v)/∂v(si)
∞	v(s1)+v(sS)/2	v(s1)-v(sS)/2	{1/2 if i=1 -1/2 if i=S 0 o.w.
2	∑i=1S v(si)/S	√∑i=1S (v(si)-ω2(v))2	v(si)-ω2(v)/κ2(v)
1	v(sn1)+v(snu)/2	∑i=1n1(v(si)-v(sn-i))	{1 if i<nl -1 if i>nu 0 o.w.
With a slight abuse of notation, we can similarly define a Bellman operator over  $\mathbf{Q}$ -values as
$$
T _ {(P, R)} ^ {\pi} Q (s, a) := R (s, a) + \gamma \sum_ {(s ^ {\prime}, a ^ {\prime}) \in \mathcal {X}} P (s ^ {\prime} | s, a) \pi_ {s ^ {\prime}} (a ^ {\prime}) Q (s ^ {\prime}, a ^ {\prime}), \quad \forall (s, a) \in \mathcal {X}.
$$
3.2 Robust Markov Decision Processes
In a robust MDP setting, we assume that  $(P,R)\in \mathcal{U}$  and aim to maximize return under the worst model from the set. We denote the robust performance of a policy  $\pi \in \Pi$  by  $\rho_{\mathcal{U}}^{\pi}:= \min_{(P,R)\in \mathcal{U}}\rho_{(P,R)}^{\pi}$ . It is maximal when it reaches  $\rho_{\mathcal{U}}^{*}:= \max_{\pi \in \Pi}\rho_{\mathcal{U}}^{\pi}$  at an optimal robust policy  $\pi_{\mathcal{U}}^{*}\in \arg \max_{\pi \in \Pi}\rho_{\mathcal{U}}^{\pi}$ . When considering the robust value function  $v_{\mathcal{U}}^{\pi}:= \min_{(P,R)\in \mathcal{U}}v_{(P,R)}^{\pi}$ , we further need to assume that  $\mathcal{U}$  is convex and rectangular so that an optimal robust policy realizing  $v_{\mathcal{U}}^{*}:= \max_{\pi}v_{\mathcal{U}}^{\pi}$  can be computed in polynomial time [33]. We thus assume  $\mathcal{U}$  to be convex and rectangular in the remainder of this work. Specifically, we denote an  $(s,a)$ -rectangular uncertainty set by  $\mathcal{U}^{\mathrm{sa}}:= \times_{(s,a)\in \mathcal{X}}(\mathcal{P}_{(s,a)},\mathcal{R}_{(s,a)})$ . It represents a particular case of  $s$ -rectangular uncertainty which we similarly denote by  $\mathcal{U}^{\mathrm{s}}:= \times_{s\in S}(\mathcal{P}_s,\mathcal{R}_s)$ . In both cases, there exists an optimal robust policy that is stationary, although all optimal ones may be stochastic [33].
Similarly to non-robust MDPs, robust MDPs can be solved through Bellman recursion. Indeed, the robust value function  $v_{\mathcal{U}}^{\pi}$  (resp., optimal robust value function  $v_{\mathcal{U}}^{*}$ ) is known to be the unique fixed point of the robust Bellman operator  $T_{\mathcal{U}}^{\pi}v \coloneqq \min_{(P,R)\in \mathcal{U}}T_{(P,R)}^{\pi}v$  (resp., the optimal robust Bellman operator  $T_{\mathcal{U}}^{*}v \coloneqq \max_{\pi \in \Pi}T_{\mathcal{U}}^{\pi}v$ ,  $\forall v \in \mathbb{R}^{S}$ , both being  $\gamma$ -contractions for the sup-norm. Although this ensures linear convergence of robust value iteration, the evaluation of each Bellman operator can still be prohibitive for practical use.
3.2.1 Ball Constrained Uncertainty set
To facilitate the computation of robust Bellman updates, we consider uncertainty sets that are centered around a nominal model  $(P_0,R_0)$ , i.e., of the form  $\mathcal{U} = (P_0,R_0) + (\mathcal{P},\mathcal{R})$ , and constrained according
to  $\ell_p$ -norm balls [3, 14, 9, 1]. In the  $(s,a)$ -rectangular case, the corresponding uncertainty set is denoted by  $\mathcal{U}_p^{\mathrm{sa}} \coloneqq \mathcal{R}_p^{\mathrm{sa}} \times \mathcal{P}_p^{\mathrm{sa}} = \times_{(s,a) \in \mathcal{X}} (\mathcal{P}_{(s,a)}, \mathcal{R}_{(s,a)})$  where for any  $(s,a) \in \mathcal{X}$ ,
$$
\mathcal {R} _ {(s, a)} = \left\{r \in \mathbb {R} \mid | r | \leq \alpha_ {s, a} \right\}, \quad \text {a n d} \quad \mathcal {P} _ {(s, a)} = \left\{p \in \mathbb {R} ^ {\mathcal {S}} \mid \langle p, \mathbf {1} \rangle_ {\mathcal {S}} = 0, \| p \| _ {p} \leq \beta_ {s, a} \right\}.
$$
Similarly, an  $s$ -rectangular norm-constrained uncertainty is denoted by  $\mathcal{U}_p^{\mathrm{s}} \coloneqq \times_{s \in S} (\mathcal{P}_s, \mathcal{R}_s)$  where for any  $s \in S$ ,
$$
\mathcal {R} _ {s} = \left\{r \in \mathbb {R} ^ {\mathcal {A}} \mid \| r \| _ {p} \leq \alpha_ {s} \right\}, \quad \text {a n d} \quad \mathcal {P} _ {s} = \left\{p \in \mathbb {R} ^ {\mathcal {X}} \mid \langle p (\cdot , a), \mathbf {1} \rangle_ {\mathcal {S}} = 0 \quad \forall a \in \mathcal {A}, \| p \| _ {p} \leq \beta_ {s} \right\}.
$$
In both cases, the kernel uncertainty set conceals linear constraints ensuring all entries in  $P_0 + \mathcal{P}$  are non-negative. Indeed, we generally ignore  $P_0$  to satisfy these constraints in practice [23]. Although it may include absurd models and unnecessarily lead to conservative policies, this proxy region is appropriate for model-free robust learning. Moreover, the norm-ball structure on uncertainty sets above enables us to compute robust Bellman updates with similar time complexity as non-robust ones using regularization [3, 14]. We formalize this below.
Proposition 3.1. ([14, Thm. 2-3].) For any policy  $\pi \in \Pi$  and any rectangular  $\ell_p$ -ball-constraint uncertainty set, the robust Bellman operator is equivalent to its regularized form:
$$
(T _ {\mathcal {U}} ^ {\pi} v) (s) = T _ {(P _ {0}, R _ {0})} ^ {\pi} v (s) + \Omega_ {q} (\alpha , \beta , v),
$$
where  $\Omega_q(\alpha, \beta, v) := -\langle \pi_s, \alpha_{s'} + \gamma \kappa_q(v) \beta_{s'}^P \rangle_A$  for  $(s, a)$ -rectangular uncertainty  $\mathcal{U}_p^{sa}$ , and  $\Omega_q(\alpha, \beta, v) := -(\alpha_s + \gamma \beta_s \kappa_q(v)) \| \pi_s \|_q$  for  $s$ -rectangular uncertainty  $\mathcal{U}_p^s$ .
In the following, we leverage the regularized formulation of robust value functions to explicitly derive RPG for rectangular  $\ell_p$ -ball uncertainty sets.
3.2.2 Robust Gradient Method
Since the robust return can be non-differentiable, we need to follow the projected sub-gradient ascent rule in order to optimize the robust return, namely, update  $\pi_{k + 1} \coloneqq \mathbf{proj}_{\Pi}(\pi_k + \eta \partial_\pi \rho_{\mathcal{U}}^{\pi_k})$  where
$$
\left. \partial_ {\pi} \rho_ {\mathcal {U}} ^ {\pi} := \nabla_ {\pi} \rho_ {(P, R)} ^ {\pi} \right| _ {(P, R) = \left(P _ {\mathcal {U}} ^ {\pi}, R _ {\mathcal {U}} ^ {\pi}\right)}, \tag {1}
$$
$\eta$  is the learning rate,  $\mathbf{proj}_{\Pi}$  denotes the orthogonal projection on  $\Pi$ , and  $(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})$  is the worst model associated with  $\pi \in \Pi$  and  $\mathcal{U}$ , i.e.,  $(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi}) \in \arg \inf_{(P,R) \in \mathcal{U}} \rho_{(P,R)}^{\pi}$ .
Given oracle access to sub-gradient  $\partial \rho_{\mathcal{U}}^{\pi}$ , projected gradient ascent converges to an  $\epsilon$ -optimal policy  $\pi_{\mathcal{U}}^{*}$ . Moreover, under similar conditions as in the non-robust setting, projected gradient ascent holds an iteration complexity of  $O(S^4 A^2 \epsilon^{-4})$  [30]. Yet, the sub-gradient in (1) is generally intractable, particularly because general convex uncertainty sets may yield NP-hard complexity. Instead, we propose to focus on ball-constrained uncertainty sets in order to efficiently compute RPG updates.
4 Towards RPG: Expressing the worst quantities
In this section, we provide all the ingredients needed for deriving RPG. Before diving into the gradient expression, we first settle on the general framework of policy gradient. Secondly, in Sec. 4.1, we focus on expressing the worst model according to the nominal explicitly. Surprisingly, we find that the worst transition kernel is a rank-one perturbation of the nominal. This finding enables us to derive the robust occupancy measure, i.e., the visitation frequency of the worst kernel in Sec. 4.2. As a last piece, in Sec. 4.3, we propose an alternative definition of robust Q-value and show that it can be estimated from a specific Bellman recursion.
Consider again the projected gradient ascent rule:
$$
\pi_ {k + 1} := \mathbf {p r o j} _ {\Pi} \left(\pi_ {k} + \eta \partial_ {\pi} \rho_ {\mathcal {U}} ^ {\pi_ {k}}\right).
$$
By definition of the sub-gradient in (1) and applying the standard PG theorem [26], it holds that:
$$
\partial_ {\pi} \rho_ {\mathcal {U}} ^ {\pi} = \sum_ {(s, a) \in \mathcal {X}} d _ {\mathcal {U}} ^ {\pi} (s) Q _ {\mathcal {U}} ^ {\pi} (s, a) \nabla \pi_ {s} (a),
$$
where  $Q_{\mathcal{U}}^{\pi} \coloneqq Q_{(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})}^{\pi}$  is the Q-value associated with the worst-case model, and  $d_{\mathcal{U}}^{\pi} \coloneqq d_{P_{\mathcal{U}}^{\pi}}^{\pi}$  the occupation measure of the worst transition kernel. In fact, for the uncertainty sets we focus on in this work, the worst Q-value  $Q_{\mathcal{U}}^{\pi}$  retrieves the common definition of robust Q-value [20, 28] (see the appendix for a detailed discussion). Therefore, for conciseness and with a slight abuse, we shall designate  $Q_{\mathcal{U}}^{\pi}$  by the robust Q-value, and  $d_{\mathcal{U}}^{\pi}$  by the robust visitation frequency. The remaining question is how to compute these quantities and in particular, can we efficiently find the worst parameters  $(P_{\mathcal{U}}^{\pi}, R_{\mathcal{U}}^{\pi})$ ? The following part of our study aims to address these questions.
Given an uncertainty set  $\mathcal{U}$ , let first define the normalized and balanced robust value function as:
$$
u _ {\mathcal {U}} ^ {\pi} (s) := \frac {\operatorname {s i g n} \left(v _ {\mathcal {U}} ^ {\pi} (s) - \omega_ {q} \left(v _ {\mathcal {U}} ^ {\pi}\right)\right) \left\| v _ {\mathcal {U}} ^ {\pi} (s) - \omega_ {q} \left(v _ {\mathcal {U}} ^ {\pi}\right) \right\| ^ {q - 1}}{\kappa_ {q} \left(v _ {\mathcal {U}} ^ {\pi}\right) ^ {q - 1}}. \tag {2}
$$