SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Using duality results for coherent risk measures [42], we show that the RAMU Bellman operators  $\mathcal{T}_{\rho^{+},r}^{\pi}$  and  $\mathcal{T}_{\rho ,c}^{\pi}$  are equivalent to distributionally robust Bellman operators [53, 56] with ambiguity sets of distributions  $\mathcal{U}^+$  and  $\mathcal{U}$ , respectively. The RAMU Q functions are the respective fixed points of these Bellman operators, so they can be written as distributionally robust Q functions. Finally, by averaging over initial states and actions, we see that (3) is equivalent to (6). See the Appendix for details.
Theorem 1 shows that the application of  $\rho^{+}$  and  $\rho$  at every timestep are equivalent to solving distributionally robust optimization problems over the ambiguity sets of distributions  $\mathcal{U}^{+}$  and  $\mathcal{U}$ , respectively. This can be interpreted as adversarially reweighting  $\mu_{s,a}$  with  $\xi_{s,a}$  at every state-action pair. Note that worst-case distributions appear in both the objective and constraint of (6), so any policy trained with our RAMU framework is guaranteed to deliver robust performance and robust safety. The level of robustness depends on the choice of  $\rho^{+}$  and  $\rho$ , which determine the structure and size of the corresponding ambiguity sets based on their dual representations [42].
In addition, because (3) is equivalent to a distributionally robust safe RL problem according to Theorem 1, we can leverage existing results for distributionally robust MDPs [53, 56] to show that  $\mathcal{T}_{\rho^{+},r}^{\pi}$  and  $\mathcal{T}_{\rho ,c}^{\pi}$  are contraction operators.
Corollary 1. The RAMU Bellman operators  $\mathcal{T}_{\rho^{+},r}^{\pi}$  and  $\mathcal{T}_{\rho ,c}^{\pi}$  are  $\gamma$  -contractions in the sup-norm.
Proof. Apply results from Xu and Mannor [53] and Yu and Xu [56]. See the Appendix for details.
Therefore, we have that  $Q_{\rho^{+},r}^{\pi}(s,a)$  and  $Q_{\rho ,c}^{\pi}(s,a)$  can be interpreted as distributionally robust Q functions by Theorem 1, and we can apply standard temporal difference methods to learn these RAMU Q functions as a result of Corollary 1. Importantly, Theorem 1 demonstrates the robustness properties of our RAMU framework, but it is not used to implement our approach. Directly implementing (6) would require solving for adversarial distributions over transition models throughout training. Instead, our framework provides the same robustness, but the use of risk measures leads to an efficient deep RL implementation as we describe in the following section.
6 Model-free implementation with a single training environment
The RAMU policy update in (4) takes the same form as the standard safe RL update in (2), except for the use of  $Q_{\rho^{+},r}^{\pi}(s,a)$  and  $Q_{\rho ,c}^{\pi}(s,a)$ . Because our RAMU Bellman operators are contractions, we can learn these RAMU Q functions by applying standard temporal difference loss functions that are used throughout deep RL. In particular, we consider parameterized critics  $Q_{\theta_r}$  and  $Q_{\theta_c}$ , and we optimize their parameters during training to minimize the loss functions
$$
\mathcal {L} ^ {+} (\theta_ {r}) = \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \left[ \left(Q _ {\theta_ {r}} (s, a) - \hat {\mathcal {T}} _ {\rho^ {+}, r} ^ {\pi} \bar {Q} _ {\theta_ {r}} (s, a)\right) ^ {2} \right],
$$
$$
\mathcal {L} (\theta_ {c}) = \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \left[ \left(Q _ {\theta_ {c}} (s, a) - \hat {\mathcal {T}} _ {\rho , c} ^ {\pi} \bar {Q} _ {\theta_ {c}} (s, a)\right) ^ {2} \right],
$$
where  $\hat{T}_{\rho^{+},r}^{\pi}$  and  $\hat{T}_{\rho,c}^{\pi}$  represent sample-based estimates of the RAMU Bellman operators applied to target Q functions denoted by  $\bar{Q}$ . Therefore, we must be able to efficiently estimate the RAMU Bellman targets, which involve calculating coherent distortion risk measures that depend on the distribution  $\mu_{s,a}$ .
Sample-based estimation of risk measures Using the formulation of our RAMU Bellman operators from Lemma 1, we can leverage properties of distortion risk measures to efficiently estimate the results in (5) using sample-based weighted averages of standard Bellman targets. For  $n$  transition models  $p_{s,a}^{(i)}, i = 1,\dots ,n$ , sampled independently from  $\mu_{s,a}$  and sorted according to their corresponding Bellman targets, consider the weights
$$
w _ {\rho} ^ {(i)} = g \left(\frac {i}{n}\right) - g \left(\frac {i - 1}{n}\right),
$$
Figure 1: Coherent distortion risk measures used in RAMU experiments. Left: Distortion function  $g$ . Right: Weights for sample-based estimates in (7) when  $n = 5$ .
where  $g$  defines the distortion risk measure  $\rho$  according to Definition 2. See Figure 1 for the distortion functions and weights associated with the risk measures used in our experiments. Then, from Jones and Zitikis [23] we have that
$$
\sum_ {i = 1} ^ {n} w _ {\rho^ {+}} ^ {(i)} \mathcal {T} _ {p ^ {(i)}, r} ^ {\pi} Q (s, a), \quad \sum_ {i = 1} ^ {n} w _ {\rho} ^ {(i)} \mathcal {T} _ {p ^ {(i)}, c} ^ {\pi} Q (s, a),
$$
are consistent estimators of the results in (5), where  $\mathcal{T}_{p^{(i)},r}^{\pi}Q(s,a)$  are sorted in ascending order and  $\mathcal{T}_{p^{(i)},c}^{\pi}Q(s,a)$  are sorted in descending order. Finally, we can replace  $\mathcal{T}_{p^{(i)},r}^{\pi}Q(s,a)$  and  $\mathcal{T}_{p^{(i)},c}^{\pi}Q(s,a)$  with the standard unbiased sample-based estimates
$$
\hat {\mathcal {T}} _ {p ^ {(i)}, r} ^ {\pi} Q (s, a) = r (s, a) + \gamma Q (s ^ {\prime}, a ^ {\prime}), \quad \hat {\mathcal {T}} _ {p ^ {(i)}, c} ^ {\pi} Q (s, a) = c (s, a) + \gamma Q (s ^ {\prime}, a ^ {\prime}),
$$
where  $s' \sim p_{s,a}^{(i)}$  and  $a' \sim \pi(\cdot | s')$ . This leads to the sample-based estimates
$$
\hat {\mathcal {T}} _ {\rho^ {+}, r} ^ {\pi} Q (s, a) = \sum_ {i = 1} ^ {n} w _ {\rho^ {+}} ^ {(i)} \hat {\mathcal {T}} _ {p ^ {(i)}, r} ^ {\pi} Q (s, a), \quad \hat {\mathcal {T}} _ {\rho , c} ^ {\pi} Q (s, a) = \sum_ {i = 1} ^ {n} w _ {\rho} ^ {(i)} \hat {\mathcal {T}} _ {p ^ {(i)}, c} ^ {\pi} Q (s, a), \tag {7}
$$
which we use to train our RAMU Q functions. Note that the estimates in (7) can be computed very efficiently, which is a major benefit of our RAMU framework compared to robust RL methods. Next, we describe how we can sample models  $p_{s,a}^{(i)}, i = 1,\dots,n$ , from  $\mu_{s,a}$ , and generate state transitions from these models to use in the calculation of our sample-based Bellman targets in (7).
Generative distribution of transition models Note that our RAMU framework can be applied using any choice of distribution  $\mu$ , provided we can sample transition models  $p_{s,a}^{(i)} \sim \mu_{s,a}$  and corresponding next states  $s' \sim p_{s,a}^{(i)}$ . In this work, we define the distribution  $\mu$  over perturbed versions of a single training environment  $p^{\mathrm{train}}$ , and we propose a generative approach to sampling transition models and corresponding next states that only requires data collected from  $p^{\mathrm{train}}$ . By doing so, our RAMU framework achieves robust performance and safety with minimal assumptions on the training process, and can even be applied to settings that require real-world data collection for training.
We consider a latent variable  $x \sim X$ , and we define a transition model  $p_{s,a}(x)$  for every  $x \sim X$  that shifts the probability of  $s'$  under  $p_{s,a}^{\mathrm{train}}$  according to a perturbation function  $f_x: \mathcal{S} \times \mathcal{S} \to \mathcal{S}$ . This perturbation function takes as input a state transition  $(s, s')$ , and outputs a perturbed next state  $\tilde{s}'$  that depends on the latent variable  $x \sim X$ . Therefore, a distribution over latent space implicitly defines a distribution  $\mu_{s,a}$  over perturbed versions of  $p_{s,a}^{\mathrm{train}}$ . In order to obtain the next state samples needed to compute the Bellman target estimates in (7), we sample latent variables  $x \sim X$  and apply  $f_x$  to the state transition observed in the training environment. We have that  $s' \sim p_{s,a}^{\mathrm{train}}$  for data collected in the training environment, so  $\tilde{s}' = f_x(s, s')$  represents the corresponding sample from the perturbed transition model  $p_{s,a}(x)$ .
In our experiments, we consider a simple implementation for the common case where  $S = \mathbb{R}^d$ . We use uniformly distributed latent variables  $x \sim U([-2\epsilon, 2\epsilon]^d)$ , and we define the perturbation function as
$$
f _ {x} (s, s ^ {\prime}) = s + (s ^ {\prime} - s) (1 + x),
$$
Algorithm 1 Risk-Averse Model Uncertainty for Safe RL
Input: policy  $\pi_0$  , critics  $Q_{\theta_r}$ $Q_{\theta_c}$  , risk measures  $\rho^{+},\rho$  , latent random variable  $X$    
for  $k = 0,1,2,\ldots$  do Collect data  $\tau \sim (\pi_k,p^{\mathrm{train}})$  and store it in  $\mathcal{D}$    
for  $K$  updates do Sample batch of data  $(s,a,r,c,s^{\prime})\sim \mathcal{D}$  Sample  $n$  latent variables  $x_{i}\sim X$  per data point, and compute next state samples  $f_{x_i}(s,s')$  Calculate Bellman targets in (7), and update critics  $Q_{\theta_r}$ $Q_{\theta_c}$  to minimize  $\mathcal{L}^{+}(\theta_{r}),\mathcal{L}(\theta_{c})$  Update policy  $\pi$  according to (4)   
end for   
end for
where all operations are performed per-coordinate. Therefore, the latent variable  $x \sim U([-2\epsilon, 2\epsilon]^d)$  can be interpreted as the percentage change in each dimension of a state transition observed in the training environment, where the average magnitude of the percentage change is  $\epsilon$ . The hyperparameter  $\epsilon$  determines the distribution  $\mu_{s,a}$  over transition models, where a larger value of  $\epsilon$  leads to transition models that vary more significantly from the training environment. The structure of  $f_x$  provides an intuitive, scale-invariant meaning for the hyperparameter  $\epsilon$ , which makes it easy to tune in practice. This choice of distribution  $\mu_{s,a}$  captures general uncertainty in the training environment, without requiring specific domain knowledge of potential disturbances.
Algorithm We summarize the implementation of our RAMU framework in Algorithm 1. Given data collected in a single training environment, we can efficiently calculate the sample-based RAMU Bellman targets in (7) by (i) sampling from a latent variable  $x \sim X$ , (ii) computing the corresponding next state samples  $f_{x}(s,s^{\prime})$ , and (iii) sorting the standard Bellman estimates that correspond to these sampled transition models. Given the sample-based RAMU Bellman targets, updates of the critics and policy have the same form as in standard deep safe RL algorithms. Therefore, our RAMU framework can be easily combined with many popular safe RL algorithms to incorporate model uncertainty with robustness guarantees, using only a minor change to the estimation of Bellman targets that is efficient to implement in practice.
In order to evaluate the performance and safety of our RAMU framework, we conduct experiments on 5 continuous control tasks with safety constraints from the Real-World RL Suite [18, 19]: Cartpole Swingup, Walker Walk, Walker Run, Quadruped Walk, and Quadruped Run. Each task has a horizon length of 1,000 with  $r(s,a) \in [0,1]$  and  $c(s,a) \in \{0,1\}$ , and we consider a safety budget of  $B = 100$ . Unless noted otherwise, we train these tasks on a single training environment for 1 million steps across 5 random seeds, and we evaluate performance of the learned policies across a range of perturbed test environments via 10 trajectory rollouts. See the Appendix for information on the safety constraints and environment perturbations that we consider.
Our RAMU framework can be combined with several choices of safe RL algorithms. We consider the safe RL algorithm Constraint-Rectified Policy Optimization (CRPO) [55], and we use Maximum a Posteriori Policy Optimization (MPO) [1] as the unconstrained policy optimization algorithm in CRPO. For a fair comparison, we apply this choice of safe RL policy update in every method we consider in our experiments. We use a multivariate Gaussian policy with learned mean and diagonal covariance at each state, along with separate reward and cost critics. We parameterize our policy and critics using neural networks. See the Appendix for implementation details. $^{1}$
We summarize the performance and safety of our RAMU framework in Table 1 and Figure 2, compared to several baseline algorithms that we discuss next. We include detailed experimental results across all perturbed test environments in the Appendix. We apply our RAMU framework using the Wang transform with  $\eta = 0.75$  as the risk measure in both the objective and constraint. In order
$^{1}$ Code is publicly available at https://github.com/jqueeney/robust-safe-r1.
Table 1: Aggregate performance summary
Algorithm	% Safe†	Normalized Ave.‡	Rollouts Require*
Reward	Cost	Adversary	Simulator
Safe RL	51%	1.00	1.00	No	No
RAMU (Wang 0.75)	80%	1.08	0.51	No	No
RAMU (Expectation)	74%	1.05	0.67	No	No
Domain Randomization	76%	1.14	0.72	No	Yes
Domain Randomization (OOD)	55%	1.02	1.02	No	Yes
Adversarial RL	82%	1.05	0.48	Yes	No
† Percentage of policies that satisfy the safety constraint across all tasks and test environments.
${}^{ \ddagger  }$  Normalized relative to the average performance of standard safe RL for each task and test environment.
* Denotes need for adversary or simulator during data collection (i.e., trajectory rollouts) for training.
Figure 2: Performance summary by task, aggregated across perturbed test environments. Performance of adversarial RL is evaluated without adversarial interventions. Top: Total rewards averaged across test environments. Bottom: Percentage of policies across test environments that satisfy the safety constraint.
to understand the impact of being risk-averse to model uncertainty, we also consider the risk-neutral special case of our framework where expectations are applied to the objective and constraint. For our RAMU results in Table 1 and Figure 2, we specify the risk measure in parentheses. Finally, we consider  $n = 5$  samples of transition models with latent variable hyperparameter  $\epsilon = 0.10$  in order to calculate Bellman targets in our RAMU framework.
Comparison to safe reinforcement learning First, we analyze the impact of our RAMU framework compared to standard safe RL. In both cases, we train policies using data collected from a single training environment, so the only difference comes from our use of risk-averse model uncertainty to learn RAMU Q functions. By evaluating the learned policies in perturbed test environments different from the training environment, we see that our RAMU framework provides robustness in terms of both total rewards and safety. In particular, the risk-averse implementation of our algorithm leads to safety constraint satisfaction in  $80\%$  of test environments, compared to only  $51\%$  with standard safe RL. In addition, this implementation results in higher total rewards (1.08x) and lower total costs (0.51x), on average. We see in Table 1 that the use of expectations over model uncertainty (i.e., a risk-neutral approach) also improves robustness in both the objective and constraint, on average, compared to standard safe RL. However, we further improve upon the benefits observed in the risk-neutral case by instead applying a risk-averse perspective.
Comparison to domain randomization Next, we compare our RAMU framework to domain randomization, a popular approach that also represents model uncertainty using a distribution  $\mu$  over models. Note that domain randomization considers parametric uncertainty and has the benefit of training on a range of simulated environments, while our method only collects data from a single training environment. In order to evaluate the importance of domain knowledge for defining the training distribution in domain randomization, we consider two different cases: an in-distribution
version that trains on a subset of the perturbed test environments, and an out-of-distribution (OOD) version that randomizes over a different perturbation parameter than the one varied at test time.
The results in Table 1 and Figure 2 show the importance of domain knowledge: in-distribution domain randomization leads to improved robustness compared to standard safe RL and the highest normalized average rewards  $(1.14\mathrm{x})$ , while the out-of-distribution version provides little benefit. In both cases, however, domain randomization achieves lower levels of safety, on average, than our risk-averse formulation. In fact, we see in Figure 2 that the safety constraint satisfaction of our risk-averse formulation is at least as strong as both versions of domain randomization in 4 out of 5 tasks, despite only training on a single environment with no specific knowledge about the disturbances at test time. This demonstrates the key benefit of our risk-averse approach to model uncertainty.
Comparison to adversarial reinforcement learning Finally, we compare our approach to adversarial RL using the action-robust PR-MDP framework [46], which randomly applies worst-case actions a percentage of the time during data collection. Although adversarial RL only collects data from a single training environment, it requires potentially dangerous adversarial interventions during training in order to provide robustness at test time. In order to apply this method to the safe RL setting, we train an adversary to maximize costs and consider a  $5\%$  probability of intervention during training. The performance of adversarial RL is typically evaluated without adversarial interventions, which requires a clear distinction between training and testing.
We see in Figure 2 that adversarial RL learns policies that achieve robust safety constraint satisfaction at test time in the Quadruped tasks. Our risk-averse formulation, on the other hand, achieves higher levels of safety in the remaining 3 out of 5 tasks, and similar levels of safety on average. Unlike adversarial RL, our RAMU framework achieves robust safety in a way that (i) does not alter the data collection process, (ii) does not require training an adversary in a minimax formulation, and (iii) does not require different implementations during training and testing. In addition, our use of a distribution over models represents a less conservative approach than adversarial RL, resulting in higher normalized average rewards as shown in Table 1.
We have presented a framework for safe RL in the presence of model uncertainty, an important setting for many real-world decision making applications. Compared to existing approaches to model uncertainty in deep RL, our formulation applies a risk-averse perspective through the use of coherent distortion risk measures. We show that this results in robustness guarantees, while still leading to an efficient deep RL implementation that does not involve minimax optimization problems. Importantly, our method only requires data collected from a single training environment, so it can be applied to real-world domains where high-fidelity simulators are not readily available or are computationally expensive. Therefore, our framework represents an attractive approach to safe decision making under model uncertainty that can be deployed across a range of applications.
Prior to potential deployment, it is important to understand the limitations of our proposed methodology. The robustness and safety of our RAMU framework depend on the user-defined choices of model distribution  $\mu$  and risk measure  $\rho$ . The distribution  $\mu$  defines the uncertainty over transition models, and the risk measure  $\rho$  defines the level of robustness to this choice of  $\mu$ . In addition, our approach only considers robustness with respect to model uncertainty and safety as defined by expected total cost constraints. It would be interesting to extend our techniques to address other forms of uncertainty and other definitions of safety, including epistemic uncertainty in model-based RL, observational uncertainty, and safety-critical formulations based on sets of unsafe states.
[1] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum a posteriori policy optimisation. In Sixth International Conference on Learning Representations, 2018.
[2] A. Abdolmaleki, S. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view on multi-objective policy
[54] J. Xu, S. Kim, T. Chen, A. Rodriguez, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In The Conference on Robot Learning (CoRL), 2022.
[55] T. Xu, Y. Liang, and G. Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In Proceedings of the 38th International Conference on Machine Learning, pages 11480-11491. PMLR, 2021.
[56] P. Yu and H. Xu. Distributionally robust counterpart in Markov decision processes. IEEE Transactions on Automatic Control, 61(9):2538-2543, 2016. doi: 10.1109/TAC.2015.2495174.
A Properties of coherent distortion risk measures
Majumdar and Pavone [30] proposed a set of six axioms to characterize desirable properties of risk measures in the context of robotics.
A1. Monotonicity: If  $Z, Z' \in \mathcal{Z}$  and  $Z \leq Z'$  almost everywhere, then  $\rho(Z) \leq \rho(Z')$ .
A2. Translation invariance: If  $\alpha \in \mathbb{R}$  and  $Z\in \mathcal{Z}$ , then  $\rho (Z + \alpha) = \rho (Z) + \alpha$
A3. Positive homogeneity: If  $\tau \geq 0$  and  $Z\in \mathcal{Z}$ , then  $\rho (\tau Z) = \tau \rho (Z)$ .
A4. Convexity: If  $\lambda \in [0,1]$  and  $Z,Z^{\prime}\in \mathcal{Z}$ , then  $\rho (\lambda Z + (1 - \lambda)Z^{\prime})\leq \lambda \rho (Z) + (1 - \lambda)\rho (Z^{\prime})$
A5. Comonotonic additivity: If  $Z, Z' \in \mathcal{Z}$  are comonotonic, then  $\rho(Z + Z') = \rho(Z) + \rho(Z')$ .
A6. Law invariance: If  $Z, Z' \in \mathcal{Z}$  are identically distributed, then  $\rho(Z) = \rho(Z')$ .
See Majumdar and Pavone [30] for a discussion on the intuition behind these axioms. Note that coherent risk measures [6] satisfy Axioms A1-A4, distortion risk measures [17, 50] satisfy Axioms A1-A3 and Axioms A5-A6, and coherent distortion risk measures satisfy all six axioms.
The properties of coherent risk measures also lead to a useful dual representation.
Lemma 2 (Shapiro et al. [42]). Let  $\rho$  be a proper, real-valued coherent risk measure. Then, for any  $Z\in \mathcal{Z}$  we have that
$$
\rho (Z) = \sup_{\beta_{s,a}\in \mathcal{U}_{s,a}}\mathbb{E}_{\beta_{s,a}}\left[ \begin{array}{c}Z \end{array} \right],
$$
where  $\mathbb{E}_{\beta_{s,a}}[\cdot ]$  represents expectation with respect to the probability measure  $\beta_{s,a}\in P(\mathcal{M})$ , and
$$
\mathcal {U} _ {s, a} \subseteq \left\{\beta_ {s, a} \in P (\mathcal {M}) \mid \beta_ {s, a} = \xi_ {s, a} \mu_ {s, a}, \xi_ {s, a} \in \mathcal {Z} ^ {*} \right\}
$$
is a convex, bounded, and weakly* closed set that depends on  $\rho$ .
See Shapiro et al. [42] for a general treatment of this result.
In this section, we prove all results related to the RAMU cost Bellman operator  $\mathcal{T}_{\rho,c}^{\pi}$ . Using the fact that  $\rho^{+}(Z) = -\rho(-Z)$  for a coherent distortion risk measure  $\rho$  on a cost random variable, all results related to the RAMU reward Bellman operator follow by an appropriate change in sign.
Proof. Starting from the definition of  $\mathcal{T}_{\rho,c}^{\pi}$  in Definition 3, we have that