SlowGuess/NeurIPS_2023_OCR huggingface dataset:
James Queeney*  
Division of Systems Engineering  
Boston University  
jqueeney@bu.edu
Mouhacine Benosman Mitsubishi Electric Research Laboratories benosman@merl.com
Many real-world domains require safe decision making in uncertain environments. In this work, we introduce a deep reinforcement learning framework for approaching this important problem. We consider a distribution over transition models, and apply a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures. We provide robustness guarantees for this framework by showing it is equivalent to a specific class of distributionally robust safe reinforcement learning problems. Unlike existing approaches to robustness in deep reinforcement learning, however, our formulation does not involve minimax optimization. This leads to an efficient, model-free implementation of our approach that only requires standard data collection from a single training environment. In experiments on continuous control tasks with safety constraints, we demonstrate that our framework produces robust performance and safety at deployment time across a range of perturbed test environments.
In many real-world decision making applications, it is important to satisfy safety requirements while achieving a desired goal. In addition, real-world environments often involve uncertain or changing conditions. Therefore, in order to reliably deploy data-driven decision making methods such as deep reinforcement learning (RL) in these settings, they must deliver robust performance and safety even in the presence of uncertainty. Recently, techniques have been developed to handle safety constraints within the deep RL framework [3, 27, 28, 41, 44, 47, 55], but these safe RL algorithms only focus on performance and safety in the training environment. They do not consider uncertainty about the true environment at deployment time due to unknown disturbances or irreducible modeling errors, which we refer to as model uncertainty. In this work, we introduce a framework that incorporates model uncertainty into safe RL. In order for our framework to be useful, we emphasize the importance of (i) an efficient deep RL implementation during training and (ii) robustness guarantees on performance and safety upon deployment.
Existing robust RL methods address the issue of model uncertainty, but they can be difficult to implement and are not always suitable in real-world decision making settings. Robust RL focuses on worst-case environments in an uncertainty set, which requires solving complex minimax optimization problems throughout training. This is typically approximated in a deep RL setting through direct interventions with a learned adversary [38, 46, 49], or through the use of parametric uncertainty with multiple simulated training environments [31, 32, 39]. However, we do not always have access to fast, high-fidelity simulators for training [10, 33, 54]. In these cases, we must be able to incorporate robustness to model uncertainty without relying on multiple training environments or potentially dangerous adversarial interventions, as real-world data collection may be necessary.
*Work partly done during an internship at Mitsubishi Electric Research Laboratories.
A more informative way to represent model uncertainty is to instead consider a distribution over potential environments. Domain randomization [37] collects training data from a range of environments by randomizing across parameter values in a simulator, and optimizes for average performance. This approach to model uncertainty avoids minimax formulations and works well in practice [5], but lacks robustness guarantees. In addition, domain randomization focuses on parametric uncertainty, which still requires detailed simulator access and domain knowledge to define the training distribution.
In this work, we introduce a general approach to safe RL in the presence of model uncertainty that addresses the main shortcomings of existing methods. In particular, we consider a distribution over potential environments, and apply a risk-averse perspective towards model uncertainty. Through the use of coherent distortion risk measures, this leads to a safe RL framework with robustness guarantees that does not involve difficult minimax formulations. Using this framework, we show how we can learn safe policies that are robust to model uncertainty, without the need for detailed simulator access or adversarial interventions during training. Our main contributions are as follows:
1. We reformulate the safe RL problem to incorporate a risk-averse perspective towards model uncertainty through the use of coherent distortion risk measures, and we introduce the corresponding Bellman operators.
2. From a theoretical standpoint, we provide robustness guarantees for our framework by showing it is equivalent to a specific class of distributionally robust safe RL problems.
3. We propose an efficient deep RL implementation that avoids the difficult minimax formulation present in robust RL and only uses data collected from a single training environment.
4. We demonstrate the robust performance and safety of our framework through experiments on continuous control tasks with safety constraints in the Real-World RL Suite [18, 19].
Safe reinforcement learning In this work, we consider RL in the presence of safety constraints. We model this sequential decision making problem as an infinite-horizon, discounted Constrained Markov Decision Process (CMDP) [4] defined by the tuple  $(S, \mathcal{A}, p, r, c, d_0, \gamma)$ , where  $S$  is the set of states,  $\mathcal{A}$  is the set of actions,  $p: S \times \mathcal{A} \to P(S)$  is the transition model where  $P(S)$  represents the space of probability measures over  $S$ ,  $r, c: S \times \mathcal{A} \to \mathbb{R}$  are the reward function and cost function used to define the objective and constraint, respectively,  $d_0 \in P(S)$  is the initial state distribution, and  $\gamma$  is the discount rate. We focus on the setting with a single constraint, but all results can be extended to the case of multiple constraints.
We model the agent's decisions as a stationary policy  $\pi : S \to P(\mathcal{A})$ . For a given CMDP and policy  $\pi$ , we write the expected total discounted rewards and costs as  $J_{p,r}(\pi) = \mathbb{E}_{\tau \sim (\pi, p)} \left[ \sum_{t=0}^{\infty} \gamma^{t} r(s_{t}, a_{t}) \right]$  and  $J_{p,c}(\pi) = \mathbb{E}_{\tau \sim (\pi, p)} \left[ \sum_{t=0}^{\infty} \gamma^{t} c(s_{t}, a_{t}) \right]$ , respectively, where  $\tau \sim (\pi, p)$  represents a trajectory sampled according to  $s_{0} \sim d_{0}$ ,  $a_{t} \sim \pi(\cdot | s_{t})$ , and  $s_{t+1} \sim p(\cdot | s_{t}, a_{t})$ . The goal of safe RL is to find a policy  $\pi$  that maximizes the constrained optimization problem
$$
\max  _ {\pi} J _ {p, r} (\pi) \quad \text {s . t .} \quad J _ {p, c} (\pi) \leq B, \tag {1}
$$
where  $B$  is a safety budget on expected total discounted costs.
We write the corresponding state-action value functions (i.e.,  $Q$  functions) for a given transition model  $p$  and policy  $\pi$  as  $Q_{p,r}^{\pi}(s,a)$  and  $Q_{p,c}^{\pi}(s,a)$ , respectively. Off-policy optimization techniques [28, 55] find a policy that maximizes (1) by solving at each iteration the related optimization problem
$$
\max  _ {\pi} \mathbb {E} _ {s \sim \mathcal {D}} \left[ \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ Q _ {p, r} ^ {\pi_ {k}} (s, a) \right] \right] \quad \text {s . t .} \quad \mathbb {E} _ {s \sim \mathcal {D}} \left[ \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ Q _ {p, c} ^ {\pi_ {k}} (s, a) \right] \right] \leq B, \tag {2}
$$
where  $\pi_{k}$  is the current policy and  $\mathcal{D}$  is a replay buffer containing data collected in the training environment. Note that  $Q_{p,r}^{\pi}(s,a)$  and  $Q_{p,c}^{\pi}(s,a)$  are the respective fixed points of the Bellman operators
$$
\mathcal {T} _ {p, r} ^ {\pi} Q (s, a) := r (s, a) + \gamma_ {s ^ {\prime} \sim p _ {s, a}} \mathbb {E} _ {\left[ \begin{array}{c} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ Q (s ^ {\prime}, a ^ {\prime}) ] \end{array} \right]},
$$
$$
\mathcal {T} _ {p, c} ^ {\pi} Q (s, a) := c (s, a) + \gamma \underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} [ Q (s ^ {\prime}, a ^ {\prime}) ] \right].
$$
Model uncertainty in reinforcement learning Rather than focusing on a single CMDP with transition model  $p$ , we incorporate uncertainty about the transition model by considering a distribution  $\mu$  over models. We focus on distributions of the form  $\mu = \prod_{(s,a)\in S\times A}\mu_{s,a}$ , where  $\mu_{s,a}$  represents a distribution over transition models  $p_{s,a} = p(\cdot | s,a)\in P(\mathcal{S})$  at a given state-action pair and  $\mu$  is the product over all  $\mu_{s,a}$ . This is known as rectangularity, and is a common assumption in the literature [11, 15, 16, 53, 56]. Note that  $\mu_{s,a}\in P(\mathcal{M})$ , where we write  $\mathcal{M} = P(\mathcal{S})$  to denote model space. Compared to robust RL methods that apply uncertainty sets over transition models, the use of a distribution  $\mu$  over transition models is a more informative way to represent model uncertainty that does not require solving for worst-case environments (i.e., does not introduce a minimax formulation).
In order to incorporate robustness to the choice of  $\mu$ , distributionally robust MDPs [53, 56] consider an ambiguity set  $\mathcal{U} = \bigotimes_{(s,a) \in S \times \mathcal{A}} \mathcal{U}_{s,a}$  of distributions over transition models, where  $\mu_{s,a} \in \mathcal{U}_{s,a} \subseteq P(\mathcal{M})$ . The goal of distributionally robust RL is to optimize the worst-case average performance across all distributions contained in  $\mathcal{U}$ . In this work, we will show that a risk-averse perspective towards model uncertainty defined by  $\mu$  is equivalent to distributionally robust RL for appropriate choices of ambiguity sets in the objective and constraint of a CMDP. However, our use of risk measures avoids the need to solve for worst-case distributions in  $\mathcal{U}$  throughout training.
Risk measures Consider the probability space  $(\mathcal{M},\mathcal{F},\mu_{s,a})$ , where  $\mathcal{F}$  is a  $\sigma$ -algebra on  $\mathcal{M}$  and  $\mu_{s,a} \in P(\mathcal{M})$  defines a probability measure over  $\mathcal{M}$ . Let  $\mathcal{Z}$  be a space of random variables defined on this probability space, and let  $\mathcal{Z}^*$  be its corresponding dual space. A real-valued risk measure  $\rho: \mathcal{Z} \to \mathbb{R}$  summarizes a random variable as a value on the real line. In this section, we consider cost random variables  $Z \in \mathcal{Z}$  where a lower value of  $\rho(Z)$  is better. We can define a corresponding risk measure  $\rho^+$  for reward random variables through an appropriate change in sign, where  $\rho^+(Z) = -\rho(-Z)$ . Risk-sensitive methods typically focus on classes of risk measures with desirable properties [30], such as coherent risk measures [6] and distortion risk measures [17, 50].
Definition 1 (Coherent risk measure). A risk measure  $\rho$  is a coherent risk measure if it satisfies monotonicity, translation invariance, positive homogeneity, and convexity.
Definition 2 (Distortion risk measure). Let  $g:[0,1]\to [0,1]$  be a non-decreasing, left-continuous function with  $g(0) = 0$  and  $g(1) = 1$ . A distortion risk measure with respect to  $g$  is defined as
$$
\rho (Z) = \int_ {0} ^ {1} F _ {Z} ^ {- 1} (u) d \tilde {g} (u),
$$
where  $F_Z^{-1}$  is the inverse cumulative distribution function of  $Z$  and  $\tilde{g}(u) = 1 - g(1 - u)$ .
A distortion risk measure is coherent if and only if  $g$  is concave [52]. In this work, we focus on the class of coherent distortion risk measures. We will leverage properties of coherent risk measures to provide robustness guarantees for our framework, and we will leverage properties of distortion risk measures to propose an efficient, model-free implementation that does not involve minimax optimization. See the Appendix for additional details on the properties of coherent distortion risk measures. Many commonly used risk measures belong to this class, including expectation, conditional value-at-risk (CVaR), and the Wang transform [51] for  $\eta \geq 0$  which is defined by the distortion function  $g_{\eta}(u) = \Phi (\Phi^{-1}(u) + \eta)$ , where  $\Phi$  is the standard Normal cumulative distribution function.
Safe reinforcement learning The CMDP framework is the most popular approach to safety in RL, and several deep RL algorithms have been developed to solve the constrained optimization problem in (1). These include primal-dual methods that consider the Lagrangian relaxation of (1) [41, 44, 47], algorithms that compute closed-form solutions to related or approximate versions of (1) [3, 28], and direct methods for constraint satisfaction such as the use of barriers [27] or immediate switching between the objective and constraint [55]. All of these approaches are designed to satisfy expected cost constraints for a single CMDP observed during training. In our work, on the other hand, we consider a distribution over possible transition models.
Uncertainty in reinforcement learning Our work focuses on irreducible uncertainty about the true environment at deployment time, which we refer to as model uncertainty and represent using a
distribution  $\mu$  over transition models. The most popular approach that incorporates model uncertainty in this way is domain randomization [37, 48], which randomizes across parameter values in a simulator and trains a policy to maximize average performance over this training distribution. This represents a risk-neutral attitude towards model uncertainty, which has been referred to as a soft-robust approach [16]. Distributionally robust MDPs incorporate robustness to the choice of  $\mu$  by instead considering a set of distributions [11, 15, 53, 56], but application of this distributionally robust framework has remained limited in deep RL as it leads to a difficult minimax formulation that requires solving for worst-case distributions over transition models.
Robust RL represents an alternative approach to model uncertainty that considers uncertainty sets of transition models [21, 34]. A major drawback of robust RL is the need to calculate worst-case environments during training, which is typically approximated through the use of parametric uncertainty with multiple training environments [31, 32, 39] or a trained adversary that directly intervenes during trajectory rollouts [38, 46, 49]. Unlike these methods, we propose a robust approach to model uncertainty based on a distribution  $\mu$  over models, which does not require access to a range of simulated training environments, does not impact data collection during training, and does not involve minimax optimization problems.
In contrast to irreducible model uncertainty, epistemic uncertainty captures estimation error that can be reduced during training through data collection. Epistemic uncertainty has been considered in the estimation of Q functions [9, 35, 36] and learned transition models [7, 13, 22, 25, 40], and has been applied to promote both exploration and safety in a fixed MDP. Finally, risk-sensitive methods typically focus on the aleatoric uncertainty in RL, which refers to the range of stochastic outcomes within a single MDP. Rather than considering the standard expected value objective, they learn risk-sensitive policies over this distribution of possible outcomes in a fixed MDP [12, 24, 26, 43, 45]. Distributional RL [8] trains critics that estimate the full distribution of future returns due to aleatoric uncertainty, and risk measures can be applied to these distributional critics for risk-sensitive learning [14, 29]. We also consider the use of risk measures in our work, but different from standard risk-sensitive RL methods we apply a risk measure over model uncertainty instead of aleatoric uncertainty.
4 Risk-averse model uncertainty for safe reinforcement learning
The standard safe RL problem in (1) focuses on performance and safety in a single environment with fixed transition model  $p$ . In this work, however, we are interested in a distribution of possible transition models  $p \sim \mu$  rather than a fixed transition model. The distribution  $\mu$  provides a natural way to capture our uncertainty about the unknown transition model at deployment time. Next, we must incorporate this model uncertainty into our problem formulation. Prior methods have done this by applying the expectation operator over  $\mu_{s,a}$  at every transition [16]. Instead, we adopt a risk-averse view towards model uncertainty in order to learn policies with robust performance and safety. We accomplish this by applying a coherent distortion risk measure  $\rho$  with respect to model uncertainty at every transition.
We consider the risk-averse model uncertainty (RAMU) safe RL problem
$$
\max  _ {\pi} J _ {\rho^ {+}, r} (\pi) \quad \text {s . t .} \quad J _ {\rho , c} (\pi) \leq B, \tag {3}
$$
where we use  $\rho^{+}$  and  $\rho$  to account for reward and cost random variables, respectively, and we apply these coherent distortion risk measures over  $p_{s,a} \sim \mu_{s,a}$  at every transition to define
$$
\begin{array}{l} J _ {\rho^ {+}, r} (\pi) := \underset {s \sim d _ {0}} {\mathbb {E}} \left[ \underset {a \sim \pi (\cdot | s)} {\mathbb {E}} \left[ r (s, a) + \gamma \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho^ {+}} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} [ r (s ^ {\prime}, a ^ {\prime}) + \ldots ]\right)\right) \right] \right], \\ J _ {\rho , c} (\pi) := \underset {s \sim d _ {0}} {\mathbb {E}} \left[ \underset {a \sim \pi (\cdot | s)} {\mathbb {E}} \left[ c (s, a) + \gamma \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} \left[ c (s ^ {\prime}, a ^ {\prime}) + \ldots \right]\right)\right) \right] \right]. \\ \end{array}
$$
The notation  $\rho_{p_{s,a} \sim \mu_{s,a}}^{+}(\cdot)$  and  $\rho_{p_{s,a} \sim \mu_{s,a}}(\cdot)$  emphasize that the stochasticity of the random variables are with respect to the transition models sampled from  $\mu_{s,a}$ . Note that we still apply expectations over the aleatoric uncertainty of the CMDP (i.e., the randomness associated with a stochastic transition model and stochastic policy), while being risk-averse with respect to model uncertainty. Because we are interested in learning policies that achieve robust performance and robust safety at deployment
time, we apply this risk-averse perspective to model uncertainty in both the objective and constraint of (3).
We write the corresponding RAMU reward and cost  $Q$  functions as  $Q_{\rho^{+},r}^{\pi}(s,a)$  and  $Q_{\rho ,c}^{\pi}(s,a)$ , respectively. Similar to the standard safe RL setting, we can apply off-policy techniques to solve the RAMU safe RL problem in (3) by iteratively optimizing
$$
\max  _ {\pi} \mathbb {E} _ {s \sim \mathcal {D}} \left[ \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ Q _ {\rho^ {+}, r} ^ {\pi_ {k}} (s, a) \right] \right] \quad \text {s . t .} \quad \mathbb {E} _ {s \sim \mathcal {D}} \left[ \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ Q _ {\rho , c} ^ {\pi_ {k}} (s, a) \right] \right] \leq B. \tag {4}
$$
Therefore, we have replaced the standard Q functions for a fixed transition model  $p$  in (2) with our RAMU Q functions in (4).
We can write the RAMU Q functions recursively as
$$
Q _ {\rho^ {+}, r} ^ {\pi} (s, a) = r (s, a) + \gamma \rho_ {p _ {s, a} \sim \mu_ {s, a}} ^ {+} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} \left[ Q _ {\rho^ {+}, r} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) \right] \right]\right),
$$
$$
Q _ {\rho , c} ^ {\pi} (s, a) = c (s, a) + \gamma \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} \left[ Q _ {\rho , c} ^ {\pi} (s ^ {\prime}, a ^ {\prime}) \right] \right]\right).
$$
These recursive definitions motivate corresponding RAMU Bellman operators.
Definition 3 (RAMU Bellman operators). For a given policy  $\pi$ , the RAMU Bellman operators are defined as
$$
\mathcal {T} _ {\rho^ {+}, r} ^ {\pi} Q (s, a) := r (s, a) + \gamma \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho^ {+}} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} [ Q (s ^ {\prime}, a ^ {\prime}) ] \right]\right),
$$
$$
\mathcal {T} _ {\rho , c} ^ {\pi} Q (s, a) := c (s, a) + \gamma \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho} \left(\underset {s ^ {\prime} \sim p _ {s, a}} {\mathbb {E}} \left[ \underset {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} {\mathbb {E}} [ Q (s ^ {\prime}, a ^ {\prime}) ] \right]\right).
$$
Note that the RAMU Bellman operators can also be interpreted as applying a coherent distortion risk measure over standard Bellman targets, which are random variables with respect to the transition model  $p_{s,a} \sim \mu_{s,a}$  for a given state-action pair.
Lemma 1. The RAMU Bellman operators can be written in terms of standard Bellman operators as
$$
\mathcal {T} _ {\rho^ {+}, r} ^ {\pi} Q (s, a) = \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho^ {+}} \left(\mathcal {T} _ {p, r} ^ {\pi} Q (s, a)\right), \quad \mathcal {T} _ {\rho , c} ^ {\pi} Q (s, a) = \underset {p _ {s, a} \sim \mu_ {s, a}} {\rho} \left(\mathcal {T} _ {p, c} ^ {\pi} Q (s, a)\right). \tag {5}
$$
Proof. The results follow from the definitions of  $\mathcal{T}_{p,r}^{\pi}$  and  $\mathcal{T}_{p,c}^{\pi}$ , along with the translation invariance and positive homogeneity of coherent distortion risk measures. See the Appendix for details.
In the next section, we show that  $\mathcal{T}_{\rho^{+},r}^{\pi}$  and  $\mathcal{T}_{\rho ,c}^{\pi}$  are contraction operators, so we can apply standard temporal difference learning techniques to learn the RAMU Q functions  $Q_{\rho^{+},r}^{\pi}(s,a)$  and  $Q_{\rho ,c}^{\pi}(s,a)$  that are needed for our RAMU policy update in (4).
5 Robustness guarantees
Intuitively, our risk-averse perspective places more emphasis on potential transition models that result in higher costs or lower rewards under the current policy, which should result in learning safe policies that are robust to model uncertainty. Next, we formalize the robustness guarantees of our RAMU framework by showing it is equivalent to a distributionally robust safe RL problem for appropriate choices of ambiguity sets.
Theorem 1. The RAMU safe RL problem in (3) is equivalent to the distributionally robust safe RL problem
$$
\max  _ {\pi} \inf  _ {\beta \in \mathcal {U} ^ {+}} \underset {p \sim \beta} {\mathbb {E}} [ J _ {p, r} (\pi) ] \quad \text {s . t .} \quad \sup  _ {\beta \in \mathcal {U}} \underset {p \sim \beta} {\mathbb {E}} [ J _ {p, c} (\pi) ] \leq B \tag {6}
$$
with ambiguity sets  $\mathcal{U}^{+} = \bigotimes_{(s,a)\in \mathcal{S}\times \mathcal{A}}\mathcal{U}_{s,a}^{+}$  and  $\mathcal{U} = \bigotimes_{(s,a)\in \mathcal{S}\times \mathcal{A}}\mathcal{U}_{s,a}$ , where
$$
\mathcal {U} _ {s, a} ^ {+}, \mathcal {U} _ {s, a} \subseteq \left\{\beta_ {s, a} \in P (\mathcal {M}) \mid \beta_ {s, a} = \xi_ {s, a} \mu_ {s, a}, \xi_ {s, a} \in \mathcal {Z} ^ {*} \right\}
$$
are sets of feasible reweightings of  $\mu_{s,a}$  with  $\xi_{s,a}$  that depend on the choice of  $\rho^{+}$  and  $\rho$ , respectively.