SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yihang Yao\*, Zuxin Liu\*, Zhepeng Cen\*, Jiacheng Zhu\*,3, Wenhao Yu\*, Tingnan Zhang\*, Ding Zhao\*
$^{1}$  Carnegie Mellon University,  $^{2}$  Google DeepMind,  $^{3}$  Massachusetts Institute of Technology * Equal contribution, {yihangya, zuxinl} @andrew.cmu.edu
Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Constraint-Conditioned Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance, while preserving zero-shot adaptation capabilities to different constraint thresholds data-efficiently. This makes our approach suitable for real-world dynamic applications.
Safe reinforcement learning (RL) has emerged as a promising approach to address the challenges faced by agents operating in complex, real-world environments [1], such as autonomous driving [2], home service [3], and UAV locomotion [4]. Safe RL aims to learn a reward-maximizing policy within a constrained policy set [5-10]. By explicitly accounting for safety constraints during policy learning, agents can better reason about the trade-off between task performance and safety constraints, making them well-suited for safety-critic applications [11].
Despite the advances in safe RL, the development of a versatile policy that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored area. Investigating versatile safe RL is crucial due to the inherent trade-off between task reward and safety requirement [12, 13]: stricter constraints typically lead to more conservative behavior and lower task rewards. For example, an autonomous vehicle can adapt to different thresholds for driving on an empty highway and crowded urban area to maximize transportation efficiency. Consequently, learning a versatile policy allows agents to efficiently adapt to diverse constraint conditions, enhancing their applicability and effectiveness in real-world scenarios [14].
This paper studies the problem of training a versatile safe RL policy capable of adapting to tasks with different cost thresholds. The primary challenges are two-fold:
(1) Training efficiency. A straightforward approach is to train multiple policies under different constraint thresholds, then the agent can switch between policies for different safety requirements. However, this method is sampling inefficient, making it unsuitable for most practical applications, as the agent may only collect data under a limited number of thresholds during training.
(2) Zero-shot adaptation capability. Constrained optimization-based safe RL approaches rely on fixed thresholds during training [15], while recovery-based safe RL methods require a pre-defined backup policy to correct agent's unsafe behaviors [5-7]. Therefore, current safe RL training paradigms face challenges in adapting the learned policy to accommodate unseen safety thresholds.
To tackle the challenges outlined above, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, a sampling-efficient algorithm for versatile safe reinforcement learning that achieves zero-shot generalization to unseen cost thresholds during deployment. Our method consists of two integrated components: Versatile Value Estimation (VVE) and Conditioned Variational Inference (CVI). The first VVE module is inspired by transfer learning in RL [16, 17], which utilizes value function representation learning to estimate value functions for the versatile policy under unseen threshold conditions. The second CVI module aims to encode arbitrary threshold conditions during policy training. Our key contributions are summarized as follows:
1. We frame safe RL beyond pre-defined constraint thresholds as a versatile learning problem. This perspective highlights the limitations of most existing constrained-optimization-based approaches and motivates the development of CCPO based on conditional variational inference. Importantly, CCPO can generalize to diverse unseen constraint thresholds without retraining the policy.
2. We introduce two key techniques, VVE and CVI, for safe and versatile policy learning. To the best of our knowledge, our method is the first successful online safe RL approach capable of achieving zero-shot adaptation for unseen thresholds while preserving safety. Our theoretical analysis further provides insights into our approach's data efficiency and safety guarantees.
3. We conduct comprehensive evaluations of our method on various safe RL tasks. The results demonstrate that CCPO outperforms baseline methods in terms of both safety and task performance for varying constraint conditions. The performance gap is notably larger in tasks with the high-dimensional state and action space, wherein all baseline methods fail to realize safe adaptation.
Safe RL has been approached through various methods. Some techniques leverage domain knowledge of the target problem to enhance the safety of an RL agent [18-26]. Another line of work employs constrained optimization techniques to learn a constraint-satisfaction policy [27, 11, 28], such as the Lagrangian-based approach [29-31], where the Lagrange multipliers can be optimized via gradient descent along with the policy parameters [32, 33, 12]. Alternatively, other works approximate the nonconvex constrained optimization problem with low-order Taylor expansions [15] or through variational inference [34], then solve for the dual variable using convex optimization [35-38]. However, most existing approaches consider a fixed constraint threshold during training, which can hardly be deployed for different threshold conditions after training.
Transfer learning in RL. The concept of transfer learning, also recognized as knowledge transfer, denotes a sophisticated technique that exploits external knowledge harnessed from various domains to enhance the learning trajectory of a specified target task [39]. Transfer learning in RL can be categorized from multiple perspectives, such as skill composition for novel tasks [40-43], parameter transfer [44], and feature representation transfer [45-47]. Among these, the methodologies leveraging Successor Features (SFs) [48, 17] are particularly relevant to this work. These methodologies operate under the assumption that the reward function can be deconstructed into a linear combination of features, and they further extend the successor representation to decouple environmental dynamics from rewards [16, 49]. However, most existing works using SFs consider transfer learning problems among tasks with different reward functions but not with different task conditions.
3.1 Safe RL with Constrained Markov Decision Process
Constrained Markov Decision Process (CMDP)  $\mathcal{M}$  is defined by the tuple  $(S, \mathcal{A}, \mathcal{P}, r, c, \mu_0)$  [50], where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $\mathcal{P}: S \times \mathcal{A} \times S \to [0,1]$  is the transition function,  $r: S \times \mathcal{A} \times S \to \mathbb{R}$  is the reward function, and  $\mu_0: S \to [0,1]$  is the initial state distribution. CMDP augments MDP with an additional element  $c: S \times \mathcal{A} \times S \to \mathbb{R}^+$  to characterize the cost of violating the constraint. Note that in this work we use a single constraint for ease of demonstration.
Figure 1: Proposed framework
A safe RL problem is specified by a CMDP and a constraint threshold  $\epsilon \rightarrow [0, +\infty)$ . Let  $\pi : S \times \mathcal{A} \to [0,1]$  denote the policy and  $\tau = \{s_1, a_1, \ldots\}$  denote the trajectory. We use shorthand  $\mathbf{f}_t = \mathbf{f}(s_t, a_t, s_{t+1}), \mathbf{f} \in \{r, c\}$  for simplicity. The value function is  $V_{\mathbf{f}}^{\pi}(\mu_0) = \mathbb{E}_{\tau \sim \pi, s_0 \sim \mu_0} [\sum_{t=0}^{\infty} \gamma^t \mathbf{f}_t]$ ,  $\mathbf{f} \in \{r, c\}$ , which is the expectation of discounted return under the policy  $\pi$  and the initial state distribution  $\mu_0$ . The goal of safe RL is to find the policy that maximizes the reward return while limiting the cost return under the pre-defined threshold  $\epsilon$ :
$$
\pi^ {*} = \arg \max  _ {\pi} V _ {r} ^ {\pi} (\mu_ {0}), \quad s. t. \quad V _ {c} ^ {\pi} (\mu_ {0}) \leq \epsilon . \tag {1}
$$
3.2 Versatile Safe RL beyond a Single Threshold
We consider the versatile safe RL problem beyond a single pre-defined constraint threshold. Specifically, we consider a set of thresholds  $\epsilon \in \mathcal{E}$  and a constraint-conditioned policy space:  $\pi (\cdot |\epsilon):S\times \mathcal{A}\times \mathcal{E}\to [0,1]$ . We can then formulate the versatile safe RL problem as finding the optimal versatile policy  $\pi^{*}(\cdot |\epsilon)$  that maximizes the reward while sticking to the corresponding threshold condition on a range of constraint thresholds  $\epsilon \in \mathcal{E}$ , i.e.,
$$
\pi^ {*} (\cdot | \epsilon) = \arg \max  _ {\pi} V _ {r} ^ {\pi} \left(\mu_ {0}\right), \quad s. t. \quad V _ {c} ^ {\pi} \left(\mu_ {0}\right) \leq \epsilon , \quad \forall \epsilon \in \mathcal {E}. \tag {2}
$$
The generated action is subsequently based on the state and threshold:  $a \sim \pi(s|\epsilon)$ . A successful versatile training algorithm should satisfy as least two core properties:
(1) Thresholds Sampling Efficiency: The training dataset  $D = \bigcup_{i=1}^{N} D_i$  is collected through a limited set of thresholds  $D_i \sim \pi(\cdot|\tilde{\epsilon}_i), \forall \epsilon_i \in \tilde{\mathcal{E}}$ , with  $\tilde{\mathcal{E}} \subset \mathcal{E}, |\tilde{\mathcal{E}}| = N$ , where  $N$  denotes the number of behavior policies with pre-specified constraint conditions during training.
(2) Safety for Varying Thresholds: Given that the agent is trained under a restricted range of threshold conditions, ensuring safety when adapting to unseen thresholds is of significant importance.
We identify two key challenges to meet these requirements: (1) Q function estimation for unseen threshold conditions with limited behavior policy data and (2) encoding arbitrary safety constraint conditions in the versatile policy training. To address these challenges, we propose the Constraint-Conditioned Policy Optimization (CCPO) method as follows.
As illustrated in Figure. [1], CCPO is comprised of two parts. The first versatile critic learning part involves data collection via several behavior agents, each under their respective target constraint thresholds. The goal is to learn feature representations for both the state-action pair feature and the target thresholds, facilitating linear disentanglement of Q values and hence enabling generalization to unseen thresholds. This step is inspired by the concept of successor feature in RL transfer learning [16], and we term it Versatile Value Estimation (VVE).
In the second versatile agent training part, we train the policy to be responsive to a range of unseen thresholds based on the well-trained value functions. Our key insight is to adopt the variational inference safe RL framework [34], which allows us to compute the optimal non-parametric policy
distributions for various thresholds through convex optimization, and, subsequently, use them to train the constraint-conditioned policy via supervised learning. With this Conditional Variational Inference (CVI) step, the policy can achieve zero-shot adaptation to unseen thresholds without the necessity for behavior agents to collect data under corresponding conditions. We introduce each step as follows.
4.1 Versatile value estimation
It has been shown that inherent trade-offs exist between the reward and cost values for the optimal policy under varying cost thresholds [13]. Typically, stricter safety requirements tend to be accompanied by reduced rewards. Hence, the estimation of Q-value functions becomes increasingly crucial when dealing with unseen thresholds that are not encountered by the behavior agents. To this end, we propose the versatile critics learning module, which disentangles observations and target thresholds within a latent feature space. The assumption regarding the decomposition is as follows.
Assumption 1 (Critics linear decomposition). The versatile  $Q$  functions  $Q_{\mathbf{f}}^{*}$  with respect to the optimal versatile policy  $\pi^{*}$  can be represented as:
$$
Q _ {\mathbf {f}} ^ {*} (s, a | \epsilon) = \psi_ {\mathbf {f}} (s, a) ^ {\top} z _ {\mathbf {f}} ^ {*} (\epsilon), \quad \mathbf {f} \in \{r, c \}, \tag {3}
$$
where  $||\psi_{\mathbf{f}}(s,a)||_{\infty}\leq K_{\mathbf{f}}$  is the feature for the function of the state-action pair  $(s,a)$ ,  $K_{\mathbf{f}}$  are constants, and  $z_{\mathbf{f}}^{*}$  is the optimal constraint-conditioned policy feature, which only depends on the policy condition  $\epsilon$  for a specified task. The dimension of  $\psi_{\mathbf{f}}(s,a)$  and  $z_{\mathbf{f}}^{*}(\epsilon)$  are both  $M$ .
Note that Assumption  $\square$  is reasonable and widely accepted in the RL transfer learning literature with successor features [17], as it is reasonable to find a high-dimensional feature space to decompose the Q functions into the product of feature functions  $\psi_{\mathbf{f}}(s,a)$  and the latent vectors  $z_{\mathbf{f}}(\epsilon)$  [51]. As shown in Theorem 1, by learning  $\psi_{\mathbf{f}}(s,a)$  and  $z_{\mathbf{f}}(\epsilon)$  jointly and adding norm constraints on the feature function  $\| \psi_{\mathbf{f}}(s,a)\|_{\infty} \leq K_{\mathbf{f}}$ , we can efficiently encode the threshold information  $\epsilon$  into the Q functions and achieve accurate estimations for unseen thresholds. This is the basis for our method's data-efficient training. To further facilitate theoretical analysis, we assume that the optimal constraint-conditioned policy feature  $z_{\mathbf{f}}^{*}(\epsilon)$  can be approximated by polynomial functions:
Assumption 2 (Polynomial feature space). The optimal constraint-conditioned policy feature  $\mathbf{z}_{\mathbf{f}}^{*}$  can be approximated by  $\mathbf{z}_{\mathbf{f}}^{*}(\epsilon) = \text{Poly}(\epsilon, p) + e$ , meaning each element of  $\mathbf{z}_{\mathbf{f}}^{*}$  is a  $p$ -degree polynomial of  $\epsilon$ , and  $e$  is the remainder. Each component for  $e$  follows  $e_j \sim \mathcal{N}(0, \sigma_j^2)$ ,  $j = i, \dots, M$ , and denote  $\sigma = \max_j \sigma_j$ .
Note that the degree  $p$  corresponds to the  $z(\epsilon)$  model representation capability. Based on the above assumptions, we can derive the Q function estimation error bound as follows.
Theorem 1 (Bounded estimation error). Denote  $\epsilon_{L}$  and  $\epsilon_{H}$  are the lower and upper bound of the target threshold interval for  $\mathcal{E}$ . Suppose the threshold conditions  $\{\tilde{\epsilon}_i\}_{i = 1,2,\dots,N}$  for behavior policies are selected to divide the interval  $[\epsilon_L,\epsilon_H]$  evenly, then with confidence level  $1 - \alpha$ , the estimation error of versatile  $Q$  functions conditioned on arbitrary  $\epsilon \in [\epsilon_L,\epsilon_H]$  can be bounded by:
$$
\left| \left| \hat {Q} _ {\mathbf {f}} (s, a | \epsilon) - Q _ {\mathbf {f}} ^ {*} (s, a | \epsilon) \right| \right| \leq \frac {z _ {\alpha / 2} B (p)}{N ^ {\beta (p)}} \sqrt {\sigma^ {2} K _ {\mathbf {f}} ^ {2} M}, \tag {4}
$$
where  $B(p)$  and  $\beta (p)$  are both functions of the polynomial degree  $p$ , and  $z_{\alpha /2}$  is the upper alpha quantile for the standard Gaussian distribution. The proof and detailed discussion of Theorem I and functions  $B(p),\beta (p)$  are shown in Appendix B.1 It is worth noting that we normalize the threshold conditions  $\epsilon \in [\epsilon_L,\epsilon_H]$  to the interval [0, 1] by  $\epsilon = (\epsilon -\epsilon_L) / \epsilon_H$  for numerical stability. Theorem I establishes that by decomposing the Q function into the product of  $\psi (s,a)$  and  $z(\epsilon)$  and jointly learning these components, we can guarantee a bounded estimation error for Q functions under unseen threshold conditions. Furthermore, we can derive the bounded cost violation for unseen thresholds and  $\epsilon$ -sample complexity analysis based on the theorem, both of which are discussed in proposition I and remark I. We also provide empirical verification of Q function estimation in Appendix D.1
4.2 Conditioned variational inference
Given well-trained versatile Q functions in the previous VVE module, we aim to encode arbitrary threshold constraints during policy learning in the versatile policy learning part. We utilize the safe
RL as inference framework to achieve this goal, as it decomposes safe RL to a convex optimization followed by supervised learning, both stages readily accommodating varying target thresholds. In contrast to the classical view of safe RL aiming to find the most-rewarding actions while satisfying the constraints, the probabilistic inference perspective finds the feasible (constraint-satisfying) actions most likely to have been taken given future success in maximizing task rewards [34].
Following the RL as inference literature [52, 53], we consider an infinite discounted reward formulation. In the condition that the constraint threshold is  $\epsilon_{i}$ , we denote  $O = O(s,a)$  as the optimality variable of a state-action pair  $(s,a)$ , which indicates the reward-maximizing (optimal) event by choosing an action  $a$  at a state  $s$ . Then for a given trajectory  $\tau$ , the likelihood of being optimal is proportional to the exponential of the discounted cumulative reward:  $p(O = 1|\tau)\propto \exp (\sum_t\gamma^t r_t / \alpha)$ , where  $\alpha$  is a temperature parameter. Since the probability of getting a trajectory  $\tau$  under the conditioned policy  $\pi (\cdot |\epsilon_i)$  can be expressed as  $p_{\pi (\cdot |\epsilon_i)}(\tau) = p(s_0)\prod_{t\geq 0}p(s_{t + 1}|s_t,a_t)\pi (a_t|s_t,\epsilon_i)$ , the lower bound for the log-likelihood of optimality given the conditioned policy  $\pi (\cdot |\epsilon_i)$  is:
$$
\begin{array}{l} \log p _ {\pi (\cdot | \epsilon_ {i})} (O = 1) = \log \mathbb {E} _ {\tau \sim q (\cdot | \epsilon_ {i})} \frac {p (O = 1 | \tau) p _ {\pi} (\tau | \epsilon_ {i})}{q (\tau | \epsilon_ {i})} \\ \geq \mathbb {E} _ {\tau \sim q (\cdot | \epsilon_ {i})} \log \frac {p (O = 1 | \tau) p _ {\pi (\cdot | \epsilon_ {i})} (\tau)}{q (\tau | \epsilon_ {i})} \tag {5} \\ \propto \mathbb {E} _ {\tau \sim q (\cdot | \epsilon_ {i})} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} ] - \alpha D _ {\mathrm {K L}} (q (\tau | \epsilon_ {i}) \| p _ {\pi (\cdot | \epsilon_ {i})} (\tau)) := \mathcal {J} (q, \pi | \epsilon_ {i}), \\ \end{array}
$$
where the inequality follows Jensen's inequality, and  $q(\tau |\epsilon_i)$  is an auxiliary trajectory-wise variational distribution conditioned on  $\epsilon_{i}$ .  $\mathcal{J}(q,\pi |\epsilon_i)$  in equation (5) is the evidence lower bound (ELBO) to reach the reward optimality under condition  $\epsilon_{i}$ . Since  $q(\tau |\epsilon_i) = p(s_0)\prod_{t\geq 0}p(s_{t + 1}|s_t,a_t)q(a_t|s_t,\epsilon_i)$ , we have the following ELBO over the state and constraint conditioned action distribution  $q(a|s,\epsilon_i)$ :
$$
\mathcal {J} (q, \theta | \epsilon_ {i}) = \mathbb {E} _ {\rho_ {q} (\cdot | \epsilon_ {i})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r _ {t} - \alpha D _ {\mathrm {K L}} (q (\cdot | \epsilon_ {i}) \| \pi_ {\theta} (\cdot | \epsilon_ {i})) \right] + \log p (\theta) \tag {6}
$$
where  $\rho_q(s|\epsilon_i)$  is the stationary state distribution induced by  $q(\cdot |s,\epsilon_i)$  and  $\rho_0,\theta$  refers to the parameters for policy  $\pi$ , and  $p(\theta)$  is a prior distribution over the parameters. Note we overload  $q$  by using it both in  $q(a|s,\epsilon_i)$  and  $q(\tau |\epsilon_i)$ .
We utilize the Expectation-Maximization (EM)-based RL algorithms, which alternate to improve  $\mathcal{J}(q,\pi |\epsilon_i)$  in terms of  $q(\tau |\epsilon_i)$  and  $p_{\pi (\cdot |\epsilon_i)}(\tau)$  to improve the likelihood of optimality [54, 55, 34]. Denote the feasible distribution family for the conditioned variational distribution  $q(\cdot |s,\epsilon_i)$  as:
$$
\Pi_ {Q} ^ {\epsilon_ {i}} := \left\{q (a | s, \epsilon_ {i}): \mathbb {E} _ {\tau \sim q (\cdot | \epsilon_ {i})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c _ {t} \right] \leq \epsilon_ {i}, a \in \mathcal {A}, s \in \mathcal {S} \right\}, \tag {7}
$$
which is a set of all the state-conditioned action distributions that satisfy the safety constraint specified by  $\epsilon_{i}$ . Then the E-step optimizes  $q(\tau|\epsilon_{i})$  to maximize the reward return within the trust region of the old policy and within  $\Pi_{\mathcal{Q}}^{\epsilon_{i}}$ , while the M-step aims to minimize the KL divergence between  $p_{\pi(\cdot|\epsilon_{i})}(\tau)$  and  $q(\tau|\epsilon_{i})$  by updating the parametrized policy in a supervised learning fashion. Since Off-policy deep RL techniques can be used during training, the EM updating steps are more data efficient.
The key strength of using the variational inference framework lies in its ability to encode arbitrary threshold conditions during policy learning, as shown in (7), a feat that is challenging for other methods, such as those based on primal-dual algorithms. Optimizing the factorized lower bound  $\mathcal{J}(q,\theta |\epsilon_i)$  w.r.t  $q(\cdot |\epsilon_{i})$  within the feasible distribution family and the policy parameter  $\theta$  iteratively with one single threshold  $\epsilon_{i}$  via EM yields the CVPO method [34], which is the basis of our method. We introduce the modified constraint-conditioned E-step and M-step as follows.
Constraint-Conditioned E-step: The conditioned E-step aims to find the optimal variational distribution  $q(\cdot |\epsilon_i) \in \Pi_Q^{\epsilon_i}$  that maximizes the reward return while satisfying the safety condition defined by  $\epsilon_{i}$ . At the  $j$ -th iteration, We can write the ELBO objective w.r.t  $q$  as a constrained
optimization problem (see Appendix B for proofs):
$$
\max  _ {q (a | s, \epsilon_ {i})} \mathbb {E} _ {\rho_ {q}} \left[ \int q (a | s, \epsilon_ {i}) \hat {Q} _ {r} ^ {\pi_ {\theta j}} (s, a | \epsilon_ {i}) d a \right]
$$
$$
\text {s . t .} \mathbb {E} _ {\rho_ {q}} \left[ \int q (a | s, \epsilon_ {i}) \hat {Q} _ {c} ^ {\pi_ {\theta_ {j}}} (s, a | \epsilon_ {i}) d a \right] \leq \epsilon_ {i}, \tag {8}
$$
$$
\mathbb {E} _ {\rho_ {q}} \left[ D _ {\mathrm {K L}} \left(q (a | s, \epsilon_ {i}) \| \pi_ {\theta_ {j}} (\cdot | \epsilon_ {i})\right) \right] \leq \kappa ;
$$
where  $\hat{Q}_{\mathbf{f}}(\cdot |\epsilon_i)$  is the versatile Q functions as introduced in section 4.1, the first inequality constraint represents the constraint defined in (7) and the last term in the constraint is the trust region with the old policy defined by KL distance  $\kappa$ . Inspired by [54], we use the solution of the optimal variational distribution  $q_{i}^{*} = q_{i}^{*}(a|s,\epsilon_{i})$  for arbitrary safety constraint  $\epsilon_{i}$ , which has the closed form:
$$
q _ {i} ^ {*} = \frac {\pi_ {\theta_ {j}} (\cdot | \epsilon_ {i})}{Z (s , \epsilon_ {i})} \exp \left(\frac {\hat {Q} _ {r} ^ {\pi_ {\theta_ {j}}} (\cdot | \epsilon_ {i}) - \lambda_ {i} ^ {*} \hat {Q} _ {c} ^ {\pi_ {\theta_ {j}}} (\cdot | \epsilon_ {i})}{\eta_ {i} ^ {*}}\right), \tag {9}
$$
where  $Z(s, \epsilon_i)$  is a normalizer to make sure  $q_i^*$  is a valid distribution, and the dual variables  $\eta_i^*$  and  $\lambda_i^*$  are the solutions of the following convex optimization problem (see Appendix B for details):