SAFER: Data-Efficient and Safe Reinforcement Learning Through Skill Acquisition
Authors: Dylan Z Slack
Yinlam Chow
Bo Dai
Nevan Wichers
Venue: ICLR 2022 Submitted (2022)
Keywords: safety
reinforcement learning
behavioral priors
skill primitives

Abstract: Though many reinforcement learning (RL) problems involve learning policies in settings that are difficult to specify safety constraints and sparse rewards, current methods struggle to rapidly and safely acquire successful policies. Behavioral priors, which extract useful policy primitives for learning from offline datasets, have recently shown considerable promise at accelerating RL in more complex problems.  However, we discover that current behavioral priors may not be well-equipped for safe policy learning, and in some settings, may promote unsafe behavior, due to their tendency to ignore data from undesirable behaviors.  To overcome these issues, we propose SAFEty skill pRiors (SAFER), a behavioral prior learning algorithm that accelerates policy learning on complex control tasks, under safety constraints. Through principled contrastive training on safe and unsafe data,  SAFER  learns to extract a safety variable from offline data that encodes safety requirements, as well as the safe primitive skills over abstract actions in different scenarios.  In the inference stage, SAFER composes a safe and successful policy from the safety skills according to the inferred safety variable and abstract action. We demonstrate its effectiveness on several complex safety-critical robotic grasping tasks inspired by the game Operation, in which SAFER not only out-performs baseline methods in learning successful policies but also enforces safety more effectively.

Under review as a conference paper at ICLR 2022
SAFER: DATA-EFFICIENT AND SAFE REINFORCE- MENT LEARNING THROUGH SKILL ACQUISITION
Anonymous authors Paper under double-blind review
ABSTRACT
Though many reinforcement learning (RL) problems involve learning policies in settings that are difficult to specify safety constraints and sparse rewards, current methods struggle to rapidly and safely acquire successful policies. Behavioral priors, which extract useful policy primitives for learning from offline datasets, have recently shown considerable promise at accelerating RL in more complex problems. However, we discover that current behavioral priors may not be well- equipped for safe policy learning, and in some settings, may promote unsafe be- havior, due their tendency to ignore data from undesirable behaviors. To over- come these issues, we propose SAFEry skill pRiors (SAFER), a behavioral prior learning algorithm that accelerates policy learning on complex control tasks, under safety constraints. Through principled contrastive training on safe and unsafe data, SAFER learns extract a safety variable from offline data that encodes safety re- quirements, well as the safe primitive skills over abstract actions in different scemarios. the inference stage, SAFER composes a safe and successful policy from the safety skills according to the inferred safety variable and abstract action. We demonstrate its effectiveness on several complex safety-critical robotic grasp- ing tasks inspired by the game Operation, in which SAFER outperforms baseline methods in learning successful policies and enforcing safety.
INTRODUCTION
Reinforcement learning (RL) has demonstrated impressive performance at solving complex control tasks. However, RL algorithms still require large amounts of exploration and data collection in order to acquire successful policies. For many complex safety-critical applications (i.e, autonomous driving, healthcare, factory robotics) extensive interactions with an environment are not possible due to potential dangers associated with exploration or the material costs of online data collection. These challenges are further complicated by the fact that is difficult to manually specify safety constraints in these, due to the complexities of the environments. Nevertheless, relatively few existing safe reinforcement learning algorithms can rapidly and safely solve challenging, high- nigh-dimensional RL problems, with difficult to specify safety constraints.
One promising route is the behavioral prior (Singh et al., 2021) for offline skill discovery (Pertsch et al., 2020; 2021; Ajay et al., 2021). These methods use offline datasets to learn representations of useful actions behaviors through generative models, such normalizing flow model or variational autoencoders (VAH). Specifically, they treat the latent space of the generative model as the abstract action space of higher level actions (i.e., skills). Equipped with behavioral prior, for each downstream task, an RL agent trained to map states onto the abstract action space of skills. This approach can greatly accelerate policy learning because it first learns useful primitives from dataset. This structure simplifies the action space for RL "Dulac-Arool et al., 2015).
Intuitively, if trained on datasets consisting of trajectories which are both safe and successful, be- havioral priors should capture safe and useful behaviors and thus encourage the rapid acquisition of safe policies when used to learn future tasks (downstream learning). For example, when trained on demonstrations of common house hold tasks, behavioral priors should capture behaviors that successfully and safely accomplish similar tasks, such as handling objects carefully and avoiding animals/people in the environment. However, we find many existing behavioral priors may not be
Phttps://ent http//nwwwwkkkk www.wiiiiiiiaaa.. org/ /w wi /pperat onGame

---

Under review as a conference paper at ICLR 2022
well suited for safe policy learning. This is mainly because current state-of-the -f-hh--rtt behavioral pri- ors (Singh et al., 2021) cannot distinguish between safe and unsafe actions and are unable to avoid the concentration of unsafe actions in high likelihood regions of the abstract action space (see Fig- ure 1). When the behavioral priors (oftentimes modeled with deep generative models) are trained only with safe experiences, the unsafe data is out of the training distribution. It is well known that deep generative models have problems generalizing to out of distribution data, which indeed increases the likelihood of unsafe actions in this case (Nalisnick al., 2018; Fetaya et al., 2020; Kirichenko et al., 2020). For the household robotics example, the behavioral prior may lead to unsafe behavior (breaking objects or harming animals/people).
In this work, we propose SAFER: safety skill priors, which possesses both desiderata: accel- erating reinforcement learning with safe oper- ations. (An overview of SAFER is provided in Figure 2.) To acquire safe behavioral pri- ors, SAFER i) uses a contrasive loss to distin- guish safe date from unsafe ones and ii) learns posterior sampling distribution of a latent safety variable, that captures different safety contexts. With this in hand, SAFER maps the abstract ac- tion space onto the set of safe behavioral ac- tions. These safe actions are parameterized by the safety context variable, which makes the be- havioral prior more adaptable to different tasks and safety constraints. To further establish the safety assurance of SAFER, we propose tech- nique to adjust the abstract action space, such that at any state 8, at most (1 <)% of ac- tions generated by the safety prior is unsafe. As shown in Figure 1, SAFER assigns much lower likelihood to unsafe states and actions,
% Unsafe (state, action)
40
30-
SAFER
PARROT
254.59 - -7.09
4.61
4.59
Log. ligllikelhhood lower bound
Figure 1: Evaluating the concentration of un- safe data in high likelihood regions by comput- ing the % of unsafe state-action pairs in a hold- out dataset of a safe robotic grasping task. PAR- ROT assigns high likelihoods to unsafe data, i.c., does not encourage safety, while SAFER has much lower likelihood in unsafe data.
indicating that it will better promote safe behaviors when applied to downstream RL. For house- hold robotics, this means SAFER is much better equipped to Iearn behaviors that are both safe and useful for downstream tasks (handling objects safely and not harming animals/people) compared to the alternatives because it may be difficult to manually specify a constraint function, but could easier to access demonstrations of unsafe behavior. To demonstrate the effectiveness of SAFER, we evaluate it a set of complex safety-critical robotic grasping tasks. When compared with other baseline methods, SAFER policies have higher success rate and less safety violations.
2 BACKGROUND
In a setting with different tasks, for each task T, the agent's interaction is modeled as Markov decision process (MDP). A MDP is a tuple T, su). In the MDP, S and A are the state and action spaces, T(-|s, a) is the transition probabilities, r(s, a) is the reward function, 7 E [o, 1) the discount factor, and so E S is the initial state. A number of different safe RL formulations exist in the literature (Garefa et al., 2015). In this work, safety measured by the safety violation function, w(s, a) E {0. 1). that indicates whether the current state and action lead to safety violation (1) or no safety violation (0). Consequently, the safety MDP is given as a tuple (S, A, T, 80, w). Note, that the safety violation function w(s, a) is task specific. This formulation is desirable because different tasks likely have different safety criterion. For instance, in robotic grasping, different tasks likely contain different objects that have various sensitivities being gripped limits to how quickly they can move without breaking. Observe that we utilize a binary notion of safety violation because are concerned with complex RL problems, (e.g., learning from pixels), where it can be difficult to specify continuous notions of safety.
We now formalize the safety MDP problem for each task in the environment. Let A be the set of Markovian stationary policies, i.c., A = {n x A + [0, 1]. P(a 1}. Given policy E A, we define the expected return as Rs := E r'((se, at) so] and at each given state E S the safety constraint function (i.e., expected safety violation) as (.(s) E[w(s, s). The safety constraint is then defined as (s)ss) S e, where E [0. 1] is the tolerable threshold of
2

---

Under review as a conference paper at ICLR 2022
violation. For each task the goal in safety MDP is to solve the constrained optimization problem
Je" E arg max (s.) : (s) S e, Vs E S}.
(I)
PEA
It has been shown that if the feasibility set is non-empty, then there exists an optimal policy in the class of stationary Markovian policies A (Altman, 1999, Theorem 3.1). Similar to policy gradient (PG) algorithms, to effectively solve for the optimal policy we parameterize the station- ary Markovian policy by a s-dimensional vector v/ so the space of policies can be written as (ay, & E v c J. In the next section we will further exploit the connections of different tasks to design parameterization structure with policy primitives that leads to effective and safe learning.
To effectivelyy solve the safety MDP for each task, similar to the recent work of primitive discovery (Singh et al., 2021; Ajay et al., 2021), we propose the specifie policy structure Hy = fo (z; s) and z ~ To(z)s), where vb = (é, 0). In this parameterization, the mapping (with learnable parameters Φ)
Behavioral Priors
fa 3 x 8 A
is denoted as the behavioral prior (Singh et al., 2021), which maps from the abstract action space Z and state space S to the action space A. The fask-dependent, high-level policy Ti S - ((2)) maps any state E S to the corresponding distribution of abstract actions in Z. Notice that the behavioral prior fe is independent of any tasks. To train this action mapping more effectively, one can use an offline dataset D. which consists of state-action rollouts т = {su, ag .... St, at} collected across different tasks. As long as these rollouts are generated by following a diverse set of policies, they will likely contain information for learning different low-level action mappings from 2 x S to A that leads to useful behavior for the downstream tasks. Typically, simple distribution is chosen as 2 (e.g., the unit normal distribution) to make controlling the primitives straightforward.
Policies trained with behavioral priors use 2 as the action space and optimize the parameters @ in the policy Te. In the context of the safety MDP, the behavioral prior provides a simplified action space Z which only contains actions in the action space A that will satisfy the safety threshold (x. == E[w(s, a) I μ. s] and maximize rewards (so). For example, in a robotic grasping task, behavioral prior may learn to reach toward objects or grip nearby objects. These properties make behavioral priors highly appealing for solving complex control tasks because learning successful policies from high level behaviors typically requires much less interaction with the environment than learning policies from scratch (Singh et al., 2021). As such, different ways to express the behavior prior mapping have been proposed. For instance, Ajay et al. (2021) optimizes the likelihood of actions, conditioned on the state and abstract action space, log To (a(as. z). Singh et al. (2021) directly optimizes the conditional log- likelihood policy, log p(a[s). and fix an invertible mapping through the use of conditional normalizing flow (Dinh et al., 2017) between the abstract action space 2 and the distribution over useful actions p(a) (s) to retain control of the learned behaviors.
Shortcomings of Behavioral Priors for Safe RL Though current behavioral priors are highly useful at accelerating learning, they only increase of likelihood of useful actions. Thus, when applied to safety MDP problem, data containing unsafe or unsuccessful data should not be used because it is counter-intuitive to increase the likelihood of these actions (Singh et al., 2021). Consequently, unsafe states and actions may be out of distribution (OOD). It is well established in the literature on deep generative models (including the behavioral prior models) that OOD data is handled poorly and, in some cases, might have higher likelihood than in-distribution data (Nalismick et al., 2018; 88; Fetaya et al., 2020; Kirichenko et al., 2020). As we see in Figure I, these observations hold true for current behavioral priors where unsafe data has high likelihood, indicating that they may encourage unsafe behavior in the presence of OOD states. Since the proposed policy structure with behavioral priors relies on high likelihood actions from the prior (Ajay et al., 2021; Singh et al., 2021), using the aforementioned behavior priors to solve safety MDP problems will generally be problematic.
SAFER: SAFETY SKILL PRIORS
Considering the shortcomings mentioned in Section 2 of existing behavioral priors and the need for methods that can learn complex safety constraints, ideally a behavioral prior that encourages safety should i) be capable of learning complex safety constraints by sufficiently exploiting the data, thereby avoiding the OOD issue; ii) permit the specification undesirable behaviors through data; and iii) accelerate the learning of successful policies. Motivated by these requirements, in this section,
3

---

Under review as a conference paper at ICLR 2022
Figure 2: Overview of SAFER:
Safety Variable Posterior 4p.c)
SAFER optimizes the posterior over a latent safery variable (top of figure) that encodes safety in- formation about the environment. SAFER uses the safety variable to learn an abstract action space Z that maps to safe and useful behav- iors through fa through a normal- izing flow (lower left). SAFER celerates policy learning on down- stream tasks by learning to take actions in Z using policy (=(2)8) (lower right).
Previuu
Multhheed Attention
Satey Carible
Py ligie
N Image
Add Laper Norn
Add Lager Norm
Training SAFER
Safe x.00
Unauf a.4)
Acceferating Learning with SAFER
Sample xaie uncete dats
Inttolize policy Tyl #// states window for
Compute safaty variable
Ely N]
Sample safitty variablee
C CpCC
~ np(2 (
action using
fa(z;
Mazimiza contractive
objective (Equation
Updale policcy maximbe researdi
(8:2:
pla.3 c.
introduce SAFER, behavioral prior that circumvents the aforementioned shortcomings and is specifically designed for safety MDPs.
3.1 LATENT SAFETY VARIABLE
To address all of the above criteria, we introduce an additional latent variable as the input to the behavioral prior called the safety variable c C, i.e.,
fø Z x S A.
The safety variable captures information for the behavioral prior so that it can safely and rapidly generalize when used in downstream tasks. This construction is useful because information beyond the current state 8 can be encoded into the task variable, to better help the behavioral prior model complex per task safety dynamics. For example, the safety variable could encode the locations of other vehicles in autonomous driving or animals/people in robotics application. Because we do not assume the task variable is provided from the environment, we train a network to infer it.
3.2 LEARNING THE SAFETY VARIABLE
In order to train the behavioral prior and posterior over the safety variable, we adopt a variational inference (VT) approach. We jointly train an invertible conditional normalizing flow fg (Dinh et al., 2017) as the behavior prior and posterior over the safety variable using VI. At each state safety variable e E C, the flow model fg maps a unit Normal abstract action 2 E Z (i.e., samples = f (al,8, s, e) of the inverse flow model follow the distribution pz(2 (+) == (0, I)) onto the action space A of safe behaviors, and therefore the corresponding prior action distribution given by (o c) := p2(f (a; 8; c)) |det(Of, (a; 8; c))aa))
S and
(3)
The flow model is good choice for the behavioral prior because it allows computing exact log likelihoods. Further, yields a mapping such that actions taken in the abstract action space Z can easily be transformed into useful ones a fz(2; c). However, since VI approximates the lower bound of maximum likelihood, does not explicitly enforce the safety requirements in the safety variable c. To overcome this issue, we encode safety to by formulating the learning problem as chance constrained optimization (Charnes and Cooper, 1959).
Chance Constrained Objective Formally, our objective arises from optimizing neural network to infer the posterior over the safety variable C using amortized variational inference (Zhang al., 2019). In particular, we parameterize the posterior over the safety variable as (c A), where is the safety variable, and A is information from which infer the variable. In practice, we set as a sliding window of states, such that if s, is the current state at time t and w is the window size, then the information is given by A [s,, St-1, w--w]- We infer the safety variable from the sliding window of states A because For example, in a robotics setting where the observations are images, previous states may contain useful information concerning the locations of objects to avoid, which may be unobserved in the current state. We write the evidence-lower bound (ELBO) of our model as follows,
expect this to contain useful information concerning safe learning.
Be-g,(c/A) [log (a s, c) D (g, (cA))p(((
(4)
where a is the action that will not result in safety violations (i.e, w(a s) = 0) and p(c) is prior over the safety variable c. To ensure that unsafe actions are low probability, we add chance constraint
4

---

Under review as a conference paper at ICLR 2022
about the likelihood of unsafe actions (Nemirovski and Shapiro, 2007) to the ELBO optimization,
max (ela)(cls) llog Po(a)s, s, c) DKL (9p(cIA)llp(c)) s.t. Pa(pp (Aunsatee (s, c) > c) S 5. (5) p.u
where the constraint states that in most cases (at least with probability <) with the safety variable c drawn from C the distribution of the corresponding unsafe actions (i.e., w (nnusaae s) = 1) is always
less than the safety threshold e. Intuitively, this objective enforces that the safety variable should make safe actions likely as possible while minimizing the probability of unsafe actions.
Tractable Lower Bound Due to the difficulty in optimizing the chance constrained ELBO objec- tive function in high- nigh-dimensional settings, we instead consider optimizing an unconstrained surro- gate lower bound (Nemirovski and Shapiro. 2007).
Proposition 3.1 Assuming the chance constrained ELBO is written as in Equation 5, we write the surrogate lower bound as,
max (eo-)g)e))) [log p c) X log po po(aumafe 8, c) - DxL (xx(go(c/A)lll(c)) p.O,A
(6)
Proof: We rewrite the optimization 5 into the following form (with the term + +X'g dropped because it is independent of the decision variables),
max Beogg(ele) [log Pa Pa 8, c) DXL P.
- X'Pe(Po (Aussafe (s, c) > c).
where > is a E-dependent penalty parameter that is connected to the Langrange multiplier and is chosen to enforce the constraint. Using the Markov inequality we have
Ee PPo 8, c]]
Pel Pa (Aunsate/ss (s, c) e) S
(8)
such that the following objective function is lower bound of that in Equation 5:
X'
max E (elg)()ls) log po(als, c)- DKL (ka((,(cAA))llc p.O.
- Ee pa a 8 ci]
For convenience, we write X' as the single hyperparameter À and optimize the log of p (Gunsate (s, c) for better numerical stability. We finally have the lower bound surrogate objective in Equation 6. L
We denote this objective as the SAFER Contrastive Objective. Beyond rigorous derivations, this ELBO objective function is intuitively interpretable. The first two terms together act as a contrastive loss that encourages safe actions (high likelihood) while discourages unsafe ones (low likelihood). Together with the final term, the task variable e then forced to only contain useful information that does not violate safety. Thus the objective satisfies the earlier goals, allowing for the inference of difficult-oo-specffy safety constraints through the task variable and discouraging unsafe behaviors. Finally, since SAFER can increase the likelihood of any safe behaviors, the final criteria that the behavioral prior can accelerate downstream policy learning can be met by using safe and successful trajectory data during SAFER training.
Parametization Choices To parameterize the behavioral prior in SAFER, we use the Real NVP conditional normalizing flow, proposed by Dinh al. (2017), due to being highly expressive and allowing exact log- likelioeld calculations. Next, we parameterize the posterior distribution q, (cl A) over the safety variable as a diagonal Gaussian. This choice allows computing the KL efficiently, while enabling an expressive task variable latent space. We use transformer architecture to model the sequential dependency between Gaussian safety variable e and the window of previous states A (Vaswani et al., 2017). Finally, because the state space is image pixel space, we also encode each observation to a vector using a CNN. An overview of the architecture is given in Figure 2.
Training It is necessary to use the reparameterization trick to compute gradients across the objec- tive in Equation 6 (Kingma and Welling, 2014). Second, optimizing Equation 6 involves minimizing an unbounded log-likelihood in the second term of the objective. This term can lead to numerical in- stabilities, if p.o (Gumsatel ( e) becomes and freeze this term if starts to diverge. Psuedo code of the procedure to train SAFER is provided Appendix D in Algorithm and hyperparameter details are provided in Appendix B. Last, note the objective requires access to unsafe data. We don't I advise running an unsafe policy in the real
small. To overcome these issues, we use gradient clipping
2 can be optimized via gradient descent (Chow et al., 2017) or treated a hyper-parameter.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Accelerating Safe Reinforcement Learning with SAFER
Require: SAFER Prior fø. Safety Posterior (p(clA), Safety bound 17. Task T, Window A { for step k= == -,,,.., K do
8k current state
Cj < (9p(- [c]
D. Compute mean of safety variable D Sample abstract action from policy Compute action using SAFER
2x ~ Ta (2|8k)
a f- fa(zz; 81; Ck
8k+1, Tk, w/ +- Perform 0g in task T
Update TT [2) 8) using (Sk, 2k, Sk+1, rk) maximize R2 (sp) > Update A in FIFO order
Update A with 8k
end for Return: Policy T=(=]s) for task T
world to collect unsafe data. Rather, our method should be applied in domain where unsafe data already exists or where unsafe data can be collected in simulation (Srinivasan et al., 2020).
3.3 ACCELERATING SAFE REINFORCEMENT LEARNING WITH SAFER
When using SAFER on a safe RL task, the goal is to accelerate safe learning by leveraging the behavioral prior in the hierarchical policy Me -Ia (z; s; (>)T)(z(s) where the pol- icy parameters of the behavioral prior & are fixed and the parameters 0 need to be opti- mized (Psuedo code of the procedure is pro- vided in Algorithm 1). The policy (z)ss) can be learned by any standard RL methods (e.g., SAC (Haarnoja et al., 2018)) that produces con- tinuous actions. To leverage SAFER at infer- ence time for timestep the current RL pol- icy takes an action in the abstract action space zt (2(s
Algorithm 2 SAFER Safety Assurances
Require: Bound y), SAFER prior fax safe dataset Date, unsafe dataset Dunsafe
bound + (-1),7)
define get.total(D):
total (-
for (8, a, A) D do
e (. [c]. + 1o 8;;8;
if z within bound then
total +=
end if end for return total get tor.tl
st). Using the sliding window of states A. the safety variable posterior com- putes the distribution over the safety variable c. Because a single safety variable value e is required, we fix it at its mean, E/c,] J (c) At). Finally, SAFER computes the action a; = fa s;; Efc,]). the action is taken the environment, and the reward r(st, ar) and safety violations wi w(8 ax) are returned. The action 2, and reward r; are added to the replay buffer for subsequent RL training.
return
3.4 SAFETY ASSURANCES
To provide concrete and tunable safety assurances under SAFER, for any given bound in the abstract space Z we develop technique that estimates the corresponding safety threshold where at most c portion all actions data is unsafe (psuedo-code provided in Algorithm 2). Notice that actions that more likely to be safe and successful are closer to the mean of Z. Therefore, construct the bound around the mean of Z using an offline data set of safe and unsafe (s, a) pairs. In particular, by fixing any arbitrary range (-1, ')) each component of the Z space the corresponding percentage of unsafe actions within the dataset. We take this value as the upper bound of e, the portion of unsafe actions under SAFER. Since the normalizing flow in SAFER is invertible, it allows computing the values in Z of every (8, a) pair, and thus calculating the upper bound straightforward. Suppose the offline dataset contains sufficiently diverse state-action data that covers most situations encountered by SAFER. Then we would expect the above safety threshold be rather generalizable and statistically significant (Kääriäinen and Langford, 2005).
compute
RELATED WORK
Safe Exploration A number of related works focus on safe exploration in RL when there is ac- cess to known constraint function (Wachi and Sui, 2020; Achiam et al., 2017: Dalal et al., 2018;
If there are insufficient states to compute a task window of size w (e.g., the beginning of the rollout), pad the available states with 0's in order construct window of states.

---

Under review as a conference paper at ICLR 2022
Figure An example of task where the robot success- fully and safely grasping an object (top row). Here, the robot reaches into the container and extracts the object with- out touching the container. On the bottom row, the robot per- forms the same task but com- mits safety violations by touch- ing the container.
Successful Safe Trajectory
Unsafe Trajectory
Safety Violations!
Bharadhwaj et al., 2021; Narasimhan, 2020; Yang et al., 2021; Chow et al., 2018a;b; Achiam and Amodei, 2019; Berkenkamp et al., 2017: El Chamie et al., 2016; Turchetta et al., 2020). In our work, we focus on the setting where the constraint function cannot be casily specified and must be inferred from data, which is critical for scaling safe RL methods to the real world. To this end, few works consider a similiar setting where the constraints must be inferred from data (Yang et al., 2017: Thananjeyan et al., 2021a). However, these works mainly focused on constrainted exploration and do not consider accelerating learning through learning useful behaviors within the space of safe (s, a) pairs, as is possible with SAFER.
Demonstrations for safe RL Using demonstrations to enable safe RL has received interest in the literature (Rosolia and Borrelli, 2018; Thananjeyan et al., 2020; 2021b; Driessens and Dzeroski, 2004; Smart and Kaelbling, 2000; Srinivasan et al., 2020). Though these works leverage demonstra- tions to improve safety, they do not handle data from a variety of tasks and enable the acquisition of diverse sets of skills.
Various works consider learning skills in an online fashion (Eysenbach et al., 2019; Nachum al., 2019; Sharma et al., 2020; Xie et al., 2021; Konidaris and Barto, 2009). These works learn skills that are used for planning (Sharma et al., 2020) or online RL (Eysenbach et al., 2019; Nachum et al., 2019). In contrast, we focus on a setting with access to an offline dataset, from which the primitives these to downstream learning (Pertsch et al., 2020; 2021; Ajay et al., 2021). However, these methods do not consider safety.
Skill Discovery
learned. Further works use offline datasets to extract skills, and transfer
Hierarchical RL Numerous works have found it beneficial for policy learning to learn high level primitives using auxiliary models and control these with RL (Singh et al., 2021: 1: Peng et al., 2019; Chandak et al., 2019; Nachum et al., 2018; Hausman et al., 2017; Florensa et al., 2017; Fox et al., 2017a; Dietterich, 1 1998; Rakelly et al., 2019). Though these works propose methods that are capable of accelerating the acquisition of successful policies, they do not specifically consider accelerating learning with safety constraints and, in some cases, could be susceptible to the issues in Section 2. SAFER considers building a hierarchical policy in the form behavioral prior that handles safety through encouraging safe actions while discouraging unsafe ones.
5 EXPERIMENTS
We evaluate the calibration of the safety assurances introduced in Section 3.4 and how well SAFER encourages both safe and successful policy learning compared to baselines.
5.1 EXPERIMENTS SETUP
Recall that SAFER improves data efficiency by first learning a behavioral prior from offline data. To evaluate SAFER, we introduce suite of safety-critical robotic grasping tasks that are inspired by the game Operation'.
Safety-eritical Robotic Grasping Tasks Inspired by the game Operation, whose primary goal is to extract objects from different containers without touching the container, we construct set of 40 different grasping tasks, each consisting of a container and object defined in PyBullet (Coumans and Bai, 2016-2021). We collect data from all these tasks to train SAFER but choose 6 of the more complex tasks for evaluation. each of these tasks, the objects are randomly selected from ones available in PyBullet package, and the containers are generated to fit each object, whose dimensions
hhttps://en wi wwwipediaa.coom org wikkiikkkk /pperat: onGame
7

---

Under review as a conference paper at ICLR 2022
Table 1: Training RL with SAFER, we give the mean + SD success rate and cumulative safety violations across different tasks and initializations. SAFER produces the lowest cumulative safety violations throughout training and outperforms the baseline methods in terms of success rate. Meth- ods without the use of behavioral prior, namely SAC, are not able to learn during training (of 50, 000 steps). These results demonstrate SAFER is highly effective at encouraging safe learning.
Success Rate (%)
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
SAC PARROT
0.0
0.0
0.0 0.0
0.0
0.0 0.0
2.0
0.0
12.8 0.2
25.7 0.2
0..
33.9
6.3 0.1
Context PAR.
0.0
21.2 0.2
27.0 0.3
0.7
0.3
12.0 0.2
Prior Explore
0.0
1.5 0.0
3.0 0.0
1.8
1.1 0.0
1.0
SAFER
0.1
87.4 0.2
89.3 + 0.0
28.1 = 0.2
54.4 1:44000
83.3 = 0.0
Total Number of Safety Violations (Out of 50,000 Steps)
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
SAC
2045 236
876 +: 117
1055 +: 216
2736 +: 147
2188 1. 405
756 1: 293
PARROT
6332 = 3026
307 291
13 2: 21
541 + 461
2414 = 314
932 1 844
Context PAR.
5929 = 2964
1576 = 1208
1039 = 777
5056 = 1778
2796 = 624
2085 = 1951
Prior Explore
6203 551
2240 634
2867 = 853
4525 +: 826
4G69 542
2596 + 703
SAFER
610 - 184
51 + 61
10 14
455 + 470
1707 = 292
7 9
(heights and widths) generated randomly. Our agent controls the 5DoF robotic arm and gripper, for which a positive reward (r(8,a) = 1) is received when an object is extracted from the box, while negative reward (r(s, a) 1) is incurred every time step before the task is complete. The agent incurs a safety violation (w(s, a) = 1) if the robotic arm touches the box (examples of safe/unsafe trajectories given in Figure 3, examples of the tasks provided in Appendix Figure 10).
The states are 48 x 48 pixel image observations of the scene collected from fixed camera.
Offline Data Collection To generate the offline data for the SAFER training algorithm, for each robot grasping task we use the scripted policy from Singh et al. (2021) to collect trajecto- ries with a total of 1, 000, 000 steps. The scripted policy controls the robotic arm grasp the object generally by minimizing the absolute distance between objects and the robot. To obtain more diverse/exploratory trajectories. it adds random actuation noise. After collecting the tra- jectories, for each state-action pair (s, a) in the dataset we provide labels for i) safety violation w(s,a) a) E (0. 13. and ii) whether the pair (s, a.) is part of successful rollout (i..e, (s, a) such that B|r(sT, ( ate the state window A for SAFER training, for each (8, a) in the data buffer we save the previous states. One can utilize these labels to categorize safe versus unsafe data train SAFER.
Ag a) = 1, where T is the trajectory length random variable). To cre-
Baseline Comparisons We compare against baselines that leverage demonstrations to accelerate learning, including RL from scratch using SAC, PARROT (Singh et al., 2021), a contextual version of PARROT (Context. PAR) that uses a latent variable to help accelerate learning, and Prior Explore, method that samples from SAFER to help with data collection during training. Because in our problem setting, we infer safety constraints from data and do not have access to such constraints during training, do not compare against safe RL works that make this assumption. Full details surrounding the basclines are given in Appendix C.
5.2 RESULTS DISCUSSION
Effectiveness of RL training with SAFER In Table we compare SAFER with the baseline methods both in terms of cumulative safety violations and success rate. We choose a SAFER policy primitive with a safety assurance upper bound that guarantees at most 15% unsafe actions, which empirically maintains good balance between success and safety. For each downstream task, we then train the RL agent Ty with SAC for only 50, 000 steps because we are more interested to evalu- ate the power of the behavioral prior. Overall, see that SAFER has the lowest cumulative safety violations, indicating that it is the most effective method in promoting safe policy learning. Interest- ingly, SAFER also consistently outperforms other methods in encouraging successful learning. The strong success rate results of SAFER could be due to the fact that discouraging unsafe behaviors may help refine the space of useful behaviors, better enabling the learning of successful policies.
Safety Assurance Calibration We evaluate whether the safe abstract action bound of SAFER computed in Section 3.4 is well calibrated, i.e., the empirical percent of unsafe actions should be
8

---

Under review as a conference paper at ICLR 2022
Empirical Unsafe Actions
Task
Task 2
Task 3
Upper Bound
40
30
20
10
Empirical % Unsafe Actions
Task 4
Task 5
Task 6
50
30
20
10
0
30 45
15
30
15
Upper Bound Unsafe Actions
Upper Bound Unsafe Actions
Upper Bound Actions
Figure 4: Assessing the calibration of the SAFER safety assurances by randomly sampling ac- tions from the prior with various safety upper bounds across different evaluation tasks. Each dot corresponds to the empirical percent of unsafe (s, a) pairs from single rollout on the task. Overall, we see that the SAFER safety assurances are quite well calibrated.
0.6
Figure 5: Assessing the tradeoff be- tween success and safety varying the safety assurances bound the abstract action space Z, (referred to as v in Al- gorithm 2). There is an sweet spot where success rate is high and safety violations low.
Safety Vialations
Success Rate
100
0.6 0.4 0.2
0.6
0.4 0.2
Upper Bound
Upper Bound
less than the upper bound. To study this, compute the Z-action bound (-17, 11) corresponding to an upper bound of 0%, 15%. 30% and 45% unsafe actions for SAFER. We compute the percent of unsafe actions by randomly sampling actions from SAFER each of the evaluation tasks and report the results in Figure 4, which shows that SAFER bounds are indeed well calibrated.
Success Safety Tradeoff In Figure 5 we assess the tradeoff between success and safety by varying the Z-action bound in Algorithm 2. We sweep over different bounds and compute both the success rate and safety violations the end of training for Task We see that there sweet spot with high success rate and low safety violations when the safety assurances bound is close to 15%. Interestingly when the bound too tight (corresponding small z values), both the safety violation and success rate become low, indicating SAFER cannot solve the task without sufficient exploration.
Impact of latent safety variable We train SAFER on Tasks 2 and using the contrastive objective in Equation 6 but without the safety variable. In this case, the success rate never exceeds 10% and the safety violations are quite high (see Appendix A for the results for Task 2). In comparisons, with the safety variable the SAFER method has success rate of least 60% both tasks (see Table for details). This suggests that the latent safety variable crucial for success and safety.
6 CONCLUSION
In this paper, we introduced SAFER, behavioral prior that improves the data efficiency of safe RL when there is access to both safe and unsafe data examples. This is particularly important because most existing safe RL algorithms are very data hungry. We proposed a set of complex safety-critical robotic grasping tasks to evaluate SAFER, investigated limitations of state-of-the-an behavioral priors in safety settings, and demonstrated that SAFER achieves better success rates and enforces safety, and has the following assurance: at state at most (1 <)% of actions will be unsafe. Future work includes extending the behavioral priors to enforce safety on cumulative or worst case constraints, and applying SAFER to larger-scale problems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. ICLR, 2021.
Karl Pertsch, Youngwoon Lec, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. CoRL, 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J Lim. Guided reinforcement learning with learned skills. In Self-Supervision for Reinforcement Learning Workshop-ICL.8 2021, 2021.
Amurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. ICLR, abs/2010.13611, 2021.
Gabriel Dulac-Amold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep rein- forcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679, 2015.
Eric Nalisnick, Akíhiro Matsukawa, Yee Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? ICLR, 10 2018.
Ethan Fetaya, Beern-Herrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the lim- itations of conditional generative models. ICIR, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect -fttt--distributtin data. arXiv, 2020.
Javier García, Fern, and Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015. URL http::tt//// / /mjlr.olorrrrrr/o papers/vll 6/ //gaaaacaall. 5a .tmll.
Eitan Altman. Constrained markov decision processes, 1999.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: /ppprevvee..eeeeeee Sicrummiiiiii wwt.iiikkkkkbb...........
Abraham Charnes and William W Cooper. Chance-constrained programming. Management science, 6(1):73-79, 1959.
C. Zhang, Judith Bütepage, H. Kjellström, and S. Mandt. Advances variational inference, IEEE Transactions Pattern Analysis and Machine Intelligence, 41:2008-2022 2019.
Arkadi Nemirovski and Alexander Shapiro. Convex approximations of chance constrained pro- grams. SIAM Journal on Optimization, 17(4):969 996, 2007. doi: 10.1137/050622328, URL https:// /dops: 10. httg///...117/002222333
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates. Inc., 2017. URL https: httts:://proceeddngs.ooo neurips............p............ wttpriss.cc/ppppe///00/ cc/papeer201//iiee 3f5ee243547de Fbd053c1144845aa-22222
pdf.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Krishna Parasuram Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Leamn ing to be safe: Deep rl with safety critic. ArXiv, abs/2010.14600) 2020.
10

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. In JCML, 2018
Matti Kääriäinen and John Langford. A comparison of tight generalization error bounds. In Pro- ceedings of the 22nd international conference on Machine learning, pages 409-416, 2005.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained Markov decision pro- cesses. In Hal Daumé IIII and Aarti Singh, editors, Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9797-9806. PMLR, 13-18 Jul 2020. URL https:/ htaps:://rrceedd: iag.nngsmm... mlr press/v199 wachi20a. html.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 22-31. PMLR, 06-11 Aug 2017. URL https: htttps::ppocceedinngggeeeoe achiami 7a html..
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. CoRR, htt//1801.0875,, 2018. URL http:
//arxiv. org. wwwg/aaae abs / 801 08757.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety crities for exploration. ICLR. 2021.
Karthik Narasimhan. Projection-based constrained policy optimization. ICLR, abs/2010.03152, 2020.
Tsung- Yen Yang, Justinian P. Rosca, Karthik Narasimhan, and Peter J. Ramadge. Accelerating safe reinforcement learning with constraint-mismattee policies. ICML, abs/2006. 11455, 2021.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning. Proceedings of the 32nd International Con- ference Neural Information Processing Systems, NIPS' 18, page 8103-8112, Red Hook, NY, USA, 2018a. Curran Associates Inc.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning. In S. Bengio. H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc., 201 8b. URL https: http:////////e///pe///oo/o neurips. pdf.
cc/paper/200
9b527 5bdde 6tt.e//e00ee.e6eeeeeeeeeeeeeeee Sbcdessbeeisff677400--aaaa
Joshua Achiam and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. 2019.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig. and Andreas Krause. Safe model-based reinforcement learning with stability guarantees.
I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Asso-
ciates, Inc., 2017. URL https: httttp::/pprocceeiinee ngs neuriss.................. http:///gggrrr...s..p/.///////// pctpppp//////////1111ee (eneeed5622ee001705555 (tte3177006103333ac33222223 htwp:eSddwaaaaaapp.ppppm
Mahmoud El Chamie, Yue Yu, and Behçet Açıkmeşe. Convex synthesis of randomized policies for controlled markov chains with density safety upper bound constraints. In 2016 American Control Conference (ACC), pages 6290-62999 2016.
doi: 10. 109AACC.22211...5265555
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. In H. Larochelle, M. Ranzato, R. Hadsell, M. E. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12151-12162. Curran Associates, Inc., 2020. URL httpss://poccceinggggg.. neurips. cc/ cc/pappr/2000/ E1 htt.aa/aaaa.aaaaaaaaaaaaaaaaaaa.

---

Under review as a conference paper at ICLR 2022
Tsung- Yen Yang, Michael Hu, Yinlam Chow, Peter J. Ramadge, and Karthik Narasimhan. Safe reinforcement learning with natural language constraints. CoRR, abs/2010.05150, 2017. URL
https://arxix: org/ httg//b//////o///////////////a//
05150.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishna Parasuram Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6:4915 4922, 2021a.
Ugo Rosolia and Francesco Borrelli. Learning model predictive control for iterative tasks, a data- driven control framework. IEEE Transactions Automatic Control, 63:1883-1896, 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph Gon- zalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estima- tion from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks. IEEE Robotics and Automation Letters, 5:3612-3619, 2020.
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Joseph Gonzalez, Aaron D. Ames, and Ken Goldherg- Abc-Impe: Safe sample-based learning mpc for stochastic nonlinear dynamical systems with adjustable boundary conditions. In WAFR, 2021b.
K. Driessens and S. Dzeroski. Integrating guidance into relational reinforcement learning. Machine Learning. 57:271-304 2004.
William D. Smart and Leslie Pack Kaclbling. Practical reinforcement learning in continuous spaces. In Proceedings of the Seventeenth International Conference on Machine Learning. ICML '00, page 903-910, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. ICLR, 2019.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: /openreieeeee net/ / forum? Hit.HiemssssseF
Archit Sharma, Shixiang Shane Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. ArXiv, abs/1907.01657 2020.
Kevin Xie, Homanga Bharadhwaj. Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. ICLR, 2021.
George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning do- mains using skill chaining. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Cu- lotta, editors, Advances in Neural Information Processing Systems, volume 22. Curran Asso- ciates, Inc., 2009. URL ht.ps:::tp...............:. htttps:://roceeeeee/ee httpp//ooeeenn...eeeeo www.eeeeddinns..eeiinie.ee newrips. ctt.ccc/pppppe htcpp/pppe/2200/E1// .dd..
62666666992 SFE11f1711ddebbbbbbbb
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. Curran Associates Inc., Red Hook, NY, USA, 2019.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learning action representations for reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov, editors, Proceedings the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 941-950. PMLR, 09-15 Jun 2019. URL https: httpss///poeeeeieeee ing...niii....... ml r. p.rress v97 chandak 9a htmll
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. and R. Garettt editors, Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018 URL https: httttpss/pppccceeingg neurips. pee.rippa.ccppape/2/...... .pdf.
fi file/e6384711 71 3d2 w1w.1111111111133332011133 htw.
12

---

Under review as a conference paper at ICLR 2022
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph Lim. Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets. In Proceedings the 31st International Conference Neural Information Processing Sys- tems. NIPS 17, page 1235-1245, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. ICLR, 2017.
Roy Fox, Sanjay Krishnan, lon Stoica, and Ken Goldberg. Multi-level discovery of deep options. ArXiv, abs/1703.dm 0.2294 2017a.
Thomas G. Dietterich. The maxq method for hierarchical reinforcement learning. In Proceedings of the Fifteenth International Conference Machine Learning. ICML '98, page 118-126 1--26, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 1558605568.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-po reta--eiinorrcement learning via probabilistic context variables. In Kamalika Chaudhuri and Rus- lan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learn- ing, volume of Proceedings of Machine Learning Research, pages 5331-5340. PMLR, 09-15 Jun 2019. URL https: http//eeeonngsss////ee/eeeees.se .wl....mlll.....m............mm. mlr. www..mmww........mmmmmm.m.mm...e /v97 rakell 11lllllyyllll... html.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. ht http: http::/ppybullet.orrr 2016-2021.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference Machine Learning, pages 8624-8633. PMLR, 2020.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv: arXiv: 7703.0824, 2017b.
Ali Ghadirzadeh, Petra Poklukar, Ville Kyrki, Danica Kragic, and Märten Björkman. Data-efficient visuomotor policy training using reinforcement learning and generative models. arXiv preprint arXiv:2007.13133 13134, 2020.
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Ef Kokiopoulou, Luciano Sbaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. https: //gp:::://// thub. comc toonntensorfff agentss, 2018. URL. https: coobb wwwpe//rts..o.oooootatte...o.... agentss. [Online; accessed 25-June- 2019].
13


Reviews:
[{"review_id": "fG839w-0Q_b", "reviewer": "Reviewer_dr7N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is a modification of existing latent skill-learning algorithms, in particular PARROT (Singh et al.) to incorporate safe learning at test-time by learning a behavioral prior from safe and unsafe offline data. The approach is intuitive, easy to understand, and performs better than two external baselines - PARROT and SAC in terms of exhibiting safer behavior after training. \n\nThere are several crucial issues with the paper, that I've listed below:\n\n1. The main drawback of the approach is that it requires an offline data of *both* safe and unsafe behavior for training the latent skill prior. This is fundamentally limiting and goes against the entire principle of safe RL - collecting such a dataset requires multiple failures and hence requires being *unsafe* multiple times during data collection. If the aim of safety is to prevent the robot + it's environment from unintended consequences, implementing this approach on a real robot would lead to multiple failures during data collection and thus by definition be unsafe. \n\n2. Several prior works (both old and new) on safe RL and skill learning are not cited / discussed. Some relevant papers are listed below [a-e].   The papers [a], [b], [c], and those cited in the paper in section 4 emphasize the non-trivial nature of safe RL by describing how it is difficult to collect data while being safe (i.e. safety during training). This paper completely omits discussion of this point and proposes a safe RL method by collecting an offline dataset while being unsafe. \n\n3. In light of the points in 1 and 2, the results on safety violations in Table 1 need to be updated to account for all the safety violations during offline data collection. \n\n4. The paper does not compare with *any* external safe RL baselines. Why are PARROT and SAC the only external baselines compared against? It is important to compare with safe RL papers in the experiments since the main claim of the paper is number of safety violations. Again, for fairness of comparison, the number of safety violations for SAFER should include the violations during data collection.\n\n5. There are no guarantees of any sort on safe behavior. It is important to provide some theoretical guarantees on safe behavior - either during or after convergence. In the current framework, during training would not make sense as the offline dataset collection necessarily involves multiple failures. So, some guarantees on the behavioral prior would help understand the generalization of the method beyond the environments evaluated in the paper. \n\n6. In equation 9, it is unclear how the objective corresponds to a form of contrastive learning. It will be helpful to clarify what is being contrasted with what and why is that helpful, compared to other possible lower bounds on the ELBO. \n\n7. How is it ensured that the policy conditioned on the skill prior, simply doesn't ignore the prior during learning? Is there any constraint on the optimization process to ensure that this does not happen?\n\n\nThere are interesting ideas in the paper around learning a safe behavior prior for latent skill-conditioned policy learning. However, as I pointed out above, there are fundamental issue and assumptions on offline data collection that require multiple safety violations in the data collection process, thereby defeating the purpose of the proposed safe RL approach.\n\n[a] Srinivasan, K., Eysenbach, B., Ha, S., Tan, J. and Finn, C., 2020. Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010.14603.\n\n[b] Bharadhwaj, H., Kumar, A., Rhinehart, N., Levine, S., Shkurti, F. and Garg, A., 2020, September. Conservative Safety Critics for Exploration. In International Conference on Learning Representations.\n\n[c] Brunke, L., Greeff, M., Hall, A.W., Yuan, Z., Zhou, S., Panerati, J. and Schoellig, A.P., 2021. Safe learning in robotics: From learning-based control to safe reinforcement learning. arXiv preprint arXiv:2108.06266.\n\n[d] Xie, K., Bharadhwaj, H., Hafner, D., Garg, A. and Shkurti, F., 2020, September. Latent Skill Planning for Exploration and Transfer. In International Conference on Learning Representations.\n\n[e] Konidaris, G. and Barto, A., 2009. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 22, pp.1015-1023.", "summary_of_the_paper": "This paper proposes an approach for safe policy learning with a behavioral prior conditioned policy. The approach relies on learning a behavior prior from a dataset of safe and unsafe trajectories with chance-constrained variational inference, and using this prior as a latent skill distribution to condition the policy during reinforcement learning. The proposed approach is evaluated against prior skill learning approaches on simulated robot manipulation tasks.", "summary_of_the_review": "There are interesting ideas in the paper around learning a safe behavior prior for latent skill-conditioned policy learning. However, as I pointed out above, there are fundamental issue and assumptions on offline data collection that require multiple safety violations in the data collection process, thereby defeating the purpose of the proposed safe RL approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gkr8PfmUc_v", "reviewer": "Reviewer_GszC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> We rewrite the optimization 5 into its equivalent\n\nRegarding the expression \"equivalent\": Eq. 5 is a mathematical program, while eq 6 is Lagrangian-like term. Eq. 6 is not a mathematical program per-se. You may want to write it as a saddle-point problem if you want to talk about \"equivalent optimization\", or say something related to the KKT conditions.  \n\n> Equation 8 and 9 \n\nYou use $\\theta$ while I think you mean to use $\\phi$. \n\n> we propose the specific policy structure µψ = fφ ◦ πθ, where fφ ◦ πθ(a|s) := R z∈Z fφ(z, s)dπθ(z|s), and ψ = (φ, θ)\n\nI think that talking about function composition is appropriate here because the underlying model is inherently probabilistic. All this is saying is that you have a mixture model, which is a clear concept when explained in those terms. \n\nAlso, in your math expression, I don't understand why you have \"a\" on the lhs but not the rhs. \n\n# Actionable feedback \n\nBe clear about the problem setup. \"Safety\" can mean a lot of things depending on the framework, and the flavor considered in this work is not a usual one of the kind that we have in CMDPs for example. I was personally struggling to grasp the motivation and the real-world grounding of the problem setup. The operation game for the robot is concrete, but arguably a rather artificial safety-critical setup. What would be a real-real world scenario where your framework would make sense (compared to the alternatives)? I suggest you expand on this in the intro.", "summary_of_the_paper": "This paper is an extension of \"PARROT\" by Singh et al. (2021), which does a form of hierarchical pretaining from offline datasets to a setting where preferences need to be expressed on the kinds of behaviors that ought to be generalized. The authors use the expression \"safety\" to refer to what their work tries to accomplish, but I think that a more direct description of their work is in terms of preference specification via binary variables. \n\nMy reference point to understand this work is to think about hierarchical reinforcement learning, especially the probabilistic viewpoint laid out by Daniel et al. (2016). In this perspective, we think of the \"termination functions\" (or initiation) in the options framework in terms of the sequence of Bernoulli variables that they induce. Similarly, the proposed SAFER approach introduces an auxiliary \"safety\" variable \"c\" which needs to be inferred at execution time.  In the proposed framework, it is as if we were to learn a mixture model where instead of having a categorical distribution over options (discrete), we would instead be using a continuous (Gaussian) distribution to represent our distribution over latent options. The particularity of this line of work (PARROT) is that this Gaussian latent variable is learned in an unsupervised manner over a dataset of successful solutions across different tasks, rather than a purely downstream-oriented manner.\n\nThe new component that this paper adds to PARROT is this additional latent \"safety\" latent variable. (Note: this paper is not very self-contained: I had to read carefully the PARROT, which I didn't know beforehand, to make sense of the proposal.) One complication that arises from this modification is to now enforce that this new \"safety\" latent behaves the way that is intended to: that is, to steer/control the induced behavior to act \"safely\" when told to do so. The solution proposed by the authors is to view this as a chance-constrained optimization problem. This allows them to write the safety satisfaction requirement in probabilistic terms (constraint violation), which they then turn into a constrained program where the probabilistic constraint becomes a deterministic one via Markov's inequality. The authors then *do not solve* the resulting problem as a constrained problem, but rather choose to work with a regularized unconstrained approximation coming from the Lagrangian. \n\nA last important conceptual idea of the paper is that of calibration (which they call \"safety assurance\") of that safety variable. The problem arises from the fact that in adding this safety knob, we need to ensure that the constraint violation coefficient specified in the chance-constrained relaxation does indeed coincide empirically with the level set by the designer. Here, the authors leverage the properties of normalizing flows to estimate this threshold from historical data. ", "summary_of_the_review": "My main criticism of this paper is that it presents a fairly specific idea, that completely hinges on very recent prior work by Singh et al. 2021: to the point that the submitted paper is undecipherable unless you read Singh et al. (2021) first (which I had to do). That being said, the paper presents an interesting new perspective and builds on chance-constrained optimization and constrained optimization (but the authors choose not to go deeper in this direction, perhaps because of technical difficulties that are not mentioned in the paper).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Be8LEqn-MBZ", "reviewer": "Reviewer_LYgh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a clear extension of behavior priors to safety domains. The paper argues that the typical BP setup is not adequate for safety contexts because of how they are typically trained vs. out of distribution samples. One question I have here is about the relation between the quality of the dataset (safety of the demonstrated behaviors, and completeness of the demonstrations wrt the full behavioral repertoire) and how large the problem of OOP actually is in this context. I wish more discussion would appear in the paper about this. Conceptually, one could think of creating a better dataset such that all OOP actions would be really outside the \"safe\" distribution in the dataset. Here the authors choose for a more explicit solution of labeling (automatically) safe and unsafe behaviors and predicting a separate safety variable as a useful signal to learn the difference between safe and unsafe behaviors. I think it makes a lot of sense, although I am not sure a single binary (or graded, but in the end only the mean of a distribution is chosen) will be sufficient for all safety scenarios. Some discussion on the limits of this approach would be appreciated. I am not knowledgeable enough about the specifics of the learning approach (and the contrastive way of optimizing the desired loss) whether there is a contribution here, or whether this is the \"smartest\" way of doing it. Nevertheless, it seems this is the first time this (BP + safety) is considered so any solution would be sufficient.\n\nI have another issue with the paper when it comes to the \"abstract\" action space in the function composition of the policy and the BP. I think some of the terminology is confusing a bit, and maybe too close to the deep learning formulation. What I mean by this, is that abstractions in the action space are used very often in MDP contexts, but typically such abstractions are given/modeled/learned (e.g. an abstraction putON(X,Y) abstracting real actions like putON(a,b), putON(b,c), putON(d,a) etc.) or they are part of a behavioral hierarchy (such as in hierarchical RL, with move-to-roomX, with low-level actions in the MDP such as north, south, west, etc). The paper mentions terms like \"downstream task\", and hierarchical, and abstract, but the definition of Z wrt the MDP (or POMDP for that matter) is not made expliciet (enough). Also, it is unclear how the paper (and BPs in general) relate to many other works in continuous spaces where abstract actions are used which have to be \"instantiated\" or \"grounded\" when actually used (e.g. to choose parameters and specific values). To make the paper better, a better description of BPs and how they relate to the problem ((PO)MDP) would be needed, and only after that one should move to a deep learning solutions where Z happens to be positioned in a (relatively random) latent space. This would also make things more clear in terms of learning high-level behavioral choices (Z) and \"mappings\" from Z to actual actions in the underlying problem space. I checked Singh et al 2021 and the description of BPs there is longer and better, but also lacks a better grounding of concepts in the underlying problem. To make the paper better, and to connect better to related literature, these things need to be described better and more in full.\n\nOverall, a well-done paper with a clear contribution and several good experiments showing the benefits of the approach. The description of the approach can be improved (see issues above, and below, related work, overall story, bibliography, typos, etc).\n\n- \"inspired by the game Operation\"; I was disappointed it was not about the real game itself :)\n- p1:par2:line3 \"use a generic offline datasets\" (grammar)\n- (see also above): explain the non-standard \"downstream\"\n- Figure 2 (and especially the caption text) is not useful at all for me.\n- The safety variable can be motivated better\n- title section 3.2: \"learning THE safety variable\"?\n- Maybe the chance constrained optimization needs more explanation (technical, especially the relaxation)\n- I like the approach in section 3.4. although it also sounds like a practical hack (no offense)\n- The related work is ok, but could say more about the safety in RL literature (beyond safe exploration)\n- sec5.1:par2:line1 \"whose\" -> \"which\"(?)\n- sec5.1:par2:line3 \"the PyBullet\" (grammar, and sentence ends strangely)\n- Experiment:table1 I think it is strange that all the baselines perform so badly on the task (success rate). This table says that all methods are not capable at all, and only SAFER works (but only sometimes). This sounds weak. Can you elaborate?\n- Experiment:table1 The results for safety violations show large variations over methods and tasks. Is there any pattern here (other than SAFER is more robust)?\n- I like the experiments in Fig 5, since they show how safety can hinder raw performance in various ways. Maybe this could be discussed more in full, and earlier in the paper. I think it is relevant.\n- Conclusions/future work are quite short, and the bibliography needs polishing.\n", "summary_of_the_paper": "This paper builds on previous ideas of behavioral priors in reinforcement learning (RL), more in particular the paper by Singh et al 2021. The main idea is to have a behavioral prior BP function Z x S -> A and a policy over an abstract space Z such that the composition of both will deliver actions that are somehow constrained by the likelihood of actions occurring in a dataset. This dataset is used to obtain the BP in an offline learning sessions, after which standard RL algorithms can learn a policy over Z. Overall this speeds up RL. In this paper this approach is extended with methods that learn the BP in the context of safety conditions (by learning and using an extra safety indicator variable), such that the overall behavior is learned more quickly again but also makes better distinction between safe and unsafe action choices. Much of the paper is about the learning setting (and loss function) to accomplish this. The paper also shows a mechanism to balance performance and safety violations. The approach is tested on a continuous/robotic task of manipulating objects and shown effective, also against some baselines.", "summary_of_the_review": "Overall, a well-done paper with a clear contribution and several good experiments showing the benefits of the approach. The description of the approach can be improved (see issues above, and below, related work, overall story, bibliography, typos, etc).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5OxrF4AwtON", "reviewer": "Reviewer_31VF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe problem is important and interesting.\nThe paper was well written and easy to understand. \nThe experimental results also show the methods have better performance than the benchmarks.\n\nWeakness:\nThe key components of the methods, including using conditional normalizing flow to represent the behavioral prior, using chance constraints to represent unsafe constraints, using ELBO to optimize the chance constraints are all well-known standard treatments in the field of reinforcement learning. The authors provided heuristics on the choices of these tools but lack rigorous analysis on the proposed methods. Given the authors only one environment to evaluate the proposed method, it is unknown how well the methods could generalize to other problems. \n\nIt is suggested that authors add more theoretical analysis to unveil the novelty and insight of the method and more diverse experiments to show the strength and limits of the methods in a more comprehensive way.", "summary_of_the_paper": "This paper proposes a new safe RL method, SAFER, that uses a behavioral prior learning algorithm to accelerate policy learning under safety constraints. SAFER learns to extract a safety variable from offline data and the safe primitive skills over abstract actions in different scenarios via contrastive training. The authors demonstrate the effectiveness of the methods with several complex safety-critical robotic grasping tasks inspired by game Operation. Results show that SAFER no only outperforms baseline methods in learning successful policies but also enforces safety more effectively. \n\n", "summary_of_the_review": "It is an interesting paper but lacks insight into the novelty and the application scope of the proposed method", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an algorithmic framework to accelerate RL through Behavioral Priors, while having some notion of safety incorporated. The reviewers are divided about this paper:

On the positive side, some of the reviewers consider the problem important, and the experimental results reasonable and promising.

On the negative side, reviewers raised issues such as
1) The paper is on a heuristic side.
2) No formal guarantee on the safety is provided.
3) The paper is not as self-contained as it should be, as it relies much on Singh et al. (2021).
4) The algorithm requires access to unsafe offline data.

I do not give the same weights to all these concerns. For example, even though (d) is an issue in some applications, it is alright for others. What concern me most are (1) and (2). 

A method for safety that is only evaluated empirically and does not have any formal guarantee cannot be used for safety critical tasks. I realize that some other published papers may have the same issue. But given that this is a real concern, and that two out of four reviewers believe that the paper should not be accepted, unfortunately I cannot recommend acceptance of this paper, especially given the competitiveness of this conference.

P.S: I also noticed that in the proof of Proposition 3.1, an expectation term $E[p_\phi(a|s,c)]$ in Eq. (9) is replaced by a $\log p_\phi(a|s,c)$. This requires more justifications.
Decision: Reject