ConBaT: Control Barrier Transformer for Safety-Critical Policy Learning
Authors: Yue Meng
Sai Vemprala
Rogerio Bonatti
Chuchu Fan
Ashish Kapoor
Venue: Submitted to ICLR 2023 (2023)
Keywords: Learning from demonstration
Control barrier functions
Transformer models

Abstract: Large-scale self-supervised models have recently revolutionized our ability to perform a variety of tasks within the vision and language domains. However, using such models for autonomous systems is challenging because of safety requirements: besides executing correct actions, an autonomous agent needs to also avoid high cost and potentially fatal critical mistakes. Traditionally, self-supervised training mostly focuses on imitating previously observed behaviors, and the training demonstrations carry no notion of which behaviors should be explicitly avoided. In this work, we propose Control Barrier Transformer (ConBaT), an approach that learns safe behaviors from demonstrations in a self-supervised fashion. ConBaT is inspired by the concept of control barrier functions in control theory and uses a causal transformer that learns to predict safe robot actions autoregressively using a critic that requires minimal safety data labeling. During deployment, we employ a lightweight online optimization to find actions that can ensure future states lie within the safe set. We apply our approach to different simulated control tasks and show that our method results in safer control policies compared to other classical and learning-based methods.

Under review as a conference paper at ICLR 2023
CONBAT: CONTROL BARRIER TRANSFORMER FOR SAFETY-CRITICAL POLICY LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale self- models have recently revolutionized our ability to per- form a variety of tasks within the vision and language domains. However, us- ing such models for autonomous systems is challenging because of safety re- quirements: besides executing correct actions, an autonomous agent needs to also avoid high cost and potentially fatal critical mistakes. Traditionally, self- supervised training mostly focuses on imitating previously observed behaviors, and the training demonstrations carry notion of which behaviors should be ex- plicitly avoided. In this work, we propose Control Barrier Transformer (ConBaT), approach that learns safe behaviors from demonstrations in sule-super fashion. ConBaT is inspired by the concept of control barrier functions in con- trol theory and uses causal transformer that learns predict safe robot actions autoregressively using critic that requires minimal safety data labeling. During deployment, we employ lightweight online optimization to find actions that can ensure future states lie within the safe set. We apply our approach to different simulated control tasks and show that our method results safer control policies compared to other classical and barning-based methods.
INTRODUCTION
Mobile robots are finding increasing use in complex environments through tasks such as autonomous navigation. delivery, and inspection (Ning al.) [021] Gillula et al.] [01I]) Any unsafe behavior such as collisions in the real world carries great amount of risk while potentially resulting in catastrophic outcomes. Hence, robots are expected to execute their actions a safe, reliable manner while achieving the desired tasks. Yet, learning safe behaviors such as navigation comes with several challenges. Primarily, notions of safety are often indirect and only implicitly found in datasets, as it is customary to show examples of optimal actions (what the robot should do) opposed to giving examples of failures (what to avoid). In fact, defining explicit safety criteria in most real- world scenarios is complex task and requires deep domain knowledge (Gressenbuch & Althoff] 2021 Braga et al. 2021 Kreutzmann et al. 2013). In addition, learning algorithms can struggle to directly infer safety constructs from nigh-dimensional observations, as most robots do not operate with global ground truth state information.
We find examples of safe navigation using both classical and learning-based methods. Classical methods often rely on carefully crafted models and safety constraints expressed as optimization problems, and require expensive tuning of parameters for each scenario (Zhou et al.) 2017 Van den Berg et al., 2008 Trautman & Krause, 2010). The challenges in translating safety definitions into rules make challenging to deploy classical methods in complex settings. The mathematical struc- ture of such planners can also make them prone to adversarial attacks (Vemprala & Kapoor 202
Within the domain of safe learning- approaches we see instances of reinforcement and im- itation learning leveraging safety methods (Brunke et al.] 2022) Turchetta et al., 2020). and also learning applied towards reachability analysis and control barrier functions (Herbert et al. [021] Luo et al. 2022). The application of learning-based approaches to safe navigation is significantly hindered by the fact that while expert demonstrations may reveal one way to solve a certain task, they do not often reflect which types of unsafe behaviors should be avoided by the agent. We can draw similarities and differences with other domains: natural language (NL) and vision models can learn how to generate grammatically correct text temporally consistent future image frames by

---

Under review as a conference paper at ICLR 2023
Typical learning from demonstrations
ConBaT: learn from safe & unsafe demonstrations
Safety critic
Control policy
Control policy
Experi demonstrations
deployment: imperfect policy
Safe and unsafe demonstrations
deployment:
policy update using critie
Figure I: (Left) An agent trained to imitate expert demonstrations may just focus on the end result of the task without explicit notions of safety. (Right) Our proposed method ConBaT learns safety critic top of the control policy and uses the critic's control barrier to actively optimize the policy for safe actions.
following patterns consistent with the training corpus. However, for control tasks, notions of safety are less evident from demonstrations. While the cost of mistake is not fatal in NL and vision, when comes autonomous navigation we find that states that disobey the safety rules can have significant negative consequences to a physical system. Within this context, our paper aims to take a step toward answering a fundamental question: how can we use agent demonstrations to learn a policy that is both effective for the desired task and also respects safety-critical constraints?
Recently, the success large language models (Vaswaní et al. 2017 Brown et al. 2020) has in- spired the development of a class of Transformer-basee models for decision decision-making which uses ruto-regressive losses over sequences of demonstrated state and action pairs (Reed et al. [022] Bonatti et al. 2022). While such models able to Iearn task-specific policies from expert data, they lack a clear notion of safety and are unable to explicitly avoid unsafe actions. Our work builds upon this paradigm of large autoregressive Transformers applied perception-action sequences, and introduces methods to learn policies in a safety-critical fashion.
Our method, named Control Barrier Transformer (ConBaT), takes inspiration in barrier functions from control theory (Ames et al., 2019). Our architecture consists of causal Transformer aug- mented with safety critic that implicitly models control barrier function evaluate states for safety. Instead of relying on a complex set of hand-defined safety rules, our proposed critic only requires binary label of whether certain demonstration is deemed control barrier critic then learns to map observations to continuous safety score, inferring safety constraints
safe or unsafe. This
supf-super way. If proposed action from the policy is deemed unsafe by the critic, ConBaT attempts to compute better action that ensures the safety of reachable states. A lightweight optimization scheme operates on the critic values to minimally modify the proposed ac- tion and result in a safer alternative, process inspired by optimal control methods and enabled by the fully differentiable fabric of the model. Unlike conventional formulations of the control barrier function, our critic operates in the embedding space of the transformer opposed to ground truth states, making applicable to a wide variety of systems. We list our contributions below:
We propose the Control Barrier Transformer (ConBaT)) architecture, built upon causal Transform- ers with the addition of differentiable safety critic inspired by control barrier functions. Our model can be trained auto-regressively with state-action pairs and can be applied to safety-critical applications such as safe navigation. We present loss formulation that enables the critic to map latent embeddings to continuous safety value, and during deployment couple the critic with a lightweight optimization scheme over possible actions to keep future states within the safe set.
We apply our method to two simulated environments: simplified Fl car simulator upon which perform several analyses, and simulated LiDAR-bases vehicle navigation scenario. We compare method with imitation learning and reinforcement learning baselines, as well classical prediccve control method. We show that ConBaT results relatively lower collision rates and longer safe trajectory lengths.
We show that novel safety definitions (beyond collision avoidance) can be quickly learned by ConBaT with minimal data labeling. Using new demonstrations can finetune the safety critic towards new constraints such as "avoid driving in straight lines", and the adapted policy shifts towards the desired distribution.
2

---

Under review as a conference paper at ICLR 2023
2 METHODS
We formulate the problem of learning safe control from observations as partially observable Markov decision process that involves the agent interacting with an environment to receive high- dimensional observations which are used to take continuous-valued actions a. For simplicity, we treat the observations as equivalent to states s while noting that actual agent states are implicit and not evident from the data. We define a trajectory т as set of state-action tuples (s,, at) over discrete-tine cerrete-timeeeeee finite horizon t E TT. We assume that we have access to demonstrations from two sets of trajectories: which lead to an unsafe terminal state. Our goal is to learn a safe policy Tsaae 8 -> a that results in actions that mimic the action distribution from good demonstrations Σ,, while avoiding sequences of actions that lead to the unsafe terminal states of Cg.
Es, which obey the desired safety constraints at all time steps, and T E En.
2.1 BASE ARCHITECTURE: PERCEPTION-ACTIN CAUSAL TRANSFORMER
The base transformer architecture for our method is derived from the Perception-Action Causal Transformer (PACT) proposed in Bonatti et al. (0022). PACT is primarily focused on creating a pretrained representation that can be finetuned towards a diverse set of tasks for a mobile agent. Its pretraining stage uses a history of action pairs from expert demonstrations autoregressively train both world model and a policy network, using imitation learning for its training objectives. The main functions from PACT that we employ are:
Tokenizer:
The state and action tokenizers operate on raw high dimensional observation and action data, and learn to represent them compact tokens: (st) - 84, (a;)) - af, where s',a' E RRd and d is fixed length of the token embedding. Т, and Ta are learned functions, and we detail their architectures when describing concrete problem settings in subsequent sections.
Causal Transformer: A set of Transformer blocks X operates upon sequence of state and action tokens. A causal attention mask is applied so that the transformer learns a distribution over the token at index given the history of tokens from [0, 1]. The output of this module is sequence of state and action embeddings, X ($6: dg. s dj ... s ar) (st ag ....T,,..... af), where st at E Rd.
Policy model: An action prediction head acts as policy and operates the output state embed- ding st at a given timestep and predicts the appropriate action: π(,;") à,
This module operates on the state and action output embeddings from the current timestep. and predicts the state embedding: ((st, - 84+1- This module functions as a regularizing mechanism during training, and can optionally be utilized during the safe policy roll-out as explained next.
World model:
2.2 CONTROL BARRIER TRANSFORMER
On top of the original PACT architecture, we introduce important modifications that allow ConBaT generate safe actions. While the original policy head does not explicitly distinguish between safe and unsafe behavior, which could result in sub-optimal action predictions, we use our policy head as an action proposal network which later refined by the critic.
2.2.1 CONTROL BARRIER CRITIC
We augment the transformer backbone with two trainable critic modules, which predict safety scores for the current and future expected states. The first critic C. s & E R maps the current state embedding s to real-valued safety score éz The second critic Cy (s a,,) + õ estimates the future state safety score & based on the current state and action embeddings st at Here Cj can be interpreted conjoining both world model and safety critic within single network. We display our full ConBaT architecture along with the prediction heads in Figure
To train these critics, draw inspiration from the control barrier functions from classical controls literature (Ames et al. 2019). We assume our system to be of the form š = f(s, a) with f locally Lipschitz continuous, state and action A. Let denote the safe set of states S, C S.
3

---

Under review as a conference paper at ICLR 2023
Cost
πI
Output embeddings
Tokenizer Transformer
Causal Transformer:t N
Input takens
S113
R2
Action
Row data
Osservation
proposal
Optimal action
Environment
(A)
(b)
Figure 2: (a) The ConBaT architecture causal Transformer operates state and action tokens (s',a') produce embeddings (s a²)) policy head computes actions given state embeddings, and current state critic compules a safety score. Both state action embeddings are processed by world model to compute the future state token, and by the future critic Cj to produce future safety score. (b) The deployment process for ConBaT involves feedback loop. The future critic evaluates action proposals from the policy head check safety of resultant states. The red arrows show the flow of gradients that allow optimizing for the safe action that results desired cost characteristic. The optimal action a" used the final command.
If there exists function h S - IR and policy n² S + A satisfying:
h(s) 2 E
(1)
1(()) < 0, Vs S., = S/S,
h(s)) = ah(s)/Os f(s, " (s)) 2 -ah(s) with a 0,
(3)
then h is defined be control barrier function (CBF), and the policy x* ensures any initial state starting from S, will always stay in S, (forward invariant). The proof can be found in (Ames et al. 2014) 1). Since S, denotes the safe set, the policy x² will guarantee all the states initialized from the safe set to be always safe.
In practice, it challenging to find the perfect safe policy x² for all states due to input constraints, imperfect policy learning, or disturbances (Qin et al., [221]] Dawson et al. (0226). One alternative to learn the CBF on top of an imperfect policy π and then use an optimization routine (quadratic program (Ames et al.] 2016), second-order cone program (Buch et al. 2021), gradient descent) to steer π to satisfy CBF constraints, thus achieving safety. ConBaT follows a similar philosophy where the crities are learnt approximate the CBF for given (imperfect) policy и (from PACT) so that the CBF conditions can satisfied in most cases, and then use back sock-propagation to rectify π to satisfy the CBF condition during deployment. In our method, the supervision signal needed to learn the critic values is just the binary labels indicating a state is safe or not, bypassing the need for more complex hand-designed signals. We call C and Cj Control Barrier-like crities (CBC) and detail the learning process below.
During CBC learning, instead of considering all possible state embeddings, we only enforce the state embeddings from the demonstrations Σ, and Σg to satisfy the conditions above. In order for Equations (1)-33) to hold, we expect the critic values to be positive on the collected safe state embeddings S. negative on the collected unsafe state embeddings St and not decrease too quickly for all collected state embeddings 5- 5: S For St we include all the states from the trajectories in Σ, and the states from the first (L 27) time steps from the trajectories in En, where the expert trajectory length and T the time horizon observed by Transformer sequence. For St we only consider the terminal states from the trajectories in E., (because those are the only 'unsafe' states we are certain). Note that S and Š will be highly imbalanced, since have more safe demonstrations than the unsafe ones, and also only pick state from each unsafe trajectory. But as shown empirically in future sections, we find out that the critic can be learnt effectively even with limited data using our formulation.
4

---

Under review as a conference paper at ICLR 2023
Training the CBC involves three loss terms. First, we employ a classification loss Le to enable the CBC to learn the safe set boundary:
E S
o+ (7- C(+t))
(++()C((5)))
(4)
where o (z) max(æ, 0) and y is margin factor that ensures numerical stability in training. The second loss enforces smoothness on the CBC values over time:
C.
1o+ ((1 (()=a)C((t) C(st1)))
(5)
where controls the local decay rate. Note that this loss is asymmetrical as it only penalizes fast score decays but permits instantaneous increases, as fast-improving safety level does not pose problem. The final loss ensures consistency between the predictions of both critics C and Cy:
Cj
E S [[Cr(s], aals:aa)= C(st+1)]]
(6)
Theoretically, one could use single critic C' coupled with world model ó generate oss aa) 8141 and then estimate future CBC score as (() We found empirically helpful to use a separate critic head Cj to predict future CBC scores directly from the output embeddings, as it facilitates the action optimization process described in Section 2.2.2 The total training loss is LCB = deLe + XgC, Xy with relative weights X.
2.2.2 OPTIMIZING ACTIONS TO IMPROVE SAFETY
As noted earlier, our goal learn policy x² s - a that imitates the good demonstrations while always keeping the agent within the safe set. We compute n² through two-step approach:
First, we use the output of the default prediction action head as a reasonable action proposal à. This action is then propagated through the Transformer and the future state control barrier critic, and we compute the next state's safety score: čf, = Cy(s ,a;).
Action proposal:
Action optimization: If the future state is found to be in violation of the desired safety constraint, i.e. ef.+ < O, we need to override the default action so as to keep 8 within the safe set S We intend to minimally modify the default action to result in safe state, hence we express the optimal action as à + Aa", and solve the following optimization problem for the modification Aa:
a arg min Al|Aa|| + (oxx--C(ss at Aa),
Aa
Given the fully differentiable model, can easily compute gradients of the cost with respect to the action through the control barrier function. We use a lightweight gradient descent optimization routine that results in the smallest action difference with respect to the original policy to keep the agent safe. We then take a step in the environment, collect new observations, and repeat the process. We show visualization of the optimization step in Figure
2.3 TRAINING PROCEDURE
We train ConBaT in a two- two-phase approach. Phase is analogous to the original pretraining scheme for PACT (Bonatti et al. (022), and focuses on training the policy head, world model, tokenizers and transformer blocks. From this phase, we aim to learn reasonable agent behavior from demonstra- tions. For phase II, we freeze the base network weights and add both control barrier critic modules and train with the combination of safe and unsafe demonstrations. By decoupling both training stages, we allow user to potentially adapt single base policy π from pretrained model towards different definitions of safety. Optionally, unsafe demonstrations can also be included phase I to allow the world model to learn from the additional distribution of states, but the policy head is always trained only on safe samples. We empirically find that training the world model with data from unsafe demonstrations in the first phase results in better final performance of the model.
5

---

Under review as a conference paper at ICLR 2023
3 EXPERIMENTAL RESULTS
We apply ConBaTT to two domains in simulation, which we visualize in Figure
F1/10 race car. We tracks (O Kelllly et al. 2020). The car receives observations at every step comprised of the distance and the angle relative the center line. The action corresponds the steering angle, and the goal is to learn how to drive safely without colliding with the track's ' edges. This is a toy scenario used demonstrate the functionalities of our method. upon which we conduct several ablation studies; MuSHR car. Inspired by the MuSHR car (Srinivasa et al.) 2019), we use vehicle simulation in a more realistic navigation setting. The car equipped with 2D range sensor, returns LiDAR scans observations, and takes steering angles actions. We use 2D map scanned from real office space of approximately 30 x 70 meters of area, for both data collection and safe navigation deployment without collisions.
simulation of a racing car that navigates within multiple 2-D racing
100
Male
75
-20
50-
50
ymm)
y
40
-2
-25
-60
-50
50
0
160
50 <m))
100
-70
660
40
-20
(m)
(m)
(a) F1/10 (playground)
(h) F1/10 (Silverstone)
(c) F1/10 (Austin)
(d) MuSHR environment
Figure Simulation environment visualization.
We collect sets of safe and unsafe trajectories in both domains. Each trajectory has a discrete binary safety label, which is used to train the safety critic. Further details about the domains and data collection can be found in Appendix A
We evaluate our models in both domains by deploying the learned policies and evalu- ating the rollout trajectories. We measure the following metrics: (1) collision rate, measured percentage of trajectories the test set that end a crash within the cut-off time horizon; and (2) av- erage trajectory length (ATL), which corresponds to the average length of deployment trajectories, expressed in number of time steps before crashing or time-out if no crash occurs.
Metrics:
3.1 SAFE NAVIGATION ANALYSIS
We first examine the safe navigation performance of ConBaT the F1/10 simulator, and compare it with PACT. We train the models with IK demonstrations, each 100 timesteps long, from the Playground track (Figure (a)) During deployment, we roll out 128 trajectories for each model for maximum of 1000 timesteps. ConBaT achieves 0% collision rate whereas PACT collides at some point in every instance.
Next, apply the ConBaT model trained Playground more challenging tracks such Sil- verstone and Austin (Figure BiggueBBb).co)) (c).).. We compare our model's s performance against a frozen PACT model (PACT- trained only Playground) well against version finetuned these tracks (PACT-FT). In Table [ we observe that ConBaT is able to leam the safety concepts effectively enough to generalize new tracks, outperforming both PACT and PACT-FT. In Figure we take closer look at the performance on Silverstone and show histogram of the trajectory lengths across the test dataset. A PACT model trained on Playground registers fairly low ATL values, whereas PACT-FT performs better, sometimes even reaching the maximum length of 1000. However Con- BaT significantly outperforms both, with all the trajectories being collision-free.
3.2 COMPARISON WITH BASELINE METHODS
In the MuSHR simulation collect 10K trajectories to create dataset upon which we train differ- ent baseline safety models for comparison against ConBaT. We compare with baselines from three different categories: imitation learning (II.), for which we use naive behavior cloning approach

---

Under review as a conference paper at ICLR 2023
PACT
PACT-FT
ConBaT
PACT
PACT-FT
ConBaT
Playground
100
0.0
Playground
175.45
1000
Silverstone
100
96.88
0.0
Silverstone
61.57
439.28
1000
Austin
100
100
61.7
Austin
57.11
165.12
678.14
(a) Collision Rate (%) lower is better
(b) Avg. Trajectory Length higher better
Table 1: Comparison of PACT and ConBaT for the F1/10 task. ConBaT outperforms PACT
(frzzen)
PACT (uuee)
ConBaT
125
100
80
25
500
1000
500
1000
500
1000
Trajectory length
Trajectory length
Trajectory length
Figure 4: Histogram of trajectory lengths in the Silverstone track: ConBaT generalizes efficiently to a new track, achieving 100% success rate, whereas both frozen and finetuned PACT fall short.
(BC), PACT, and Generative Adversarial Imitation Learning (GAIL) (Ho Ermon 2016); rein- forcement learning, for which evaluate PPO (Schulman et al. 2017). TRPO (Schulman al. 2015), and SAC (Haarnoja et al. 2018). We also implement a model predictive control baseline via the CasADi solver (Andersson al. Press, 2018). Further details about the implementations of
B During simulation deployment, we run 128 trajecto- ries for each model, each for maximum of 5000 timesteps. Figure (S) shows ConBaT compared against all baselines. Even on this complex setting using nigh-dimensional LiDAR observations in a realistic map, we observe that ConBaT achieves the lowest collision rate and the highest ATL.
these baselines can be found in Appendix
3.3 LEARNING A NEW SAFETY DEFINITION
An important feature for safety critic is be able learn different definitions of safety without major architectural changes or hand-crafting. Therefore we evaluate ConBaT's ability to learn new safety constraints beyond simple notions of collision-free navigation. We consider the hypothetical situation of vehicle agent where traveling in straight line is undesirable, and only curved motions are allowed (due to dynamical constraints, or perhaps impress other agents in the environment). We generate new dataset with unsafe labels for every straight trajectory segment, and train new critic which we call ConBaT-NS (not-straight).
As seen in Figure 6 ConBaT-NS succeeds in moving the vehicle through curved trajectories only, even in straight hallways, showing that new constraints can be incorporated with ease. Figure/6c|also displays how this new constraint is reflected in the policy's 's action distribution output. We note that both ConBaT and ConBaT-NS are finetuned over the same base PACT model, which demonstrates the potential for training multiple critics mapping to distinct safety constraints.
3.4 ABLATION STUDIES
Variations in critic architecture: As discussed in Section 2 the control barrier critic module could be architected different ways. We compare different architectural choices for the critic: CBC-NW: Both C and Cy are used, but there is no world model in the architecture.
CBC-CW: Only the current state critic is used, and the output the world model fed into it for the next state's i cost.
CBC-TF: All the modules C, Cy, and @ are used. The input to Cy is the output state embedding and the action input token (s+ a.). CBC-EF: All the modules C, Cy. and are used. The input to Cý is the output state embedding and the output action embedding , a.).

---

Under review as a conference paper at ICLR 2023
Irsjeclories
Average Trajectory Length
Trajectory (limestoga
4000
Collaion
40
1000
INCTConBaT
Figure 5: ConBaTT outperforms classical MPC and several learning-basse baarin methods on safe navigation in the 2D MuSIIR car domain.
length=1000
length=1000
ConBaT
ConBat-NS
-10
1ra
-10
trajs prempt
gramis
-20
-20
-30
-30
Density
-0.5
-0.5 0.0
-10
Action (rad)
Action (rad)
(a) ConBaT Trajectory
ConBaT-NS Trajectory
(c) Comparison of action distributions
Figure 6: Demonstration of ConBaT being used for new safety constraint where actions mapping to the vehicle going straight are deemed unsafe.
We compare all methods in Table 2a We note that the world model is indeed useful, as evidenced by the lowest performance of CBC-NW. Using future state critic results in slightly better performance than only using a current state critic that takes world model simulated states, but higher computa- tional effort. Lastly, CBC-EF achieves higher performance than CBC-TE, indicating that the critic benefits from the context that plays a part in the computation of the action embedding, as opposed the single token. The CBC-EF architecture is used for all the other experiments in the paper.
Arch
Collision (%)
ATL (# steps)
Runtime (s)
& Trjis
Collision (58
ATL (# steps)
Runtime (s)
PACT
100
175.45
63.328
5
5.47
944.67
120.427
CBC NW
4.69
952.47
129.595
10
3.91
960.18
132.079
CBC-CW
1.56
983,48
200.153
25
1.56
983.47
130.049
CBC.TF
3.91
972.42
78.159
50
960.26
127.275
CRC-EF
0.78
991.22
130.032
100
0.78
991.22
127.073
(a) Different architectures for online optimization.
(b) Amount of unsafe training trajectories used.
Table 2: Ablation study over critic architecture and amount of unsafe trajectories needed for Phase training
The existence of demonstrations that include unsafe terminal states critical for ConBaT to be able to encode the safety constraints. Yet, given that simulating and collecting large amount unsafe demonstrations can be challenging (and costly real world conditions), we perform analysis of how ConBaT performs in low-data regimes when training the critic (phase II). Table 2b) reports performance levels for varying unsafe dataset sizes. We observe that ConBaT effective achieving reasonable safe navigation even with small amounts of unsafe demonstrations.
Data Requirements:
4 BACKGROUND
Learning Policies from Data: Imitation Learning (IL) is popular method for policy learning, and
requires expert demonstrations (Schaal) 1996 Atkeson & Schaal 1997: Pastor et al. 2009 Behba-
hani et al. 2019 Billard & Grollman 2013 Ekvall & Kragic 2008). The most naive IL approach is behavioral cloning (BC) (Bain & Sammut] 1995), which directly learns the mapping from states to
actions using supervised learning from expert data. Such simple applications of II. cannot generalize
8

---

Under review as a conference paper at ICLR 2023
to the
scenarios induced by on on-policy deployment, and often requires additional
training routines (Ross et al. 2011). Inverse reinforcement learning (IRL) (g et al. 2000 Natara- Jan et al. 2010) mitigate this challenge, it seeks to recover the original expert cost function.
Alternatively, one may use generative adversarial imitation learning (GAIL.) (Ho & Ermon 2016]
expert's policy from other behaviors generated by a policy network. When it comes to safety, however, none of the above methods incorporate the concept of safety directly into their learning procedures.
Bhattacharyya et al.] 2018), which employs discriminator differentiate
Reinforcement learning can enable safe policy learning through rewards that not only encourage good behaviors but also penalize unsafe states. |Bhattchhryya et al. (2019) use an augmented re- ward for traffic simulation to guide safe imitation, and we find numerous examples of safe policy design (Liu et al. 2020, Berkenkamp et al., 2017 2017) [Ceng et al. 2019 & Belta (019). However, manual reward shaping is laborious and nontrivial, often leading to undesired consequences. In comparison, our approach with learned control barrier functions uses minimal safety labels.
Safe policy learning using control barrier functions: Control barrier function (CBF) is a classical concept to ensure system safety (Prajna & Jadbabaie 2004, Wieland & Allgöwer 2007 Ames
2014 2019). Recently, several works have demonstrated learning safe policies based on existing CBFs (Chen et al. 2017; Borrmann et al. 2015). constructing CBF jointly with the policy learning via Sum-of-Squnee Squares <Wang al. 2018)) leveraging SVMs (Srinivasan al.) 2020) or
et al.
Neural Networks (Ferlez et al., 2020; Qin et al. 20011 Meng et al. 2021 Dawson et al.] 2022a b). However, these methods often assume access raw state inputs and ground truth safety labels from the environment. Instead, ConBaT works on the embedding space akin to models that plan using imagination (Okada & Taniguchi 202 Wu al.) 2022). learns purely from offline data, and employs an online policy rectification process to achieve safety.
ConBaT is aligned with the Mode-2 proposal in LeCun (022). where the agent makes action proposals, evaluates the costs from future predic- tions, and then plans for the next action. Predicting the next state based on the current state and action is fundamental to model predictive control (Bryson & Ho 2018). The idea of predicting the future costs via the world model can be traced back to (Schmidhuber 1990). When it comes to high-dimension state/observation space, latent embedding dynamics are learned to help the agent to achieve high performance reinforcement learning (Okada & Taniguchi [02 Wu et al. 2022), and sequence predictions (Giuliari et al. 2021: Chen et al. [021] Micheli et al. (022). In our case, we learn a predictive critic on the embedding space with CBF conditions as guidance.
Safe policy learning with predictive world models:
5 CONCLUSIONS
In this work, we propose ConBaT, a framework that learns safe navigation directly from demonstra- tions. Our method leverages causal Transformers coupled with differentiable safety critic that is inspired by control barrier functions in control theory. The control barrier critic module implicitly builds safe set for states from discrete safety labels, bypassing the need for complex mathematical formulations of safety constraints. We apply our method to two simulated domains and show that our method outperforms existing classical and learning-basee safe navigation approaches. Furthermore, we show that ConBaT can be quickly adapted to new safety constraints from limited demonstrations. This paradigm of learning safety constraints implicitly from minimal labels reduces training effort in creating safe agents and makes it easier to adapt to new definitions of safety.
Though empirically ConBaT can learn safer policy and quickly adapts to new safety concepts, there are some limitations. Our method might not guarantee safety if ConBaT makes wrong CBF score predictions due to out-of-distribution data, or the online optimization falls into a local minimum. One solution is to follow paradigm like DAGGER (Ross et al.) 2011) to keep exploring the envi- ronment and collect new states and labels to update our CBF critics and world model. The runtime for ConBaT in evaluation is also 0.2 ~ 1.0x higher than other learning- based methods due to the back- pooc-prpaggttion step in online optimization. We discuss different ConBaT architectures in Sec. [3.4] to weigh the trade-off between the safety performance and the runtime. Further runtime improve- ment can be made by faster back proppaaronn algorithms optimized for GPU hardware. Currently, our framework was only applied in simulated environments. In the future, we wish to extend our approach to image-based control in 3D and toward more complex robotic platforms in simulation and the real world.
baased
9

---

Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
ConBaT is a tool that learns safety concepts with a neural approximation control barrier func- tion. Given the finite nature of the datasets used to train our model, and despite performing an optimization routine on top of the safety critic's judgement, we may still encounter failure models the critic imperfectly approximates the safety function. Therefore, ConBaT may give users false sense of security during deployment. We recommend the use of backup methods such as emergency maneuvers if ConBaT is used in safety-critical real-world applications, besides extensive testing in simulation and controlled realistic scenarios. Additionally, since ConBaT reproduces patterns found its training datasets that contain safe and unsafe trajectories, it may reproduce data biases during deployment.
7 REPRODUCIBILITY STATEMENT
We are working towards providing an open-sourced implementation of our datasets, simulators, models and evaluation framework in the near future. We will make the links available for the camera- ready version of this work.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017. 20 21
Aaron D Ames, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE Conference Decision and Control, pp. 6271-6278. IEEE, 2014.
Aaron Ames, Xiangru Xu, Jessy Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions Automatic Control, 62(8): 3861-3876, 2016.
Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pp. 3420-3431 -3313. IEEE, 2019.
Joel A E Andersson, Joris Gillis, Greg Hor,, James B Rawlings, and Moritz Diehl. CasADi - A software framework for nonlinear optimization and optimal control. Mathematical Programming Computation, In Press, 2018. 7
Christopher G Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, volume 97, pp. 12-20, 1997. g
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995. g
Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, Joao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, al. Learning from demonstration in the wild. In 2019 International Conference on Robotics and Automation (ICRA), pp. 775-781.
IEEE, 2019. 8
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances neural information processing sys-
tems, 30, 2017. g
Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel Kochenderfer. Multi-agent imitation learning for driving simulation. In 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), 1534-1539, IEEE, 2018. 9
Raunak P Bhattacharyya, Derek J Phillips, Changliu Liu, Jayesh K Gupta, Katherine Driggs- Campbell, and Mykel J Kochenderfer. Simulating emergent properties of human driving behavior using multi aaeet reward augmented imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 789-795. IEEF, 2019.
10

---

Under review as a conference paper at ICLR 2023
Aude Billard and Daniel Grollman. Robot learning by demonstration. Scholarpedia, 8(12):3824, 2013. g
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, and Ashish Kapoor. Pact: Perception-action causal transformer for autoregressive robotics pre-training. arXiv preprint arXiv:2209. / 133, 2022. 2 5
Urs Borrmann, Li Wang, Aaron D Ames, and Magnus Egerstedt. Control barrier certificates for safe swarm behavior. Japrr-apes 48(2)))) 73, 2015. |
Rafael Gomes Braga, Sina Karimi, Ulrich Dah-Achinanon, Ivanka lordanova, and David St- St-Onge. Semantic navigation with domain knowledge. arXiv preprint arXiv:2106. 10220, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411 1-44, 2022. [
Arthur E Bryson and Yu-Chi Ho. Applied optimal control: optimization, estimation, and control. Routledge, 2018.
Jyot Buch, Shih-Ch Liao, and Peter Seiler. Robust control barrier functions with sector-bounded uncertainties. IEEE Control Systems Letters, 6:1994-1999, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 15084-10977 2021. |
Yuxiao Chen, Huei Peng, and Jessy Grizzle. Obstacle avoidance for low-speed autonomous vehicles with barrier function. IEEE Transactions on Control Systems Technology, 26(1):194-266, 2017.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3387-3395, 2019.
Charles Dawson, Bethany Lowenkamp, Dylan Goff, and Chuchu Fan. Learning safe, generalizable perception-based hybrid control with certificates. IEEE Robotics and Automation Letters, 7(2):
1904-1911, 2022a.
9
Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Conference on Robot Learning, pp. 1725-1735. PMLR, 2022b. 9
Staffan Ekvall and Danica Kragic. Robot learning from demonstration: task-level planning ap- proach. International Journal of Advanced Robotic Systems, 5(3):33, 2008.
James Ferlez, Mahmoud Elnaggar, Yasser Shoukry, and Cody Fleming. Shieldnn: A provably safe nn filter for unsafe nn controllers. arXiv preprint arXiv:2006. 09564, 2020.
Jeremy H Gillula, Gabriel M Hoffmann, Haomiao Huang, Michael P Vitus, and Claire J Tomlin. Applications of hybrid reachability analysis to robotic aerial vehicles. The International Journal of Robotics Research, 30(3):335-355 2011.
Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting, In 2020 25th international conference on pattern recognition (ICPR), pp. 10335-10342. IEEE, 2021.
Luis Gressenbuch and Matthias Althoff. Predictive monitoring of traffic rules. In 2021 IEEE Inter- national Intelligent Transportation Systems Conference (ITSC), pp. 915-922, IEEE, 2021.
11

---

Under review as a conference paper at ICLR 2023
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer-
on machine learning, pp. 1861-1870. PMLR, 2018. 17
Sylvia Herbert, Jason Choi, Suvansh Sanjeev, Marsalis Gibson, Koushil Sreenath, and Claire J Tomlin. Scalable learning of safety guarantees for autonomous systems using hamilton-jacobi reachability. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp- 5914-5920. IEEE, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016. 7 9 17
Arne Kreutzmann, Diedrich Wolter, Frank Dylla, and Jae Hee Lee. Towards safe navigation by formalizing navigation nules. TransNav: International Journal on Marine Navigation and Safety of Sea Transportation, 7(2):611-66, 2013.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. arxiv, 2022. g
Xiao Li and Calin Belta. Temporal logic guided safe reinforcement learning using control barrier functions. arXiv preprint arXiv:/903.09883, 2019. 9
Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. In Learning for Dynamics and Control, pp. 608-619. PMLR, 2020.
Wenhao Luo, Wen Sun, and Ashish Kapoor. Sample-efficient safe learning for online nonlinear control with control barrier functions. arXiv preprint arXiv:2207. 14419, 2022. |
Yue Meng. Zengyi Qin, and Chuchu Fan. Reactive and safe road user simulations using neural barrier certificates. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6299-6306. IEEE, 2021.
9
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.00588 2022. g
Sriraam Natarajan, Gautam Kunapuli, Kshitij Judah, Prasad Tadepalli, Kristian Kersting, and Jude Shavlik. Multi-agent inverse reinforcement learning. In 2010 ninth international conference on machine learning and applications, pp. 395-400. IEEE, 2010. 9
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2. 2000. 9
Huansheng Ning, Rui Yin. Ata Ullah, and Feifei Shi. A survey on hybrid gence for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2021.
intelli-
Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4209-4215 IEEE, 2021.
Matthew O'Kelly, Hongrui Zheng, Dhruv Karthik, and Rahul Mangharam. Fltenth: An open-
source evaluation environment for continuous control and reinforcement learning. In NeurIPS 2019 Competition and Demonstration Track, pp. 77-89. PMLR, 2020. 15
Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pp. 763-768. IEEE, 2009. g
Stephen Prajna and Ali Jadbabaie. Safety verification of hybrid systems using barrier certificates. In International Workshop on Hybrid Systems: Computation and Control, pp. 477-477492. Springer, 2004.
g
Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi- agent control with decentralized neural barrier certificates. arXiv preprint arXiv:2101.05436,
2021. 4 g
12

---

Under review as a conference paper at ICLR 2023
Zengyi Qin, Dawei Sun, and Chuchu Fan. Sablas: Learning safe control for black-box dynamical
systems. IEEE Robotics and Automation Letters, 1(2):1928-193)) 2022. 20 21
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprini arXiv:2205.061
2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996. 8
Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Institut für Informatik, Technische Universität München. Technical Report FKI- 126, 90, 1990. [
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1899. PMLR, 2015.
17
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017. 7
Siddhartha S. Srinivasa, Patrick Lancaster, Johan Michalove, Matt Schmittle, Colin Summers, Matthew Rockett, Joshua R. Smith, Sanjiban Chouhury, Christoforos Mavrogiannis, and Fereshteh Sadeghi. MuSHR: low-cost, open-source robotic racecar for education and research. CoRR, abs//1908088
2019. [
Mohit Srinivasan, Amogh Dabholkar, Samuel Coogan, and Patricio A Vela. Synthesis of control barrier functions using a supervised machine learning approach. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7139-7145. IEEE, 2020. 9
Peter Trautman and Andreas Krause. Unfreezing the robot: Navigation in dense, interacting crowds. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 797-803. IEEE, 2010. [
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. Advances in Neural Information Processing Sys-
tems, 33: 11111111111222 2020. [
Jur Van den Berg, Ming Lin, and Dinesh Manocha. Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 IEEE international conference on robotics and automation, pp-
Ieee, 2008. [
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Éukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017. 2
Sai Vemprala and Ashish Kapoor. Adversarial attacks on optimization based planners. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 9943-9949. IEEE, 2021. [
Li Wang, Dongkun Han, and Magnus Egerstedt. Permissive barrier certificates for safe stabilization using sum-of-squares. In 2018 annual American control conference (ACC), pp. 585-590. IEEE, 2018. g
Peter Wieland and Frank Allgöwer. Constructive safety using control barrier functions. IFAC Pro- ceedings Volumes, 40( (011))462-466 2007.
Philipp WU, Alejandro Escontrela. Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning, arXiv preprint ar arxiv:2206. 14/76, 2022. 9
13

---

Under review as a conference paper at ICLR 2023
Dingjiang Zhou, Zijian Wang, Saptarshi Bandyopadhyay, and Mac Schwager. Fast, on-line collision avoidance for dynamic vehicles using buffered voronoi cells. IEEE Robotics and Automation Letters, 102::0477-1044, 2017. T
14


Reviews:
[{"review_id": "KMtJMGizeK", "reviewer": "Reviewer_tYu8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a method to learn from both safe and unsafe demonstrations by constructing a critic that assigns positive values to safe demonstrations and negative values to unsafe demonstrations. The proposed model leverages a transformer backbone. Experiments in simulated environments demonstrate the proposed method outperforms the baseline that does not leverage failed demonstrations.", "summary_of_the_review": "This paper proposed an incrementally novel idea with little conceptual novelty. Experiments are limited in simulated domain and only one weak baseline is included. I do not recommend accepting this paper. ", "strength_and_weaknesses": "This paper aims to build safe systems, which is important for learning-based models. The paper motivated the problem by emphasizing the criticalness of ensuring safety for autonomous agents but the proposed method does not guarantee safe behavior. While the method is claimed to be inspired by control barrier functions, the proposed method just use failed demonstrations as examples of states/state-actions with negative values (assigned by the critic). No benefit of control barrier functions is actually seen here. At the same time, there are many RL methods that leverage failed demonstrations or learn from failed experiences but there is no such baseline compared in the experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "KIuA3jxNFnW", "reviewer": "Reviewer_iZzR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of safety-critical imitation learning, where the learned policy needs to avoid unsafe actions. To this end, the paper builds on the Perception-Action Causal Transformer and augments the framework to learn from safe and unsafe demonstrations. While the experiments show that the derived policy performs reasonably well in two simulation environments, the experimental setups are unclear and the ablation study is missing. I believe this work extends previous work to a practical safety-critical scenario, but the experiments must be improved to support the proposed method.\n", "summary_of_the_review": "The author introduces a convincing learning framework for a practical safety-critical imitation learning problem. However, the experimental setup is unclear and the ablation study is missing. Also, the comparisons to the related works are not entirely fair. In sum, I am leaning toward rejecting this paper because the provided experimental results are not solid enough to support the effectiveness of the proposed method.", "strength_and_weaknesses": "## Paper strengths and contributions\n**Motivation and intuition**\nThe motivation for studying safety-critical imitation learning is convincing.\n\n**Novelty**\nConsidering safety constraints for imitation learning problems is intuitive and convincing. This paper presents an effective way to implement this idea.\n\n**Technical contribution**\nLearning the Control Barrier-like critics (CBC) to predict the value of safety seems effective, especially when the unsafe annotated trajectories are available for training.\n\n**Clarity**\n- The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 1 clearly shows the scenario of the safety-critical imitation learning problem.\n- The paper gives clear descriptions in intuitive ways. The notations, formulations, and designs of the learning framework are well-explained. \n\n**Related work**\nThe proposed method is based on prior work (i.e., Perception-Action Causal Transformer). The authors clearly describe the prior work and how they extend it to address the safety-critical problem. Also, sufficient relevant literature is discussed to help readers understand the context.\n\n**Reproducibility**\nThe paper states that the authors are working towards providing an open-sourced implementation of the proposed method.\n\n## Paper weaknesses and questions\n**Trade-off between safety and performance**\nSafety-critical policy learning is mentioned as a strength in the paper, but this comes at the cost of performance. For a car agent that aims to reach a goal from a starting point, it may choose a safer but longer path with the proposed algorithm. The experiments use the collision rate and the average trajectory length as metrics, so the policy only needs to ensure no crash occurs. I would also like to see experiments that consider the policy's optimality.\n\n**Experiment**\n\n- The description of the experimental setups is not clear to me. While the claims seem to be promising, the amount of unsafe training trajectories is not provided, which is very important to determine how practical the proposed method is. To be more specific, for the F1/10 race car environment, the authors train the model with 1K demonstrations (Sec 3.1) where the expert trajectory is generated from a search-based model predictive control (Supp. A.1). How many trajectories are unsafe among the 1K training samples? How to generate and collect such unsafe trajectories? The statistic of the training data is not specified for the MuSHR environment.\n- The comparisons to the previous works may be unfair. First, comparing the proposed method with reinforcement learning methods such as PPO, TRPO, and SAC, is unreasonable. The proposed method learns from demonstrations generated by a search-based expert model. In contrast, the RL algorithms learn from a pre-defined reward function (Supp. B.2). Comparing methods with different input signals is improper. Second, the authors use the same expert trajectories containing unsafe samples for ConBat and other imitation learning algorithms. The imitation learning algorithms such as BC and GAIL learn from the expert data, so inferior results during deployment are expected. Since the annotations of the trajectories are available, it is more reasonable to use collision-free samples to learn those imitation learning baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HRc0qkWCdU", "reviewer": "Reviewer_89Hu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for safe policy learning that relies on a causal transformer architecture that proposes actions, and a pair of critics that predict a current safety score and a future safety value. The method is inspired by control barrier functions, and critics trained using binary supervision (safe and unsafe trajectories.) Results on two simulated navigation environments show that the method is effective and outperforms baselines that (do not?) consider safe learning.", "summary_of_the_review": "This paper proposes an interesting model and approach to learn safe policies, so I recommend acceptance. However, this approach is only applicable in certain situations where examples of unsafe and safe trajectories can be obtained, which limits the applicability of this work somewhat. I'd also value some more precise language to avoid giving readers the impression that this approach allows for safe policy learning. ", "strength_and_weaknesses": "Strengths:\n\nThe idea makes sense, the paper is well written and motivated, and the proposal to use two critics is an interesting one, with ablations  exploring this valuable.\n\nWeaknesses:\n\nBaselines. As far as I can tell, none of these baselines consider \"safe\" policy learning, so while the comparison is useful to show that the proposed approach works as well as those that don't consider safety, I feel it would be valuable to put this into perspective with some of the approaches in Brunke 22 or those mentioned in related work.\n\nIt is also unclear to me whether these approaches had the benefit of the additional safe/unsafe training data, which would presumably be quite informative to the policy. It looks like they don't get this information, which does not really make the comparison very fair?\n\nThe proposed approach also needs unsafe demonstrations to learn, which isn't always practical (i.e. I didn't learn to drive by crashing a lot and being told that sequence of states and actions was \"bad\"). Is it fair to term this a \"safety-critical policy learning\" approach, if it needs to violate safety constraints to learn? As I understand it the resultant policy may be safe, but to me the primary purpose of a barrier function is to prevent any violation of the safety constraints, both during learning and after so I found the link to control barrier functions a little tenuous.\n\nAblations show that relatively large amounts unsafe data would be required to obtain an acceptable safety level.\n\nAlong these lines, I think the term \"safe policy learning\"  is ambiguious and potentially misleading (although I recognise the precedent in this field). The key distinction is that the learning process is clearly not safe, although the resultant policy may be safe(-ish). I think this needs improvement (eg. learning safe policies)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H6nIE52GJOR", "reviewer": "Reviewer_okow", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to learn a \"control barrier critic\" (CBC) on top of PACT, an autoregressive world model and policy trained through imitation learning. The CBC is loosely inspired by control barrier functions (CBFs), a scalar function of state which if can be found for a system and policy guarantees that if the policy starts in a safe state will always stay in the set of safe states. The CBC is trained using expert demonstrations along with safe/unsafe labels, and is optimized to satisfy the conditions of true CBFs.\n\nOnce a CBC (safety score) has been optimized, it can be used to improve the safety of the policy at inference time. Because the safety score should be positive, if an action taken by the policy were to achieve a negative safety score a minimum perturbation could be applied to that action to ensure the safety score is non-negative in the next time-step. This minimum perturbation is found through gradient descent.\n\nThe authors present experiments in two simulated environments: F1/10, a 2D racing simulation, and MuSHR, a more complex simulation of a toy car navigating an office space with LIDAR. ConBaT is shown to be safer than PACT or finetuned PACT, and beats various baselines on the MuSHR environment.", "summary_of_the_review": "The paper is well written, and the idea is novel and well motivated. However, the lack of convincing experiments concerns me (see \"weaknesses\" section). It is hard to compare ConBaT against baselines, and as a result it's not clear if the method is actually effective.", "strength_and_weaknesses": "Strengths:\n\nThe idea is pretty clever, and very well motivated by control barrier functions. In addition, I quite enjoyed reading about the ConBaT-NS experiment.\n\nWeaknesses:\n\nThe biggest weakness is the lack of experiments, and the fact that they are unconvincing.\n\nThe first experiment compares ConBaT to PACT and PACT-FT. This experiment seems more of a proof of concept, as neither PACT nor PACT-FT are given access to safe/unsafe trajectory labels (as far as I can tell). Of course ConBaT should outperform PACT here. \n\nThe second experiment compares ConBaT to a variety of imitation learning and RL baselines. The number of baselines is high, but their implementation is not clear to me. From the appendix I see that the RL baselines have a hand-designed reward function discouraging collisions. This makes comparing ConBaT to the RL baselines difficult as one is given a reward function and one is given expert demonstrations with safe/unsafe labels. The imitation learning baselines also do not seem to get access to the the safe/unsafe labels (but please correct me if I'm wrong), making comparison difficult. What this paper could benefit from is baselines that are more comparable to ConBaT's setup.\n\nSome other things I think this paper could benefit from:\n\nAnalysis of how often and when the CBF conditions are satisfied by the critics. This could be interesting to look at: for example, see if unsafe regions really do have negative safety score, or how consistent the critics are in different regions.\n\nOn a related note, are all the CBF conditions necessary? Do we need to optimize all three losses? ConBaT may be inspired by CBFs but perhaps the method could be simplified.\n\nAn aside: I don't quite understand the emphasis on the transformer aspect of this method. As far as I can tell these critics can be trained on a variety of architectures, and emphasizing transformers detracts from the core idea. Perhaps this versatility could be a benefit.\n\nAnother aside: I don't understand why the RL baselines don't just learn to drive in circles. Does the turning radius prevent this?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject