ABNet: Attention BarrierNet for Safe and Scalable Robot Learning

## Abstract

Abstract Safe learning is central to AI-enabled robots where a single failure may lead to catastrophic results. Barrier-based method is one of the dominant approaches for safe robot learning. However, this method is not scalable, hard to train, and tends to generate unstable signals under noisy inputs that are challenging to be deployed for robots. To address these challenges, we propose a novel Attention BarrierNet (ABNet) that is scalable to build larger foundational safe models in an incremental manner. Each head of BarrierNet in the ABNet could learn safe robot control policies from different features and focus on specific part of the observation. In this way, we do not need to one-shotly construct a large model for complex tasks, which significantly facilitates the training of the model while ensuring its stable output. Most importantly, we can still formally prove the safety guarantees of the ABNet. We demonstrate the strength of ABNet in 2D robot obstacle avoidance, safe robot manipulation, and vision-based end-to-end autonomous driving, with results showing much better robustness and guarantees over existing models. 1 1 1 Code will be available once approved: https://github.com/Weixy21/ABNet

## 1 Introduction

Figure: Figure 1: The proposed ABNet that is robust, scalable and generates stable output while guaranteeing safety for robots. Each head of BarrierNet in the model could learn safe control policies with attention on different observation feature in a scalable or one-shot manner.
Refer to caption: x1.png

Robot learning usually requires to leverage scalable training and vast amount of data. There are many large models for complex robotic tasks including manipulation, locomotion, autonomous driving [] [] []. However, these models are not trustworthy and have no safety guarantees. Existing methods that incorporate guarantees or certificates into neural networks are not scalable and hard to train [] [] []. It is desirable to merge these safe models for complicated robot tasks. Traditional mixture of expert methods [] [] [] or other merging approaches [] [] [] are hard to retain the safety of the models. In this work, we explore to leverage the collective power of many safety-critical models to handle complex tasks while preserving the safety of the merged models.

There are various definitions of safety for robotics and autonomy, and safety can be basically defined as something bad never happens. Mathematically, safety can be defined as a continuously differentiable constraint with respect to the system state and it can be further captured by the forward invariance of the safe set over such a constraint [] [] []. In other words, we can use different constraints and approaches to enforce safety. The way we learn such safety enforcement methods may depend on the focused observation feature, which corresponds to the attention mechanism. For instance, some human drivers may focus on the left lane boundary in driving in order to achieve safe lane keeping, while others may focus on the right lane boundary, as shown in Fig. 1. Both attention mechanisms can achieve similar purpose. Merging these models or attention mechanisms enables us to build robust and powerful learning models. However, retaining safety is non-trivial.

In the literature, barrier-based learning methods [] [] [] [], such as the BarrierNet [] [] [], are widely used to equip deep learning systems with safety guarantees. We may incorporate control-theoretic based optimizations into learning systems in the form of differentiable quadratic programs (dQPs) []. There are several limitations of these barrier-based learning methods: $(i)$ it can only implement a single safety enforcement method as the last layer of the neural network, which is not scalable to larger safe learning models; $(ii)$ the model is not robust such that it is hard to be trained to work for complicated robotic applications; $(iii)$ these methods tend to generate unstable output under noisy observation, which is intractable to be deployed for robots.

In this paper, we propose a novel Attention BarrierNet (ABNet) to merge many safety-critical models while preserving the safety guarantees.The ABNet is scalable, robust to noise, and easy to be trained in an incremental manner. As shown in Fig. 1, we may build multi-head BarrierNets within the ABNet. Each head of the BarrierNet may pay attention to different observation features to generate a safe control policy. We linearly combine the outputs of all the BarrierNets in a way that is provably safe. The weights of this combination quantify the importance of each head of BarrierNet, and they are trainable. The structure of the ABNet allows us to build larger foundational safe models for various and complicated robotic applications as we can incrementally train safe models corresponding to different robot skills and this will simply increase the head $h$ of BarrierNets.

In summary, we make the following new contributions:

- •
We propose a novel ABNet that merges many safety-critical learning models, and this new model is scalable, robust, and easy to be trained.
- •
We formally prove the safety guarantees of the proposed ABNet.
- •
We demonstrate the strength and effectiveness of our model on a variety of robot control tasks, including 2D robot obstacle avoidance, safe robot manipulation, and vision-based end-to-end autonomous driving in an open dataset. We also show that existing models/policies merging could make safety worse in complicated tasks (such as in vision-based driving).

## 2 Preliminaries and Problem Formulation

In this section, we present background on the forward invariance with High-Order Control Barrier Functions (HOCBFs) that is widely used to enforce safety, as well as introduce the BarrierNet.

Forward Invariance with HOCBFs.
Consider an affine control system defined as:

$$ $\dot{\bm{x}}=f(\bm{x})+g(\bm{x})\bm{u}$ (1) $$

where $\bm{x}\in\mathbb{R}^{n}$ is the system state, $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$
and $g:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times q}$ are locally
Lipschitz, and $\bm{u}\in U\subset\mathbb{R}^{q}$, where $U$ denotes a control constraint set. $\dot{\bm{x}}$
denotes the time derivative of state $\bm{x}$.

We begin with some definitions before introducing the HOCBF.

###### Definition 2.1 .

(Forward invariance []):
A set $C\subset\mathbb{R}^{n}$ is forward invariant for system (1) if its solutions for some $\bm{u}\in U$ starting at any $\bm{x}(0)\in C$ satisfy $\bm{x}(t)\in C,$ $\forall t\geq 0$.

###### Definition 2.2 .

(Class $\mathcal{K}$ function []):
A Lipschitz continuous function $\alpha:[0,a)\rightarrow[0,\infty),a>0$ belongs to class $\mathcal{K}$ if it is strictly increasing and $\alpha(0)=0$.

###### Definition 2.3 .

(Relative degree []): The relative degree of a (sufficiently many times) differentiable function $b:\mathbb{R}^{n}\rightarrow\mathbb{R}$ with respect to system (1) is defined as the number of times that we need to differentiate $b$ along the system (1) until any component of the control $\bm{u}$ explicitly shows up in the corresponding derivative.

Since a function $b(\bm{x})$ is used to defined a safety constraint $b(\bm{x})\geq 0$, we refer to the relative degree of the constraint as the relative degree of the function.
Consider a safety constraint $b(\bm{x})\geq 0$ with relative degree $m$ for system (1), where $b:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is continuously differentiable, we recursively define a sequence of CBFs $\psi_{i}:\mathbb{R}^{n}\rightarrow\mathbb{R},i\in\{1,\dots,m\}$ in the form:

$$ $\psi_{i}(\bm{x}):=\dot{\psi}_{i-1}(\bm{x})+\alpha_{i}(\psi_{i-1}(\bm{x})),i\in \{1,\dots,m\},$ (2) $$

where $\psi_{0}(\bm{x}):=b(\bm{x})$, and $\alpha_{i},i\in\{1,\dots,m\}$ are class $\mathcal{K}$ functions.

We further define a sequence of safe sets $C_{i},i\in\{1\dots,m\}$ corresponding to (2) in the form:

$$ $C_{i}:=\{\bm{x}\in\mathbb{R}^{n}:\psi_{i-1}(\bm{x})\geq 0\},i\in\{1,\dots,m\}.$ (3) $$

###### Definition 2.4 .

(High Order Control Barrier Function (HOCBF) []):
Let $C_{i},i\in\{1,\dots,m\}$ and $\psi_{i},i\in\{1,\dots,m\}$ be defined by (3) and (2), respectively.
A function $b:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a HOCBF if there exist class $\mathcal{K}$ functions $\alpha_{i},i\in\{1\dots,m\}$ such that

$$ $\sup_{\bm{u}\in U}[L_{f}\psi_{m-1}(\bm{x})+[L_{g}\psi_{m-1}(\bm{x})]\bm{u}+ \alpha_{m}(\psi_{m-1}(\bm{x}))]\geq 0,$ (4) $$

for all $\bm{x}\in\cap_{i=1}^{m}C_{i}$. $L_{f}$ and $L_{g}$ denote Lie derivatives w.r.t. $\bm{x}$ along $f$ and $g$, respectively.

###### Theorem 2.5 ( [ 38 ] ) .

Given a HOCBF $b(\bm{x})$ from Def. 2.4, if $\bm{x}(0)\in\cap_{i=1}^{m}C_{i}$, then any Lipschitz continuous controller $\bm{u}(t)$ that satisfies the constraint in (4), $\forall t\geq 0$ renders $\cap_{i=1}^{m}C_{i}$ forward invariant for system (1).

The HOCBF is a general form of the CBF [] [], i.e., setting the relative degree $m=1$ of a safety constraint $b(\bm{x})\geq 0$ will reduce a HOCBF to a CBF. CBFs/HOCBFs are widely used to transform nonlinear optimal control problems into a sequence of Quadratic Programs (QPs) that are very efficient to solve while preserving the safety guarantees of the system.

BarrierNet. The BarrierNet [] is a neural network layer that incorporates CBF/HOCBF-based QPs as differentiable QPs (dQPs) [], in which all the CBFs/HOCBFs are differentiable in terms of their parameters (such as those in class $\mathcal{K}$ functions). Those parameters are crucial to the system conservativeness or performance in guaranteeing safety. In summary, the BarrierNet frees us from handing-tuning all the parameters in safety-critical controls, and simply uses data to optimize them. Referring to Fig. 1, a BarrierNet only has a single head in the model (i.e., $h=1$) and it is placed as the last layer of the model when used in conjunction with other neural networks (such as CNN and LSTM).

In this paper, we consider the following safe learning problem:

Problem. Given (a) a system with dynamics in the form of (1); (b) a state-feedback nominal controller $\pi^{*}(\bm{x})=\bm{u}^{*}$ (such as a model predictive controller) that provides the training label; (c) a set of safety constraints $b_{j}(\bm{x})\geq 0,j\in S$ ($b_{j}$ is continuously differentiable, $S$ is a constraint set); (d) a neural network controller $\pi(\bm{x},\bm{z}|\theta)=\bm{u}$ parameterized by $\theta$ (under observation $\bm{z}$);

Our goal is to find the optimal parameter

$$ $\theta^{*}=\arg\min_{\theta}\mathbb{E}_{\bm{x},\bm{z}}[\ell(\pi^{*}(\bm{x}), \pi(\bm{x},\bm{z}|\theta))],$ (5) $$

while satisfying all the safety constraints in (c) and the dynamics constraint (a). $\mathbb{E}$ is the expectation, and $\ell$ is a loss function.

###### Definition 2.1 .

###### Definition 2.2 .

###### Definition 2.3 .

###### Definition 2.4 .

###### Theorem 2.5 ( [ 38 ] ) .

## 3 Attention BarrierNet

In this section, we present the architecture of the Attention BarrierNet (ABNet) and formally prove its safety guarantees in learning systems.

### 3.1 Multi-head BarrierNets

We can use a BarrierNet to transform the constrained optimal control in the considered problem into the following differentiable QP, which forms a head of BarrierNet in the model:

$$ $\bm{u}_{k}=\arg\min_{\bm{u}(t)\in U}\frac{1}{2}\bm{u}(t)^{T}H(\bm{z}_{k}| \theta_{h,k})\bm{u}(t)+F^{T}(\bm{z}_{k}|\theta_{f,k})\bm{u}(t)$ (6) $$

s.t.

$$ $\displaystyle L_{f}$ $\displaystyle\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+[L_{g}\psi_{j,m-1}(\bm{x}, \bm{z}|\theta_{p})]\bm{u}+p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m})\alpha_{j,m}( \psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p}))\geq 0,j\in S,$ (7) $\displaystyle\psi_{j,i}(\bm{x},\bm{z}|\theta_{p})=\dot{\psi}_{j,i-1}(\bm{x}, \bm{z}|\theta_{p})+p_{i}(\bm{z}|\theta_{p}^{i})\alpha_{j,i}(\psi_{j,i-1}(\bm{x },\bm{z}|\theta_{p})),i\in\{1,\dots,m-1\},j\in S,$ $\displaystyle\psi_{j,0}(\bm{x},\bm{z}|\theta_{p})=b_{j}(\bm{x}),j\in S,\quad \quad t=\omega\Delta t+t_{0},\omega\in\{0,1,\dots\},$ $$

where $k\in\{1,\dots,h\}$, and $h$ is the number of heads of BarrierNet (as shown in Fig. 1). $p_{i}\geq 0,i\in\{1,\dots,m-1\},p_{m,k}\geq 0$ are penalty functions on the class $\mathcal{K}$ functions $\alpha_{j,i},i\in\{1,\dots,m\},j\in S$ that address the conservativeness of the model (e.g., how far away the system state should stay form the unsafe set bound in order to maintain safety). All the HOCBFs corresponding to the safety constraints share the same penalty functions, but they may use different ones in which case $p_{i}$ and $p_{m,k}$ will be dependent on $j,j\in S$. The derivatives of the observation $\bm{z}$ in the above are omitted, as shown in []. $H(\bm{z}_{k}|\theta_{h,k})\in\mathbb{R}^{q\times q}$ is positive definite, and $H^{-1}(\bm{z}_{k}|\theta_{h,k})F(\bm{z}_{k}|\theta_{f,k})$ can be interpreted as a reference control (the output of previous network layers). $\theta:=(\theta_{h},k,\theta_{f,k},\theta_{p,k}^{m},\theta_{p}),k\in\{1,\dots,h\}$, where $\theta_{p}:=(\theta_{p}^{1},\dots,\theta_{p}^{m-1})$ are all trainable parameters of the neural network. $\bm{z}_{k}$ is the observation of the BarrierNet head $k,k\in\{1,\dots,h\}$, and it is possible that all heads share the same observation, i.e. $\bm{z}_{k}=\bm{z},\forall k\in\{1,\dots,h\}$. $\Delta t>0$ is the discretized time interval, and $t_{0}$ is the initial time.

Figure: Figure 2: Architecture of multi-head BarrierNets (i.e., ABNet). The ABNet is usually used in conjunction with any other neural networks and can be implemented in parallel. The parameters (inputs) of each head of BarrierNet are the outputs of previous layers (such as CNN or LSTM).
Refer to caption: x2.png

Attention mechanism. Each head of BarrierNet may learn safe self-attention even if all the BarrierNets have the same observation $\bm{z}$. The parameter $p_{m,k}^{m}$ may be learned from different input features via random initialization, and it determines the conservativeness of the model in guaranteeing safety. On the other hand, we may also make each head of BarrierNet focus on different observations $\bm{z}_{k}$. The observation $\bm{z}_{k}$ may come from different parts of the sensor observation (such as the left lane boundary and right lane boundary in driving shown in Fig. 1), or even different perceptions (such as vision, lidar, etc.)

Cross connection. It can be noted from (7) that each head of BarrierNet $k\in\{1,\dots,h\}$ has some cross connection with other heads, as also shown in Fig. 1. In other words, $\psi_{j,i}(\bm{x},\bm{z}|\theta_{p}),i\in\{1,\dots,m-1\},j\in S$ are formulated in the same way through the shared parameter $\theta_{p}$ (independent from $k$). This structure is to ensure (i) the construction for provable safety (as shown later), and (ii) some shared information across different heads of BarrierNet as they all generate safe control policies for the same system (1).

Fusion. Another important consideration is how should we fuse all these controls $\bm{u}_{k},k\in\{1,\dots,h\}$ while preserving the safety property of each head of the BarrierNet. We propose the following form:

$$ $\bm{u}=\sum_{k=1}^{h}w_{k}\bm{u}_{k},\quad\text{where }\sum_{k=1}^{h}w_{k}=1.$ (8) $$

In the above, $w_{k}\geq 0,k\in\{1,\dots,h\}$ are trainable parameters. The composition of all the heads of BarrierNet (6) s.t., (7) in the form of (8) is our proposed ABNet, as shown in Fig. 2. The safety guarantees of the ABNet is shown in the following theorem:

###### Theorem 3.1 .

(Safety of ABNets)
Given the multi-head BarrierNets formulated as in (6) s.t. (7). If the system (1) is initially safe (i.e., $b_{j}(\bm{x}(t_{0}))\geq 0,\forall j\in S$), then a control policy $\bm{u}$ from the ABNet output (8) guarantees the safety of system (1), i.e., $b_{j}(\bm{x}(t))\geq 0,\forall j\in S,\forall t\geq t_{0}$.

All the proofs for theorems are given in Appendix A. If the system is not initially safe (i.e., $b_{j}(\bm{x}(t_{0}))<0,\exists j\in S$), then the system state $\bm{x}$ of (1) will be driven to the safe side of the state space due to the Lyapunov property of CBF/HOCBFs [] []. This enables the possibility of utilizing data that violates safety to conduct adversary training of the ABNet.

Natural noise filter. The ABNet is a natural noise filter since $w_{k}\in[0,1],\forall k\in\{1,\dots,h\}$ in (8). This can ensure that the output $\bm{u}$ of the model is stable with a large enough head number $h$ if all the BarrierNets have different observation $\bm{z}_{k}$ for the current environment. This feature makes ABNet a very robust controller for robotic systems, and thus, ABNet can generate smooth signals.

###### Theorem 3.2 .

(Safety of merging of ABNets) Given two ABNets with each formulated as in (8) and (6) s.t. (7), the merged model using the form as in (8) again guarantees the safety of system (1).

###### Theorem 3.1 .

###### Theorem 3.2 .

### 3.2 Model Training

The ABNet can be trained incrementally or in one-shot. This is due to the fact that each head of BarrierNet can generate a control policy that is applicable to system (1). The linear combination weights $w_{k},k\in\{1,\dots,h\}$ in the ABNet denote the importance of the corresponding control policies.

Scalable training. In ABNet, we may train each head $k,k\in\{1,\dots,h\}$ of the BarrierNet in a scalable way as we wish to minimize the loss between their output $\bm{u}_{k}$ and the label $\bm{u}^{*}$ as well. The training can be done using the batch QP training method proposed in []. There are some cross connections via $p_{i}(\bm{z}|\theta_{p})$ between BarrierNets in the ABNet that may prevent the implementation of the training. We may address this by training a $p_{i}(\bm{z}|\theta_{p})$ for each head of the BarrierNet. After we train all heads of the BarrierNet, we may fix the parameters of those models, choose a $p_{i}(\bm{z}|\theta_{p})$ from one of the BarrierNets (or take an average of all $p_{i}(\bm{z}|\theta_{p})$ among the BarrierNets) to build the cross connection, and train the weights $w_{i}$ for some more iterations. Another way is to fuse these BarrierNets by their testing loss. In other words, the weight $w_{k},k\in\{1,\dots,h\}$ can be determined by:

$$ $w_{k}=\frac{\frac{1}{\ell_{k}(\bm{u}_{k},\bm{u}^{*})}}{\sum_{k=1}^{h}\frac{1}{ \ell_{k}(\bm{u}_{k},\bm{u}^{*})}},$ (9) $$

where $\ell_{k}$ is a loss function. We may also use some exponential functions of the losses to determine $w_{k}$ similarly as in the above equation.

Figure: Algorithm 1 Construction and training of ABNet

If we already have some trained ABNet, and we wish to add some new capabilities (such as safe driving by only focusing on the left lane boundary) to the model, then we can train some heads of BarrierNets based on the new data we have. Finally, we can fuse the models similarly with safety guarantees as shown in Thm. 3.2. This shows the scalability of the proposed ABNet that allows us to build larger foundational safe models in an incremental way.

One-shot training. The one-shot training of the ABNet can be directly done using the traditional reverse mode automatic differentiation. In addition to the loss between the eventual output $\bm{u}$ of the ABNet and the label $\bm{u}^{*}$, we may also consider the losses on $\bm{u}_{k},k\in\{1,\dots,h\}$, as well as on the reference controls $H^{-1}(\bm{z}_{k}|\theta_{h,k})F(\bm{z}_{k}|\theta_{f,k})$, in order to improve the training performance.

The construction and training of the ABNet involve the formulation of each head of BarrierNet as in (6) s.t. (7), the BarrierNet fusion as in (8), and the scalable or one-shot training as shown above. We summarize this process in Alg. 1.

Figure: Figure 3: 2D robot obstacle avoidance closed-loop testing control profiles (left) and ABNet performance with the increasing of BarrierNet heads using scalable training (right). This scalable training for ABNet is with safety guarantees. The controls are subject to input noise, and thus are non-smooth.
Refer to caption: x3.png

## 4 Experiments

**Table 1: 2D robot obstacle avoidance closed-loop testing under noisy input and comparisons with benchmarks.**
| Model | MSE($\downarrow$) | Safety ($\geq 0$) | Conser. ($\geq 0\&\downarrow$) | $u_{1}$ Uncertainty ($\downarrow$) | $u_{2}$ Uncertainty ($\downarrow$) | Theoret. guar. |
| --- | --- | --- | --- | --- | --- | --- |
| E2E [] | ${0.007{\small\pm 0.004}}$ | -14.140 | $-2.976{\small\pm 3.770}$ | 0.063 | 0.049 | $\times$ |
| E2Es-MCD [] | ${\color[rgb]{1,0,0}0.004{\small\pm 0.001}}$ | -2.087 | $-1.341{\small\pm 0.824}$ | 0.041 | 0.026 | $\times$ |
| E2Es-DR [] | ${0.080{\small\pm 0.006}}$ | -35.130 | $-3.176{\small\pm 4.299}$ | ${\color[rgb]{1,0,0}0.032}$ | 0.020 | $\times$ |
| DFB [] | $0.013{\small\pm 0.003}$ | ${\color[rgb]{1,0,0}36.659}$ | $47.810{\small\pm 4.377}$ | 0.062 | 0.052 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet [] | $0.014{\small\pm 0.006}$ | ${\color[rgb]{1,0,0}5.045}$ | $7.966{\small\pm 1.287}$ | 0.074 | 0.047 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet-UP [] | $0.008{\small\pm 0.004}$ | ${\color[rgb]{1,0,0}5.988}$ | $8.573{\small\pm 1.738}$ | 0.054 | 0.028 | $\times$ |
| ABNet-10-sc (ours) | $0.011{\small\pm 0.007}$ | ${\color[rgb]{1,0,0}5.731}$ | ${\color[rgb]{1,0,0}6.269{\small\pm 0.319}}$ | 0.065 | 0.027 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-10 (ours) | $0.008{\small\pm 0.005}$ | ${\color[rgb]{1,0,0}12.639}$ | $13.887{\small\pm 1.323}$ | ${0.049}$ | 0.030 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-100 (ours) | $0.012{\small\pm 0.006}$ | ${\color[rgb]{1,0,0}10.122}$ | $11.729{\small\pm 0.816}$ | ${0.049}$ | ${\color[rgb]{1,0,0}0.013}$ | ${\color[rgb]{1,0,0}\surd}$ |

In this section, we conduct several experiments to answer the following questions:

- •
Does our method match the theoretic results in experiments and guarantee the safety of robots in various tasks quantitatively, qualitatively and is it scalable?
- •
How does our method compare with state-of-the-art models (baseline E2E, safety-guaranteed models, policies merging, models merging) in enforcing safety constraints?
- •
The benefit of models/policies merging and the robustness of our models in safety and smoothness?

Benchmark models: We compare with (i) baseline: Tables 1, 2–single end-to-end learning model (E2E) [] and Table 3–single vanilla end-to-end (V-E2E) model [], (ii) safety guaranteed models: single BarrierNet (BNet) [], Deep forward and backward (DFB) model [], (iii) policies merging: BarrierNet policies merged with uncertainty propagation (BNet-UP) [] that employs Gaussian kernels with Scott’s rule [] to select the bandwidth, (iv) models merging: E2Es merged with Monte-Carlo Dropout (E2Es-MCD) [], E2Es merged with Deep Resembles (E2Es-DR) [].

Our models: Sec. 4.1 and 4.2: ABNet trained in a scalable way with 10 heads (ABNET-10-SC), ABNet trained in one shot with 10 heads (ABNET-10), ABNet trained in one shot with 100 heads (ABNET-100). Sec. 4.3: our ABNet trained in one shot with 10 heads using the same input images (ABNET), ABNet with attention images and 10 heads (ABNET-ATT), our ABNet first trained with ABNET scaled/augmented by ABNET-ATT (20 heads, ABNET-SC).

Evaluation metrics: The evaluation metrics in all the tables are defined as follows: mean square error of the model testing (MSE), satisfaction of safety constraints where non-negative values mean safety guarantees (SAFETY), system conservativeness (CONSER.), steering control $u_{1}$ uncertainty ($u_{1}$ UNCERTAINTY), acceleration control $u_{2}$ uncertainty ($u_{2}$ UNCERTAINTY), and theoretical safety guarantees (THEORET. GUAR.) respectively. All the metrics are explicitly defined in Appendx B.

### 4.1 2D Robot Obstacle Avoidance

We aim to find a neural network controller for a 2D robot that can drive the robot from an initial location to an arbitrary destination while avoiding crash onto the obstacle. All the models (h copies/heads) have the same input (with uniformly distributed noise, $10\%$ of the input magnitude in testing). The detailed problem setup and model introductions are given in Appendix B.1.

Models/policies merging can improve the performance as shown by the MSE metrics in Table 1 and the scalable training in Fig. 3. Note that our scalable training for ABNets has safety guarantees. The DFB tends to be very conservative as the CBFs within which are not differentiable, which presents a high conservative value shown in Table 1. Our proposed ABNets can significantly reduce the uncertainty of the outputs (controls) under noisy input while guaranteeing safety, and this uncertainty decreases as the increases of the BarrierNet heads in the ABNets, as shown by the last two and three columns in Table 1, as well as shown in Fig. 3 and 6 of Appendix B.1 where the control uncertainty of ABNet-100 is lower than the one of BNet. The smoothness of the controls also increases with the increase of BarrierNet heads (e.g., blue from ABNet v.s. red from BNet in Fig. 6). In terms of performance, our proposed ABNets can also improve the testing errors compared to BNet and DFB, as shown by the MSE in Table 1. The E2Es-MCD model can achieve the best performance, but this is at the cost of safety (the SAFETY metric in Table 1 is negative, which implies violated safety).

**Table 2: Robot manipulation closed-loop testing under noisy input and comparisons with benchmarks.**
| Model | MSE($\downarrow$) | Safety ($\geq 0$) | Conser. ($\geq 0\&\downarrow$) | $u_{1}$ Uncertainty ($\downarrow$) | $u_{2}$ Uncertainty ($\downarrow$) | Theoret. guar. |
| --- | --- | --- | --- | --- | --- | --- |
| E2E [] | ${3.6e\text{-}4{\small\pm 1.7e\text{-}4}}$ | -11.027 | $-1.082{\small\pm 2.992}$ | 0.013 | 0.009 | $\times$ |
| E2Es-MCD [] | ${1.1e\text{-}4{\small\pm 7.3e\text{-}5}}$ | -11.827 | $0.162{\small\pm 2.085}$ | 0.008 | 0.005 | $\times$ |
| E2Es-DR [] | ${1.3e\text{-}4{\small\pm 8.5e\text{-}5}}$ | -11.381 | $-0.958{\small\pm 1.875}$ | ${0.007}$ | 0.005 | $\times$ |
| DFB [] | ${8.7e\text{-}4{\small\pm 1.9e\text{-}4}}$ | ${\color[rgb]{1,0,0}2.905}$ | $6.023{\small\pm 3.110}$ | 0.019 | 0.018 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet [] | ${2.3e\text{-}4{\small\pm 1.2e\text{-}4}}$ | ${\color[rgb]{1,0,0}0.147}$ | $0.745{\small\pm 0.505}$ | 0.010 | 0.009 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet-UP [] | ${\color[rgb]{1,0,0}5.2e\text{-}5{\small\pm 3.2e\text{-}5}}$ | ${\color[rgb]{1,0,0}0.206}$ | $0.346{\small\pm 0.098}$ | ${\color[rgb]{1,0,0}0.005}$ | 0.005 | $\times$ |
| ABNet-10-sc (ours) | ${5.9e\text{-}5{\small\pm 5.5e\text{-}5}}$ | ${\color[rgb]{1,0,0}0.233}$ | ${0.570{\small\pm 0.360}}$ | 0.006 | 0.005 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-10 (ours) | ${1.2e\text{-}4{\small\pm 9.6e\text{-}5}}$ | ${\color[rgb]{1,0,0}0.039}$ | $0.272{\small\pm 0.443}$ | ${0.008}$ | 0.007 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-100 (ours) | ${1.1e\text{-}4{\small\pm 4.4e\text{-}5}}$ | ${\color[rgb]{1,0,0}0.053}$ | ${\color[rgb]{1,0,0}0.123{\small\pm 0.177}}$ | ${\color[rgb]{1,0,0}0.005}$ | ${\color[rgb]{1,0,0}0.004}$ | ${\color[rgb]{1,0,0}\surd}$ |

Figure: Figure 4: Robot manipulation closed-loop end-effector trajectories (left) and ABNet performance with the increasing of BarrierNet heads using scalable training (right). This scalable training for ABNet is with safety guarantees. The transparent red and blue trajectories in the left figure are corresponding to BNet and ABNet-100 models in all runs, respectively.
Refer to caption: x5.png

### 4.2 Safe Robot Manipulation

In robot manipulation, we employ a two-link planar robot manipulator to grasp an object from an arbitrary point to an arbitrary destination while avoiding crashing onto obstacles. All the models (h copies/heads) have the same input (with uniformly distributed noise, $10\%$ of the input magnitude in testing). We compare our proposed ABNets with the same benchmark models as in the last subsection. More detailed problem setup and model introductions are given in Appendix B.2.

Again, models/policies merging can improve the performance as shown by the MSE metrics in Table 2 and the sclable training in Fig. 4.
All the E2E-related models are not robust to noise and violate safety constraints (i.e., crash onto obstacles) under noisy input since there are no formal guarantees, and such an example is shown by the magenta trajectory curve of the end-effector in Fig. 4. As shown in Table 2, the proposed ABNet-100 model is the least conservative one with the lowest control uncertainties as well under noisy inputs (significantly improved compared with BNet and DFB), which demonstrates its advantage over other models. This uncertainty improvement is also shown by the control distributions in Fig. 7 in Appendix B.2 (BNet: red area v.s. ABNet-100: blue area). The BNet-UP achieves the best performance without safety guarantees.

### 4.3 Vision-based End-to-End Autonomous Driving

We finally test our models in a more complicated and realistic task: vision-based driving, using an open dataset and benchmark from the VISTA [].
One of ABNets, named ABNet-att, is constructed such that different heads of BarrierNets focus on different parts of the image (left lane boundary, right lane boundary, etc., the corresponding images are shown in Fig 8 of Appendix B.3). For more experiment and model details, please refer to Appendix B.3.

**Table 3: Vision-based end-to-end autonomous driving closed-loop testing and comparisons with benchmarks. New items are short for obstacle crash rate (CRASH), obstacle passing rate (PASS).**
| Model | Crash ($\downarrow$) | Pass ($\uparrow$) | Safety ($\geq 0$) | Conser. ($\geq 0\&\downarrow$) | $u_{1}$ Uncertainty ($\downarrow$) | $u_{2}$ Uncertainty ($\downarrow$) | Theoret. guar. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| V-E2E [] | 6% | 94% | -60.297 | $-0.610{\small\pm 21.165}$ | 0.443 | 0.222 | $\times$ |
| E2Es-MCD [] | 8% | 92% | -60.566 | $-2.211{\small\pm 22.343}$ | 0.429 | 0.227 | $\times$ |
| E2Es-DR [] | 9% | 91% | -60.572 | $-1.499{\small\pm 21.500}$ | 0.431 | 0.224 | $\times$ |
| DFB [] | 4% | 39% | -18.114 | $-0.828{\small\pm 5.444}$ | 0.513 | ${\color[rgb]{1,0,0}0.125}$ | ${\color[rgb]{1,0,0}\surd}$ |
| BNet [] | 3% | 33% | -16.694 | $-4.882{\small\pm 4.817}$ | 0.724 | 0.385 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet-UP [] | 2% | 35% | -23.252 | $-5.190{\small\pm 4.920}$ | 0.726 | 0.532 | $\times$ |
| ABNet (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}1.455}$ | ${\color[rgb]{1,0,0}6.132{\small\pm 2.181}}$ | ${0.168}$ | 0.316 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-att (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}4.198}$ | $8.053{\small\pm 1.449}$ | $0.172$ | $0.269$ | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-sc (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}2.221}$ | $7.224{\small\pm 1.667}$ | ${\color[rgb]{1,0,0}0.130}$ | $0.256$ | ${\color[rgb]{1,0,0}\surd}$ |

Figure: Figure 5: Vision-based end-to-end autonomous driving closed-loop testing trajectories in VISTA (left) and ABNet performance with the increasing of BarrierNet heads using scalable training (right). This scalable training is done by both the ABNet and ABNet-att in Table 3 with safety guarantees.
Refer to caption: x7.png

As shown in Table 3, the proposed ABNets can avoid crash onto obstacles with $100\%$ obstacle passing rate, including the ABNet-sc that is trained in a scalable way with two ABNets (also shown by the scalable training in Fig. 5). This is because the ABNets can learn the correct steering control (the blue and green sine waves shown in Fig. 9 (right) in Appendix B.3) to avoid the obstacle without stopping in front of it. The DFB and BNet-related models learn a significant deceleration control (shown in Fig. 9) to avoid crashing onto obstacles, which explains why the corresponding obstacle passing rates are low compared to other models in Table 3 and why the blue trajectories (BNet) terminate near the obstacle in Fig. 5 (left). Nonetheless, there are still some crash cases in DFB and BNet models due to badly learned CBF parameters that make the inter-sampling effect (i.e., safety violation between discretized times) serious. Most importantly, our proposed ABNet can learn less uncertain controls for this complicated task, as shown in Table 3, the scalable training in Fig. 5, and Fig. 9 (e.g., ABNet:blue or ABNet-att:green area v.s. BNet: red area). The ABNet-att can learn more consistent autonomous driving behavior than the ABNet due to the image attention setting, as shown by the magenta (ABNet-att) and cyan (ABNet) trajectories in Fig. 5 (left) and the green (ABNet-att) and blue (ABNet) areas in Fig. 9. Ablation studies on the robustness of our ABNets in terms of safety under high-noisy inputs (50% noise level) are given in Table 4 of Appendix B.3.

## 5 Related Works

Scalability, merging and uncertainty in learning for robot control. Machine learning techniques have been widely used in robot control [] [] []. Mixture of expert methods [] [] [] are scalable but hard to retain the property (such as safety) of the models. The uncertainty resulting from noisy model input or dataset is preventing the deployment to real robots [] []. To address this, predictive uncertainty quantification [] [], also a model merging approach, has been widely adopted. It has been shown to work well in vision-based autonomous driving under noisy input [] using the Gaussian kernel with Scott’s rule [] to select bandwidth. The main challenge of this technique is that it may make the system lose performance guarantees, such as safety. Other model merging approaches [] [] [] do not preserve safety either. We address the uncertainty and scalablibity problem for robot control using the proposed ABNets with provable safety guarantees.

CBFs and set invariance.
In control theory, the set invariance has been widely adopted to prove and enforce the safety of dynamical systems [] [] [] [] []. The Control Barrier Function (CBF) [] [] is such a state of the art technique that can enforce set invariance [], [], [], and transforms a nonlinear optimization problem to a quadratic problem that is very efficient to solve. CBFs originates from barrier functions that are originally used in optimization problems
[]. However, the CBF method tends to make the system conservative (i.e., at the cost of performance) in order to enforce safety, and it is not scalable to build large safe filters in neural networks. Our proposed ABNet can address all these limitations.

Safety in neural networks.
Safety is usually enforced using optimizations. Recently,
differentiable optimizations show great potential for learning-based control with safety guarantees [, , , ]. The quadratic program (QP) can be employed as a layer in the neural network, i.e., the OptNet []. The OptNet has been used with CBFs in neural networks as a safe filter controls [], in which CBFs themselves are not trainable, which can significantly limiting the learning capability. Neural network controllers with safety certificate have been learned through verification-in-the-loop training [, , ]. However, this verification method cannot ensure to cover the whole state space. CBFs are also used in neural ODEs to equip them with specification guarantees []. None of these methods are scalable to larger models, and are subject to uncertainty, which the proposed ABNet can address.

## 6 Conclusions, Limitations and Future Work

We propose a novel Attention BarrierNet (ABNet) that merge many safety-critical learning models while preserving the safety in this paper. The proposed ABNet is scalable to larger safe learning models, can achieve better performance, and is robust to input noise. We have demonstrated the effectiveness of the model on a series of robot control tasks.
Nonetheless, our model (and all the other barrier-based learning models [] []) still have a few limitations motivating for further research.

Limitations. First, the ABNet depends on the system/robot dynamics to strictly enforce safety guarantees. We may use neural ODEs [] to simultaneously learn the dynamics in the ABNet if they are unknown. Second, the ABNet also depends on accurate system/robot state that is hard to estimate from high-dimensional observations. We will explore to use foundation models [] in conjunction with ABNet to address such a challenge in the future. Finally, the ABNet also requires safety specifications that may be unknown in some robot control tasks, we may learn the safety specifications from data [], [], and this can also be done in conjunction with ABNet.

## 7 Acknowledgement

The research was supported in part by Capgemini Engineering. It was also partially sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This research was also supported in part by the AI2050 program at Schmidt Futures (Grant G-
965 22-63172).

## Appendix A Proof of Theorems

Theorem 3.1. (Safety of ABNets) Given the multi-head BarrierNets formulated as in (6) s.t. (7). If the system (1) is initially safe (i.e., $b_{j}(\bm{x}(t_{0}))\geq 0,\forall j\in S$), then a control policy $\bm{u}$ from the ABNet output (8) guarantees the safety of system (1), i.e., $b_{j}(\bm{x}(t))\geq 0,\forall j\in S,\forall t\geq t_{0}$.

Proof: The proof outline is to first show the existence of new HOCBF constraints (corresponding to all the safety specifications) that are defined over the output of the ABNet. Then, we can use Nagumo’s theorem [] to recursively show the forward invariance of each safety set in the HOCBFs, and this can eventually imply the satisfaction of the safety specifications $b_{j}(\bm{x})\geq 0,\forall j\in S$.

Since each control $\bm{u}_{k},k\in\{1,\dots,h\}$ in the ABNet is obtained from solving the QP (6) s.t. (7), we have that the following constraint is satisfied:

$$ $L_{f}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+[L_{g}\psi_{j,m-1}(\bm{x},\bm{z}| \theta_{p})]\bm{u}_{k}+p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m})\alpha_{j,m}(\psi_{ j,m-1}(\bm{x},\bm{z}|\theta_{p}))\geq 0,j\in S,$ (10) $$

Multiplying the weight $w_{k}\geq 0$ to the last equation, we have

$$ $w_{k}L_{f}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+w_{k}[L_{g}\psi_{j,m-1}(\bm{x },\bm{z}|\theta_{p})]\bm{u}_{k}+w_{k}p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m}) \alpha_{j,m}(\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p}))\geq 0,j\in S,$ (11) $$

Taking a summation of the last equation over all $k\in\{1,\dots,h\}$, the following equation establishes:

$$ $\displaystyle\sum_{k=1}^{h}w_{k}L_{f}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+ \sum_{k=1}^{h}w_{k}[L_{g}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})]\bm{u}_{k}$ (12) $\displaystyle+\sum_{k=1}^{h}w_{k}p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m})\alpha_{j ,m}(\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p}))\geq 0,j\in S,$ $$

Since $L_{g}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p}$ is a vector that is independent of $k$ and $\sum_{k=1}^{h}w_{k}=1$, the last equation can be rewritten as:

$$ $\displaystyle L_{f}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+L_{g}\psi_{j,m-1}( \bm{x},\bm{z}|\theta_{p})\left(\sum_{k=1}^{h}w_{k}\bm{u}_{k}\right)$ (13) $\displaystyle+\sum_{k=1}^{h}w_{k}p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m})\alpha_{j ,m}(\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p}))\geq 0,j\in S,$ $$

The summation of class $\mathcal{K}$ functions is also a class $\mathcal{K}$ function. Since $\alpha_{j,m}$ are class $\mathcal{K}$ functions, the $\sum_{k=1}^{h}w_{k}p_{m,k}(\bm{z}_{k}|\theta_{p,k}^{m})\alpha_{j,m}(\psi_{j,m-
1}(\bm{x},\bm{z}|\theta_{p}))$ is also a class $\mathcal{K}$ function over $\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})$. Therefore, equations (13) are the new HOCBF constraints defined over the output of the ABNet, i.e., $\sum_{k=1}^{h}w_{k}\bm{u}_{k}$. In other words, whenever $\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})=0$, we have

$$ $\displaystyle L_{f}\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})+L_{g}\psi_{j,m-1}( \bm{x},\bm{z}|\theta_{p})\left(\sum_{k=1}^{h}w_{k}\bm{u}_{k}\right)\geq 0,j\in S,$ (14) $$

The controls (outputs of the ABNet) $\sum_{k=1}^{h}w_{k}\bm{u}_{k}\equiv\bm{u}$ are directly used to drive the system (1), and $\bm{z}$ is taken as a piece-wise constant within discretized time intervals []. Therefore, the last equation can be rewritten as

$$ $\displaystyle\frac{\partial\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})}{\partial\bm {x}}(f(\bm{x})+g(\bm{x})\bm{u})=\frac{\partial\psi_{j,m-1}(\bm{x},\bm{z}| \theta_{p})}{\partial\bm{x}}\dot{\bm{x}}=\dot{\psi}_{j,m-1}(\bm{x},\bm{z}| \theta_{p})\geq 0,j\in S,$ (15) $$

Since $b_{j}(\bm{x}(t_{0}))\geq 0$, we can always initialize the HOCBF definition such that $\dot{\psi}_{j,m-1}(\bm{x},\bm{z}|\theta_{p})\geq 0$ is satisfied at $t_{0}$ []. By Nagumo’s theorem [] and (13)-(15), we have that $\psi_{j,m-1}(\bm{x},\bm{z}|\theta_{p})\geq 0,\forall t\geq t_{0}$.

Recursively, we can show that $\psi_{j,i}(\bm{x},\bm{z}|\theta_{p})\geq 0,\forall t\geq t_{0},\forall i\in\{0
,\dots,m-1\}$ from $i=m-1$ to $i=0$. Since $b_{j}(\bm{x})=\psi_{j,0}(\bm{x},\bm{z}|\theta_{p})$ by (2), we have that $b_{j}(\bm{x}(t))\geq 0,\forall t\geq t_{0},\forall j\in S$, which the safety guarantees of the ABNet for system (1). $\hfill\blacksquare$

Theorem 3.2.
(Safety of merging of ABNets) Given two ABNets with each formulated as in (8) and (6) s.t. (7), the merged model using the form as in (8) again guarantees the safety of system (1).

Proof: The proof outline is similar to that of Theorem 3.1. From each ABNet, we can show the existence of new HOCBF constraints (corresponding to all the safety specifications) that are defined over the output of each ABNet. Then we can again show the existence of another set of new HOCBF constraints (corresponding to all the safety specifications) that are defined over the output of the merged ABNet. Finally, we can also use Nagumo’s theorem [] to recursively show the forward invariance of each safety set in the HOCBFs, and this can eventually imply the satisfaction of the safety specifications $b_{j}(\bm{x})\geq 0,\forall j\in S$.

The mathematical proof is similar to that of Theorem 3.1, and thus is omitted.

## Appendix B Experiment Details

Metrics used in all the tables.
The SAFETY metric is defined as:

$$ $\text{SAFETY}=\min_{k}\{\min_{t\in[t_{0},T]}b(\bm{x}(t)\}_{k},k\in\{1,\dots,N\},$ (16) $$

where $N$ is the number of testing runs ($N=100$ in this case). $T$ is the final time of each run. $b(\bm{x})\geq 0$ is the safety constraint that is given explicitly in each experiment below.

The CONSER. metric is defined as

$$ CONSER. mean $\displaystyle=\underset{k}{\text{mean}}\{\min_{t\in[t_{0},T]}b(\bm{x}(t)\}_{k} ,k\in\{1,\dots,N\},$ (17) CONSER. std $\displaystyle=\underset{k}{\text{std}}\{\min_{t\in[t_{0},T]}b(\bm{x}(t)\}_{k}, k\in\{1,\dots,N\}.$ $$

The UNCERTAINTY metric for both controls are calculated by:

$$ $u_{i}\text{ UNCERTAINTY}=\underset{t\in[t_{0},T]}{\text{mean}}\{\underset{k}{ \text{std}}\{u_{i}(t)\}_{k},k\in\{1,\dots,N\}\},i\in\{1,2\}.$ (18) $$

### B.1 2D Robot Obstacle Avoidance

Models. All the models include fully connected layers of shape [5, 128, 32, 32, 2] with RELU as activation functions. There are some additional layers of differentiable QPs in other models (other than E2E-related models). The model input is the system state and the goal.

Training and Dataset. The dataset includes 100 trajectories, and each trajectory has 137 trajectory points. The ground truth controls (i.e., training labels) are obtained via solving HOCBF-based QPs []. We use Adam as the optimizer to train the model with a MSE loss function and a learning rate $0.001$. We use the QPFunction from the OptNet [] to solve the dQPs. The training time of the ABNet is about 1 hour for 20 epochs on a RTX-3090 computer.

Robot dynamics and safety constraints. We employ the bicycle model as the robot dynamics:

$$ $\underbrace{\left[\begin{array}[c]{c}\dot{x}(t)\\ \dot{y}(t)\\ \dot{\theta}(t)\\ \dot{v}(t)\end{array}\right]}_{\dot{\bm{x}}(t)}=\underbrace{\left[\begin{array }[c]{c}v(t)\cos\theta(t)\\ v(t)\sin\theta(t)\\ 0\\ 0\end{array}\right]}_{f(\bm{x})}+\underbrace{\left[\begin{array}[c]{cc}0&0\\ 0&0\\ 1&0\\ 0&1\end{array}\right]}_{g(\bm{x})}\underbrace{\left[\begin{array}[c]{c}u_{1}(t )\\ u_{2}(t)\end{array}\right]}_{\bm{u}}$ (19) $$

where $(x,y)\in\mathbb{R}^{2}$ denotes the 2D location of the robot, $\theta\in\mathbb{R}$ is the heading angle of the robot, $v\in\mathbb{R}$ is the linear speed of the robot. $u_{1},u_{2}$ are the angular speed and acceleration controls, respectively.

The safety constraint of the robot is defined as:

$$ $b(\bm{x})=(x-x_{0})^{2}+(y-y_{0})^{2}-R^{2}\geq 0,$ (20) $$

where $(x_{0},y_{0})\in\mathbb{R}^{2}$ is the 2D location of the obstacle, and $R>0$ is its size.

Acceleration control profiles. We show the acceleration control profiles in Fig. 6. The corresponding uncertainty is also significantly decreased with the proposed ABNet.

Figure: Figure 6: 2D robot obstacle avoidance acceleration control profiles and their distributions. The controls are subject to input noise, and thus are non-smooth. All the testings are done in a closed-loop fashion, i.e., the model outputs are directly used to control the robot.
Refer to caption: x9.png

### B.2 Safe Robot Manipulation

Models. All the models include fully connected layers of shape [6, 128, 256, 128, 128, 32, 32, 2] with RELU as activation functions. There are some additional layers of differentiable QPs in other models (other than E2E-related models). The model input is the system state and the goal.

Training and Dataset. The dataset includes 1000 trajectories, and each trajectory has about 350 trajectory points. The ground truth controls (i.e., training labels) are obtained via solving HOCBF-based QPs []. We use Adam as the optimizer to train the model with a MSE loss function and a learning rate $0.001$. We use the QPFunction from the OptNet [] to solve the dQPs. The training time of the ABNet is about 2 hours for 10 epochs on a RTX-3090 computer.

Robot dynamics and safety constraints. We employ the following model as the manipulator dynamics:

$$ $\underbrace{\left[\begin{array}[c]{c}\dot{\theta_{1}}\\ \dot{\omega_{1}}\\ \dot{\theta}_{2}\\ \dot{\omega}_{2}\end{array}\right]}_{\dot{\bm{x}}}=\underbrace{\left[\begin{ array}[c]{c}\omega_{1}\\ 0\\ \omega_{2}\\ 0\end{array}\right]}_{f(\bm{x})}+\underbrace{\left[\begin{array}[c]{cc}0&0\\ 1&0\\ 0&0\\ 0&1\end{array}\right]}_{g(\bm{x})}\underbrace{\left[\begin{array}[c]{c}u_{1}\\ u_{2}\end{array}\right]}_{\bm{u}}$ (21) $$

where $(\theta_{1},\theta_{2})\in\mathbb{R}^{2}$ denotes the angles of the two-link manipulator joints, $(\omega_{1},\omega_{2})\in\mathbb{R}^{2}$ is the angular speed of the two-link manipulator joints, $u_{1},u_{2}$ are the angular acceleration controls corresponding to the two joints, respectively.

The safety constraint of the robot is defined as:

$$ $b(\bm{x})=(l_{1}\cos\theta_{1}+l_{2}\cos\theta_{2}-x_{0})^{2}+(l_{1}\sin\theta _{1}+l_{2}\sin\theta_{2}-y_{0})^{2}-R^{2}\geq 0,$ (22) $$

where $(x_{0},y_{0})\in\mathbb{R}^{2}$ is the location of the obstacle, and $R>0$ is its size. $l_{1}>0,l_{2}>0$ are the length of the two links of the manipulator, respectively. In the current setting, the non-collision of the end-effector implies the non-collision of the link. Therefore, we only need to consider the safety of the end-effector. We show both the $u_{1},u_{2}$ control profiles in Fig. 7 to demonstrate the advantage of the proposed ABNet. The metric definitions are
the same as in the 2D robot obstacle avoidance, and the number of testing runs is $N=100$.

Figure: Figure 7: Robot manipulation joint control profiles and their distributions. The controls are subject to input noise, and thus are non-smooth. All the testings are done in a closed-loop fashion, i.e., the model outputs are directly used to control the manipulator.
Refer to caption: x10.png

### B.3 Vision-based End-to-End Autonomous Driving

Models. All the models include CNN ([[3, 24, 5, 2, 2], [24, 36, 5, 2, 2], [36, 48, 3, 2, 1], [48, 64, 3, 1, 1], [64, 64, 3, 1, 1]]) and LSTM layers (size: 64) and some fully connected layers of shape [32, 32, 2] $\times 2$ with RELU as activation functions. The dropout rates for both CNN and fully connected layers are 0.3. There are some additional layers of differentiable QPs in other models (other than E2E-related models). The model input is the front-view RGB images (shape: $3\times 45\times 155$) of the ego vehicle, and the outputs are the steering rate and acceleration controls of the vehicle.

Training and Dataset. The dataset is open-sourced including 0.4 million image-control pairs from a closed-road sim-to-real driving field. Static and parked cars of different types and colors are used as obstacles in the dataset. The dataset is collected from the VISTA simulator []. The ground truth controls (i.e., training labels) are obtained via solving a nonlinear model predictive control (NMPC). We use Adam as the optimizer to train the model with a MSE loss function and a learning rate $0.001$. We use the QPFunction from the OptNet [] to solve the dQPs. The training time of the ABNet is about 15 hours for 5 epochs on a RTX-3090 computer.

Brief introduction to VISTA. VISTA is a sim-to-real driving simulator that can generate driving scenarios from real driving data []. The VISTA allows us to train our model with guided policy learning. This learning method has been shown to work for model transfer to a full-scale real autonomous vehicle. There three steps to generate the data: (i) In VISTA, we randomly initialize the locations and poses of ego- and ado-cars that are associated with the real driving data; (ii) we use NMPC to collect ground-truth controls (training labels) with corresponding states, and (iii) we collect front-view RGB images along the trajectories generated from NMPC.

Vehicle dynamics and safety constraints. The vehicle dynamics are specified with respect to a reference trajectory [], such as the lane center line. The two most important states are the along-trajectory progress $s\in\mathbb{R}$ and the lateral offset distance $d\in\mathbb{R}$ of the vehicle center with respect to the trajectory. The dynamics are defined as:

$$ $\underbrace{\left[\begin{array}[c]{c}\dot{s}\\ \dot{d}\\ \dot{\mu}\\ \dot{v}\\ \dot{\delta}\end{array}\right]}_{\dot{\bm{x}}}=\underbrace{\left[\begin{array} [c]{c}\frac{v\cos(\mu+\beta)}{1-d\kappa}\\ v\sin(\mu+\beta)\\ \frac{v}{l_{r}}\sin\beta-\kappa\frac{v\cos(\mu+\beta)}{1-d\kappa}\\ 0\\ 0\end{array}\right]}_{f(\bm{x})}+\underbrace{\left[\begin{array}[c]{cc}0&0\\ 0&0\\ 0&0\\ 1&0\\ 0&1\end{array}\right]}_{g(\bm{x})}\underbrace{\left[\begin{array}[c]{c}u_{1}\\ u_{2}\end{array}\right]}_{\bm{u}},\vspace{-2pt}$ (23) $$

where $\mu$ is the local heading error of the vehicle with respect to the reference trajectory, $v$ is the linear speed of the vehicle, $\kappa$ is the curvature of the trajectory at the progess $s$. $l_{r}$ is the length of the vehicle from the tail to the center, $\beta=\arctan\left(\frac{l_{r}}{l_{r}+l_{f}}\tan\delta\right)$, where $l_{f}$ is the length of the vehicle from the head to the center. $u_{1},u_{2}$ are the steering rate and acceleration controls of the vehicle, respectively.

Figure: Figure 8: Attention-based image observations for the ABNet-att model. From left to right and top to down: attentions on full image, left-most part, left lane boundary, lane center, right lane boundary, and right-most part.
Refer to caption: extracted/5676570/fig/test.png

The safety constraint of the vehicle is defined as:

$$ $b(\bm{x})=(s-s_{0})^{2}+(d-d_{0})^{2}-R^{2}\geq 0,$ (24) $$

where $(s_{0},d_{0})\in\mathbb{R}^{2}$ is the location of the obstacle in the curvi-linear frame (i.e., defined with respect to the reference trajectory), and $R>0$ defines its size that is chosen such that the satisfaction of the above constraint can make the ego vehicle avoid crashing onto the obstacle.

Closed-loop testing. We test all of our models in a closed-loop manner in VISTA. In other words, at each time step, we get the front-view RGB image observation from VISTA. Then, the model generates a control based on the image. Finally, the control is used to drive the “virtual” vehicle in VISTA. This process is done recursively until the final time. The total number of testing runs is $N=100$ for all the tables. The obstacles are randomly initialized (in uniform probability distribution) with lateral distance $d_{0}$ ranges from $\pm 0.1m$ to $\pm 1.5m$. In Figs. 5 and 9, the ego vehicle is randomly initialized with $d\in[-0.5,0.5]m$ (in uniform probability distribution).

Image observations for the ABNet-att model. We generate the attention-based observations as shown in Fig. 8. Each of the attention images may play an important role in a specific driving scenario (e.g., attention on the left-most part may be crucial for sharp-left turn).

Figure: Figure 9: Vision-based end-to-end autonomous driving closed-loop testing control profiles. The models directly take images as inputs, and output controls for the vehicle. All the testings are done in closed-loop in VISTA.
Refer to caption: x12.png

**Table 4: Ablation study: vision-based end-to-end autonomous driving closed-loop testing under noise and comparisons with benchmarks. Items in the first row are short for obstacle crash rate (CRASH), Obstacle passing rate (PASS), satisfaction of safety constraints where non-negative values mean safety guarantees (SAFETY), system conservativeness (CONSER.), acceleration control $u_{1}$ uncertainty ($u_{1}$ UNCERTAINTY), steering rate control $u_{2}$ uncertainty ($u_{2}$ UNCERTAINTY), and theoretical safety guarantees (THEORET. GUAR.) respectively. In the model column, items are short for single vanilla end-to-end driving model (V-E2E), E2Es merged with Monte-Carlo Dropout (E2Es-MCD), E2Es merged with deep resembles (E2Es-MERG), deep forward and backward model (DFB), single BarrierNet (BNET), BarrierNet policies with uncertainty propagation (BNET-UP), ABNet with 10 heads (ABNET), ABNet with attention images and 10 heads (ABNET-ATT), ABNET-SC denotes our ABNet first trained with ABNET-ATT scaled by ABNET (20 heads)respectively. The safety metric is defined as the minimum value of the safety specification $b_{j}(\bm{x}),j\in S$ among all runs. The conservativeness metric is defined as the mean (with std) of the minimum value (in each run) of the safety specification $b_{j}(\bm{x}),j\in S$ among all runs. The uncertainty metrics for both $u_{1}$ and $u_{2}$ are measured by the standard deviations of the model outputs (two controls) among all runs.**
| Model | Crash ($\downarrow$) | Pass ($\uparrow$) | Safety ($\geq 0$) | Conser. ($\geq 0\&\downarrow$) | $u_{1}$ Uncertainty ($\downarrow$) | $u_{2}$ Uncertainty ($\downarrow$) | Theoret. guar. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| V-E2E [] | 31% | 69% | -59.455 | $-8.932{\small\pm 19.741}$ | 0.529 | 0.239 | $\times$ |
| E2Es-MCD [] | 28% | 72% | -58.405 | $-8.116{\small\pm 20.802}$ | 0.524 | 0.232 | $\times$ |
| E2Es-DR [] | 27% | 73% | -60.267 | $-8.781{\small\pm 20.910}$ | 0.512 | 0.225 | $\times$ |
| DFB [] | 1% | 37% | -13.281 | $-0.256{\small\pm 4.348}$ | 0.482 | ${\color[rgb]{1,0,0}0.127}$ | ${\color[rgb]{1,0,0}\surd}$ |
| BNet [] | 23% | 37% | -45.415 | $-9.114{\small\pm 13.382}$ | 0.730 | 0.316 | ${\color[rgb]{1,0,0}\surd}$ |
| BNet-UP [] | 24% | 39% | -44.634 | $-8.866{\small\pm 13.167}$ | 0.747 | 0.278 | $\times$ |
| ABNet (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}4.268}$ | ${8.315{\small\pm 2.147}}$ | ${0.151}$ | 0.326 | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-att (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}5.986}$ | ${\color[rgb]{1,0,0}7.032{\small\pm 0.405}}$ | ${\color[rgb]{1,0,0}0.118}$ | $0.213$ | ${\color[rgb]{1,0,0}\surd}$ |
| ABNet-sc (ours) | ${\color[rgb]{1,0,0}0\%}$ | ${\color[rgb]{1,0,0}100\%}$ | ${\color[rgb]{1,0,0}4.118}$ | ${7.515{\small\pm 1.120}}$ | ${0.128}$ | $0.255$ | ${\color[rgb]{1,0,0}\surd}$ |

Acceleration control profiles. We present both the acceleration control and steering rate control profiles in Fig. 9. Both the BNet and BNet-UP models have forced the ego vehicle to have a large deceleration instead of making it to pass the obstacle using the steering control when the vehicle approaches the obstacle. This can make the ego vehicle get stuck at the obstacles, and thus, the obstacle passing rate (as shown in Table 3) is low in these two models.

Ablation studies on the model robustness in terms of safety under noisy input. To further test the model safety robustness, we add random noise (50% magnitude of the image values) to all the image observations. The results are presented in Table 4. Our proposed ABNets can still guarantee the safety of the vehicle under noisy input (0% crash rate), while the crash rates using other models significantly increase except the DFB model. This is because the HOCBFs in the DFB model are not trainable, and the corresponding parameters are fixed. Badly trained HOCBFs could make the method fail to guarantee safety due to the inter-sampling effect.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does ABNet handle situations where the safety constraints of different heads conflict? For example, if one head focuses on obstacle avoidance and another on goal attainment, how are these constraints balanced?

2. Could you provide further clarification on the choice of penalty functions in the HOCBFs? Specifically, how sensitive is the model's performance to different choices of these functions?

3. Has any analysis been done on the computational cost of scaling up ABNet to a large number of heads, especially in real-time applications like autonomous driving?
soundness: 3
strengths: 1. The paper rigorously evaluates ABNet across multiple domains, using well-defined metrics and benchmarks to validate its performance and safety claims.

2. The integration of attention mechanisms into barrier-based models for robot learning is novel. By focusing each "head" on specific aspects of the input, ABNet allows for scalable, efficient training without needing to construct a large model from scratch, a unique approach in safe robot learning.

3. ABNet's scalability and robustness to noise make it highly valuable for real-world robot applications, particularly in safety-critical scenarios like autonomous driving.
summary: This paper proposes a model for safe and scalable robot learning. It addresses the limitations of traditional barrier-based methods, which are not scalable and may produce unstable outputs under noisy inputs. The proposed ABNet incorporates multiple "heads," each focusing on different aspects of the input to generate control policies safely and efficiently. This architecture enables incremental, scalable training without sacrificing safety guarantees. The paper presents theoretical proofs of ABNet’s safety and demonstrates its effectiveness across various robotic tasks, including 2D obstacle avoidance, safe manipulation, and vision-based autonomous driving, showing improved performance and robustness compared to existing methods.
weaknesses: 1. The mathematical explanations, especially regarding HOCBFs and the attention mechanisms, are dense and may be challenging for some readers to follow. More intuitive explanations or diagrams could be better.

2. All heads in the ABNet share the same safety constraints. Expanding the model to allow for varying constraints in different heads could make it even more flexible and applicable to a wider array of tasks.

3. While ABNet’s scalability is demonstrated in 2D tasks and a simulated autonomous driving environment, it would benefit from testing in real-world environments to further validate its robustness and generalizability.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1.  Would the method still be effective in environments that require sophisticated contact handling? How might this approach be extended to humanoid robots, especially in scenarios involving complex external interactions, such as full-body dynamics with external force feedback?

2. To my knowledge, large transformer models are increasingly applied in autonomous driving scenarios involving dynamic external agents, as referenced in [3]. This approach uses transformers as a backbone for safety planning based on sampled trajectories, ensuring collision-free paths while closely resembling NMPC planners or human driving behaviors. How do the authors view the online planning, and could barrier functions be integrated to enhance safety?

3. The method’s applicability appears limited by the requirement for a known, differentiable constraint function. In many real-world scenarios, defining such a function can be highly complex or may even resemble a black-box system. For example, ensuring that large language models avoid generating outputs contrary to human intentions necessitates defining a "human-safe" constraint function, which is inherently difficult.

[3] Planning-oriented Autonomous Driving
soundness: 3
strengths: 1. The approach explicitly incorporates barrier functions into neural network training, ensuring safety constraints are satisfied.
2. The modular architecture of ABNet with multiple attention heads allows scalable, incremental learning, which is promising for building complex, safe models in stages.
3. The method demonstrates robustness to noise, yielding lower variance in performance.
summary: The paper introduces ABNet, a novel framework that leverages attention mechanisms to address varying input patterns while integrating barrier functions to ensure the system state remains within a safety set, achieving forward invariance. The proposed method aims to enhance scalability and robustness in robot learning by allowing each head of BarrierNet to focus on different aspects of the observation space, potentially enabling safe control policies in diverse environments.
weaknesses: 1. **Lack of Optimality Guarantees**: The method does not appear to ensure optimal task performance. As stated in the paper, "we use NMPC to collect ground-truth controls (training labels) with corresponding states," implying that the upper limit of ABNet's task performance is constrained by the performance of NMPC (e.g., minimum time to reach a target). Additionally, optimality does not seem to be the primary focus in training. By employing imitation learning with barrier functions, safety appears to be prioritized, which could further impact task performance. Is there a mechanism to balance task performance and safety? Furthermore, the criteria for selecting the penalty functions $\alpha$ in Equation 7 are not well explained. How are these values chosen, and how does the method balance conservatism and performance optimality?

2. **Simplicity of Scenarios Considered**: The experimental scenarios are relatively simplistic, involving static environments with limited dynamics, and lack sufficient qualitative analysis, such as video demonstrations comparing the method with baselines. Additionally, Figure 5 does not clearly illustrate the performance differences between ABNet and methods like MPC or BNet. Moreover, while Figure 5 includes results from MPC, Table 3 does not provide a corresponding quantitative comparison. For the method to be applicable to more complex, real-world scenarios—such as autonomous driving with dynamic obstacles like vehicles or pedestrians—consideration of the environment's external dynamics is crucial. This raises concerns about the scalability of ABNet. The authors could consider testing in more sophisticated simulators, such as [1-2], to better demonstrate the method's adaptability and robustness in dynamic environments.

3. **Contribution of Attention Mechanism**: The primary contribution appears to be the integration of attention into BarrierNet. The authors are encouraged to provide a detailed explanation of why this integration is non-trivial. Replacing BarrierNet’s structure with an attention mechanism might seem straightforward, as it could be perceived as a simple network architecture change. Were there specific challenges or technical hurdles encountered when integrating attention into BarrierNet? Did any particular techniques make this integration effective, ? Discussing these aspects would clarify the unique benefits of the proposed approach.

[1] CARLA: An open urban driving simulator

[2] Nuscenes: A multimodal dataset for autonomous driving

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: ## Theoretical Concerns

1.  Linear Combination of Policy Outputs
    
    ABNet generates its final control output by linearly combining the outputs from multiple heads. However, there are concerns about whether such a linear combination is effective for complex nonlinear problems. Many real-world environments involve significant nonlinearity and complex interactions, which may not be fully captured by linear combinations.
    
2.  Generalization of Safety Guarantees
    
    While the paper provides theoretical guarantees that the combined outputs ensure safety, there is insufficient explanation regarding whether these guarantees hold in novel situations not covered by the training data. If the safety constraints do not generalize across all scenarios, the model may malfunction in unexpected environments.
    
3.  Gap Between Mathematical Models and Reality
    
    ABNet relies on mathematical optimization (specifically Quadratic Programming, or QP) to implement safety control. However, real-world uncertainties, noise, and sensor errors may reduce the model’s performance. There is a concern about whether the mathematical models, which do not account for such uncertainties, can achieve similar performance in practice.
    

----------

## Experimental Concerns

1.  Gap Between Simulation and Real-World Environments
    
    Most of the experiments in the paper were conducted in simulated environments. However, there are significant differences between simulations and real-world conditions. For example, sensor errors and unexpected scenarios, which may not be fully replicated in simulations, could affect safety in practice.
    
2.  Interpretation of Performance Metrics
    
    The paper presents metrics on rewards and safety but may have overlooked potential trade-offs between safety and performance in specific tasks. For instance, to achieve high rewards, the model might subtly violate some safety constraints, but this trade-off has not been analyzed in detail.
    
3.  Limitations of Comparative Experiments
    
    While ABNet's performance was compared with several existing models, there is a possibility that the comparison was not exhaustive. The inclusion of more recent reinforcement learning or control models would enhance the reliability of the results.
    
4.  Overfitting Risks
    
    Since ABNet trains each head individually and combines them linearly, there is a potential risk of overfitting to the training data. This could limit the model’s generalization to unseen scenarios, leading to performance degradation in untested environments.
    

----------

## Comprehensive Concerns

-   Generalization and Data Limitations: If the training data and safety constraints are limited, the model might struggle to handle the wide range of situations encountered in real-world applications.
-   Real-Time Control Challenges: The QP-based optimization required for safety control may not be suitable for real-time control due to computational delays. The paper lacks a detailed discussion of latency issues, which are critical for real-time robotic control.
-   Probabilistic Safety Guarantees: If safety is guaranteed only probabilistically or as an expected value, rare but critical safety violations could occur. The paper does not adequately address how to handle these rare but potentially catastrophic violations.

These theoretical and experimental concerns are crucial for evaluating the reliability of ABNet’s proposed methodology and results. Future research should focus on conducting experiments in more diverse environments and incorporating complex real-world scenarios. Additionally, further studies are needed to ensure real-time performance and provide clearer safety guarantees.
soundness: 2
strengths: 1.  Scalability: ABNet utilizes multiple heads, allowing for the incremental construction of large-scale models suitable for complex tasks.
2.  Safety Assurance: The outputs from each head are combined in a way that ensures mathematical safety guarantees.
3.  Robustness to Noise: ABNet provides stable outputs even with noisy input data, as evidenced by smooth signals in testing.
4.  Parallel Learning: Each head can be trained independently, maximizing learning efficiency.
summary: This paper addresses two critical aspects of robot learning: safety and scalability. While traditional barrier-based models have been utilized to ensure safety in robotic systems, they present the following challenges: lack of scalability, learning stability, slow training speed and sensitivity to noise. To overcome these issues, the paper proposes the Attention BarrierNet (ABNet). This model guarantees the necessary safety for robotic control while being scalable and capable of incremental learning. ABNet offers a novel approach that addresses the limitations of existing barrier-based models by balancing safety and performance. Experimental results demonstrate the superiority of ABNet across various tasks, particularly highlighting its ability to maintain robust control signals even in noisy environments.
weaknesses: 1.  Uniform Safety Constraints: All heads currently operate under the same safety constraints, limiting the model's ability to incorporate diverse constraint types.
2.  Uncertainty in Safety Specifications: In certain robotic control tasks, safety specifications are not clearly defined, indicating the need for further research.
3.  Output Space Combination: The model currently combines outputs only in the output space. Further research is required to explore safe combinations within the parameter space.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. About the definition of attention

1a. How is the self-attention used for ABNet? Usually, self-attention refers to the one in the transformer. I think, in general, ABNet is not restricted to being a transformer. So how was it used for the ABNet? Is it used for both state-based task and vision-based task?

1b. A follow-up question: Is Fusion also part of the attention? If so, I think it is not the usual self-attention. Normally, self-attention calculates the score based on the softmax between K and Q. In this paper, it is simply a weighting coefficient. The fusion looks more similar to model ensemble rather than self-attention.

2. About the effectiveness of attention

2a. I wonder whether the approach's effectiveness comes mainly from enlarging model size instead of using this attention mechanism. Can we have some experiments that enlarge the size of the BNet to the same size as those best models, e.g., ABNet-10 or ABNet-100, and compare this large BNet to the same-size ABNet, w.r.t. the existing benchmarks? It would be appreciated if the model parameter size for each method could be a column in the table.

3. About the experiment setting

3a. Is imitation learning used for the experiment? If so, do we assume a safe enough reference controller should be known?

3b. The vision-based task seems the most interesting one, and I wish to know more about it. If I understand correctly, the x space here is the image space. How are the dynamics defined (or are they even unknown), and are they control-affine?

3c. I notice in the setting, "with uniformly distributed noise, 10% of the input magnitude in testing". Why are they needed for the testing? Is it for the claim of robustness of the control? I wonder what the performance will be if we simply make the training and testing dynamics the same.

4. About theoretical guarantee

4a. A general problem with the learning-based CBF is that they are guaranteed to be a CBF only when they are trained very well to satisfy the forward invariance property. If out-of-distribution states happen, or the neural CBF fails to fit on the training dataset, then it is not a theoretical CBF anymore. Just as a discussion, does this work circumvent this problem? Meanwhile, I don't think this is a big issue since almost all neural CBFs have this problem.
soundness: 2
strengths: 1. Provable theoretical safety guarantee using forward invariance.

2. Better safety performance across multiple benchmarks.
summary: The paper introduces ABNet (Attention BarrierNet), a novel neural network architecture designed for safe and scalable robot learning.
- ABNet combines multiple "BarrierNet" heads that can each learn safe control policies while focusing on different aspects of the input observation
- The architecture allows for merging multiple safety-critical models while preserving safety guarantees
- It's scalable and can be trained incrementally to build larger foundational safe models
weaknesses: Most of the weaknesses are well explained in the limitation section. In addition to that:

1. It is not well explained how attention is used in the ABNet. See question 1.

2. It is uncertain how effective this self-attention is rather than the contribution of scaling model size. See question 2.

3. The control affine assumption seems strong. The approach heavily relies on the assumption, because it linearly interpolates the controls.

Meta-review: This paper proposed a 'safe learning' approach to learning robot control policies. The reviewers found a variety of issues with the paper, and the authors did not respond to these issues. In light of that, and the fact that no reviewer strongly advocated acceptance, I recommend that the paper be rejected in its current form (and that the authors try to address the concerns of the reviewers to improve the paper for a potential resubmission).

Decision: Reject