HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
Authors: H M Sabbir Ahmad
Ehsan Sabouni
Alexander Wasilkoff
Param Budhraja
Zijian Guo
Songyuan Zhang
Chuchu Fan
Christos Cassandras
Wenchao Li
Venue: NeurIPS 2025 poster ()
Keywords: Hierarchical RL
Multi-Agent RL
Safe RL
Control Barrier Functions
Safety-Critical Systems
Primary area: reinforcement_learning

Abstract: We address the problem of safe policy learning in multi-agent safety-critical autonomous systems.
In such systems, it is necessary for each agent to meet the safety requirements at all times while also cooperating with other agents to accomplish the task. Toward this end, we propose a safe Hierarchical Multi-Agent Reinforcement Learning (HMARL) approach based on Control Barrier Functions (CBFs). Our proposed hierarchical approach decomposes the overall reinforcement learning problem into two levels –- learning joint cooperative behavior at the higher level and learning safe individual behavior at the lower or agent
level conditioned on the high-level policy. Specifically, we propose a skill-based HMARL-CBF algorithm in which the higher-level problem involves learning a joint policy over the skills for all the agents and the lower-level problem involves
learning policies to execute the skills safely with CBFs. We validate our approach on challenging environment scenarios whereby a large number of agents have to safely navigate through conflicting road networks. Compared with existing state-of-the-art methods, our approach significantly improves the safety achieving near perfect (within $5\%$) success/safety rate while also improving performance across all the environments.

HMARL-CBF Hierarchical Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
CHARR-CBB--Hirarchicaaal Multi-Agent
H. M. Sabbir Ahmad', Ehsan Sabouni', Alexander Wasilkoff', Param Budhraja', Zijian Guo Songyuan Zhang", Chuchu Fan", Christos Cassandras', Wenchao Li
'oston University, Maasschussetts Institute of Technology
(sabbir92, esabouni, awasilkoff, paramb, 2 jguo, ege, wenchao] {szhang21, chuchu) edu
Abstract
We address the problem of safe policy learning in multi-agent safety-critical au- tonomous systems. In such systems, is necessary for each agent to meet the safety requirements all times while also cooperating with other agents to accomplish the task. Toward this end, we propose a safe Hierarchical Multi-Agent Reinforcement Learning (HMARL) approach based Control Barrier Functions (CBFs). Our proposed hierarchical approach decomposes the overall reinforcement learning problem into two levels learning joint cooperative behavior at the higher level and learning safe individual behavior at the lower or agent level, conditioned on the high-level policy. Specifically, we propose skill-based HMARL-CHE algorithm in which the higher-level problem involves learning joint policy over the skills for all the agents, and the lower-level problem involves learning policies execute the skills safely with CBFs. We validate our approach in challenging environment sce- narios, whereby large number of agents have to safely navigate through conflicting road networks. Compared with existing state-of-the-an methods, our approach significantly improves the safety, achieving near-pertee sear-perfect (2 95%) success/safety rate while improving performance across all the environments
Introduction
Safety-critical multi-agent systems are rapidly expanding across diverse domains, including self- driving cars [37 [00 unmanned acrial vehicles (UAVs) 184, 54), swarm robotics [41], soft robotics [53 72 31, and autonomous underwater vehicles (AUVs) (55) As these systems operate complex and often unpredictable environments, failures can lead to catastrophic consequences-eedangeee human lives, causing environmental damage, or resulting in severe economic losses. Hence, is crucial to design cooperative control policies that enable agents to achieve their objectives while ensuring safety with respect to both other agents and the environment [83] 77).
Learning a flat policy using existing Multi-Agent Reinforcement Learning (MARL) 40
64
algorithms by adjoining the safety constraints the reward function provides one possible approach. However, these approaches suffer from scalability and high sample complexity as the number of agents grows, and most importantly, they lack formal safety guarantees. The problem is further exacerbated in partially observable settings. Safety in both single- and multi-agent reinforcement learning has been studied within the constrained Markov Decision Process (CMDP) framework [89 22]. In this setting, constraints are typically expressed as expectations of cumulative (discounted) costs over entire trajectories, thereby ensuring safety only in an average or statistical sense on trajectories.
Project website: https::/bau-dppendd http:/ddddeeen-aaaa.........aaee github. cochuu..oor HIAAR--CEEE
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, safety-critictt systems necessitate ppintwise-in--iieg guarantees -iee. constraints that must hold at every time step along each trajectory- rather than merely in expectation over trajectories.
hierarchical approach, as proposed in 67 20 can alleviate sample complexity, but existing methods do not tackle safety. To bridge these gaps, we propose hierarchical multi-agent rein- forcement learning framework based on CBFs. The higher-level policy coordinates cooperative behaviors across agents, while the lower-level policy enforces safety through CBFs. The high-level and low-level policies are trained jointly to enable safe cooperation at scale, We summarize our main contributions below:
1. We introduce HMARL-CBF, a novel centralized -ertaliized-triiinggee training- deceentraalizze--xeccutooo (CTDE) hierarchical multi-agent reinforcement learning framework that guarantees safety using Con- trol Barrier Functions (CBFs). This approach enforces pointwise-in-tine safety constraints throughout both training and execution across each trajectory.
2. Our hierarchical structure leverages skill-based decomposition, where the high-level policy coordinates cooperative behaviors across agents, while the low-level policy learns and executes safe skills guided by CBF-based safety constraints. This design guarantees safety both during training and in real-world deployment.
3. We validate our proposed approach on challenging conflicting environment scenarios where a large number of agents must each travel safely from their origin to their destination without colliding with other agents. Simulation results demonstrate superior performance and safety compliance, achieving near-perfect (z 95%) success/safety rate compared to existing benchmark methods.
2 Related Works
Hierarchical Multi-Agent RL (HMARL). Hicrarchical RL structures decision-making making hierarchi-
cally, accelerating learning [5T] Classical methods include HAM 48, MAXQ [15, options 665 52
and feudal architectures 13 Deep HRL extends this with subgoal-based 669 44, option-based
5 29,, and skill-based frameworks 36 19). HMARL introduces hierarchy in MARL for enhanced
coordination and exploration. Notable methods include value decomposition (QTRAN 63). MAXQ
[1]. HAVEN 771, 20)), feudal structures M1, and option-based hierarchies 68 Temporal abstrac-
tion is explored in 167 while meta-policy learning appears in [69]. Skill discovery in HRL has explored mutual information maximization (2]. trajectory clustering 43 and hierarchical skill identification [79], option learning (28), imitation [61], and adaptive refinement via Skill-Critic [27).
However, these works define skills as fixed- henghhh or variable-length sequences that lack interpretable semantics and safety guarantees, limiting applicability to safety-critical systems.
Safe RL. Safety in multi-agent systems is commonly addressed using the Constrained Markov Decision Process (CMDP) framework 89 and primal-dual methods [3
2 22 88
Approaches include primal methods 10 11 39
16 32 While CMDPs ensure safety over trajectories, they do not
consider pointwise time constraints critical for safety-critical systems. Multi-agent scenarios further complicate this, the number of constraints scales with agents. Primal-dual methods [3 2 also face issues with training instability, slow convergence, and hyperparameter sensitivity. Additionally, works addressing reach-avoid problems 62
82 87 are primarily limited single-agent fully observable settings. Previous works on RL-CBFs have harnessed exact or approximate models [59 6 g to learn safe policies. The existing methods tackle safety for single-agent systems [8] 50). and multi-agent systems [18]. [59] considers a multi-agent cooperative setting conditioned on fixed coordination policy, which can be suboptimal. In [8) solved constrained RL problem with CBF-based chance constraints using the augmented Lagrangian, which nullifies the merits of CBF filters. Additionally, existing works require nominal policy which is learned [7] however, this can be challenging for MAS. Finally, [86] 85] proposed methods for multi multi-agent systems, but such methods suffer from high sample complexity and scalability due to the flat policy structure.
3 Background
3.1 Semi-Markov Decision Process
Semi-Markov Decision Process (SMDP) is a generalization of MDP that allows actions to take variable amounts of time steps before transitioning to the next state. Building on that, similar to [2]

---

we model a Multi-Agent Comi-Markov Decision Process (MSMDP) as tuple (N.S, S. P.
described follows: N is finite set of N agents, with each agent N having an individual action set 2 The joint action space 2 [ 2' consists of joint actions = (z ...... ;,",,,),)),,............. representing the joint actions of the agents. The joint state space denoted by S IT St xS where S is the environment state where S is the state space of agent i. The reward function is denoted by R. :
N + R where R[,, 2, 2,) corresponds to the reward accrued executing action & in state in k steps. Y is the discount factor, and the multistep transition probability function S N x S + [0. 1J gives the probability of transitioning from state 8 to state s' in & time steps as result of the joint action Since the joint actions may have varying termination times, P influenced by the duration of executing an action from the state it is initiated until transitioning to the next state (also referred to as the decision epoch) and a termination scheme T.
Three termination strategies for the temporally extended joint actions, Tany: Tall, and Teontinue: are analyzed in [58, Figure 1J. First, we define the decision epoch as the time when either subset or all the agents select new actions. Synchronous schemes, Tany and Tall. set decision epochs when either any or all actions within a joint action terminate, with all agents selecting new actions. In contrast, the asynchronous scheme Tcontimue sets the decision epoch when any of the actions within the joint action terminate. Then let those actions that did not terminate naturally continue running while initiating new actions for the agents that completed their action. Our decentralized approach adopts this asynchronous action update strategy.
The equation corresponding to the state value function (*(s) and state-action value function (*)()2) 2) are defined below for MSMDP:
(s) R. [Q (s, 2)] Bz,,, [R(s z, k)) (V
(s) = Eg, [R(s, 2, h) ( Ex, (Rss,2. 2,) (Q" 2')]]
(2)
where & represents the time interval until agent(s) update their action, and R(s, z, k) represents the reward obtained by taking the joint action z in state s over the random time interval k. The learning ob- jective is to find the joint policy æ for all the agents that maximizes: Ex [x 7 - ke R(s;, žt, kz k.)
3.2 Control Barrier Functions
Consider the following agent dynamics (with the agent index dropped for simplicity):
& f(s g())a,
(3)
where 8 e I2 is the state of the agent, a s R" are the primitive actions, and f R + R² and
9 R. 12ax are locally Lipschitz.
Definition 3.1 (Class IC function). A continuous function a [0, B) - [0, | > 0 is said to belong to class IC if it is strictly increasing and a0)) 0.
Definition 3.2. A set C is forward invariant for system 3 if for every s(0) e C, we have s(t) e C, for all ( 2 0. Definition 3.3 (Control barrier function (3.. Given a continuously differentiable function b 12" R
and the set C := {s e R" b(s) 2 0}, b(s) is a candidate control barrier function (CBF) for the system if there exists class function a such that
sup (()) L.g(())a + a(b(s)) 0,
aeA
for all s e C, where T.f, Tng denote the Lie derivatives along f and g, respectively.
Theorem 3.4 (3)). Given a constraint b(s(t)) with the associated set C, any Lipschitz continuous
controller al),, that satisfies 4 V1 ty renders the set C forward invariant for the control system in 3
Definition 3.5 (Control Lyapunov function (CLF) 3).. A continuously differentiable function V R is globally and exponentially stabilizing CLF for (3) if there exists constants 6 E Rss, i 1,2, such that e: |||l||2 s Vss) c2]|s|| ; and the following inequality holds
inf Vs) Lg Vs)a (())] se,
asA
where e makes this soft constraint.
Additional details on Higher Order CBFs (HOCBFs) as well relaxation of the affine dynamics assumption are included in Appendix A.2
3

---

Problem Formulation
((tt)) - (+)
[ligh Level Policy
Low Level Policy
sampless conditioned
Agent samples skill af ~ x (
Portall dover oble
Polley Network
High Level
Low Level
CBT--Based Policy
a² ~
x²C lot,
Agest k
Network 1, g
Ageui N' samples achion conditionad skill al nz (.10
Agont N samples skill z" ~ mbl (.]o')
High-Level Polliyy Natwoek Apent my
Low-Levvll Policy Parametor
CDE Dased Policy
d
Skill: Cruising
skill: Left
chanoino
skill:
Skill BevebbratiingSSlwwie
Antelorating Spontingg
Craising
dewai
Figure 1: the higher level, agents choose skills in decentralized manner based лн conditioned on their own observation, whereas at the lower level every agent executes these skills by learning parametric CBF-based policy pigi conditioned on their observation and the skill. The extrinsic trajectory return RH (T) is used to jointly learn the cooperative policy for all the agents centrally, and the intrinsic trajectory return 12i((T) for agent i's trajectory 7' is used to learn the policies corresponding to the skills. The high-level and low-level policy networks are shared among all agents.
We consider cooperative multi-agent setting among set of agents. Let the set of agents be denoted by N (1, (I.....,N] .N. We consider that every agent e N has dynamics as in (3). We partition the state space into two disjoint spaces corresponding to the agent states and the environment states. The environment state denoted by 8f e S The state of an agent control input vector is denoted by a' A [aminy amarl c R2 with A' the control input constraint set for the agent i, and amin Qmax IR', Then, the joint state space is denoted by S Πle x S and the joint action space of the agents is denoted by A ΠRN A'. In setting, there is performance-related cost function 1(8, a) associated with the system specification, where S (more specifically Thevs and A. Without loss of generality, in our cooperative setting, the stage cost is defined in the form 1(8, a) Σien (ss a²), where (' at) denotes the stage cost associated with agent N.
denoted by s² e S c IR" and the
We consider a partially observable setting where each agent perceives only a subset of the global state. Let Oi ç S denote the observation space of agent and the joint observation space is O = Nex Oi Associated with every agent we define a set of safety-related functions {b)))en((*)) where by O IR, and (s)s) is the index set of the constraints dependent the state. Our safety-constrained problem is expressed as a Stochastic Optimal Control Problem (SOCP) as follows:
min Eysn Σ E)t((s,).;))
subject to Egsep [min
(o b, = g(s:)] 0, V, jeN((ss)) Vtz O
(6)
where π (-|s) II is the joint state feedback policy of the agents, i.e., ग :: (AA), where A is the space distributions over the support
random variable, and y maps state to agent's 's observation. It is noteworthy that this problem requires that the constraints are satisfied pointwise in time for every trajectory generated by æ. The observation comprises of the agent's own state, states of observable agents and the environment states. Hence, the constraints can be alternatively expressed

---

in pairs as (s'"", s*) O where s is the state of another agent, or the environment. We assume that the constraints are initially satisfied upon their introduction.
Cooperative setting with pointwise-in-time hard constraints vs. CMDP setting: As shown in 89 CMDPs address safety in fully cooperative multi-agent settings where agents share common reward and seek to minimize the expected cumulative cost:
min E TI Σ (s)( a.)
subject to
I Σn (st,
sajiivj
While structurally similar to pointwise-in-tee constraints in the single-agent setting 73 the multi- agent CMDP formulation requires policies to depend on the joint state 8t, making decentralized execution infeasible in partially observable environments and linearly increasing constraint complexity with the number of agents. Our formulation overcomes these limitations by enabling decentralized safety guarantees in partially observable, safety-critical multi-agent systems.
5 Safe Hierarchical Reinforcement Learning (HMARL-CBF)
We propose novel Safe Hierarchical Multi-Agent Reinforcement Learning (HMARL.) approach illustrated in Figure I to tackle the aforementioned problem in (6). Specifically, we decompose the problem in (6) into a bi-level RL problem. The higher-level problem objective concerns the optimization of the joint performance of the agents. The lower-level optimization problem addresses the safety of the agents defined using the second term in the stage cost in (TO). Our proposed HMARL method is based on the idea of skills, which defined as follows.
Safe agent skills. Inspired by [66] 12), we define canety-constraiied skill =² for an agent as seven- tuple: (I, T: (ini)) ) T maany Di Ne (o'), r (s' aa), where Z3 S' is the initiation set for the skill respectively, o E I and imi E S are the observation of agent and environment state at the time of initiation of the skill, Ti( siitt = O' is the termination set of the skill, and O x N + (0,1) } is the termination function as defined below:
11 if e To iiiit: (mit) or, t T max
(' () 10, otherwise
(8)
A skill is terminated as a function of the initiation state of the skill sy in one of the two ways: i. either upon observing that the skill has been executed or, ii interruption due to a timeout for exceeding the maximum allowed duration of execution, T Tiax N. ('((') ) is the set of safety constraints for agent i that has to be satisfied during the execution of the skill. We define a reward function rs a' corresponding action a' in state s' which is used learn the skill. Finally, Tे. O² J (AA')
is the safe skill policy expressed as a SOCP as follows:
max Eg aitwr x subject Es, [mi b) 0, st J. N(s)(ss), =t= 0......,
(ss aj)s stmst
The set of skills for an agent is defined as Z and the joint set of skills of the N agents is defined as Z UseN Zt. We define a high-level policy for the N agents over their set of skills denoted by TH : 0 + A(Z) where ти 1g. which becomes the high-level RL problem. Subsequently, we define n= O J= + ((A) as the joint policy for the joint skill 2 ε Z of the N agents, where J= = U Learning the safe skills of the agents becomes our low-level RL problem. Next, we detail our proposed HMARL-CBF method.
5.1 Multi-Agent High-Level Policy Learning
As mentioned above, the high-level policy is formulated as joint centralized optimization problem involving all the agents over their set of skills. The optimization problem can be solved using existing CTDE schemes, either using policy gradients or value decomposition. The higher level problem can be modeled as an MSMDP and the corresponding extrinsic reward for state s and skill z executed in steps using policy मæ can be defined as: R(s, 2, k) = Ea,- -T, [E+=]- yr a )|80 s). We call this extrinsic as it is tpsc-specific and, hence, chosen to be the environment reward. This is revised by
5

---

incorporating safety corresponding to (6) rendering the following extrinsic reward:
( ((s,a)- Σ Σ TT s²) 0)6(6', (eN
(10)
where 8 S, a A. p/ e Rso and is an indicator function defined as follows.
1, if for some i,
I(x)
0, otherwise.
The minimization problem is thus mapped maximization problem of accumulated reward. Then, the problem in (6) can be mapped to the following problem:
J ( (r-(HHoN=) (R (T) (7-(FHOT=)
Σ 1'=0
tv Fu, TH
kyr-
E B
Es Be.a.rzz Mi
Σ
(==0
TH arg max ( ryelll
(II)
where F 11 is the space of the high-level policies, T. E 1lg the space the joint skills policy of the agents and is trajectory sampled from the joint composition of иH and πz. The policy ai given a joint skills policy of the agents т₃ optimizes the trajectories with respect to the reward in (1O). This problem we parameterize the policy with 8, the policy gradient of the parametrized policy THg as derived in 74 is given by:
be solved using any existing on-policy off policy MARL algorithm [3] 81 25
Σ of ((,)) f-0
(T nT) BT-(THH, o
Ve log THe (Z)ls))
(12)
Under CTDE scheme, in order execute the policies de-centrally, the agent policies/actor network (parameters) can be learned individually agent-wise conditioned on their observation (and history hi-1) ie., THe ( [oo, hf-1). 1). using a centralized critic network. Alternatively, the policy THe can be learnt by minimizing the following (Q-learning) loss using parametrization Qs:
((0) Eeg zu Se.1k [[R(s,, Fy, kv) + max (S(41, #r-1) (sy, 2] (13)
zell,
where Q8ot is the total state-action value as defined in (2) which can be learned using value decom- position methods based on the Individual Global Max principle proposed [6 30
5.2 CBF-Based Individual Safe Skill Learning
The low-level policy associated with exccution the skills selected by the agents based on the high-level policy. We define intrinsic step reward conditioned any skill 2¹ of any agent i as r a'|=*) to incorporate intrinsic motivation. Notice, the safety constraints are included in the extrinsic step reward rи. Additionally, to ensure safe execution of the skill, we formulate the low-level policy as a Stochastic Optimal Control Problem (SOCP) as follows:
Rir Σ Eri(si twt!
max Ell Bie ke Jed
E
subject to E., [min s))]20 (s)
e Ji
(14)
where L' is the time the skill was initiated by agent i. In order to solve the problem we formulate a parameterized constrained optimization problem based CBFs at each time step that gives us the deterministic policy as a function of the observation conditioned on the skill μ [o include control Lyapunov functions (CLFs) V as functions of the state 8 and the terminal state st, Ti assist in learning the optimal trajectory corresponding the skill and terminate by
We

---

converging to the termination set with Tmax This is similar to stabilizing the system at the terminal state. Note that 8t, need not be terminal state, but any state-related Lyapunov function that aids in learning the skill. This results in an optimization problem with quadratic cost function and linear constraints which makes a Quadratic Program (QP) that requires low computation. The optimization problem corresponding to the deterministic policy j (ojj =f; is given below.
min afe.At,e
a. ("H()))a++ F² (Øp )a 6.
(15)
subject to
(7 s) LVV s Lo (5) s )a, n(s s ):0) s e²d, =j=1,2
LT m- 1 s!)a; ((b(((s s;) / am (5m-1 ()) ) VjEN((s)
( 6. o. ..... are the parameters of the optimization problem such that each term in the vector o' belongs to Rso, H (gdim((A.xxdimmn F [d)ii(())) and is the slack term from the CLF constraints. The chosen quadratic objective, combined with the affine CBF constraints, forms a Quadratic Program (QP) at each time step 1, which is well suited for real-time safety-critic control applications.
where o' - [ p 0
In our case, H is chosen to positive definite as the objective generally includes tracking error/control effort minimization, which makes the problem strictly convex. The parameters ø' are learned as a function of the agent i's observation of and parameterized by v²: of T (o*1=z) where I differentiable w.r.t to v²_ The function can be deterministic or stochastic function of continuous random variable e using the reparameterization trick in 778.. With the reparametcrization trick, the control policy becomes stochastic, thus allowing for use of stochastic policy gradient algorithms.
{s), zú. sho 2ig si. zi...... can be represented as a sequence of states and actions indexed by time as well as states and skills indexed by time on different scale. Additionally, the forward invariant property of CBFs can guarantee that the trajectory resides in the safe set. In order to jointly learn with the high-level policy from the sampled trajectories, we map the safe skill learning problem for any agent in 14) as a trajectory optimization problem with the policy tt, given by the following objective:
agent's trajectory r² {si, ag. si ai. 82 a.....}
00 E
] (16)
('THHTTT' TT.. (E
Σ (ri
the aj Tem.
We drop the policy parameters & and superscript from y/ to avoid overloading the symbols. Here, n= is the joint skills policy of the other agents except agent i. we solve the optimization problem in (16) by learning the parameters of the QP in (5) that maximizes the total reward. The problem in (16) can be solved using policy gradient (PG) methods as detailed in Appendix A.3
Theorem 5.1. The satisfaction of the CBF constraints by in (15) guarantees safe execution of the skills, thus guaranteeing the safety of our proposed HMARRL approach.
The proofs for the theorems can be found in the appendix A.3 It should be noted that robust CBFs can be used the presence of model uncertainty following 9 to achieve similar safety guarantees. Besides that, we detail how the high-level task specific reward can be incorporated to the low-level skill learning to align the low-level learning to the task.
An example of skill in autonomous driving context is cruising. This skill causes the vehicle to move at a constant speed for a fixed distance. We choose the reward as the error norm of the velocity. And, a velocity CLF constraint in (36) is incorporated in the skill QP based policy by setting Udes Veurrent where Veurrent the current velocity of the vehicle. And the termination constraint chosen as the target distance.
6 Experiments
6.1 Environment Details
We validate our proposed method on traffic scenarios involving complex road networks with multiple agents that must ensure safety concerning neighboring agents while navigating toward their desti- nations. We use the METADRIVE simulator [35] to implement, validate, and compare method
7

---

against existing baselines in cooperative setting, where agents (autonomous vehicles) collaborate to improve the overall traffic efficiency.
The five environments from the METADRIVE, simulator are illustrated Figure] We implement environment to
algorithm in four of the originally proposed environments and design simulate an on-ramp merging roadway, replacing the parking lot environment. Since safety is less challenging in parking lots due low vehicle speeds, the problem be formulated and solved as static optimization problem with a fixed number of vehicles and parking spaces. These environments involve regions where agents can cross path with other agents. Hence, is necessary for the agents to cooperate but at the same time stay safe with each other to achieve the task i.e., reach the destination successfully and timely.
Mesging
Intarection
Roumbbout
Trllgae
Bottleneck
Figure 2: Illustration of the various METADRIVE environments.
6.2 Implementation Details
have implemented our proposed method using both on-policy and off-policy algorithms. Specifi-
cally, we implement our method based joint and group-based policy learning approach which are detailed in section A.4 The on-policy implementation was done using MAPPO [8]] with PPO
[60] for the skills. The off off-policy implementation used QMIX (56) for the higher level policy and SAC [26) for learning the agent skills. The extrinsic reward was chosen to be same as that defined in METADRIVE (which includes the energy, time, driving reward, lane deviation, collision penalty) of all the AVs in the environment. Besides that we include some regularization terms corresponding to skills avoid overfitting any specific skill(s). The algorithms were implemented using RLLib
[75 an open-source framework for MI., Further details about the implementation can be found in
A.
6.3 Results and Baseline Comparison
We compare our algorithms against the state-of-the-art baselines in METADRIVE environment namely, CoPo 44 the independent policy optimization (IPPO) method [14] using PPO (60) as the individual learners, and Mean Field MARL with centralized critic [80] and form the mean field policy optimization (MFPO). The curriculum learning (CL) 45 is also baseline. where the training was divided into phases and gradually the number of initial agents in each episode from 25% to 100% of the target number. We ran our algorithm and the benchmark algorithms for 5 seeds over 300k and 1M iterations respectively. It is important to mention that the benchmark algorithms use 2000+ hours of individual driving experience in addition to the training iterations to achieve the illustrated results. On the other hand, we jointly optimize both the RL policies from same data, thus not introducing any additional sample complexity due to the hierarchal decompoxition. The training results include the success rate and average episode length to illustrate the efficacy and validate our proposed approach.

---

CCPO
0.6
0.0
06
farttons
Figure 3: From left to right: success rate in merging, tollgate, and bottleneck environments.
683
300
Oard 0900
500
190
820
600
320
200
300
0.5
0.2
Eevations
Decasoes
Ienalioe
Figure 4: From left to right: average episode length in merging, tollgate, and bottleneck environments.
Notably, the efficiency metric defined by the authors of CoPo
calculated as the inverse of the average episode length divided by the difference between the number of successful and unsuccessful further supports the validity of our chosen metrics. We present the results of training for the Merging, Bottleneck, and Tollgate environments in Figures and 4 respectively. Additional environment results can found in appendix
49
The key observation is that our algorithm achieves almost 100 % success rate with an approximate model of the vehicle (see also appendix [A.II]] from the beginning of training, while also optimizing the joint policy evident from the average episode length. Although several baselines achieve better average episode length in the tollgate environment, the success rate is very low, implying most agents crash out, resulting in the remaining agents finishing faster, returning a lower average episode length. With the hierarchical decomposition, our algorithms converge faster (s 300k) compared the baselines which we run for 1M iterations, thus confirming the claim about expediting convergence.
Table 1: Summary of results for Merging, Intersection, Roundabout, Bottleneck, and Tollgate Environments. t: higher is better; J: lower is better. Bold: the best performance for each metric.
HMARL-CRI
HMARL-CBT
(ou-palley)
Gromping)
coff-polcyy
Grompingg)
Success
u.48.amm
0.34a012
0.99ua
0.99,a.0
0.99.0.01
0.09,0.01
Merging
Time
782 42,
600, 86
ses 25
501 BaaLLs
164.52,24t
187
199 20..0
196, 70
14.44,ass
20.41,x.
31,96
14.75mm.
Success
0.96,
0.96
Intersction
Time
725 Pa
887.60 111.00
2250.
1058.
22.17
417.41a00.2
ses 22
264.28ua2.9
250.11,12
Energy
Success 1
0.56
0.02an
0.25
0.99
0.99
0.98acca
0.9Ruaia
Roundabout
Time
004,00
$16,15,
12500,00
1304 24,
413,15
575.67,
37121
326
22..26..a.a
12.07.077
12.31
104 aa
43,76,0m
10.15
11.42,nm
Success 1
0.48
0.06
0,99u
0.99
0.99.0.01
0.99
Bottlenck
Tix:
679 Sau
1067.
1896, 2eas
202 202
$1360,
240.47 220.a
Energy
0.50.am am
00.07.0
4.80.aas
Staccoss 1
0.00
0.900am
0.99
Tollgate
Tix:
1884.96
960.32,
427.02,2
877.42,a
307. 72,88,,,m
Pnrrgy
55.05,277
5.17.0.44
895
Next, we evaluate the efficacy of our algorithm in optimizing the task objective against the baseline algorithms. Specifically, we compare against MFPO and COPO, well MAPPO-Lagrangian and the MACPO algorithm introduced in [24]. We use the success success-weighted metrics for evaluation and comparison with the baseline, ablation, and generalization experiments. The reason for weighting the metrics by success rate is that poorly performing algorithms with very low success rate (agents crashing and getting removed from the episode rapidly) can achieve superior average energy and time, which is debunked through this. Besides success rate, we use success weighted averuge energy and success weighted average time which are the average energy and episode length of all agents during an episode divided by the success rate.

---

The evaluation results are presented in Table [ As can be seen, our method achieves almost 100% success rate which significantly better than the baselines including MAPPO-Lagrangiaan and MACPO which are based on the cMDP framework. Also, we achieve better time and energy efficiency compared to the baselines across all environments. This goes to highlight that our methods learns better cooperative policy in addition to enhancing the safety of the agents.
6.4 Ablation Results
In this subsection, we present ablation studies to further highlight the merit of our proposed approach using the HMARL-CBF on-policy implementation. The results are presented in Table 2 First, we remove the hierarchical structure to implement flat MAPPO policy using various penalties on the safety constraints. However, a flat policy fails to learn these environments. Next, we replace the cooperative high-level policy with uniform random selection of skills across agents. This causes a reduction in success rates as well as performance across all environments. Subsequently, we investigate the necessity of pointwise-in time constraints via random constraints dropout- masoung road boundary constraints in the low-level policy with a Bernoulli distributed random variable (success Probability -h)chhhhhh uniformly reduces success rates and exhibits the largest performance decline in environments where agents cross paths across several zones in the environment (i.e. safety-crite -rittcccri Next, we substitute the low-level skill with an parameterized Linear Quadratic Regulator (LQR)-based (removing the CBFs) within the hierarchical MARL framework. The LQR based skill solves fixed final state control problem without incorporating safety constraints. This causes significant decline in success rate as well as in success- weighted time and energy efficiency. Finally, we examine fixed low-level policy parameterization by selecting parameters at the boundaries of the feasible space without prior knowledge, which further diminishes performance across all metrics across the environments.
Table 2: Summary of ablation results for all the environments (Pf is penalty factor).
Fovironments
Flat Policy
Flar Fulicy (PF- 5)
Policy (PI - LB)
Uniform high-keel
Constrinns
Hierantical MARI.
HMART-CRF
(PF -) l)
Dropoct
with LQQ
paranctoiintion
Success
0
0.94a
0,01aam
0
0.52,
0.99.a.01
Merging
Energy
14 17
15.30 at
22. 30 19
14.10.n.s
Time I
21 20,,00
183.
164.52.2.z
Success
0.86 os
0.80
0.14.0.22
0.97.9.04
Intersection
Energy
5. 78
5.00,
50.00.5:7
5. 5.56....oz
Time
$50.50....17
Las.es
Success
0
O 1..88an
0.R0
0
0
O 0.999...0 on
Roundshout
Energy
13 20,1
8. 6.50,s
9. 9.890...30
Time
907. 377.355.
800088 aaa:
403, Rupr.on
Sucvrsee
0
0
D. 3.65aa
0.55%amm
0
0
0.99 0.99....o
Botlleeck
Enegy
10.00 to
12.113,a on
7.97
Time I
922.07 uai iea
963. 34.44
2411.6 16.1.66...o
Succrse
0 0.922am0
0
0
0.99.a.02
Tollgate
Enegg
9.
8.400...0
Time
420 30..00.000
360.32...90
We also perform generalization experiments assess how well our policies to various environmental changes. The results for the experiments can be found in appendix A.1.3 A.1.3
7 Conclusion
In conclusion, we addressed the problem of learning safe policies in multi-agent, safety-critical autonomous systems with pointwise-in-inn safety constraints. We proposed HMARL-CBF, a novel hierarchical MARL approach leveraging Control Barrier Functions (CBFs). Our method employs skill-based hierarchy, learning a high-level policy for selecting skills and low-level policy for safely executing them. We validated our approach in complex multi-agent environments, demonstrating its effectiveness through extensive empirical results. Future work can explore jointly learning both the set of skills and their corresponding policies, along with the cooperative behavior.
A eknowwedgments
The authors thank the anony reviewers for their invaluable feedback and constructive suggestions. This work was supported in part by NSF under grants CCF-2340776, ECCS- CNS-2149511, 1, and by ARPA-E under grant DE-AR0001282.
19CC--1931M 1931600, 1600, DMS- 1664644,
10

---

References
[1] S. Ahilan and P. Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning, 2019.
[2] E. Altman. Constrained Markov decision processes. Routledge, 2021.
[3] A. D. Ames, K. Galloway, and J. W. Grizzle. Control Iyарunov functions and hybrid zero dynamics. In 2012 IEEE S1st IEEE Conference Decision and Control (CDC), pages 6837- 6842, 2012.
[4] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization as layer in neural networks, In International conference on machine learning. pages 136-145. PMLR, 2017.
[5] P.-L. Bacon, M. Harb, and D. Precup. The option-critic architecture. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[6] L. Berducci, S. Yang. R. Mangharam, and R. Grosu. Learning adaptive safety for multi-agent systems. 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2859-2865, 2023.
[7] R. Cheng, Orosz, M. Murray, and W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-Fire Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 19/EAA 19. AAAI Press, 2019.
[8] Y. Cheng, P. Zhao, and N. Hovakimyan. Safe and efficient reinforcement learning using disturbannee-bbeerer---aes control barrier functions. In Learning for Dynamics and Control Conference, pages 104-115. PMLR, 2023.
[9] J. Choi, D. Lee, K. Sreenath, C. Tomlin, and S. L. Herbert. Robust control barrier-value functions for safety-critical control. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 6814-6821. IEEE, 2021.
[10] Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances neural information processing systems, 31, 2018.
[11] Y. Chow, O. Nachum, A. Faust, F. Guzman, and M. Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv: 10031, 2019.
[12] C. Chuck, K. Black, A. Arjun, Y. Zhu, and S. Nickum. Granger causal interaction skill chains.
Transactions Machine Learning Research, 2024.
[13] Dayan and G. E. Hinton. Feudal reinforcement learning. Advances in Neural Information Processing Systems, pages 271-278, 1992.
[14] C. De Witt, Т. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson. independent learning all you need in the starcraft multi-agent challenge? arXiv preprins arXiv:2011.09533 2020.
[15] G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of Artificial Intelligence Research, 13:227-303, 2000.
[16] Z. Ding, K. Wei, and H. Lu. Upper confidence primal-dual reinforcement learning for cmdps. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[17] Y. Emam, G. Notomista, P. Glotfelter, Z. Kira, and M. Egerstedt. Safe reinforcement learning
using robust control barrier functions. IEEE Robotics and Automation Letters, (0:2886-2893, 2021.
[18] Z. Gao, G. Yang, and A. Prorok. Online control barrier functions for decentralized multi-agent navigation. In 2023 International Symposium on Multi-Robor and Multi-A Bulti-Agent Systems (MRS), pages 107-113. IEEE, 2023.

---

[19] J. Gehring, G. Synnaeve, A. Krause, and N. Usunier. Hierarchical skills for efficient exploration. Advances Neural Information Processing Systems, 34:1 14:11553-11564 2021.
[20] M. Ghavamzadeh, S. Mahadevan, and R. Makar. Hierarchical multi-agent reinforcement
learning. Autonomous Agents and Aulti-Agent Systems, 13:197-229, 09 2006.
[21] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.
[22] S. Gu, J. Grudzien Kuba, Y. Chen, Y. Du, L. Yang. A. Knoll, and Y. Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:103905, 2023.
[23] S. Gu, J. G. Kuba, Y. Chen, Y. Du, L. Yang, A. Knoll, and Y. Yang. Safe multi-agent reinforce- ment learning for multi-robot control. Artificial Intelligence, page 103905, 2023.
[24] S. Gu, J. G. Kuba, M. Wen, R. Chen, Z. Wang. Z. Tian, J. Wang, A. Knoll, and Y. Yang. Multi-agent constrained policy optimisation, 2022.
[25] Gupta and A. Dukkipati. Probabilistic view of multi-agent reinforcement learning: unified approach. 2019.
[26] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[27] C. Hao, C. Weaver, C. Tang, K. Kawamoto, M. Tomizuka, and W. Zhan. Skill-critic: Refining learned skills for reinforcement learning. arXiv preprint arXiv:2306. 20880300.008888 2023.
[28] J. Harb, P.-L. Bacon, M. Klissarov, and D. Precup. When waiting is not an option: Learning options with a deliberation cost. In AAAI Conference on Artificial Intelligence, 2018.
[29] M. Harb, P.-L. Bacon, M. Klissarov, and D. Precup. Waiting for the right time: When to commit to action in hierarchical reinforcement learning. International Conference Learning Representations (ICLR), 2018.
[30] Y. Hong. Y. Jin, and Y. Tang. Rethinking individual global max in cooperative multi-agent reinforcement learning, Advances in neural information processing systems, 35:32438-32449, 2022.
[31] S. Huang et al. Dittogym: Learning to control soft shape. In ICLR, 2024.
[32] W. Huang, J. Ji, B. Zhang, C. Xia, and Y. Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations, 2024.
[33] S. Iqbal and F. Sha. Actor-attention-cae for multi-agent reinforcement learning, In Interna- tional conference on machine learning, pages 2961-2970. PMLR, 2019.
[34] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[35] Q. Li, 7. Peng. I.. Feng, Q. Zhang, 7. Xue, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning, 2022.
[36] S. Li. R. Wang, M. Tang, and C. Zhang. Hierarchical reinforcement learning with advantage- based auxiliary rewards. Advances in Neural Information Processing Systems, 32, 2019.
[37] H. Liu, C. Wang, and B. Zhou. Reasoning multi-agent behavioral topology for interactive autonomous driving. In NeurlPS, 2024.
[38] M. Liu, T. Yang, and M. Wang. A natural policy gradient primal-dual method for constrained markov decision processes. In Advances Neural Information Processing Systems (NeurIPS), 2020.
12

---

[39] Y. Liu, J. Ding, and X. Liu. Ipo: Interior-point policy optimization under constraints. Proceed- ings of the AAAI Conference on Artificial Intelligence, 14:4940-4947, 04 2020.
[40] R. Lowe, Y. Wu, A. Tamar, J. Harb, Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperatiee-compettivy environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[41] W. Luo, W. Sun, and A. Kapoor. Multi-robot collision avoidance under uncertainty with probabilistic safety barrier certificates. In NeurIPS, 2020.
[42] R. Makar, S. Mahadevan, and M. Ghavamzadeh. Hierarchical multi-agent reinforcement learning. In Proceedings of the fifth international conference Autonomous agents, pages 246-253, 2001.
[43] D. J. Mankowitz, T. A. Mann, and S. Mannor. Adaptive skills, adaptive partitions (asap). In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[44] O. Nachum, S. Gu, H. Lee, and Levine. Data-efficient hierarchical reinforcement learning.
In Advances in Neural Information Processing Systems, pages 3303-3313, 2018.
[45] S. Narvekar, B. Peng. M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50 2020.
[46] S. Nayak, K. Choi, W. Ding, S. Dolan, K. Gopalakrishnan, and H. Balakrishnan. Scalable multi-agent reinforcement learning through intelligent information aggregation, 2023.
[47] Q. Nguyen and K. Sreenath. Exponential control barrier functions for enforcing high relative- degree safety-critical constraints. 2016 American Control Conference (ACC), pages 322-328. IEEE, 2016.
[48] R. Parr and S. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems, pages 1043-1049, 1997.
[49] Z. Peng, Q. Li, K. M. Hui, C. Liu, and B. Zhou. Learning to simulate self-driven particles system with coordinated policy optimization. Advances Neural Information Processing Systems, 34: 177884-1779,, 2021.
[50] M. A. Pereira, Z. Wang, I. Exarchos, and E. A. Theodorou. Safe optimal control using stochastic barrier functions and deep forward-backwant sdes, 2020.
[51] K. Pertsch, Y. Lee, and J. J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
[52] D. Precup and R. S. Sutton. Temporal abstraction in reinforcement learning. PhD thesis, University of Massachusetls Amhersi, 2000.
[53] Y.-L. Qiao, J. Liang, V. Koltun, and M. C. Lin. Differentiable simulation of soft multi-body systems. In NeurlPS, 2021.
[54] Z. Qin, K. Wabersich, S. Wang, and C. J. Tomlin. Learning safe multi-agent control with decentralized neural barrier certificates. In ICLR, 2021.
[55] M. Rahmati, M. Nadeem, V. Sadhu, and D. Pompili. Uw-marl: Multi-agent reinforcement learning for underwater adaptive sampling using autonomous vehicles. In Proceedings of the International Conference on Underwater Networks Systems (WUWNET2019) 2019.
[56] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Monotonic value function factorisation for deep multi-agent reinforeement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[57] T. Rashid, M. Samvelyan, and C. Schroeder. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, pages 4295-4304 2018.
13

---

[58] K. Rohanimanesh and S. Mahadevan. Learning to take concurrent actions. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15. MIT Press, 2002.
[59] E. Sabouni, H. Sabbir Ahmad, V. Giammarino, C. G. Cassandras, I. C. Paschalidis, and W. Li. Reinforcement learning-based receding horizon control using adaptive control barrier functions for safety-critical systems*. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pages 401 2024.
[60] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000 2017.
[61] A. Sharma, A. Gupta, and S. Levine. Skill-based meta- learning. In International Conference on Machine Learning (ICML), 2020.
[62] O. So, C. Ge, and C. Fan. Solving minimum-cose reach avoid using reinforcement learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, A Advances in Neural Information Processing Systems, volume 37, pages 30951-30984. Curran Associates, Inc., 2024.
[63] K. Son, D. Kim, W. J. Kang, et al. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896, 2019.
[64] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, T. Back, G. Synnaeve. et al. delmo-decomoooitionn networks for cooperative multi-agent learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[65] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1 112( (121))1811221 1999.
[66] R. S. Sutton, D. Precup, and S. Singh, Between mdps and semi semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 11111211 1999.
[67] H. Tang, J. Hao, T. Lv, Y. Chen, Z. Zhang, H. Jia, C. Ren, Y. Zheng, Z. Meng, C. Fan, and L. Wang. Hierarchical deep multiagent reinforcement learning with temporal abstraction, 2019.
[68] C. Tessler, S. Givony, T. Zahavy, et al. A deep hierarchical approach to lifelong learning in minecraft. In AAAI Conference on Artificial Intelligence, pages 1553-1561, 2017.
[69] A. Vezhnevets, S. Osindero, Schaul, et al. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549, 2017.
[70] A. Wachi et al. Safe exploration in reinforcement learning: A generalized problem and meta-algorithm. In NewrIPS, 2023.
[71] R. Wang, K. Wang, F. Xu, et al. Hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence. AAAI (LATT33AAAAT2AAAAT
AAAI Press, 2023.
[72] Y. Wang. S. Wu, H. Fu, Q. Fu, T. Zhang. Y. Chang, and Wang. Curriculum-baic based co-design of morphology and control of voxel-based soft robots. In ICLR, 2023.
[73] Y. Wang, S. S. Zhan, R. Jiao, Z. Wang, W. Jin, Z. Yang. Z. Wang, C. Huang, and Q. Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In A. Krause, F. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36593-36604. PMLR, 23-29 Jul 2023.
[74] R. J. Williams. Simple statistical gradient-followin algorithms for connectionist reinforcement learning. Machine learning, 8:229-256 1992.
14

---

[75] Z. Wu, E. Liang, M. Luo, S. Mika, J. E. Gonzalez, and I. Stoica. RLlib flow: Distributed reinforeement learning is a dataflow problem. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[76] W. Xiao and C. Belta. High-order control barrier functions. IEEE Transactions on Automatic Control, 67(7):365533666 2022.
[77] W. Xiao, C. G. Cassandras, and C. Belta. Hierarchical Optimal Control with Barrier Functions, pages 120- Springer International Publishing, Cham, 2023.
[78] M. Xu, M. Quiroz, R. Kohn, and S. A. Sisson. Variance reduction properties of the reparame- terization trick. In The 22nd international conference on artificial intelligence and statistics, pages 2711-2720. PMLR, 2019.
[79] M. Yang. Y. Yang, Z. Lu, W. Zhou, and H. Li. Hierarchical multi-agent skill discovery. In NeurlPS, 2023.
[80] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang. Mean field multi-agent reinforcement learning. In International conference on machine learning, pages 5571-5580. PMLR, 2018.
[81] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 15:24611-24622 2022.
[82] D. Yu, H. Ma. S. Li, and J. Chen. Reachability constrained reinforcement learning. In K. Chaudhuri. S. Jegelka, Song. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 22655,. PMLR, 17-23 Jul 2022.
[83] H. Yu, X. Ma, and B. Zhou. Towards safe reinforcement learning with a safety editor policy. In NeurlPS, 2022.
[84] L. Zhang, L. Li, W. Wei, H. Song, Y. Yang, and J. Liang. Scalable constrained policy optimiza- tion for safe multi-agent reinforcement learning. In NeurlPS, 2024.
[85] S. Zhang, O. So, M. Black, and C. Fan. Discrete GCBF proximal policy optimization for multi-agent safe optimal control. In The Thirteenth International Conference Learning Representations, 2025.
[86] S. Zhang, O. So, M. Black, Z. Serlin, and C. Fan. Solving multi-agent safe optimal control with distributed epigraph form MARL. In Proceedings of Robotics: Science and Systems, 2025.
[87] Z. Zhang, Y. Sun, F. Huang, and F. Miao. Safe and robust multi-agent reinforcement learning for connected autonomous vehicles under state perturbations. arXiv preprint arXiv:2309.11057 2023.
[88] Y. Zhao, Y. Yang, Z. Lu, W. Zhou, and H. Li. Multi-agent first order constrained optimization in policy space. Advances in Neural Information Processing Systems,
2023.
[89] Y. Zhao, Yang. Z. Lu, W. Zhou, and H. Li. Multi-agent first order constrained optimization in policy space. Advances in Neural Information Processing Systems, 36, 2024.
[90] B. Zhou et al. Smart: Scalable multi-agent real-time motion generation for autonomous driving.
In NeurlPS, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly present the problem and the proposed method, which is thoroughly supported by the rest of the paper including the methodology and experi- mental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations and future directions clearly discussed in both the conclusion and appendix sections.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to- to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limita- tions that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in devel- oping norms that preserve the integrity of the community. Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification:
The assumptions and proofs for all the theorems and propositions are clearly
presented. The main text contains the statements, while the detailed proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes],
Justification: We provide clear explanation of the proposed method and algorithm, ensuring transparency. Our experiments conducted using open source environments, making them easily accessible. Additionally, we include detailed implementation information, baselines and evaluation metries enabling readers reproduce the results and validate the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution
dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (c.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide for reproducibility. In the case
17

---

of closed-souree models, it may be that access to the model is limited in some way (e.g.. registered users), but it should possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We propose a novel method for solving safe coordination in multi-agent, safety- critical systems. Our approach compatible with any off-the-shelf MARL and RL algorithms (both on-policy and off-policy), as demonstrated in our experiments. We also utilize open-source environments, ensuring ease implementation. We intend to release the code upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: /at:://nipspp. cc/
sorreeSummsssion. swnnnissssooooo.ooo olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips http:/cccpclcccggiii..eeee weapguieeesoooooee SecoodeSubag Summississs.........mmm...m walmissionPooi iessonnoolcco for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the implementation details including the hyper parameters, network structure, learning details including optimizer configuration in the appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [es]..
Justification: We report the results mean values with confidence intervals, calculated from multiple runs of each experiment to ensure statistical reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, the hypothesis Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The details the computational resources utilized for producing the experimental results have been provided in the appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: We reviewed the code of ethics in the provided link and confirm work is conformal with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

Justification: The broader positive and negative societal impacts of the work has been included in a separate section prior to appendix section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many pupers will be foundational research and not tied to particular applications. let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deePfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate DeePfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: We utilize open souree environments and models for conducting the experiments on this paper. Hence, we did not share the code. Thus, our work does not require any additional safeguards against misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets utilized in this work, including code and environments, are appropriately credited in the paper. We fully comply with their terms of use and licensing agreements, and license information is clearly specified wherever applicable.
Guidelines:
The answer NA means that the paper does not existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., wehsite), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pappersi thcode com/ tasa/attaaaee datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
We do not release any new assets as part of this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work did not involve crowdsourcing or research with human subjects. Thus, this relevant our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification: This research did not involve any human subjects and thus is not subject IRB or similar approval.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, non- standard component of the core methods in this research? Note that the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were used solely for writing and editing purposes and did not contribute to the development of the core methodology or ideas of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve
LLMs any important, original, non-standard components.
http://ciisscceo/ooooaeeeeee///a LLM) for what should or should not be described.
Please refer our LLM policy
22

---

A Appendix supplemental material
A.1 Experimental Details and Additional Results
We begin this section by presenting the details of the training, and evaluation experiments. Besides that, we include results for the generalization experiments used to validate our proposed method.
In addition to the METADRIVE environment, evaluated our method on suite of lidar-based environments introduced in [85 as illustrated in F All environments are partially observable, with agents equipped with lidar sensors. In the Targer environments, each agent is assigned fixed goal and must reach it while avoiding collisions with other agents and obstacles. The primary distinction between the Target and Target-Bicycke environments lies the underlying agent dynamics; further details are provided in Figure [ In the Spread environment, agents must cooperate to cover all goal locations while avoiding both obstacles and one another.
(2) Target
(b) Spread
(c) Target bicycle
Figure 5: Illustration of the lidar based environments which include: (a) target environment, (b)
spread environment and (c) target bicycle environment.
A.1.1 Training Setup and Results.
METADRIVE Simulator. In all five environments used in our experiments, the vehicles are initialized at random spawn points, and assigned destinations to each of them randomly. The agents terminated in three situations: reaching the destinations (deemed others or driving out of the road (deemed as failure). Once an existing vehicle is terminated new vehicles spawned immediately to ensure continuous traffic flow. The terminated vehicles remains static for 10 steps (2s) in the original positions, creating impermeable obstacles. If vehicle crashes into other terminated vehicles before it is removed from the environment, is also considered as failure. The success rate is the fraction of the successful agents to the total number of spawned agents. Thus, the environment allows for the total number of vehicles vary in time and exceed the the initial number of agents which mimics real-world challenging traffic scenarios.
successful), crashing with
We trained with 15 vehicles for merging. 30 vehicles for intersection and roundabout, 25 vehicles for bottleneck and 40 vehicles for tollgate respectively. As mentioned previously, the extrinsic reward corresponding (10) was set the environment reward, besides the regularization terms. The intrinsic reward is described in subsection A.4 We present the training results for the remaining three environments from figure i.e merging, roundabout and intersection in figure
OUn 1.0 PTO
14
0.8
88
MNOO
Iaratonn
Serettee
Beratone
Figure 6: Plot of the success rate of the agents with our algorithm vs the baselines in merging, roundabout, and intersection environments respectively. CL refers curriculum learning, four different curriculum were used with varying traffic densities. The benchmark methods use 2000+ hours of individual driving data prior to the training.
23

---

As can be noticed, our algorithm achieves near perfect safety rate with both on-policy and off- aff-policy algorithms during training, and, keeping consistent with the tollgate and bottleneck environments are trained for 300k iterations. However, in order to encourage exploration in roundabout and intersection environments for learning the cooperative behavior, we use curriculum learning by varying the arrival rates, while keeping the traffic density constant. The curriculum transitions happened at 30k, 50k, 80k and 125k iterations during training.
A.1.2 Evaluation Setup and Results
METADRIVE, Simulator. As mentioned previously used success rate weighted average time and energy for evaluating the performance of our algorithms. order to compare against the baseline methods, for each environment and algorithm (e.g. on-policy, on-policy w/Grouping, off-policy and so on) we train using five seeds. Then we run five episodes for each seed and generate the evaluation results by computing the mean and standard deviation over the 25 runs. In addition the presented results in Table 2 we run the baseline algorithms by changing the penalty factors for the safety related terms (both inter-agent and environment safety) in the environment reward for the bottleneck environment. The results are presented in Table The primary takeaway is that the added emphasis on safety does render better results for the baseline algorithms compared to our method. Besides that, different algorithmss perform differently with an increase in penalty factor, thus not exhibiting any trend between penalty and success rate.
Table Summary of baseline comparison using various penalty factors for Bottleneck environment. f: higher is better; J: lower is better. Bold: the best performance for each metric.
HMARL-CBP (on-policy)
Penalty Factors
Metrics
IPPO
CL
MFPO
CoPo
Success 1
0.09 08
0.34_0.15
0.48
0.75 07
0.99
Time I
2614.44_a53.4
139.3
679.83,158.as as
651.54463522
241.16,8.33
Energy L
69.6649.77
11.61_2.02
9.95
6,93
7.37
Success
0.58
0.395
0.615
0.71620.0z
0.9920.00
5
Time I
568
646.5, 48.16
7132278.7
529. 13279
241.1625.33
Energy L
30.41.5
10.820.24
8.6420.65
7.6920.02
7.37
Success
0.1840.08
0.55
0.42,9.23
0.655,0.04
0.99.0.00
Time I
1292.17.64
606. 200.59
1414,973
514 514, 126.92
241.16,s.a5
Energy I
40.97.27
9.14.0.98
22.12,22.08
8.37
7.37.0.00
Lidar Environment Suite.
Target
Spread
1.0
Hetted
DCPPO
tyo
Pana ty1 11
Pena ty1 51
Schodule
Lagelli
Lage(S)
LearO))
3166
-1.4
-12
-1.0
-0.8
-6.6
-0.4
-02
-1.0
-0.8
-0.6
-0.4
-0.2
Figure Success/safety rate vs reward for Target and Spread environment.
Baselines. We compare our method with DGPPO (5), InforMARL 46 and MAPPO-Lagrangiaa 24 three state-of-the-a methods for multi-agent safe control. For InForMARL, we evaluate fixed and scheduled penalty variants. Further details about the baselines can be found in 85 For MAPPO- Lagrangian, we test two settings for the Lagrange multiplier and one learning rate. Performance is better the plot is located in top right corner. As can be seen, our method achieves a high success rate while also achieving high reward compared to the baselines in [85. Finally, we present the results for the bicycle target environment in Figure which shows that our method scales with number of agents.
24

---

Target
1.000
Number Agenss
0.095
Agentc
0.980
Agents
0.965
0.900
0.073
0.970
0.965
0.900
-0.7
-0.5
-0.4
Reward
Figure 8: Success/safety rate vs reward for Bicycle Target environment.
A.1.3 Generalization Results
Besides the evaluation experiments, we conducted series of experiments to assess the generalizability of our proposed approach. Firstly we systematically vary the lane widths in the environments. The corresponding success rates are reported in Table Our analysis indicates that reducing lane width has minimal impact on success rates in the roundabout and bottleneck environments. However, more comparatively greater decline is observed in the intersection environment. This performance degradation is primarily attributed to the default intersection configuration (Figure [. where two lanes are available in each direction. Agents are trained to execute lane changes from both lanes depending on their assigned routes. When lane width is reduced, agents attempting to perform lane changing from the inside lane at times gets stuck (and, eventually get eliminated from the episode due to timeout), or hit the road boundary which causes the drop success rate. Conversely, increasing lane width generally maintains enhances success rates across all environments.
Secondly, we investigated skill transferability by training lowelevel skills in an intersection environ- ment and subsequently using them to train high-level policy across multiple environments. Namely, we investigate the transferability in the merging and roundabout environments. This because the observation is different and distinet for the Bottleneck and Tollgate environments compared to the other environments. The results are presented Table This analysis reveals that skills learned in one environment can generalize effectively to others producing comparatively similar performance. In the roundabout environment, we observe a minor decline in the success rate and increase in the weighted average energy and time; yet our approach outperforms the baselines presented in Table 2
Finally, we test the generalizability of our method for varying traffic density/ number of agents. The results for this test are presented in Tables and respectively. For every environment we trained for the default density as shown in the table and tested for lower and higher values. From the results, we can conclude that the success rate is unaffected by traffic density. The average time of the agents increases with increasing traffic density as expected. Finally, the energy values show a decreasing trend with increase in traffic density across most environments. This was anticipated because energy has a negative correlation with time.
A.1.4 Computer Details.
We conducted our experiments desktop equipped with single NVIDIA RTX A5000 GPU and 32 CPU cores. Given that our environments involve a high number of agents (up to 45 agents in the tollgate environment), the primary computational cost is attributed to data collection. Training and convergence are achieved efficiently, as demonstrated in the training and evaluation results. Furthermore, we utilize shared neural network architectures across agents, which significantly reduces the overall computational cost and training time.
A.2 Background
Definition A.1 (Relative degree). The relative degree of a (sufficiently many times) differentiable function b: R - with respect to system (3) is the number of times needs to differentiated along its dynamics until the control explicitly appears in the corresponding derivative.
25

---

Table 4: Success rate across different environments for varying lane widths.
Success Rate
Lane Width (m)
Merging
Intersection
Roundabout
Bottleneck
Tollgate
3.25
4.90
0.90
0.98
0.97
0.99
3.5 (default)
0.99
0.97
0.99
0.99 20.003
0.99
4
0.99
0.97=0.015
0.99
0.99
0.99
Table 6: Summary of skill transfer results (* indicates runs done by transferring skills).
Experiments
Success
Energy
Time
Merging
0.9920.01
14.24.0.25
166.1822.44
Merging"
0.9620.016
14.4920.58
178.34,5.90
Roundabout
0.9920.01
9.89
403. 18.11821.66
Roundabout
0. 0.96
11.3940.08
404.16,18.6
Table 7: Summary of results for varying number of agents in Merging, Intersection, and Roundabout environments.
Merging 15 (default)
Intersection
Roundabout
Metric
10
20
20
30 (default)
40
20
30 (default)
40
Success
0.997_a.
0.90
0.98_a.2
0.965
0.97_o.04
0.94.0.04
0.99
0.90.o.c
0.98.a.5
Energy
14.03.5.0
14.24,0.35
14
5.85.0.07
5.82.0.00 04
5 6.94.0.05
10.48
10.32
9.95
Time
165. 18
16,.18.264
169 169.55,9.1
318.54,19
332.3,a0
361 2.1.8_20.00
389.34
401.38
415.03
Table Summary of results for varying number of agents in the Bottleneck and Tollgate environ- ments.
Bottleneck
Tollgate
Metric
20
25 (default)
30
30
40 (default)
45
Success
0.99.0.01
0.99 .0.03
0.978.0.02
a.99.0.02
0.99.0.02
0.99
Energy
7.47
7.44
7.43
8.51 20.02
2.51
8.50gu.02
Time
227.22.7.77
243.59,8.39
244.62,6.62
371.53
371.05
374.02,8.48
For constraint b(s) with relative degree m, b : R R, and Co(s) b(s), we define sequence of functions Ç R" + R,ic (l,... m..
== = (s)
(si- ie (1....,
(17)
where aa(), i (1.....,) denotes a (m thth order differentiable class K. function. We further define a sequence of sets C, iE {L...., m) associated with (17) which take the following form,
C == { (8) 0}, i mI....,m..
(18)
Definition A.2 (High Order CBF (HOCBF) 76 77 ). Let C1,..., Cm be defined by [8] and ((s), ..., Sm (s) be defined by (77 A function Function (HOCBF) of relative degree for system
R R is High Order Control Barrier 3) if there exists (m - -ith order differentiable class functions a,, i (l, m - 1} and a class function am such that
sup[L7"b(s) + L, m-1 b(s)a ((b(s)) am (mmmmmm))) asA
(19)
for all E m C. In LT and Lg denotes derivative along f and g m times and one time respectively, and S(.) denotes the remaining Lie derivative along with degree less than or equal to v 1 (omitted for simplicity, see (76)).
Note that the HOCBF in 19 a general form of the degree one CBF 3 (m 1) and exponential CBF in 47 The following theorem HOCBFs implies the forward invariance property of the
26

---

CBFs and the original safety set. The proof is omitted (see [76] for the proof). If the dynamics are of the form: $ f(s.(()))))) a), we can express it in the form:
(s's a) + (s')aa
(20)
where s' and a' are the augmented system states and new primitive actions of the system respectively,
[(s a)] and g' (Odim()) and [ddi(aaxddinaa)) Generally, B 0 dimi a)) contains ones in its diagonal entries. The redefined dynamics can be used subsequently for defining CBFs.
B
A.3 Safe Skill learning
We provide the details about the computation of policy gradients necessary to learn the skill policy parameters in this subsection. We follow [ by applying the Farush-Kuhnn Tuchnr condition on derive the gradient of the state feedback policy with respect to its parameters. Specifically, let denote the dual variables on the HOCBF and CLF constraints, let D(.) create a
the Lagrangian
diagonal matrix from vector, and let J X denote the optimal solutions of ju and X, respectively. We can then write dn and dx in the form:
[ 1 ]
H GE G D(Ga" - h)
['[
1 dim(
]
(21)
0
where G h concatenated by CyP, C.
G² I. 1 s¹): (s)
G² -1 V/ (5', : 1=
(a)':ss)):: ,)): am (6m-1 (s', hg LV +ss nss' $ j-14 1,2....
hg L7 1 (6)
(22)
As the control bounds in [5] are not trainable, they are not included in G and h. Then, the relevant gradients with respect to all the parameters are given by
V Wn (dunu + -ud Sud)).FF N = du
(23)
Vou = (D (X*) (dxu² - -D
(24)
With the hierarchical composition of the policies (TH π²). we detine the augmented state å [s', 8 ",, 8ninnit] where 8 is the state the agents other than agent and 82 is the joint state of the agents at the time of the initiation of the joint skill The initiation state is included as the joint termination of the skill depends on the initiation state. The state and action value functions for agent i corresponding to reward ri. skill policy Te. (given the high level policy TH and the skill policy other agents ) are defined as follows:
o ($)
(E( at) E., [((),
= EE (r) (s), at) E, (Q(() = s*
(25)
The state-action value function for agent i, given the hierarchical policy composition TH Tz. is defined
= = E E [V'=(
(26)
The corresponding advantage function is given by:
T ($, a') a²= (8,a')-V*5
(27)
Based on the above definition of value functions and advantage function, as well the gradient of QP controller output with respect to its parameters, standard on-policy, and off-policy algorithms be used to derive the policy gradient to learn the controller parameters.
Proof of Theorem 5.1
27

---

Proof. Given, the constraints b/'s 's are satisfied at the initial time, the satisfaction of the CBF con- straints makes the corresponding safety sets forward invariant per Theorem (3.4). thus guaranteeing
their satisfaction at all future times. This makes the policies corresponding to the skills of the agents safe, thus, guaranteeing the safety of our proposed HMARL. approach.
High-level task specific reward to low level learning. The low-level policies should be such that they optimize the entire trajectory given by the problem (6). Hence, in addition to introducing intrinsic motivation through the low-level reward, inspired by [36], we introduce an advantage term to the low-level reward resulting in the following revised low-level reward:
(s (..=a = A=M (SU, 2r) +(1 A)r (s), afl=b)
(28)
N
where À E [0. 1] and & t is the time when the joint skill at time was initiated. The high-level advantage T (sr, ae) is defined below:
A (sv, ze) QTM (sr( ~ (se)
(29)
where QT (sp and V (st) are as defined in 2 and ①T) respectively. The parameter À can be
used to balance between skill learning and optimization of the joint behavior of the agents. From [36
we have that for high level policies the following holds:
J( [ + Bagi
Σ L--0
Σr ASH (str, 2c)
(30)
Proposition A.3. Given the low level reward in 28 agent i can be expressed as:
the revised low level learning objective J for
(( N(1
E En
ATM (8) +(1 (1- (TH
(31)
Lt'00
Proof. Based on 28
we can write our low level policy's 's learning objective for agent as the following:
ky -
(T T2
E
Σ E Baf
offaff-a.
Σ
toky (s
st.
8==0
1=
AT (sv, 2x)
E
Baj afj
7 tther
+ (1 - 1)F2 aziz(e
N
00
1=0
7 (1
hy-1
Σ t==0
nn (str ze)
(1- A)E
Σ
E
Σ
(5, a/1+)
toker
1 1-y
f=0
1-0
E
Σ 7t TT (sk, 2n)
+(1 (1- (T
N(1 -7)
Similarly, the low-level reward for every agent can be revised as above to align the skill learning process with the task learning process, as the first term related the high-level policy return.
A.4 Algorithm Implementation
A.4.1 Algorithm details
As mentioned previously, our method can be implemented using both existing on-policy and off- policy RL. algorithms. The bilevel RL problems presented in (11) and (16), respectively, should be solved sequentially. However, as previous works illustrated [34], jointly optimizing both policies by updating their corresponding parameters alternatively renders similar performances, which is consistent with our observation from the experiments. Based on this observation, the algorithm is presented in I The high-level policy TH is decomposed into agent-level parameterized policies to learn decentralized policies.
or Hg
28

---

Algorithm Hierarchical Multi-Agent Reinforcement Learning with CBFs
1: Initialize randomly the high-level policy parameters and parameters of the low-level policy //network parameter sharing done between agents.
parameters Ø
2: for each episode do
3: = ~ TH (= ob). Vie
I/ sample initial skill
4: Set data buffer D 0
5: for (. +- 0,..,, do
6:
Get action for agent based on the selected skill a p'(olkzk ViEN primitive action
VieN
sample
7:
Get St-1 and FH from environment and rt, V
8:
Sample skill for agent zi41 TH, (=*)o)), Vi (a(() (IN2(()) is the list of agents who have to update their skills based on termination scheme used.
9: end for
10: Update high level policy network 0 according to 12 Or, 13 using sampled data.
Update low level policy network parameters based on (23 and (24). using sampled data
12:
end for
We implement the algorithm using two different approaches inspired by existing works in the area of MARL. Notably, we implemented the algorithm the following ways. is important to note that under all these approaches, the hierarchical policy is learned agent-wise to allow for decentralized exccution.
Joint policy learning:
This approach involves solving [I] by optimizing the high-level policies of the agents jointly by learning the policy parameters 0's.
2. Group policy learning: This approach is inspired by the idea of learning the value function by decomposing the joint value function into the value function of groups of agents. In our fully cooperative setting, the total step reward be composed as sum of the rewards of the individual agents, i.e. ( a)= EN ri (s*, a²). Let {G represent the groups with each
G¹ N such that, GO n GD Ø and V / GH. We can define the group reward as the sum over the individual reward of the agents in the group. Then, under this approach we can express the objective in (I] below (We only include the necessary symbols in the expectation avoid symbol overload):
8
J J(TH Rx) = E
Σ
E
Ba
Σ 1=
ttky TH ( (t))
8
kg-1
Ng N
E
toke
(s [)
as
1-
Ty (aS) .3)
No
00
I
Σ HH (s
(32)
Tey
3=1
t'=0
Here, @ and G are the states and actions of the agents in group G The agent skill policies are learnt independently of each other. Hence. the joint skills policies can be written as Tz = M ge 1 Πae T&f IT Tgas where ggos is the joint skills policy of the agents in the group G Then, (32) can be written as:
Ng
ky-l
((HH,Tz)
E
Σ
E
Σ
(o
e a
Gi
j-1
J +
(
No
(T GNa)
where, by now becomes the time which any agent in the group changes selects/switches to new skill. We drop the low-level policy corresponding to skills for brevity. This approach
29

---

enables better scalability as the overall problem is decomposed into one that learns policies for the groups; however, this comes at the expense of performance as the original optimization problem involves all the agents.
Note that all these implementations retain the safety guarantee as it is offered by our safe skill-learning approach.
A.4.2 Implementation Details: METADRIVE Simulator
Metadrive environments illustrated in figure by default, include 3 basic sensors: lidar, sideDetector and lanel Detecctcoor which are used for detecting moving objects, sidewalks/solid lines, and bro- ken/solid lines respectively. The lidar state observation returns state vector containing necessary information for navigation tasks. The details of the observation can be found here The environ- mental safety constraints include the road boundaries and ii. the booth for tollgate environment only. And, each agent has to stay safe to all neighboring agents which are within the lidar sensing radius. It is important to mention that we only use a dynamic bicycle model for the CBF constraints, the environment uses pybullet physics engine for simulation. The kinematic bicycle model of the vehicle provided in (33)
ccos(1b)
1 1 - dx(p))
d = vsin(w),
(33)
1/- tan(5) - -x(p)
L a.
where p represents the position of the vehicle along the reference path, d the lateral deviation of the vehicle from the reference path, v denotes the yaw angle of the vehicle (heading angle relative to the path), and is the longitudinal velocity of the vehicle. The control inputs are 6, which is the steering angle, and a, which the acceleration (or deceleration). The parameter L denotes the wheelbase of the vehicle, and k(()) is the curvature of the reference path position It is important add that, these states are provided by the environment for each ego vehicle/agent in its observation.
In order to incorporate safety with respect to other agents and environmental obstacles, we define simple circles-based constraints as illustrated in Figure 9 And, for the road boundaries we use the lateral deviation from the center lane to define safety functions. Let p' and d' denote the distance along the path and lateral deviation of another agent j with respect to agent i, located within the sensing radius of agent This information is provided in the observation of agent i.e. (d pp, ,,d)) ) E O'. Note that, for an ego vehicle p' is always 0 in its body frame. let A (p', d', p d²) denote the euclidean distance between agent and j respectively. Similarly, let and d' denote the longitudinal and lateral distance of an environmental obstacle, like a tollgate booth, as illustrated in Figure 9 from agent i, and A(p', d' pp, d²) be the euclidean distance. Then, the safety constraints corresponding to other agent j. environmental constraints for an agent i are given below:
Alp d' ,p', d))² r(u'). A(p' d' p. d²²²
and |d'l s
(34)
where r(v') is monotonically increasing function of v², e Rşo and c Rso- Consider the CBF for the constraint for inter-agent safety defined as h 1(O') App d' .p,, dd) (d)rrr))) where Appi d' d'''''' N.... ,d)) = v(p' - p2' + (d' dd then the corresponding CBF constraint given by:
1 1 xxp'' - pp² + 2dd - d') (v' sin(v ) - -d)) + 2r())) dr(w") du a²
1² cos((it)
2(p' 1²)
Lyh((O'
Lyh(Oi)ai
+ ( -p²)² (cp + (d' dd'z 1 (()))))))))))) 20
(35)
M('')
30

---

Agent j
Agent.
alpl at .pl at)
r(v")
a(p pl.p..
Agent
Agent
Figure 9: From left to right: illustration of i. inter agent safety constraints on another lane, ii inter agent safety constraints on same lane and ii environmental constraints.
Additionally, we define CLFs for state reference tracking namely velocity and heading reference tracking. As we define termination state set corresponding to each skill, we define CLFs to drive the system to the termination state to be able to successfully execute the skill. However, is important to note that the CLFs are not defined specific to skills and, hence, are generalized to the task. The CLFs corresponding to velocity and heading reference are given by the equation below, where vi and vi, are the velocities at We parameterize the corresponding CLF constraint which in case of the velocity and heading constraints becomes the following:
Vw vdes)) (v² -Wdes) (( Wdex) = (1²² (ddees)
(36)
(37)
where Udes is the desired velocity and thaes is the desired heading of the vehicle.
Skills description. We define five skills for each agent for the METADRIVE environments which are: i. cruising, ii accelerating (speed up), iii. yielding (slowing down), iv. left lane changing, and v. right lane changing.
Cruising. As the name suggests, this skill causes the vehicle to move at a constant speed for fixed distance. The velocity CLF constraint in (36) incorporated in the skill QP based policy by setting Odes Vearrent where Veurrent is the current velocity of the vehicle.
Accelerating/speeding up This skill corresponds to agent increasing its longitudinal velocity. Let the initial velocity at the time of the initiation of the skill be Up- The skill terminates when the velocity reaches vr 10 du, where de Ro,0 denotes the desired velocity increment and ur the velocity of the agent at the time of termination. Hence, the Vaes is set to op + de for this skill in the low level QP based skill policy. This skill facilitates timely task completion and helps avoid congestion by preventing agents from unnecessarily slowing down the traffic flow.
Yielding/slowing down. Yielding. This skill enables agent to reduce its longitudinal velocity. Let tp be the velocity at skill initiation. The skill terminates when the velocity decreases to by 0o - de, where du Ryo is the desired reduction and By is the velocity at the time of termination of the skill. Thus, the velocity CLF in (36) is incorporated to the QP based skill policy by setting Udes = Uy du Yielding allows agents to safely navigate dense traffic and cooperate by yielding to agents approaching from other directions, thereby facilitating efficient joint task completion.
Left Lane Changing. This skill enables an agent to switch to the adjacent left lane. Let dieft,o denote the distance to the left road boundary and do be the lateral deviation from the center lane at the initiation of the skill. Let diane denote the lane width, provided by the environment. The target lateral position is computed as dieft,.T = dieft,, diane + dg. We define the desired heading angle as a function of i.c. Pdes f(e), where = dieft,T dieft,o. and incorporate into the CLF constraint on heading in (37) via QP-based low-level skill control policy. This skill is essential for scenarios such tollgates, bottlenecks, and merging zones, where adaptive lateral movement is required to maintain flow and avoid congestion.
Right Lane Changing. This skill enables an agent to switch to the adjacent right lane. Let dright, denote the distance to the right road boundary and do be the lateral deviation from the center lane at the initiation of the skill. Let dione denote the lane width, provided by the environment. The target
31

---

lateral position is computed as dright.7 drighto--ionns dione + dy. We define the desired heading angle as a function of e, i.e., Otes f(e), where T dright.n. and incorporate into the CLF constraint on heading in (37) via QP-based low-level skill control policy. This skill is useful in scenarios such as avoiding obstacles, preparing for highway exits, or facilitating merging from the right, enabling smoother and safer lane coordination.
As mentioned previously we include a timeout to ensure finite time termination of the skills for algorithmic sanity.
Intrinsic reward The intrinsic reward serves each agent toward desirable driving Setavviors-mmothneess lane-cen,,ring, and reference- of any external task reward. By combining several negative penalties, the agent discouraged from abrupt maneuvers, drifting off its lane, or deviating from its target speed and heading.
internal shaping signal that guides
rf. = =c 02
/v' Udes
C3
Udes
ca calld"11
(38)
Udes
π
Where C1 22 C3, C1, Cg are the coefficients of each terms. The intrinsic reward combines penalties aggressive acceleration and stcering, deviations from reference speed and heading, and lateral offset from the lane center, with a small forward- proggess bonus. Together, these terms encourage smooth, centered, and efficient driving without abrupt maneuvers.
CBF-Based Safe Skill Implementation. As described earlier, we parameterize the QP-based deterministic policy u(o 2, Φ) for each skill z, as detailed in Section 5.2 The QP formulation is parameterized through the objective, as well as the CBF and CLF constraints. In particular, we introduce parameters for the class-K. function in the CBF constraint, the shape and size of the eircular safety regions, and the convergence rate of the CLF constraint. To ensure validity, all parameters are constrained to be nonnegative using box constraints of the form (0, ΦH). where Óμ is a fixed upper bound applied uniformly across all parameters.
A.4.3 Implementation Details: Lidar Suite Simulator
Target. Agents dynamics in this environment are governed by double integrator dynamies with continuous-time states æ; [P' Ply " " e TR',, where p. [p Pyy IR² denotes position and v; [v " IR? denotes velocity; control inputs u, [a a R² represent accelerations along the Cartesian axes, and the agent dynamics follow a; [u by a at with velocity and control constraints bounded in [-10, 10] and [-1, respectively.
Spread. The underlying dynamics, state representation, control parametrization, and bounds are identical to those in the Target environment,i.e. agents evolve under double integrator dynamics with four-dimenn nou--dimnssiona state vectors comprising positions and velocities, and two-dimensional control inputs that modulate accelerations; the bounded constraints on velocities and controls likewise remain [-10, 10] and [-1, 1]. respectively.
Target Bicycle In the Target Bicycle environment, agents dynamics modeled using bicy- cle model where the state vector æ; [p', P`,, cos f¹, sin '²J' e R encapsulates the agent's position, heading orientation via its sine and cosine, and scalar speed; the control input u, (6 "",''' includes the steering angle 5 and linear acceleration a', with dynamics specified as d;; [c vsin 0, -is 0 tan 8, v cos 0 lan d, a]", where v E [-10, 10] and d [-1.47, are the admissible bounds on speed and steering angle.
Similar to the METADRIVE environments, we define CBFs based distance/proximity in (34). In our formulation, we do not employ a separate heading CLF; instead, we regulate agent behavior using only the velocity CLF by aligning the desired velocity vector with the desired heading direction. Specifically, given a desired heading direction ddes, we compute a target velocity vector Uas Umag cos (cos(dess, (fdes) sin [TTes)]T where Umag is task- sisk-specffic magnitude (e.g., maximum safe speed or adaptive goal-directed speed). This Vaes is then used in the velocity CLF described below to guide both the speed and heading behavior of the agent.
For the Target and Spread environments, where the agent's longitudinal and lateral velocities vy. b explicit components of the state, we define the Control Lyapunov Function (CLF) to track
32

---

desired velocity vector ddes = dee,es Uyydes] ]" IR2 as:
v(e 1 (00) Us,des) (o) g Uyydes)
Given the double integrator dynamics, where the control inputs al and aj directly affect the velocity via of = a and oy = ayy the time derivative of the CLF becomes:
V = (v U2,des) + (u. Uyydes)
To enforce convergence, include the following CLF constraint in the QP:
(v² Uz (vy Uyydes) )ay, S -a ((v² U2,des) 2 (1t) dyydes)
where a > O is tunable rate parameter.
In the Target Bicycle environment, each agent's velocity is represented as scalar speed of IR, and the longitudinal acceleration a² R serves as the control input directly affecting the speed via of a'. To drive y' desired speed Udes IR, we define the CLF as:
v(v') -1(00 Udes)
The time derivative this CLF, using of a², is:
V= (v' )a"
To enforce convergence of the velocity the target, the following CLF constraint imposed in the QP:
(v' - Vdes )a' s -a(v" -des)
where a 0 is tunable parameter determining the exponential rate of convergence.
Skills Description. We define four interpretable low-level skills for each agent in the LiDAR suite environments, which are: speeding up. ii. slowing down, iii. turning left, and iv. turning right.
Speeding Up. This skill allows an agent to increase its translational velocity to navigate more efficiently through open space or move rapidly towards its assigned goal. Let the velocity at the start of the skill be vo E IR2 the skill is considered complete once the velocity reaches vy = 0o + Av, where Au IR, with | > 0. The CLF constraint associated with velocity tracking is enforced by setting the desired velocity Udes Vo + Av in the QP-based low-level control policy. This skill facilitates faster progression low-density scenarios while maintaining safety guarantees via CBFs.
Slowing Down. This skill enables the agent to reduce its speed when approaching dynamic obstacles, other agents, goal regions where precise navigation is required. Let the initial velocity be e IR2 and define the skill termination condition as reaching Vy = 0o Av, where A e R22 and | | > 0. To enforce this behavior, the velocity CLF is specified using Uaes uy Av. This skill is critical for smooth and safe deceleration, reducing the risk of collisions and improving maneuverability near goals.
Turning Left. This skill enables the agent to redirect its heading counterclockwise to avoid obstacles or navigate toward an alternate subgoal. Let the initial heading be 0g. and the desired heading be sas = 00+ + A6, where AO > O. Instead of using a heading CLF, we convert the desired heading into a velocity vector using fixed target magnitude Umag. and set Váss Umagl cos( (des), sin (dcs)) This desired velocity then tracked via the standard velocity CLF in the QP. The turning-left skill is useful for curvature adjustment in cluttered or multi-agent scenes.
"mag[cos((A)
Turning Right. This skill enables clockwise heading correction using the same velocity CLF formula- tion. Given initial heading 6g and desired adjustment A > O, the target heading is fae 8, - A6, which is converted to velocity vector as Uitss Umay| [cos( [ (0aes) sin(Oass)) (T The agent's 's translational motion is then regulated by the velocity CLF, enforcing convergence to the desired heading direction through velocity tracking. This skill particularly effective in obstacle-rich or conflicting zones where sharp directional changes required.
33

---

Figure 10: Visualization of agents' trajectories under the trained model: for each vehicle, its ten most recent positions shown in unique hue, with brightness fading from light (earlier points) to dark (later points) to indicate temporal progression.
Figure 11: Visualization of skills under the trained model. From left right: (a) acceleration
(speed-up), (b) yielding (slow-down), and (c) lane changing
A.5 Additional Results
Here, we present results to further demonstrate the merit our proposed algorithm. Specifically, we examine the trajectories in detail to analyze the distinct skills exhibited by the agents, offering insight into how our method sustains an extremely high success rate while also outperforming the baselines in terms of energy efficiency and episode duration. To this end, employ visualization scheme in which each agent's 's ten most recent positions are depicted a distinct color, with brightness gradually diminishing from light (earlier points) dark (later points) convey temporal progression. An example of this visualization is depicted in Figure TO By looking more closely, we can identify some of the skills that the agents exhibit during the episode. For instance, agents approach their destinations (i.e., exit points of the environment). they execute speed up skill only when safety constraints are satisfied, thereby enhancing road throughput and reducing average travel time. Several such examples are shown in Figure l.p. Yielding (i.e., controlled slow down/deceleration) is another common skill agents employ maintain safe distance from other vehicles. By anticipating the need to yield, agents avoid abrupt braking, which both enhances safety and reduces overall episode energy consumption. Several such examples are shown in Figure [pp. Another skill exhibited by agents is lane changing: when executed safely at appropriate locations, it enhances road throughput and consequently reduces the average episode duration. example of such lane changing depicted in ITe where the agent with brown color is changing its lane creating safe gap into which the grey agent then merges onto the main road.
34


Reviews:
[{"review_id": "lXYhTtdKjz", "reviewer": "Reviewer_17QD", "summary": "The paper introduces HMARL-CBF, a two-level framework for safe multi-agent reinforcement learning in partially-observable settings with hard safety constraints. At the high level a centralized-training decentralized-execution (CTDE) policy selects a joint skill for all agents. These skills are temporally-extended options that encode cooperative behaviour. At the low level each agent executes its chosen skill through a quadratic-program (QP) controller that embeds control-barrier-function (CBF) constraints, guaranteeing point-wise-in-time safety. Experiments in the MetaDrive traffic simulator show near-perfect collision-free success rates (≥ 95 %) and substantially shorter episodes than baselines like IPPO, MFPO, CoPo and a flat MAPPO variant.", "questions": "- Did you give baselines access to the same pre-trained skill library (e.g. by augmenting the action space with them) and CBF safety filter? If not, please justify. A MAPPO or IPPO baseline equipped with the identical skill set would isolate the contribution of the hierarchical credit assignment itself. This my main concern from this work.\n\n- Section 5.2 relies on known dynamics for the QP. What happens when only an approximate or learned model is available? Please include experiments with modelling errors if possible.\n\n- Five seeds are unusually low for high-variance MARL benchmarks. Please report 10–20 seeds with 95 % confidence intervals, otherwise this severely impacts the significance of the claims\n\n- Please give intuitive/informal explanations in Section 3.2. This will greatly improve accessibility for the broader NeurIPS audience.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n### Quality\nThe paper presents a technically sophisticated integration of hierarchical reinforcement learning with control-barrier-function (CBF) safety filtering. The methodology is sound: a centralized-training, decentralized-execution scheme selects joint skills, while individual agents execute them through CBF-constrained quadratic-program controllers. Empirical results in the continuous-control MetaDrive suite show dramatic gains in collision-free success rates and episode efficiency, and the authors include ablations that dissect the impact of hierarchy and safety filtering.\n\n### Clarity\nThe high-level idea and algorithmic pipeline are conveyed effectively through Figure 1 and Section 5, giving readers a coherent narrative from problem statement to implementation. An extensive appendix provides proofs and additional details, which helps knowledgeable readers follow the derivations and design choices.\n\n### Significance\nAddressing hard safety in cooperative multi-agent RL is an important open problem with clear relevance to autonomous driving and swarm robotics. Demonstrating near-perfect safety without sacrificing task efficiency could influence future research and practical deployments of MARL systems subject to stringent safety constraints.\n\n### Originality\nWhile action-space augmentation via options and CBF-based filtering are themselves established, the paper’s specific decomposition—first selecting a joint high-level skill and then letting agents independently apply CBF-filtered low-level control—appears novel. This bridges two previously separate literatures (hierarchical MARL and safe control) in a way that could inspire further hybrid approaches.\n\n\n# Weaknesses\n\n### Quality\nThe safety guarantees hinge on knowing affine system dynamics (Eq. 3); experiments likewise assume exact vehicle models, leaving robustness to modelling error unexplored. Pre-defined skills are hand-crafted and provided only to the proposed method, so performance gains may stem from privileged prior knowledge. Key baselines such as MAPPO augmented with identical skills and CBF filters are missing. Comparisons are averaged over only five seeds, and no statistical significance tests are reported.\n\n### Clarity\nDespite a clear high-level story, numerous notation and referencing issues impede readability. Symbols like $\\mathcal{U}$ in Eq. 4 are undefined; Eq. (17)/(18) are cited but absent; time-index notation is inconsistent; acronyms such as QP and LQR are never introduced or done after usage; and Section 3.2 offers little intuition for (HO)CBFs. Typos and un-hyperlinked citations and references further detract from polish. Numbers are also generally floating without being specific on whether they correspond to equations/sections/etc. \n\nSome of the notations in Section 4 are also confusing. E.g. The joint state space $\\mathcal{S}$ was defined before in line 87 as just being the joint agent states $\\mathcal{S}^i$. Then in line 127 it is defined as the joint agent states including the environment states $\\mathcal{S}^e$. Hence it is now unclear what $\\mathcal{S}^i \\subseteq \\mathcal{O}^i$ means (line 137) when the authors have that $\\mathcal{O}^i \\subseteq \\mathcal{S}$. Is $\\mathcal{S}^i$ now defined differently? What does $s^i \\in \\mathcal{O}^i$ mean? Are observations and states now the same? Is $b$ a function of observation, a single agent state, the agent joint states, or the joint states with the environment states (since sometimes it is $b(s)$, $b(o)$, $b(s^i,s^j)$)? In general the precise mathematical formulation in this section is confusing and unclear.\n\n### Significance\nThe evaluation is confined to MetaDrive traffic scenarios (and toy navigation tasks in the appendix). Generalisation to higher-dimensional platforms (e.g., MuJoCo robots like the Ant) or to settings with learned dynamics is not demonstrated. Until such breadth is shown (or at least argued theoretically) the broader impact on real-world applications remains uncertain.\n\n### Originality\nBecause both option-based action-space augmentation and CBF safety filtering have established precedents, the contribution’s novelty rests largely on their particular combination and training recipe. The paper could better differentiate itself from prior safe-HRL works to underscore what is truly unique.", "was_revised": true, "final_justification": "The authors really took some time and effort to properly address my concerns and provide a very detailed response. I am happy with their clarification and happy to increase my score to an accept accordingly.", "rebuttal": "# **Response to reviewer 17QD:**\n\nWe thank the reviewer for acknowledging our method as a sound and technically sophisticated integration of hierarchical RL with CBF safety filtering and our significant experimental results which show dramatic gains in collision-free success rates and episode efficiency. We address the issues pointed out by the reviewer below:\n\n## Summary\n\nIn brief, we address the following:\n\n- Provide additional experiments to compare our method with MAPPO and IPPO with skills and CBF filters.\n- Provide additional experiments to study the influence of modelling errors.\n- Provide additional experimental results with more seeds.\n- Provide more intuition about the use of CBFs. \n- Refine the notations. \n- Highlight our unique contributions.\n\n## Details\n\n>**Q1**: Did you give baselines access to the same pre-trained skill library (e.g. by augmenting the action space with them) and CBF safety filter? Comparison with MAPPO and IPPO with skills and the CBF safety filter.\n\nIn HMARL-CBF each skill is a stochastic, constrained optimal control problem; high-level coordination and low-level skills are learned jointly, not pre-trained. Because MAPPO and IPPO are flat, adding variable-length skills requires inserting a policy hierarchy.  Besides low level skills are learnt to optimize the entire trajectory as opposed to the trajectory corresponding to the skill in order to improve the overall performance. Our skill formulation makes them interpretable, provably safe, and transferable to related environments—an explicit contribution of our work. In Appendix A.2.3, Table 5, we show that skills learned this way are transferrable across similar environments. \n\nSection 6.4 Table 2, presents the ablation “Hierarchical MARL with parameterized LQR‑based skills,” in which skills are incorporated into MAPPO by altering the base algorithm. Each skill is formulated as a parametric LQR problem whose objective steers the agent towards the skill’s termination state. This ablation removes the safety filter from HMARL‑CBF. We also implement MAPPO and IPPO augmented with CBF filters in the bottleneck environment. The MAPPO-LQR approach only adds safety violation penalty to the high and low level rewards which fails as the LQR based skills do not consider the safety constraints. Despite incorporating safety filters both MAPPO and IPPO fails to learn the policies to complete the task which is consistent with our flat policy ablation results in Table 2. \n\n|Method|Success Rate|Avg. Travel Time|Training Steps|\n|------|-------------:|-----------------:|-------------:|\n|**HMARL-CBF**|$0.99\\pm0.00$|$214.16\\pm7.37$|1M|\n|MAPPO+Parametric LQR skills|$0.00\\pm0.00$|--|1M|\n|MAPPO+CBF filter (no skills)|$0.00\\pm0.00$|--|1M|\n|IPPO+CBF filter (no skills)|$0.00\\pm0.00$|--|1M|\n|MAPPO+Parametric LQR skills+CBF filter|$0.00\\pm0.00$|--|1M|\n\n---\n\n> **Q2**: Section 5.2 relies on known dynamics for the QP. What happens when only an approximate or learned model is available? Please include experiments with modelling errors if possible.\n\nGreat question! In fact, we require only an **approximate** model of the system for computing the time derivative that appears in the CBF constraints. In our experiments, we employ the standard kinematic bicycle model as an approximation for the MetaDrive environment, which is built on the PyBullet physics engine which simulates the vehicle dynamics. The constraints on derivatives itself adds robustness to model error, and the low-level skill policies further adapt to residual uncertainty during training by learning policies to execute the skill safely. When no simple model is available, a differentiable surrogate can be learned and used in the same way. To study more about the influence of modelling errors, we provide **an additional experiment** in the bottleneck environment with introducing up to 20% noise in state observations. The results are summarized in the table below. As can be seen, our proposed approach can learn safe policies in the presence of uncertainties.\n\n| Environment   | Success Rate | Training Steps |\n|---------------|-------------:|---------------:|\n| **Bottleneck**| 0.99 ± 0.03  | 300K |\n\n\n---\n\n> **Q3**: Five seeds are unusually low for high-variance MARL benchmarks. Please report 10–20 seeds with 95 % confidence intervals, otherwise this severely impacts the significance of the claims.\n\nWe ran the experiments with 10 seeds for 300k iterations and present results with 95% confidence intervals. The presented results are consistent with Table 1 in the results section.\n\n| Environment | Success Rate | Avg. Travel Time | Avg. Energy |\n|-------------|---------------:|-------------------:|--------------:|\n| **Merging** | 0.99 ± 0.01 | 155.48 ± 8.32 | 13.01 ± 2.32 |\n| **Bottleneck** | 0.99 ± 0.03 | 232.16 ± 41.19 | 7.46 ± 0.23 |\n| **Roundabout** | 0.99 ± 0.02 | 405 ± 33.34 | 10.00 ± 0.70 |\n| **Tollgate** | 0.99 ± 0.02 | 373 ± 26.36 | 8.79 ± 0.28 |\n| **Intersection** | 0.979 ± 0.03 | 317 ± 47.39 | 5.35 ± 0.53 |\n\n---\n\n> **Q4**: Please give intuitive/informal explanations in Section 3.2.\n\nThanks for the suggestion. We will add the following clarification at the beginning of Section 3.2: (HO)CBFs are a class of CBFs that offer sufficient conditions for ensuring safety for systems with higher relative degree. For example, for a point mass with acceleration control, the system's relative degree is 2, and only using a position-based CBF cannot ensure safety because the control term does not appear in the CBF constraint. Therefore, we need to also consider velocity in the CBF, making this CBF a HOCBF. \n\n---\n##Weaknesses##\n---\n\n**Quality**\n\n> **1**. The requirement of knowing affine system dynamics (Eq. 3). and the robustness to modelling error. \n\nPlease refer to our reply to **Q2**. In addition, for the study of the robustness to modelling error, the broader robust‑CBF theory provides means for formal treatment of bounded model errors. This is out of the scope of this paper and we leave it to future work.\n\n---\n\n> **2**. Hand-crafted pre-defined skills.\nPlease refer to our reply to **Q1**. Also, our method requires high level task prior to choose an initial skill set, and imposes no restriction on the cardinality or diversity of skills set. Thus, any set of skills can be chosen based on high level task related knowledge.\n\n---\n\n> **3**. Key baselines such as MAPPO augmented with identical skills and CBF filters are missing.\n\nPlease refer to our reply to **Q1**.\n\n---\n\n> **4**. Comparisons are averaged over only five seeds, and no statistical significance tests are reported.\n\nPlease refer to our reply to **Q3**.\n\n---\n\n**Clarity** \n\n>**1**. Symbols, notations, equation numbers, acronyms, typos and references revision.\n\nThank you for pointing these out. We will address these comments in the final submission.\n\n---\n\n> **2**. Section 3.2 offers little intuition for (HO)CBFs.\n\nPlease refer to our reply to **Q4**.\n\n---\n\n> **3**. Clarification on joint state space and agent state and observation space notations.\n\nThe joint state space is defined as $\\mathcal{S} = \\mathcal{S}^{e} \\times \\prod_{i\\in\\mathcal{N}} \\mathcal{S}^{i}$ where $\\mathcal{S}^{e}$ denotes the environment state space and $\\mathcal{S}^{i}$ the state space of agent $i$. Each agent $i$ receives an observation $o^{i} \\in \\mathcal{O}^{i} \\subseteq \\mathcal{S}$, which includes its own state $s^{i}$ together with those portions of other agents’ or environmental states that are observable to it. Safety is encoded by a control‑barrier function $b : \\mathcal{O}^{i} \\to \\mathbb{R}$; equivalently, one may write $b(s^{i}, s^{j})$ to emphasise the dependence on agent $i$’s state and the observed state $s^{j}$ of another entity. The safety constraint is expressed in the form $b(o^{i}) \\ge 0$.\n\n---\n\n**Significance**\n\n>**1**. Generalization to higher-dimensional platforms (e.g., MuJoCo robots like the Ant) or to settings with learned dynamics is not demonstrated. Until such breadth is shown (or at least argued theoretically) the broader impact on real-world applications remains uncertain.\n\nWe require only an approximate dynamics model for the CBF constraints. Emerging differentiable‑simulation frameworks (e.g., JAX + Diffrax, Brax, MJX) can learn such surrogates online, eliminating the manual dynamics‑derivation burden in future. Our experiments on MetaDrive offers manifold benefits, namely: (i) simulate large number of physical agents, (ii) incorporates hard safety‑critical interactions between agents. By contrast, standard MuJoCo multi‑agent set‑ups—Ant, Humanoid, etc.—consist of a single body whose limbs are treated as pseudo‑agents or whose policies are merely distributed; and offers limited scalability of the number of agents. Our traffic scenarios, on the other hand, allows for validating the scalability of our proposed approach, as demonstrated from results specifically table 6.\n\n---\n\n**Originality**: \n\n> The paper could better differentiate itself from prior safe-HRL works to underscore what is truly unique.\n\nBased on our survey, there are no prior works on Safe Hierarchical Multi-Agent RL (Safe-HMARL). The prior works are primarily focused on HRL and HMARL as stated in the paper. Our work is the first literature on tackling safety using Hierarchical RL for multi-agent systems. There are prior works on hierarchical control [1], and hierarchical policy refinement using adaptive chance constraints for single-agent systems [2] which is distinctly different from Safe-HMARL.\n\n---\n\n## References\n[1] L. Zhang, G. Cheng and W. Wang, \"Safe Reinforcement Learning-Based Adaptive Hierarchical Control Approach for Virtual Coupling Trains,\" (ICNSC), 2024.\n\n[2] Z. Chen et al., \"Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards,\" 2024 IEEE IROS, UAE, 2024, pp. 12378-12385."}, {"review_id": "IS4EMZnIIV", "reviewer": "Reviewer_1MFt", "summary": "This paper introduces a hierarchical multi-agent reinforcement learning framework for learning safe policies based on control barrier functions. The control problem is formulated as a partially-observable multi-agent MDP with pointwise-in-time hard constraints based on barrier functions. The proposed framework learns high-level control policies that select skills, considering performance, while the low-level control policies implement skills by selecting actions, considering strict safety constraints (policy learning is based on existing multi- and single-agent algorithms, such as MAPPO and PPO). The proposed framework is evaluated in the MetaDrive simulator with various scenarios, showing that compared to multi-agent baselines the proposed framework provides better satisfaction of safety requirements.", "questions": "* Where do skills come from?\n* Are the baseline methods SOTA in safe RL?", "limitations": "Yes.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Safety-critical multi-agent control is an important problem.\n* The proposed hierarchical decomposition into high-level and safe low-level policies is promising.\n\nWeaknesses:\n* Some aspects of the proposed framework are not clear. Where do the skills come from? The high-level policy seems to learn to select from a given set of skills, while the low-level policies seem to be learned for specific skill. Where do these skills come from in the first place?\n* The baseline approaches in the numerical experiments do not seem to be designed for safety (they seem to be generic multi-agent RL approaches). Do these really represent the SOTA in safe multi-agent RL?\n* \"While CMDPs ensure safety over trajectories, they do not consider pointwise time constraints critical for safety-critical systems.\"\nIs this really a limitation? One could define (1) a cost function that is non-zero for pointwise time-constraint violation and (2) cost constraint that enforces zero expected cost. Would that not capture pointwise constraints?\n* The description of termination strategies on lines 94 to 98 is rather informal, and it is not clear how the $\\tau_{continue}$ strategy is implemented in the model.\n* The notation on line 133 seems to be inconsistent: barrier function $b$ first takes agent-specific states as input, but then the specification of its domain includes the entire state space $\\mathcal{S}$.\n* The definition of the observation space on lines 137 and 138 is strange: the observation space is defined as a superset of the state space, but the environment is partially observable, which should be modeled as some states being indistinguishable to the agent, implying that the observation space is a subset of the state space. But this of course would prevent applying the barrier function $b$ to the observation. Can the barrier function really be evaluated on observations instead of states? This seems to contradict partial observability if the barrier function are general functions of state.\n* On lines 149 to 152, the discussion emphasizes relaxing the assumption of CMDP that policies depend on the joint state. But this does not seem significant, it is just a simple matter of definition.\n* The definition of the indicator function at the bottom of page 5 has some issues. The clause \"for some $i$\" does not make sense since $i$ is given as the running variable of the first summation of Equation (9), hence it makes not sense to refer to \"some $i$\" (also, within the definition of the indicator function the domain of $i$ would be unclear). Footnote 1 is unclear since it suggests that the indicator function would output some large value $M$, but according to the definition, it outputs 0 or 1.\n* Calling the proposed framework \"Safety Guaranteed\" might be misleading since safety is not guaranteed, it is achieved only high probability.\n* It may help readability if the distribution of random variable in the subscript of expectations is specified, e.g., in Equation (1), clarify that $z \\sim \\pi$ and $k, s' \\sim P$.", "was_revised": true, "final_justification": "The paper makes an interesting contribution to safe multi-agent reinforcement learning, and the proposed framework appears to be promising in terms of numerical results. However, the presentation of the framework is not very clear (as other reviewers also noted), and the paper would benefit from a major revision, which should also include the additional results presented in the rebuttals.", "rebuttal": "# **Response to reviewer 1MFt:**\nWe thank the reviewer for their positive comments that the addressed \"safety-critical multi agent control is an important problem\" and that our proposed \"hierarchical approach is promising\".  We also extend our gratitude for the constructive criticism. We address the issues pointed out by the reviewer below.\n\n## Summary ##\n\nThe summary of the points addressed in the reponse is provided below:\n\n- Provide explaination of the .\n- Provide additional baseline comparisons against existing safe MARL approaches\n- Provide clarifications for the reviewers comments.\n\n## Details: ##\n\n**Questions:**\n> 1. Where do skills come from?\n\nThe skills are defined from high level task related prior knowledge. However, our approach poses no limitation on the cardinality of the skill set. In our proposed HRL approach,  the low level skills are not pretrained. We define the skills as stochastic constrained optimal control problem. And, the high level coordination and the low level skills both are learned during training jointly. This allows us to define skills that are interpretable and transferable to other similar environments as evident from the generalization results in Appendix section A.2.3, Table 5. Based on the definition of the skill set, our method learns the policy over the skill sets to accomplish the task cooperatively and safely in multi-agent environment. \n\n> 2. Are the baseline methods SOTA in safe RL?\n\nBased on our survey, our paper is the first work on safe multi-agent HRL. Hence, we compared our method against the state of the art baselines in MetaDrive environment. The provided results highlight the merit of our proposed approach. Regarding safe MARL baseline: we ran MACPO [1] (Multi agent Constrained Policy Optimization) and MAPPO Langrangian [2] algorithms. The experiments were run for the bottleneck environment for 2 seeds and the results are summarized in the table below. The results highlight the sample efficiency of our proposed method as well as our safe skills based HRL approach achieves the highest success rate compared to the baselines). \n|Method|Success Rate|Avg. Travel Time (Success rate weighted)|Training Steps|\n|------|-------------:|-----------------:|-------------:|\n|**HMARL-CBF**|$0.99\\pm0.00$|$241.16\\pm7.37$|$300$K|\n|**MAPPO Lagr** (safety cost $-0.5$)  |  $0.1 \\pm 0.05$  | $-$     |  $10$M   |\n|**MAPPO Lagr** (cost threshold: $-2$)  |  $0.22 \\pm 0.02$  | $1625.47 \\pm 400$     |  $10$M   |\n|**MACPO** (cost threshold:  $-0.5$) |   $0.15 \\pm 0.03$  |  $-$   | $10$M   |\n|**MACPO** (cost threshold: $-2$) |   $0.29 \\pm 0.02$  |  $621.51 \\pm 91.31$   | $10$M   |\n\n**Weaknesses:**\n> 1. Some aspects of the proposed framework are not clear. Where do the skills come from? The high-level policy seems to learn to select from a given set of skills, while the low-level policies seem to be learned for specific skill. Where do these skills come from in the first place\n\nWe use only high-level task-related prior knowledge to define the skill set. Such prior knowledge is generally available. For example, in our MetaDrive driving environment the chosen skills are defined intuitively which include left and right lane changing, cruising, throttling and slowing down. We do not enforce any limit on the number of skills, nor, the diversity of the skills set. Finally, we jointly learn both the high- and low-level policies. \n\n\n> 2. The baseline approaches in the numerical experiments do not seem to be designed for safety (they seem to be generic multi-agent RL approaches). Do these really represent the SOTA in safe multi-agent RL?\n\nPlease refer to **Q2** for the response.\n\n\n> 3. \"While CMDPs ensure safety over trajectories, they do not consider pointwise time constraints critical for safety-critical systems.\" Is this really a limitation? One could define (1) a cost function that is non-zero for pointwise time-constraint violation and (2) cost constraint that enforces zero expected cost. Would that not capture pointwise constraints?\n\nThanks for making this great point. CMDPs consider discounted sum of costs in the infinite horizon problem scenario and applies a cost threshold on the expected sum of discounted cost. Firstly, taking the discounted sum poses less emphasis on long time cost violation. Besides that, as CMDP enforces a safe cost threshold on the expectation of trajectories, therefore, it places less emphasis on cost violation of trajectories with lower probability. Finally, this way of cost definition is susceptible to training instability particularly for primal dual methods. \n\n> 4. The description of termination strategies on lines 94 to 98 is rather informal, and it is not clear how the $\\tau_{continuous}$ strategy is implemented in the model.\n\nThe $\\tau_{continue}$ is an asynchronous skill selection scheme. Under this scheme, each agent selects and executes the skill until completion. Following that the agent picks a new skill and this process is repeated by every agent in the multi-agent setting. Thus, the decision epoch is defined as the time when any of the joint skills of the agents terminate in our setting. \n\n> 5. The notation on line 133 seems to be inconsistent: barrier function $b$ first takes agent-specific states as input, but then the specification of its domain includes the entire state space $\\mathcal{S}$. The definition of the observation space on lines 137 and 138 is strange: the observation space is defined as a superset of the state space, but the environment is partially observable, which should be modeled as some states being indistinguishable to the agent, implying that the observation space is a subset of the state space. But this of course would prevent applying the barrier function $b$ to the observation. Can the barrier function really be evaluated on observations instead of states? This seems to contradict partial observability if the barrier function are general functions of state.\n\nThank you for pointing this out. We have revised it in the paper. The joint state space is defined as $\\mathcal{S} = \\mathcal{S}^{e} \\times \\prod_{i\\in\\mathcal{N}} \\mathcal{S}^{i}$ where $\\mathcal{S}^{e}$ denotes the environment state space and $\\mathcal{S}^{i}$ the state space of agent $i$. Each agent $i$ receives an observation $o^{i} \\in \\mathcal{O}^{i} \\subseteq \\mathcal{S}$, which includes its own state $s^{i}$ together with those portions of other agents’ or environmental states that are observable to it. Safety is encoded by a control‑barrier function $b : \\mathcal{O}^{i} \\to \\mathbb{R}$; equivalently, one may write $b(s^{i}, s^{j})$ to emphasize the dependence on agent $i$’s state and the observed state $s^{j}$ of another entity. The safety constraint is expressed in the form $b(o^{i}) \\ge 0$. \n\n\n> 7. On lines 149 to 152, the discussion emphasizes relaxing the assumption of CMDP that policies depend on the joint state. But this does not seem significant, it is just a simple matter of definition.\n\nIf the constraints depend on the joint state then during execution the agents have to receive information of all other agents that are not observable to it. Executing such a policy in a real multi-agent system demands a central hub (or all-to-all communication) at each control step. \n\n> 8. The definition of the indicator function at the bottom of page 5 has some issues. The clause \"for some $i$\" does not make sense since $i$ is given as the running variable of the first summation of Equation (9), hence it makes not sense to refer to \"some $i$\" (also, within the definition of the indicator function the domain of $i$ would be unclear). Footnote 1 is unclear since it suggests that the indicator function would output some large value $M$, but according to the definition, it outputs 0 or 1.\n\nThank you for pointing this out. We will remove this footnote from the final submission.\n\n> 9. Calling the proposed framework \"Safety Guaranteed\" might be misleading since safety is not guaranteed, it is achieved only high probability.\n\nWe will revise the introduction and explicitly mention the conditions under which our proposed framework can provide safety guarantee through safe execution of the low level skills.\n\n> 10. It may help readability if the distribution of random variable in the subscript of expectations is specified, e.g., in Equation (1), clarify that $z\\sim \\pi$ and $k, s' \\sim P$.\n\nThank you for sharing this feedback. We will incorporate this comment in the final submission.\n\n---\n\n## References\n\n[1] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, R. Chen, Z. Wang, Z. Tian, J. Wang, A. Knoll, Y. Yang, \"Multi-Agent Constrained Policy Optimisation,\" arXiv preprint arXiv:2110.02793, 2022."}, {"review_id": "lMWhYaetnk", "reviewer": "Reviewer_Ysva", "summary": "This paper proposes HMARL-CBF, a hierarchical framework for cooperative safety-MARL. The approach decomposes the policy into a high-level policy and a low-level policy. The high-level policy is a policy that selects the skill to use, and the low-level policy is a policy that selects the low-level action with safety constraints conditioned on the skill. The high-level policy is trained by standard MARL methods like MAPPO or QMIX, with the environment reward with a safety penalty. The low-level policy is trained by a CBF-based method with a hand-crafted intrinsic reward. The authors evaluate the proposed method on MetaDrive, and the results show that the proposed method can achieve significant better performance than the baseline methods.", "questions": "- It seems that some baselines are not included in Related Work and Experiments sections. Can you provide more comparison with the different methods, including [1,2,3]?\n- Can you provide more experiments on different tasks, including [4] and https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_multi_agent.html.\n- HMARL-CBF employs a lot of task-specific prior knowledge or ad hoc design, such as the known dynamics of the environment, the design of intrinsic rewards and predefined skill sets. Could you please list in detail the task-related prior knowledge used in this paper? Would this make the comparison with the baseline unfair?\n- Could you visualize or at least describe the low-level skills, like what does the skills look like, how long does the skill last, etc.? This would be very helpful for understanding the method presented in this paper.\n\n[1] Berducci L, Yang S, Mangharam R, et al. Learning adaptive safety for multi-agent systems[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 2859-2865.\n\n[2] Zhang S, So O, Black M, et al. Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control[J]. arXiv preprint arXiv:2502.03640, 2025.\n\n[3] Gu S, Kuba J G, Wen M, et al. Multi-agent constrained policy optimisation[J]. arXiv preprint arXiv:2110.02793, 2021.\n\n[4] O’Kelly, M., Zheng, H., Karthik, D. &amp; Mangharam, R.. (2020). F1TENTH: An Open-source Evaluation Environment for Continuous Control and Reinforcement Learning. <i>Proceedings of the NeurIPS 2019 Competition and Demonstration Track</i>, in <i>Proceedings of Machine Learning Research</i> 123:77-89 Available from https://proceedings.mlr.press/v123/o-kelly20a.html.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The performance of the proposed method is impressive on MetaDrive.\n- Hierarchical policy is a good approach to handle the large and complex environment of MARL.\n\n### Weakness\n- The writing is not clear and has some typos, e.g.\n    - line 74: [46] is not a multi-agent RL method.\n    - line 138: $s^i, s^j$ seem to be redundant.\n    - line 140: \"hus\" should be \"Thus\".\n    - line 165: \"An skill\" should be \"A skill\".\n    - eq (8), line 182, eq (10)...: The condition should be put outside the reward function and inside the expectation, or the definition of the reward function should be changed.\n    - eq (9): $p_j^i$ is not defined.\n    - eq (9): \"for some $i$\" should be removed.\n    - footnote 1 on page 5: $\\mathbb {I}(x)$ is a binary function by definition, while in this footnote it is claimed to be a large number $M$, which is unclear.\n    - eq (14): Notations like $\\Omega, \\zeta$ are not defined.\n- The novelty to use CBF-based method to train the low-level policy in MARL is limited.\n- The approach seem to require a lot of artificial design, e.g. the hand-crafted intrinsic reward, the predefined skill set, the ad hoc Lyapunov function, etc.\n- The experimental results are impressive, but the chosen tasks are relatively few. Also, some baseline methods are not included in the comparison.", "was_revised": true, "final_justification": "The authors’ response has partially addressed my concerns. However, I still believe that:\n\n1. The generality of the task-specific designs in this paper (e.g., predefined skills, intrinsic rewards, Lyapunov functions) remains questionable. These ad hoc components seem difficult to generalize to new environments. The authors should devote more space in the main text to clearly describe the assumptions and task-specific designs they adopt, and discuss their applicability and scalability.\n\n2. As some other reviewers have also noted, the presentation of the paper could be improved.\n\nTherefore, I have decided to keep my rating 3 (borderline rejection).\n\nThe method shows promising results on the chosen benchmarks, which suggests its potential. However, I encourage the authors to conduct more in-depth studies and experiments to gain a deeper understanding of why the method works and the scope of its applicability.", "rebuttal": "# **Response to reviewer Ysva:**\nWe thank the reviewer for their positive comments and highlighting that \"hierarchical policy is a good approach to handle the large and complex environment of MARL\" and the results of our \"proposed method is impressive\". We are thankful for the constructive criticism. We address the issues pointed out by the reviewer below.\n\n## Summary\n\nWe provide the summary of the comments addressed in the response below:\n\n- Provided further baseline comparisons against SOTA Safe MARL algorithms.\n- Provided clarification on our proposed Safe Hierarchical MARL framework.\n- Provided additional clarification on the implementation of our proposed algorithm.\n\n## Questions\n\n> **Q1**. It seems that some baselines are not included in Related Work and Experiments sections. Can you provide more comparison with the different methods, including [1,2,3]?\n\nThank you for the feedback. We will incorporate additional literature in the final submission. Based on our survey, our paper is the first work on safe multi-agent HRL. Hence, we compared our method against the state of the art baselines in MetaDrive environment. Our experiments on MetaDrive offers manifold benefits, namely: (i) simulate large number of physical agents, (ii) incorporate pointwise in time safety‑critical interactions between agents. We ran MACPO [3] (Multi agent Constrained Policy Optimization) and MAPPO Langrangian [3] algorithm for the bottleneck environment in MetaDrive. The results are summarized in the table below. \n|Method|Success Rate|Avg. Travel Time (Success rate weighted)|Training Steps|\n|------|-------------:|-----------------:|-------------:|\n|**HMARL-CBF**|$0.99\\pm0.00$|$241.16\\pm7.37$|$300$K|\n|**MAPPO Lagr** (safety cost $-0.5$)  |  $0.1 \\pm 0.05$  | $-$     |  $10$M   |\n|**MAPPO Lagr** (cost threshold: $-2$)  |  $0.22 \\pm 0.02$  | $1625.47 \\pm 400$     |  $10$M   |\n|**MACPO** (cost threshold:  $-0.5$) |   $0.15 \\pm 0.03$  |  $-$   | $10$M   |\n|**MACPO** (cost threshold: $-2$) |   $0.29 \\pm 0.02$  |  $621.51 \\pm 91.31$   | $10$M   |\n\nUnfortunately, MetaDrive environment is not built using JAX which is the library used for implementing the DGPPO (Discrete Graph CBF PPO) [2] algorithm. Therefore, the experiment runs very slowly. Each rollout of 1000 steps takes approximately 20 minutes on a server equipped with RTX 4050 GPU. Therefore, it would around a week to train the algorithm for 1M steps. Thus, we run the algorithm on a suite of lidar environments provided by the authors of DGPPO [2] along with MAPPO Lagrangian algorithm [3]. The number of agents The results are summarized in the table below.\n\n|Method|LidarTarget|LidarSpread|LidarLine|Training Steps|\n|------|-------------:|-----------------:|-------------:|--------------:|\n|**DGPPO**|  $0.98\\pm 0.00$ | $0.99 \\pm 0.00$ | $0.99 \\pm 0.00$ |1M|\n|**MAPPO Lagr_5**| $0.99 \\pm 0.00$ | $0.99 \\pm 0.00$ | $1 \\pm 0.00$|1M|\n|**HMARL-CBF**| $0.99 \\pm 0.00$ | $0.99 \\pm 0.00$ | $.98 \\pm 0.02$|300k|\n\nThe presented results highlight the sample efficiency provided by our hierarchical approach. Besides that, our approach achieves outperforms the baselines in MetaDrive bottleneck environment where the number of agents was set to 25, and similar results in the LiDAR environment suite where the agent count was set to the default value of 3. Finally, [1] proposes a method for safe single-agent RL based algorithms for solving stochastic games and validates it in multi-agent settings.\n\n---\n\n> **Q2**. Can you provide more experiments on different tasks, including [4] and Safety-gymnasium.\n\nPlease refer to **Q1** for our response. We have implemented additional baselines and also ran our algorithm on environments from the recent SOTA paper [2] to validate our proposed approach. \n\n---\n\n> **Q3**. Knowledge of dynamics.\n\nGreat question! In fact, we require only an **approximate** model of the system for computing the time derivative that appears in the CBF constraints. In our experiments, we employ the standard kinematic bicycle model as an approximation for the MetaDrive environment, which is built on the PyBullet physics engine which simulates the vehicle dynamics. The presented results in table 1 validates our approach. \n\n---\n\n> **Q4** Intrinsic rewards and predefined skill sets. \n\nThe skills are defined from high level task related prior knowledge. However, our approach poses no limitation on the cardinality of the skill set. In our proposed HRL approach,  the low level skills are not pretrained. We define the skills as stochastic constrained optimal control problem. The intrinsic reward is defined as the norm of the difference between the initiation state and skill termination state; thus can be generically extended to any skill. And, the high level coordination policy and the low level skills are both learned during training jointly. This allows us to define skills that are interpretable and transferable to other similar environments as evident from the generalization results in Appendix section A.2.3, Table 5.\n\n---\n\n> **Q4**. Could you visualize or at least describe the low-level skills, like what does the skills look like, how long does the skill last, etc.? This would be very helpful for understanding the method presented in this paper.\n\nWe define the skills as fixed final state stochastic OCP which have semantic interpretation. Hence, the length of the skills is a variable. For example, in MetaDrive environment we define the skills to be left and right lane changing, cruising, speeding up and slowing down. Each of these skills take different length of time to execute. Besides that the skills policy and therefore the length of the skill is also dependent on the behavior of the other agents observable to an agent. \n\n## Weakness\n> 1. Typos and notations.\n\nThank you for pointing this out. We will revise the typos and notational issues in the final submission.\n\n---\n\n> 2. The novelty to use CBF-based method to train the low-level policy in MARL is limited.\n\nWe would like to add that based on our survey, our paper is the first work on Safe Hierarchical MARL for multi-agent safety-critical systems. Our posed hierarchical CBF based safe skill approach achieves high success rate as evident from the evaluation and baseline comparison results in safety-critical systems.\n\n---\n\n> 3. The approach seem to require a lot of artificial design, e.g. the hand-crafted intrinsic reward, the predefined skill set, the ad hoc Lyapunov function, etc.\nAdd explanation\n\nPlease refer to **Q3** and **Q4** for the response.\n\n---\n\n> 4. The experimental results are impressive, but the chosen tasks are relatively few. Also, some baseline methods are not included in the comparison.\n\nPlease refer to **Q1** for the response.\n\n## References ##\n[1] Berducci L, Yang S, Mangharam R, et al. Learning adaptive safety for multi-agent systems[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 2859-2865.\n\n[2] Zhang S, So O, Black M, et al. Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control[J]. arXiv preprint arXiv:2502.03640, 2025.\n\n[3] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, R. Chen, Z. Wang, Z. Tian, J. Wang, A. Knoll, Y. Yang, \"Multi-Agent Constrained Policy Optimisation,\" arXiv preprint arXiv:2110.02793, 2022."}, {"review_id": "GCdaHGOQdn", "reviewer": "Reviewer_MQj4", "summary": "This paper focuses on the safety of multi-agent reinforcement learning (MARL), while it emphasizes that agents have to meet safety requirements at all times. Based on Control Barrier Functions, it proposes a hierarchical approach where the higher-level problem involves learning a joint policy over the skills and lower-level problem involves learning policies to execute skills. In complex environments with a large number of agents, the method manages to present good performance in experiments.", "questions": "1.As is mentioned in previous works (such as HATRPO and MAFOCOPS), the agents will affect with each other. Although the proposed method in this paper can deal with multiple agents, it seems to mainly consider each single agent. I’m curious whether this will lead to some instability during the training.\n2.As the method relies on the CBF and hierarchical skill learning, which I think both require some prior knowledge. From the paper, I think the authors are very familiar with the experiment environments. But if a new or very complex environment is encountered, how will the method perform?\n3.In the METADRIVE experiments, the chosen baselines do not seem to be algorithms related for safety constraints. How will other methods concerned with safety perform in the environment?", "limitations": "The authors claim to discuss the limitations in both the conclusion and appendix sections but I don't find the discussion. I think the authors should analysis the computation complexity as the proposed method is based on a hierarchical framework, which tends to be more complex. Also, how the method performs if a new and complicated environment is met should as be considered.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n  The paper further promotes the safety requirements and promotes corresponding solution, which demonstrates outstanding performance.\n  The proposed method owns good scalability and is able to deal with conditions with large amounts of agents.\n\nWeakness: \n  The paper is a little difficult to understand, for example, it refers to some equations that appear later or even in appendix, which makes the reading not smooth.\n  Similar to above, maybe the content is too substantial so that many processes that help understand the work is put in appendix, and the analysis of experiments is also too brief in the main body of the paper. Even the lidar environment is not mentioned in the main body. I think the structure of the article can be improved.\n  There are also some typos, such as line 140.", "was_revised": true, "final_justification": "The response to my review solves the majority of my questions. Considering the outstanding performance of the algorithm in environments with large amount of agents, I would like to raise my score.", "rebuttal": "# **Response to reviewer MQj4:**\nWe thank the reviewer for their positive comments, specifically commending that in complex environments with a large number of agents, our method manages to achieve good performance. We address the issues pointed out by the reviewer below:\n\n---\n\n**Questions**\n\n---\n\n> **1**. As is mentioned in previous works (such as HATRPO and MAFOCOPS), the agents will affect each other. Although the proposed method in this paper can deal with multiple agents, it seems to mainly consider each single agent. I’m curious whether this will lead to some instability during training.\n\nOur proposed hierarchical approach involves learning a joint coordination policy between the agents for the high level and the individual skills of the agents at the low level. Given that we are dealing with multi agent policy learning at the high level, we adopt the Centralized Training with Decentralized Execution (CTDE) framework for our high level policy learning. In this approach, agents utilize additional global state information during training to enable centralized learning and improve stability. At execution time, however, each agent’s policy operates solely based on its own local observations, ensuring fully decentralized decision-making. Therefore this schemes ensure the stability during the training. It is worth mentioning that, despite decentralized execution of our high-level policy, our objective is to learn a joint policy for all agents to optimize the overall task objective at the high level. Subsequently, the low level policy is learned conditioned on the high-level action. \n\n> **2**. As the method relies on the CBF and hierarchical skill learning, which I think both require some prior knowledge.\n\nFor enforcing our CBF constraints, we require only an approximate dynamics model. That said, emerging differentiable‑simulation frameworks (e.g., JAX + Diffrax, Brax, MJX) can learn such surrogates online, eliminating the manual dynamics‑derivation burden in the future. For hierarchical skill learning, we require only high-level task-related knowledge to define the skill set (specifically, the termination set of a skill). Such prior knowledge is generally available. For example, in our driving environment, the chosen skills are defined intuitively.\n \n\n> **3**. From the paper, I think the authors are very familiar with the experimental environments. But if a new or very complex environment is encountered, how will the method perform?\n\nIt is noteworthy that our choice of the MetaDrive simulation environment was motivated by two factors: first, it is a widely used benchmark for multi-agent reinforcement learning, enabling fair comparisons with prior work; second, it allows us to evaluate the effectiveness of our approach on multi-agent safety-critical tasks. In particular, MetaDrive offers such features in two ways: (i) pointwise safety constraints are directly relevant to driving tasks, as there are safety constraints that must be satisfied at all times (e.g., lane-boundary constraints), and (ii) simulation of a large number of agents to assess the scalability of our proposed method. We also implemented our algorithm in a LiDAR-based environment different from the driving task (Appendix A.2), and experimental results validate the performance of our approach. Finally,  if a new and very complex environment is encountered, we anticipate a small upfront effort of defining the skill set.\n\n\n\n> **4**. In the METADRIVE experiments, the chosen baselines do not seem to be algorithms related to safety constraints. How will other methods concerned with safety perform in the environment?\n\nOur work is the first safe hierarchical MARL method (vs. safe MARL, hierarchical MARL, and safe hierarchical control for multi-agent systems). Hence, we implemented the SOTA algorithms that have been applied to MetaDrive, except TRACO [1] (whose code is not publicly available), to validate our approach. We further implemented two SOTA safe MARL algorithms, namely MAPPO-Lagrangian and Multi-Agent Constrained Policy Optimization based on [2], to compare against our approach. Their results are summarized in the table below.\n\nThe experiments were run for the bottleneck environment for 2 seeds and the results are summarized in the table below. The results highlight the sample efficiency of our proposed method as well as our safe skills based HRL approach achieves the highest success rate compared to the baselines). \n|Method|Success Rate|Avg. Travel Time (Success rate weighted)|Training Steps|\n|------|-------------:|-----------------:|-------------:|\n|**HMARL-CBF**|$0.99\\pm0.00$|$241.16\\pm7.37$|$300$K|\n|**MAPPO Lagr** (safety cost $-0.5$)  |  $0.1 \\pm 0.05$  | $-$     |  $10$M   |\n|**MAPPO Lagr** (cost threshold: $-2$)  |  $0.22 \\pm 0.02$  | $1625.47 \\pm 400$     |  $10$M   |\n|**MACPO** (cost threshold:  $-0.5$) |   $0.15 \\pm 0.03$  |  $-$   | $10$M   |\n|**MACPO** (cost threshold: $-2$) |   $0.29 \\pm 0.02$  |  $621.51 \\pm 91.31$   | $10$M   |\n\n**Weaknesses**\n\n---\n\n> **1**. The paper is a little difficult to understand; for example, it refers to some equations that appear later or even in the appendix, which makes the reading not smooth. Similar to above, maybe the content is too substantial, so many processes that help understand the work are put in the appendix.\n\nWe will revise the paper adding details of the supplementary materials (mainly derivations and proofs) and explaining how they support the main results presented in the paper. Additionally, we will bring the definition of the state value, state-action value and advantage value functions given in equations (25) - (27) to explain how the policy gradient for the low level skills can be implemented to learn the safe skills policies.\n\n---\n\n> **2**. The analysis of experiments is also too brief in the main body of the paper. Even the LiDAR environment is not mentioned in the main body. I think the structure of the article can be improved. There are also some typos, such as line 140.\n\nUnfortunately, due to the strict page limit, we had to move some materials, including the LiDAR environment, to the appendix. However, we are happy to move it back to the main body of the paper if requested. In the final submission, we will also elaborate on the experimental results, including both the evaluation and ablation studies. In particular, we will highlight the distinctions between the environments and clarify the LiDAR-based results.\n\n\n## Limitations\n\n> **1**. The authors claim to discuss the limitations in both the conclusion and appendix sections, but I don't find the discussion. \n\nApologies for this. We will expand the discussion of limitations in the conclusion of the final submission, as summarized here. At present, we assume the existence of a predefined skill set for the agents. In the future, we will investigate methods to reduce reliance on task-specific prior knowledge. To this end, we plan to explore skill discovery, allowing agents to identify task-relevant skills autonomously. Alternatively, we will examine skill pruning, where we begin with a large set of general skills and prune it down to the minimal subset required for the task.\n\n> **2** I think the authors should analyze the computational complexity, as the proposed method is based on a hierarchical framework, which tends to be more complex. Also, how the method performs if a new and complicated environment is encountered should be considered.\n\nBecause we employ a CTDE scheme, the policies are executed fully decentralized in the real world. Each agent evaluates only two neural networks per control step—the high-level skill selector and the low-level skill policy—which is computationally lightweight on modern GPU-enabled devices. For example, on our workstation equipped with an RTX 4090, we can evaluate the learned policy on up to 45 vehicles in real time, as shown in the submitted videos included with the supplementary material.\n\n## References ##\n[1] Liu, W., Jing, W., Gao, L., Guo, K., Xu, G., & Liu, Y. (2023). TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning. 7th Annual Conference on Robot Learning (CoRL 2023).\n\n[2] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, R. Chen, Z. Wang, Z. Tian, J. Wang, A. Knoll, Y. Yang, \"Multi-Agent Constrained Policy Optimisation,\" arXiv preprint arXiv:2110.02793, 2022."}]

Author rebuttal:

Meta-review: This paper presents a method for safe hierarchical multi-agent reinforcement learning (MARL) using control barrier functions. The approach learns a higher-level policy over skills for coordination while the lower-level policy executes those skills in a safe manner with control barrier functions. 

The approach is novel and performs well in terms of safety and scalability in the experiments. 

There were concerns about the experiments, the presentation, and some assumptions (e.g., knowledge of skills and approximate dynamics) but these were mostly addressed during the author response and discussion. The paper should be updated to include this new material.
Decision: Accept (poster)