Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control
Authors: Songyuan Zhang
Oswin So
Mitchell Black
Chuchu Fan
Venue: ICLR 2025 Poster (2025)
Keywords: control barrier functions
multi-agent systems
black-box systems
partial observability
reinforcement learning
Primary area: reinforcement learning
TL;DR: We propose DGPPO for solving multi-agent safe optimal control problem with unknown discrete-time dynamics, partial observability, changing neighborhoods, and input constraints, without a known performant nominal policy.

Abstract: Control policies that can achieve high task performance and satisfy safety constraints are desirable for any system, including multi-agent systems (MAS). One promising technique for ensuring the safety of MAS is distributed control barrier functions (CBF). However, it is difficult to design distributed CBF-based policies for MAS that can tackle unknown discrete-time dynamics, partial observability, changing neighborhoods, and input constraints, especially when a distributed high-performance nominal policy that can achieve the task is unavailable. To tackle these challenges, we propose **DGPPO**, a new framework that *simultaneously* learns both a *discrete* graph CBF which handles neighborhood changes and input constraints, and a distributed high-performance safe policy for MAS with unknown discrete-time dynamics.
We empirically validate our claims on a suite of multi-agent tasks spanning three different simulation engines. The results suggest that, compared with existing methods, our DGPPO framework obtains policies that achieve high task performance (matching baselines that ignore the safety constraints), and high safety rates (matching the most conservative baselines), with a *constant* set of hyperparameters across all environments.

Published as a conference paper at ICLR 2025
DISCRETE GCBF PROXIMAL POLICY OPTIMIZATION FOR MULTI-AGENT SAFE OPTIMAL CONTROL
Songyuan Zhangi Oswin Sof Mitchell Black
Chuchu Fan
DDpartment of Aeronautics and Astronautics, MIT If {szhang21, oswinso, chuchu)emit.edd
MMIT Lincoln Laboratory
"mitchel1. waitheellblackee Balacke1l.ng edu
ABSTRACT
Control policies that can achieve high task performance and satisfy safety con- straints are desirable for any system, including multi-agent systems (MAS). One promising technique for ensuring the safety of MAS is distributed control barrier functions (CBF). However, it difficult design distributed CBF-based policies for MAS that can tackle unknown discrete- dynamics, partial observability. changing neighborhoods, and input constraints, especially when a distributed nigh-performance nominal policy that can achieve the task is unavailable. To tackle these challenges, we propose DGPPO, a new framework that simultane- ously learns both a discrete graph CBF which handles neighborhood changes and input constraints, and a distributed high-performance safe policy for MAS with unknown discrete-time dynamics. We empirically validate our claims suite of multi-agent tasks spanning three different simulation engines. The results suggest that, compared with existing methods, DGPPO framework obtains policies that achieve high task performance (matching baselines that ignore the safety constraints), and high safety rates (matching the most conservative baselines), with a constant set of hyperparameters across all enviromments I
INTRODUCTION
Multi-agent systems (MAS) have gained significant attention in recent years due to their potential applications in various domains, such as warchouse robotics (Kattepur et al.) 2018), autonomous vehicles (Shale--hwartze et al. (2016), traffic routing (Wu et al. 2020), and power systems (Biagioni eet al.) 2022). However, a big challenge for MAS is designing distributed control polícies that achieve high task performance while ensuring safety, especially when the are conflicting. In the single-agent continuous-time case, control barrier functions (CBF) are an effective tool to resolve the conflict via the solution of safety filter quadratic program (QP) (Xu et al.) 2015 Ames et al. 2017), minimally modifying a given performance-orienter nominal policy to be safe. While distributed CBFs have been proposed for -agent (Wang al., (017)) and partially observable cases (Zhang et al.) (2025). they have a limitation of requiring known continuous -ime dynamics and a nominal policy that can achieve high task performance (albeit not necessarily safely).
While the assumptions above are reasonable for many applications, they do not apply when the dynamics are unknown and a performance-oriene
nominal policy is unavailable. The challenge of requiring nominal policy has been addressed by approaches that combine CBFs with reinforcement learning (RL) (Cheng et al. 2019 [maam et al. (0022), where the nominal policy is learned via an unconstrained RL algorithm to maximize task performance while the CBF is used as a safety filter to ensure safety. However, these works have only been applied to the single-agent case, and require known control-affine dynamics to ensure the resulting safety filter QP is computationally tractable, which is too strict for most systems, e.g., with contact dynamics, especially in discrete time.
A third challenge is that CBF-based methods require CBF to be known. This be challenging in the case of input constraints since not every function satisfies the CBF conditions (Chen et al.. 2021). Constructing CBF is even more challenging the case of MAS with changing neighborhoods and limited sensing (Zhang et al. 2025). ). Zhang et al. 2025 proposed a learning framework for
DDITTRIBUTINN CEETTIONNTTTTTEEEE STATEMENT A. Approved for public release. Distribution is unlimited. Projject website: https:://mmt t- real m. thub. /gdgpp//

---

Published as a conference paper at ICLR 2025
constructing graph CBF (GCBF) for MAS that guarantees safety while satisfying imput constraints. However, they assume known continuous timooussstmmeeeee dynamics and require an existing nominal policy.
In this work, we address these challenges by proposing novel framework that simultancously learns a discrete graph CBF and a hegh-peeformance safe policy for a MAS under unknown discrete-tima dynamics, changing neighborhoods, and input constraints. We summarize contributions below.
We propose a method of learning discrete CBFs (DCBF) for unknown discrete-ting dynamics and with input constraints.
We propose the discrete GCBF (DGCBF), a discrete-timg extension of the GCBF, for ensuring safety under varying neighborhoods in the limited sensing setting and extend the DCBF learning above to the case of DGCBF.
We propose Discrete Graph CBF Proximal Policy Optimization (DGPPO), a framework com- bining RL and DGCBF for solving discrete-time multi-agent safe optimal control problems for MAS with unknown dynamics and limiting sensing without a known performant nominal policy.
Through extensive simulations, we demonstrate that DGPPO outperforms existing methods and is not sensitive to hyperparameters.. Specifically, compared to existing methods that require dif- ferent choices of hyperparameters per environment, DGPPO achieves the lowest cost compared to baselines with near 100% safety rate using single set of hyperparameters.
2 RELATED WORK
Constructing Decentralized CBFs. The challenge of applying CBFs for MAS has been explored via the construction of distributed CBFs (Borrmann et al. 2015; Glotfelter et al. 2017] Wang jet al., 2017 Lindemann & Dimarogonas 2019: Black & Panagou 2023), which only take local observations as input. This simplifies the big centralized QP problem into small QP problems to be solved per agent. However, the construction is either limited to the case of unbounded control (Lnddemann & Dimarogonas 2019) or only for specific dynamics model (e.g.. double integrator)
Borrmann et al.) 2015] Glotfelter et al. 2017 Wang et al., 2017). Recent advances in learning CBFs using neural networks (Saveriano & Lee 2019 Srinivasan et al. 2020; [Lidemaan et al.) 2021 Peruffo et al. 2021 Dawson et al. [2022] So et al. 2024 Knoedler el al. (2024) has resulted in works that investigate learning distributed CBFs (Qin al. [021 Zhang al. 2023 2025 Zinage et al. 2024). Nevertheless. these approaches assume known dynamics and are only applicable to continuous-tin dynamics and hence cannot be applied to our problem setting. Moreover, it is assumed that performant nominal policy is available, which we do not consider in this work.
CBF in RL. Originally inspired by the prospect of safety during training, recent works have inte- grated CBFs into the RL training process via the safety filter (Taale et al. 2021 Hsu et al., 2023
Garg et al. 2024) for both aaggtt (Cheng et al.) [019) (Emam et al. [022]] Hailemichael al,
2023 and multi-agent (Pereira et al.) 2021; 2022) cases. Although both continuous-time (Emam et al. 2022 Hailemichael et al. 2023) and discrete-time (Cheng et al., 2019) dynamics have been considered, major limitation is the requirement of affine (D)CBFs and control-affine dynamics up to a constant disturbance term to be learned. In contrast, the problem we tackle in this work does not make any such assumptions about the safety specifications or the structure of the dynamics.
Safe Multi-agent RL. The problem of constructing safe policies for MAS has also been studied
in the RL community (Garg et al. 2024). Early works achieved safety via reward function design CCen et al. 2017b a Long et al. 2018 Everett et al. 2018 Semnani et al., 2020). However, these approaches do not guarantee the satisfaction of the safety constraints even for the optimal policy (Massiani et al. 2023] [verett et al. 2018 Long et al. 2018). More recently, in the single-agent case, methods work with constraints in the form of the constrained Markov decision process (CMDP) problem and apply techniques from constrained optimization, including primal methods (uu et al. (02T). primal-dual methods using Lagrange multipliers (Borkar) 2005 Tessler et al., 2019 He et al.
20233 HHung et al. (2024), and via trust-region-bar based approaches (Achiam et al.) He et al.
2017
approaches are the most popular due to their simplicity, leading to multi-agent extensions (Gu et al. 2023 Liu et al. 202 Ding et al.)
2023 Of these,
2023 Lu et al. 2021; Geng et al. 2023 Zhao et al. 2024).. However, Lagrangian methods for CMDPs have been observed to have unstable training and convergence to poor policies when the
constraint threshold is zero (Zanon & Gros 2020) He et al., 2023 So & Fan 2023 Ganai et al.
2024), which is the setting we target in this work.
2

---

Published as a conference paper at ICLR 2025
3 PROBLEM SETTING AND PRELIMINARIES
3.1 MULTI-AGENT CONSTRAINED OPTIMAL CONTROL PROBLEM
Consider N agent MAS. We aim to solve distributed control policies that minimize joint cost describing a desired task while staying safe. Let the state and control of agent at timestep be xf E X' and uf E U, where x contains agent i's position p P. The joint state is defined as x [af::..; ; y E X. where y y is non-agent states (e.g., obstacles, goals). The joint action is defined with a = jufi..,, ..... " E U. We assume x follows the general discrete-time dynamics
x = n²),
(1)
where / x + X' describes the joint dynamics and is unknown. We consider the setting where agents only observe objects within their sensing radius R > O. Let NG(x) {i llp; pal S R) denote the neighborhood of agent i. At timestep k; agent i only has access to local observation of = Ox(x*) E O for observation function Oj:
0,(x*) =
(000)=(((()))) (foSs)SEN. ).).
Og = (1;, of 0"
(2)
for inter-agent O" X' x X' + O₂ and non-agent O² X' x + O2 observation functions. Let the joint cost function describing the desired task be denoted as : X x u - R Each agent has an avoid set defined as A, for O h (m))) (o:) > 0, Vm} for the avoid functions h!m) : - E, m E {1,...,M, M), such that the agent is unsafe it it enters A, any time in the trajectory. We want to learn distributed policies (4; : O - 21 that minimize the joint cost / while avoiding the avoid set A at all times. Formally, denoting the joint policy by p(x)= |n (o1);
(N(oN)], we want to solve the following discrete-time distributed multi-agent safe optimal control problem (MASOCP):
min MIOMAN
E1(x ,u(x1)))
(3a)
s.t. x+1 f(x² )xxx)),
Vi E {1, ,.1....,,,... N}. k 2 0, (3b)
Mm) (ot) S 0, of Ox(x,. We (I,..,. Vm {1, (I...., M) k 0. (3c)
The main challenge in solving B is satisfying the safety constraints (3), especially in the multi- agent case with changing neighborhood, under unknown discrete- time dynamics with input con- straints. We propose tackle this challenge using the framework of DCBFs, which we review next.
3.2 DISCRETE CBF
To tackle the safety constraint of Problem 3 we review the notion of DCBF. Considering the discrete-time dynamics, we take the following definition of a DCBF from Ahmadi et al. (2019)
Definition 1. A function B X + R is discrete CBF (DCBF) for ① if there exists an extended class-k function a satisfying a--r)= > -r for all r > such that B satisfies the following property:
B(x) S 0 = inf B u)) - Bxx) + (B(xx) S O.
ueu
As shown in Ahmadi al. (2019), the following theorem holds. Theorem 1. The set == {x (xx) I 0} is control invariant under any policy i that satisfies
B(f(x, (xx))) - Bxx) + (B((x)) < 0, Vx E
(5)
Thus, IFC for the avoid set A, then the μ from Theorem I Trenders the system safe.
Safe and Performant Policies via Safety Filtering. Given DCBFs ((m) for ......,M. we
can construct safe and performant policy using the safety filter framework. Assuming nominal policy Pnom that is performant, e.g., minimizes the cost 1, but not necessarily safe, we can obtain a safe and performant policy by solving the following nonlinear optimization problem:
min uetl
||u |nom (0n-Momxx)1 (x)
(6a)
((m) (x, u) S0 ((()) u) = B(m)) u)) (x) + a( (I(m) Vm. (6b)
s.t.
"The cost function / here is not the cost in CMDP. Rather, it corresponds the negative reward in CMDP.
3

---

We address the challenge of requiring performant nominal policy in the safety filter (6) by instead directly learning a policy using RL that minimizes the joint cost function (3ab. This has been done previously in the single-agent setting via the framework of shielding for RL, where an Published as a conference paper at ICLR 2025
In the general case, even if the dynamics f are known, 66 is potentially nonlinear program that could be difficult to solve. In our problem setting, we assume ff to be unknown, which renders this approach infeasible. Moreover, applying the safety filter framework assumes access to performant, distributed nominal policy Pnom. which we do not assume available. Even if it were available, the safety filtering only minimizes the instantaneous deviation in control (6a) and can be myopic, potentially leading to liveness problems (Reis et al. 2020 and deadlocks (Jankovic et al. (023).
Note that not every function satisfies (5) and is DCBF. In continuous time with control-affine dynamics f. unbounded inputs are sufficient for any continuously differentiable function to be a CBF (Xiao Belta 2019) since the CBF condition will be linear the controls. However, in discrete time, the DCBF condition (5) is potentially nonlinear in u, making it difficult to construct a DCBF even with unbounded controls. While solutions have been proposed for the continuous-time case to construct valid CBFs under bounded inputs (Chen et al., 202T Soo et al. 2024), the same not true for DCBFs, potentially due the requirement solving a nonlinear program (6) even after such DCBF has been found.
TACKLING CHALLENGES OF DCBFs FOR MASOCP WITH DGPPO
We now address four challenges of extending DCBF-based approach to our problem setting and propose DGPPO, our framework for solving (3) All proofs can be found in Appendix A
4.1 CONSTRAINT-VALLU FUNCTION IS DCBF
To construct a DCBF, we show that the constraint-valle function of a policy a DCBF, extending the insights of So al. 2024 to discrete-time. This allows learning DCBF with policy evaluation.
R. let the avoid set be A == fx E X ((x) > 0}. For a fixed deterministic policy μa, consider the constraint- value functions V defined
For an arbitrary function Ç: X +
VSH(() max ((x*), k20
x" =x x x+1 f(x ,(x*)).
Then, VS. is DCBF, which show in the following theorem. Theorem 2 (Discrete Policy CBF). For a given μ., the constraint-valle function Vh.t is a DCBF for any extended class-k function a satisfying a(-r) > -r for all r > 0. Moreover, given l((m), f flustiin satisfies Vh""), (f((x, u)) Vhim) H (() a(V (m)) (x)) for all m {L.....M..
Theorem 2 enables the construction of DCBF by choosing any policy μ and evaluating its constraint-value function. Consequently, we can construct DCBF by learning the value function.
4.2 REMOVING THE NOMINAL POLICY WITH EXPLICIT COST OPTIMIZATION
unconstrained policy μ4l with parameters is learned using existing unconstrained RL techniques. and the (D)CBF safety filter is incorporated into the environment dynamics (Cheng al., 2019 Emam et al. 2022 Hailemichael et al. 2023). Formally, the following problem is considered:
min Aeo
E 1xx" )
s.t. u (1ottt
k=
where SafetyFilter computes the minimizer of 6 However, solving (6) in the discrete case a nonlinear program that is difficult unless mm) is linear and the dynamies affine, an assumption that we, unlike previous works (Cheng et al. 2019; Emam et al.] 2022 Hailemichacl et al., 2023), do not impose. To work around this, we constrain the leamedd policy to satisfy the DCBF conditions instead of using the safety filter framework.
control-
min
1(x (x)))))
(9a) s.t. (x*) pxx(*))) < S, Vm = (1,...,MY MY. k 0, (9b)
k=0
where (1(m) is defined in 6b This removes the need for a nominal policy and for solving the nonlinear safety filter program
4

---

Published as a conference paper at ICLR 2025
4.3 CONSTRAINED POLICY OPTIMIZATION USING DCBF UNDER UNKNOWN DYNAMICS
We now tackle the challenge of performing constrained policy optimization using DCBFs in 9 We choose to use a purely primal method inspired by constraint-rectifeed policy optimization (CRPO) (uu et al. 2021 due to its simplicity (it does not have parameters related to the dual problem but simply chooses to take different gradient steps based on the current constraint satisfaction). Specifically, if the constraint (9b) is satisfied, our algorithm takes one gradient step to minimize the objective (9)). Otherwise, if the constraint is violated, our algorithm takes one gradient step to minimize the DCBF constraint violation C.
This procedure still cannot be implemented as-is since the gradient of C' cannot be computed di- rectly without knowledge of the dynamics f. To this end, we propose to use score function gradients [Williams) 1992) to compute gradients of (9b) without knowing the dynamics f. Since this requires stochastic policy, we modify ( accordingly. For clarity, let πT (-(x])) denote the probability density function of the stochastic policy with parameters A conditioned on state It can be tempting to consider the following stochastic version of the problem, which resembles the CMDP setting.
min Excrpo,uvma (o u),,
S.L. Exxu"a (uwx((-))))))) [c()) (x, u) S O,
(10)
where QT is the Q-function and Po. ~a* are the initial and stationary state distributions. Here, we constrain the expectation of the DCBF constraint. However, this formulation is not sufficient for safety, as the expectation does not guarantee satisfaction for all states as in (6). leading to an unsafe policy. To tackle this, we modify the constraint, leading to the following problem.
min Exx uurpoo xx) (x, u)];
[Q
(11a)
s.t. Excup E Eunto()))) max {0, ((m) (x, u)}
20, Vm.
(11b)
(5m)) (xx
Here, the satisfaction of (1b guarantees that the DCBF constraint is satisfied almost surely (Appendix A.2). Applying gradient-manipulatio style primal optimization as in CRPO (Xu et al. 2021
gives us the following expression for the gradient V&L of the policy loss
V VoEx un [Q (x, u)]:
Exnpro
[c)m] (x)] 50, Vm,
VoL(O)
Expppo
[VoC (1m) (x)) for any m that violates, otherwise,
(12)
where v > 0 is a hyperparameter scaling the size of constraint minimization steps. The gradient of (m)) can be computed using score function gradients (see Appendix A.4 discussion rT in Appendix B.2).
Improving sample efficiency with Gradient Projection. One drawback of (22) is that this scheme is sample-inefficient in the sense that if only a single state x violates the DCBF constraint, then the gradient information of the total cost from all other safe states are thrown away. We propose to use this thrown-away gradient information by projecting the cost gradient to be orthogonal to the gradient direction of the violating constraints, similar to techniques from multi-objective optimiza- tion YYu et al. 2020 Liu et al. 2021a). However, this requires computing the gradient M + 1 times, which is expensive when M is large. Instead, we propose use the following informal theorem:
Informal Theorem (Approximate Gradient Projection for Decoupled Policy Parameters). Let of (m) = VoExxpp ((()) (x) denote the gradient of the m-th DCBF constraint violation for any state distribution p. Under suitable assumptions on the policy parametrization nG. modifying the gradient of the objective laa from Goriz := Exxpto Eur-mo(-)x) No log π (x, u) (x u) to
g == Exxp Be (- logog TT u) {maxm 66 (x) (x.
(13)
by multiplying the Q with an indicator function gives an approximate projection of gorie such that g (m) 0 Vm, i.e., it lies in the orthogonal complement of the construint gradients (m))
We state this more formally in Theorem A2 By Informal Theorem we can treat g as an approximate projection of the gradient of the objective (1a) that does not interfere with
5

---

Published as a conference paper at ICLR 2025
the gradients from the safety constraints. Combining ((m) (with p p=m) and g from Informal Theorem 3 gives the following gradient VoL of the policy loss L, where denotes the stop gradient operation (see Appendix B for detailed derivation and discussion about important details).
L(0) xx (p (u log no To (Q(((((x u, 0))]
(14)
@(x, u, 0) == {max.. 6)mh (x) 1 [ (x, u)) + 1 max (()) (x, u).
(15)
Although we cannot guarantee that the conditions of Informal Theorem 3 hold, we find that adding this gradient projection improves performance. We provide empirical comparisons between this
gradient projection method (14) and methods without projection (10). in Appendix [
Scheduling the weight v. Empirically, different values of result tradeoff between safety and cost minimization (potentially due to not satisfying Informal Theorem 3). To remedy this, we sched- ule y by taking
initially to encourage exploration the beginning, then doubling it after 50%
and 75% of the total update steps to emphasize safety, We investigate this further in Section 5.3
4.4 EXTENDING TO THE MULTI- CASE WITH DGCBF
Having proposed three solutions to tackle the difficulty of using CBFs in RL for unknown discrete-time dynamics, we now tackle the final challenge of changing neighborhoods due to the limited sensing radius of each agent. For this, we draw inspiration from GCBF (Zhang et al. 2025). which provides a theoretical framework for constructing distributed CBFs that can handle varying neighborhood sizes, albeit for the case of continuous-tine dynamics.
To construct a distributed DCBF B, we need B to be function of each agent's local observation o as opposed to the joint state x. We make this concrete in the following definition.
Definition 2 (Discrete GCBF). A function O + IR is a Discrete Graph CBF (DGCBF) if there exists a class-k function a with ax -r for all > 0 and a control policy μ. : O U satisfying
class-k,function
(x)z 50, 0, Vj = Bio, (x)) - ((o((x)) + a a(B(o(x))) & O, Vx X, Vi,
(16)
where of (x) Ox([(x,(['))) and denotes the resulting joint policy from each agent using μ..
Since a DCBF B is defined for a MAS with fixed size N, can not be used when N changes. Given a DGCBF B, we can construct DCBF as B(x) = max; B(o, (x)) (Appendix A.6), thus B guarantees safety. However, the same DGCBF B can also be used to guarantee safety for any N, which we show in Appendix A.7
Remark (Discontinuity due to neighborhood changes). Unlike the continuous case (Zhang al. 2025), which assumes that the GCBF unaffected by agents at the sensing-radine boundary, DGCBF does not have this requirement. This is because the proof of safety in GCBF relies on contimuity (w.rt. time) during neighborhood changes. However, in the discrete-time case, the proof of safety only looks at finite differences and hence does not require the continuity of the DGCBF.
Finding a function B that satisfies (16) is nontrivial, especially in the case of neighborhood changes due to the limited sensing radius. Though we can learn a DGCBF using Section [.1] it is unclear how the learned function satisfies (16) when the neighborhood of agents changes. One sufficient way for this to hold is to take advantage of the attention mechanism to place zero weights on the features corresponding to agents far enough away such that the value of B is not affected too much by such agents.
We state this informally in the following theorem (see Appendix A.5 for the formal version). Informal Theorem 4 (Satisfying (16) during neighborhood changes). Let Eu, &2 and Es be functions that encode the input observations into some feature space. Suppose is of the form
B(O,(x)) =
Σ w(or))E20o)) Es(o") JEN.
(17)
where On, - I. is a weighting function that goes to for observations 0zj of agents that are far enough away. Then, under technical conditions on the dynamics, B satisfies 169 and is DGCBE

---

Published as a conference paper at ICLR 2025
MSE
Stochastic
Stochastic Rollout
Fonward
Deterministic Forward
x(1xx)
Backprop
Agent Palicy
Gradient
Ai (o))
A
Projection
Deterministic Rollout
prltnn
max (mm) mt
((x)
((m) 0?
PPO loss with A
MSE
Figure I: DGPPO algorithm. In addition to the normal MAPPO path (top) using stochastic rollouts, introduce second path (bottom) that uses deterministic rollouts to learn DGCBE.
We encourage B to satisfy (16) during neighborhood changes by parameterizing the value functions using graph neural networks (GNN) with graph attention (Velićković et al.) 2017). which takes the form of (17) and hence amenable to Informal Theorem Similar to Zhang et al. (2025), the atten- tion mechanism naturally learns to place zero weights on the features corresponding to neighboring agents that are far enough away, enabling the learned B to satisfy (16) during neighborhood changes.
Finally, Theorem can be extended to DGCBF, as shown in the following Corollary.
Corollary 1 (Discrete Policy GCBF). Suppose V.hlim) 4 X - R for agent can be expressed using only local observations Og, Le., there exists some Vo 1 O - such that
Vnhlm), 10 (o)) V.h(mn) 11 (xx) = max h (m)) (o,").
(18)
k20
Then, ( is DGCBF
4.5 DGPPO: PUTTING EVERYTHING TOGETHER
Combining the proposed solutions from the previous subsections, we present DGPPO, a framework for solving the discrete-time multi-agent safe optimal control problem 33. DGPPO follows the basic structure of on- on-policy MARL algorithms such as MAPPO Yu al. 2022
1. We perform a T-step stochastic rollout with the policy πe. However, unlike MAPPO, we additionally perform a Ttst deterministic rollout using a deterministic version of πy (by taking the mode), which we denote μ., to lear the DGCBF (per Theorem
2. We update the value functions via regression the corresponding targets computed using GAE (Shhulmnn et al. 2015), where the targets for the cost-value function V uses the stochastic
rollout, and the targets for the constraint-value functions (d((m) use the deterministic rollout.
3. We update the policy na by replacing the Q-function with its GAE (Schulman et al. al. 2015)), then combining the CRPO-style decoupled policy loss (14) with the PPO clipped loss (Schulman al.)
2017) using the learned constraint- value functions ( . as the DGCBFs ((m), Specifically, we treat the expression within the expectation in 14) as a pseudo-advantage Å, for agent and (m))
(()) of Co. in (14 giving us
use single-sample estimator Co.5
(m) max f0, phlm) (O 0,,4
vh(n) 1 "(or) a(phim MoNi)
(19)
A; = A GAE
mmax Co. zo3
+ 1/ max
(m) 1
0,5 mmxx (km) 20}
(20)
where AAA denotes the GAE Schulman et al.
2015
for agent i. We then use A, in the PPO Yu et al 2022).
policy loss (Schulman et al. 2017) as done in MAPPO
We summarize DGPPO algorithm in Figure]

---

Published as a conference paper at ICLR 2025
(a) TARGET
(b) SPREAD
(C) LINE
(d) BICYCLE
Agents
Agents
Obstacle
Obstacles
Goals
Gool
(e) TRANSPORT
(f) WHEEL.
TRANSPORT2
Figure 2: Environments.
We on (top) LiDAR, (bottom) MuJoCo, and VMAS environments.
EXPERIMENTS
In this section, design experiments to answer the following research questions: (Q1) Does DGPPO learn a safe policy that also achieves low costs without hyperparameter tuning in different environments? (Q2) How stable is the training of DGPPO? (Q3) Can DGPPO maintain its perfor- mance with an increasing number of agents? (Q4) Is DGPPO sensitive to the hyperparameters?
To compare the methods, look at the cost and safety rate. The cost is the trajectory cumulative cost Σ (x**, u²). The safety rate is the ratio agents that are safe over the entire trajectory. Details on implementation, tasks, hyperparameters, and additional experiments are in Appendix
5.1 SETUP
Environments. We evaluate DGPPO in wide range of enviromments including four LiDAR environments (TARGET, SPREAD, LINE, BICYCLE) where the agents use LiDAR detect obstacles (Keyumarsi et al., (023). one MuJoCo environment TRANSPORT (Todorov et al. 2012). and two VMAS environments (TRANSPORT2, WHEEL. (Bettini et al., 2022: 2024).
Baselines. We compare DGPPO against baseline methods that can solve MASOCP under unknown discrete-time dynamics, including the state-of-the-art MARL algorithm InforMARL (Nayak et al.) 2023) and the safe MARL algorithm MAPPO-Lagrangian (Gu et al. 202 2023). For InforMARL. we add the constraint violations max(0, max, h(m)) weighted by B to the cost function for different Ais (5)) We also try a weight-scheduling scheme where starts at 0.01 and increases at 50% and 75% of the total steps (Schedule). For MAPPO-Lagrangian we use GNN backbone for fair comparison. We notice the official implementation (Gu et al. (003) uses tiny learning rate on the Lagrange multipliers (10-7), so consider two different initialization λo (Lagr(>a)). We also increase the learning rate À to 0.1 (Lagr(Ir)). We run each method for the same number of update steps, chosen be large enough such that all methods converge
5.2
MAIN RESULTS
For each environment, we each algorithm with 3 different seeds and evaluate each run 32 different initial conditions. We draw the following conclusions.
DGPPO has the best performance and is hyperparameter insensitive. We first compare the converged policies of all algorithms (Figure B) DGPPO is closest to the top left comer in all environments, indicating that it performs the best. For Penalty and Lagr, different choices of
(Q1):
*This the smallest learning rate of À that does not make the algorithm ignore the safety constraints. We also baselines with double environment steps similar DGPPO for faimess (Appendix C.6.2 C.6.2
8

---

Published as a conference paper at ICLR 2025
Target
Spread
Line
Bicycle
1.0
1.0
1.0
Sufety rate
Safety rale
Safed rale
Safedy rale
0.8
0.8
0.8
0..8
1.0
1.5
0.6
1.0
0.6
0.5
0.6
1.0
Cost
Cost
Cost
Cost
1.0
Transport
1.0
Wheel
1.0
Transport2
DGPPO (ours)
1 Penalty(0.02)
Safety rute
Safety rate
Safety rate
+
Penalty(0. I)
+ Penalty(O.S)
0.8
0.5
4 Schedule
0.6
+ Lagn(I)
4 Lagr(5)
0.2
0.4
0.5
1.0
0.0
0.50
0.75
1.00
+ Lagn(Ir)
Cost
Cost
Cost
Figure 3:
Comparison on N
3 agents. + denotes the mean + standard deviation. Methods
closer the top left yield lower costs and higher safety rates.
Spread
1.0
Spoced
Line
1.0
Line
Safdg mtto
Safdy gate
0.8
Bo.s
0.5
0.0
0.5
1.5
0.0
0.0
0.5
1.0
L.S
0.0g.n
0.5
1.5
0.0
0.5
Step
1c5
Step
Stop
Stop
DGPPO (ouss)
Schodale
Lagela)
Figure 4: Training stability. DGPPO yields smoother training curves compared to the baselines.
1.00
0.8
0.35
0.6
0,50
0.39
62
0.0
143
Stop
Stop
Skop
DIPPO (pee))
Pexalyyh
Schadola
Laght)
Lag15)
Lagrón
Figure 5: Scaling to N 7. Unlike other methods, DGPPO performs similarly with more agents.
hyperparameters result in either focusing only safety focusing only on performance. Even with a fixed hyperparameter, the performance of these two basclines also varies between environments. On the other hand, using the same set of hyperparameters achieves the lowest cost among methods with safety rate close to 100%.
all environments, DGPPO consistently
(Q2): Training of DGPPO is more stable. Next, we compare the training stability DGPPO with Schedule, due to having a weight scheduled, and Lagr(Ir). due to having non-negligible learning rate (Figure Schedule experiences an increase in cost as 3 increases throughout training. Lagr(Ir) experiences high variance and many spikes in both cost and safety rate throughout training. This is similar to previous results obtained the single-agent when the cost threshold zero (So Fan, 2023 He al. 2023). DGPPO has much smoother training curve than both. We provide training curves on other algorithms and environments Appendix
(Q3): DGPPO scales well with more agents. Finally, we test the scalability of the methods on LINE by increasing the N from 3 to 5 and 7 (Figure The same trends from before still hold, with DGPPÓ achieving the best performance and high safety rates. We also that DGPPO performs well even with more agents, but the baseline methods are more inconsistent (c.g., Schedule is mostly safe with N but not for N 7), possibly due to their hyperparameter sensitivity.

---

Published as a conference paper at ICLR 2025
1.00
1.2
1,00
0.10
1.00
S:00y 882
8 6.75
8.50
0.75
0.50
0.50
0.81
0.35
0.0
0.5
1.0
0,00
0.7
1.0
1.5
0.00
0.0
0.5
0,80
0.0
0.5
1.5
0.05
8:35
--
a- 1.0
vachadale
(a) Varying a in DGPPO.
(b) Varying in DGPPO.
1.25
1.0
1.00
0.95
1.4
Seftes are
0.75
0.50
100
0.2
0.85
0.6
8:0
0.800
Skep
Stop
Detemnisbil
Studraticc
Loxad CBF
Hasd-crahad CBF
(c) The necessity of deterministic rollouts to learn V
(d) The necessity of learning a CBF.
Figure 6: Ablations. We vary hyperparameters (top) and verify our design decisions (bottom).
5.3 ABLATION STUDIES
We now study hyperparameter sensitivity (Q4) by varying different hyperparameters in DGPPO.
Class-s function a. For the class-n: function in (16). we use a linear (r) ar with a = 0.3. We test the sensitivity of DGPPO to this by varying a (Figure 6a) on SPREAD. We observe that a 0 leads to conservative behavior with a high cost. a = 1.2 leads to an unsafe policy, which is to be expected since the a(-r) > -r condition is violated. For the other values that satisfy this condition, there is no significant difference in either cost or safety. We can thus choose any a E (0, 1).
Weight v on the gradient of C. We introduced a schedule for v, which weights the constraint SPREAD with different static schedules (Figure (bb). The safety rate is lower with v = 1, while the convergence in cost for 4, 8,8in is slower. The proposed schedule leads to faster cost convergence and high safety rate.
minimization step (Section 4.3 We test the sensitivity to
Learning 4 with a stochastic policy. Theorem 2 used to learn the DGCBF Vh" 'H, requires deterministic rollouts. Consequently, DGPPO uses double the environment samples by performing both a stochastic and deterministic rollout. We verify whether this is necessary by seeing how the type of rollout (deterministic stochastic) used to learn the DGCBF affects performance (Figure (o). which shows that using a stochastic rollout degrades both the cost and safety rate. Thus, the use of deterministic rollout to learn (m .M is necessary despite the increased data use.
Using a hand-crafted DGCBE One motivation for DGPPO that is difficult to construct a DGCBF with changing neighborhoods and input constraints. We test this by using (mm) directly as
the DGCBF (instead of the learned ph(m)) .A.), as is commonly done for CBFs, on TRANSPORT2 (Figure (dd). Using this hand-crafted "DGCBF" results in decreased safety rate (~ 15%), validating the need to learn a DGCBE. If no DGCBF used, it performs even worse (Appendix C.6.3)
CONCLUSION
We propose DGPPO to learn distributed safe policies for discrete-time MAS with unknown dynam- ics under a limited sensing range. We extend CBFs to this problem setting with DGCBFs, propose a construction using constraint-value functions, and apply CBFs to the case of unknown dynamics using score function gradients. Experimental results across three simulation engines suggest that DGPPO robust to hyperparameters and performs well, achieving a safety rate matching conservative baselines while matching the performance of the performant but unsafe baselines.
Limitations. DGPPO uses both stochastic and deterministic rollouts, decreasing the sample efficiency. Moreover, safety under stochastic dynamics has not been considered. Finally, although safety is guaranteed when the DGCBF constraints are satisfied at all states, achieving this in practice using learning is hard. We leave these limitations to future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was partly supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. In addition, Zhang, So, and Fan are supported by the MIT-DSTA program. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and don't 't necessarily reflect the views of the sponsors.
2025 Massachusetts Institute of Technology.
Delivered to the U.S. Government with Unlimited Rights, defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Govermment rights in this work defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pp. 22-31. PMLR, 2017.
Mohamadreza Ahmadi, Andrew Singletary, Joel W Burdick, and Aaron D) Ames. Safe policy syn- thesis in multi-agent pomdps via discrete-time barrier functions. 20/9 IEEE 58th Conference Decision and Control (CDC), pp. 4797-4803, IEEE, 2019.
Aaron Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions Automatic Control, 62(8): 3861-3876, 2017.
Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems, 2022.
Matteo Bettini, Amanda Prorok, and Vincent Moens. Benchmarl: Benchmarking multi-agent reinforcement learning. Journal of Machine Learning Research, 25(217):1-10, 2024.
David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rohit Chintala, Jennifer King, and Ahmed S Zamzam. Powergridworld: framework for multi-agent reinforcement learning in power systems. In Proceedings of the Thirteenth ACM International Conference Future Energy Systems, pp. 565-570, 2022.
Mitchell Black and Dimitra Panagou. Adaptation for validation of consolidated control barrier functions. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 751-757, 2023. doi: DC.109//DDC44953.20223.. 10383597.
Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & Control Letters, 54()):0772133, 2005.
Urs Borrmann, Li Wang, Aaron D Ames, and Magnus Egerstedt. Control barrier certificates for safe swarm behavior. IFAC-PaperSOnLinee 48(27):68 73, 2015.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:/ http:://qithbbe com/ coommjax-mm//a1
Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 JEEE/RSI International Conference Intelligent Robots and Systems (IROS), pp. 1343-1350. IEEE, 2017a.
Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P How. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 285-292. IEEE, 2017b.

---

Published as a conference paper at ICLR 2025
Yuxiao Chen, Mrdjan Jankovic, Mario Santillo, and Aaron D Ames. Backup control barrier functions: Formulation and comparative study. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 0835-6841. IEEE, 2021.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe rein- forcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3387-3395, 2019.
Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neu- ral lyapunov-bariier functions, In Conference on Robot Learning, pp. 1724-1735. PMLR, 2022.
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning- In Learning for Dynamics and Control Conference, pp. 315-332, PMLR, 2023.
Yousef Emam, Gennaro Notomista, Paul Glotfelter, 7solt Kira, and Magnus Egerstedt. Safe reinforcement learning using robust control barrier functions. IEFE Robotics and Automation Letters, 2022.
Michael Everett, Yu Fan Chen, and Jonathan P How. Motion planning among dynamic, decision- making agents with deep reinforcement learning. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3052-3059. IEEE, 2018.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kunal Garg, Songyuan Zhang, Oswin So, Charles Dawson, and Chuchu Fan. Learning safe control for multi-robot systems: Methods, verification, and open challenges. Annual Reviews in Control, 57:100948, 2024.
Nan Geng, Qinbo Bai, Chenyi Liu, Tian Lan, Vaneet Aggarwal, Yuan Yang, and Mingwei Xu. A reinforcement learning framework for vehicular network routing under peak and average constraints. IEEE Transactions on Vehicular Technology, 2023.
Paul Glotfelter, Jorge Cortés, and Magnus Egerstedt. Nonsmooth barrier functions with applications to multi-robot systems. IEEE control systems letters, 1(2):310-315, 2017.
Jaskaran Singh Grover, Changliu Liu, and Katia Sycara. Deadlock analysis and resolution for multi-robot systems. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop the Algorithmic Foundations of Robotics 14, pp. 292-312. Springer, 2021.
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang. Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint 1 2021.
Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:103905, 2023.
Habtamu Hailemichael, Beshah Ayalew, and Andrej Ivanco. Optimal control barrier functions for rl based safe powertrain control. PPAC-PapersOnLine, 56(3):385-390 2023.
Tairan He, Weiye Zhao, and Changliu Liu. Autocost: Evolving intrinsic cost for zero-violation reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14847-14855, 2023.
Kai-Chieh Hsu, Haimin Hu, and Jaime F Fisac The safety filter: A unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023.
Weidong Huang, Jiaming Ji. Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations, 2024. URL RRR.httss: / www.://wwwwwwweeeeeeee wtt.::/wwweeeeooouotettaaaaaaaaa
12

---

Published as a conference paper at ICLR 2025
Mrdjan Jankovic, Mario Santillo, and Yan Wang. Multiagent systems with ebf-based controllers: Collision avoidance and liveness from instability. IEEE Transactions on Control Systems Technology, 2023.
Ajay Kattepur, Hemant Kumar Rath, Anantha Simha, and Arijit Mukherjee. Distributed optimiza- tion in multi-agen robotics for industry 4.0 warehouses. In Proceedings of the 33rd Anmual ACM Symposium on Applied Computing, pp. 808-815, 2018.
Shaghayegh Keyumarsi, Made Widhi Surya Atman, and Azwirman Gusrialdi. Lidar-based online control barrier function synthesis for safe navigation in unknown environments. IEEE Robotics and Automation Letters, 2023.
Luzia Knoedler, Oswin So, Ji Yin, Mitchell Black, Zachary Serlin, Panagiotis Tsiotras, Javier Alonso-Mora, and Chuchu Fan. Rpcbf: Constructing safety filters robust model error and disturbances via policy control barrier functions. arXiv preprint, 2024.
Lars Lindemann and Dimos V Dimarogonas. Control barrier functions for multi-agent systems under conflicting local signal temporal logic tasks. IEEE control systems letters, 3(3):757-762, 2019.
Lars Lindemann, Haimin Hu, Alexander Robey, Hanwen Zhang, Dimos Dimarogonas, Stephen Tu, and Nikolai Matni. Learning hybrid control barrier functions from data. In Conference on robot learning, pp. 1351 1-300 PMLR, 2021.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021a.
Chenyi Liu, Nan Geng, Vaneet Aggarwal, Tian Lan, Yuan Yang, and Mingwei Xu. Cmix: Deep multi-agent reinforcement learning with peak and average constraints. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part 21, pp. 157-173. Springer, 2021b.
Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6252 2229. IEEE, 2018.
Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Başar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8767-8775. 2021.
Pierre-François Massiani, Steve Heim, Friedrich Solowjow, and Sebastian Trimpe. Safe value functions. IEEE Transactions on Automatic Control, 6865):743-2277, 2023.
Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, and Hamsa Balakrishnan. Scalable agett reinforcement learning through intelligent information aggregation. In International Conference on Machine Learning, pp. 25817-25833. PMLR, 2023.
Mareus Pereira, Ziyi Wang, Ioannis Exarchos, and Evangelos Theodorou. Safe optimal control using stochastic barrier functions and deep Learning, pp. 1783-1801. PMLR, 2021.
sdes. In Conference on Robot
Marcus A Pereira, Augustinos D Saravanos, Oswin So, and Evangelos A Theodorou. Decentralized safe multi-agent stochastic optimal control using deep fbsdes and admm. arXiv preprint arXiv:2202. 10658, 2022.
Andrea Peruffo, Daniele Ahmed, and Alessandro Abate. Automated and formal synthesis of neural barrier certificates for dynamical models. In International conference on tools and algorithmss for the construction and analysis of systems, pp. 370-388. Springer, 2021.
Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi-agent control with decentralized neural barrier certificates. In International Conference on Learning Representations, 2021. URL https: http:::://////////eeeee review...r.eee............. net. wttp://wwwweeeeoomummmeeeeee
13

---

Published as a conference paper at ICLR 2025
Matheus F Reis, A Pedro Aguiar, and Paulo Tabuada. Control barrier dinction-basee quadratic programs introduce undesirable asymptotically stable equilibria. IEEE Control Systems Letters, 5(2):731-736, 2020.
Matteo Saveriano and Dongheui Lee. Learning barrier functions for constrained motion planning with dynamical systems. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 112-119. IEEE, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438,, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprini 177.v01707.06377, 2017.
Samaneh Hosseini Semnani, Hugh Liu, Michael Everett, Anton De Ruiter, and Jonathan P How. Multi-agent motion planning for dense and dynamic environments via deep reinforcement learning. IEEE Robotics and Automation Letters, 5(2):3221-3223) 2020.
Shai Shalev-Shwarde Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.032955 2016.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint MiXiv:2009.03509 2020.
Oswin So and Chuchu Fan. Solving stabilize-avoic optimal control via epigraph form and deep reinforcement learning. In Proceedings of Robotics: Science and Systems, 2023.
Oswin So, Zachary Serlin, Makai Mann, Jake Gonzales, Kwesi Rutledge, Nicholas Roy, and Chuchu Fan. How to train your neural control barrier function: Learning safety filters for complex input-constrained systems. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pp. 1 11532-1 1139.. IEEE, 2024.
Mohit Srinivasan, Amogh Dabholkar, Samuel Coogan, and Patricio Vela. Synthesis of control barrier functions using supervised machine learning approach. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7139-7145. IEEE, 2020.
Ben Tearle, Kim P Wabersich, Andrea Carron, and Melanie N Zeilinger. A predictive safety filter for learning-based racing control. IEEE Robotics and Automation Letters, 6(4))7635-7642, 2021.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference Learning Representations, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10. 109//R0/ 20W.TROO...2...38866 2012.6386109.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710. 2017.
Li Wang, Aaron D Ames, and Magnus Egerstedt. Safety barrier certificates for collisions-fret multirobot systems. IEEE Transactions on Robotics, 33(3):661-674, 2017.
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine learning, 8:229-255 226,, 1992.
Tong Wu, Pan Zhou, Kai Liu, Yali Yuan, Xiumin Wang, Huawei Huang, and Dapeng Oliver Wu. Multi-agent deep reinforeement learning for urban traffic light control in vehicular networks. IEEE Transactions on Vehicular Technology, 6948))8243-8256 2020.
Wei Xiao and Calin Belta. Control barrier functions for systems with high relative degree. In 2019 IEEE 58th conference decision and control (CDC), pp. 474-479. IEEF, 2019.
14

---

Published as a conference paper at ICLR 2025
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning. pp- 11480-11499 1491. PMLR, 2021.
Xiangru Xu, Paulo Tabuada, Jessy W Grizzle, and Aaron D Ames. Robustness of control barrier functions for safety critical control. IFAC-PupersOnLinee 48(27):54-61, 2015.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-24624 2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33:5824 2020.
Mario Zanon and Sébastien Gros. Safe reinforcement learning using robust mpc. IEEE Transactions on Automatic Control, 66(8):3638-3652, 2020.
Songyuan Zhang, Kunal Garg, and Chuchu Fan. Neural graph control barrier functions guided distributed collision-avoidance multi-agent control. In Conference on Robot Learning, pp- 2373-2392. PMLR, 2023.
Songyuan Zhang. Oswin So, Kunal Garg, and Chuchu Fan. Gebf+: A neural graph control barrier function framework for distributed safe multiagent control. IEEE Transactions on Robotics, 41: 1533-1552, 2025.
Youpeng Zhao, Yaodong Yang. Zhenbo Lu, Wengang Zhou, and Houqiang Li. Multi-agent first order constrained optimization in policy space. Advances in Neural Information Processing Systems, 36, 2024.
Vrushabh Zinage, Abhishek Jha, Rohan Chandra, and Efstathios Bakolas. Decentralized safe and scalable multi-agent control under limited actuation. arXiv preprint 20X.0:240,.09573, 2024.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
A.1 PROOF OF THEOREM 2
Proof. Dynamic programming on Vho gives
(x) = max (M(x), (f(((((,
(21)
Let (x)) S O. This gives cases depending on which argument in the max is larger.
Case 1: h(x) I (f(, Here, we have that (z)( = VRn (f ((()), which implies
VM"([(s, - (hr())= -o (x)).
(22)
Case 2: h(x)= > (f*(((z) Here, we have that (z) = h(z)) > u(x))),
which implies
(f=(((z,)(z))) ( -00 (z))
(23)
Thus, Vhu (f ((x))) VhP((x) (((A (OifVh S 0, and Vh. is DCBE
A.2 PROOF THAT EQUATION
(1bb IMPLIES C u) S ALMOST SURELY
Theorem A1. Suppose
Expupo Buwno(()))) [max (0, ((m) (x, u)} so.
(24)
=06"²(x) (x)
Then, ((m) (x, u) almost surely.
Proof. Note that (24 can only be satisfied when the expectation equals 0 since max(0, : is non-negative.
Assume for contradiction that P(C(m) u) 0) < for 0, i.., P(cim) u) > 0) 2 Then,
0 Ex-p To Eunrr(
max f0, ((m) (x, u)}
(25)
P ((((m) (x, u)
Ex-p unmool max {0, ((m)) (x, 4)}
>0]
(26)
(-12 = =e Ex ,
max f0, (m) (x, u}} (m)) (x, u) > 0)
(27)
> 0.
(28)
which is contradiction. Thus, P(C(m) (x, u) S 0) and C(i) (x, u) S 0 almost surely.
A.3 FORMAL STATEMENT AND PROOF OF INFORMAL THEOREM 3
We first formally state Informal Theorem 3 below.
Theorem A2 (Approximate Gradient Projection for Decoupled Policy Parameters). Suppose that for all * X2, the parameters 0 of the stochastic policy No orthogonal, i.e., (V x1)) (VoTo(u2 x2)) = 0 for all u1, u2 E U (e.g., a finite state-space X with independent distribution at each state). Let (m) = VaBxx~p ( denote the gradient of the m-th DCBF constraint violation for any state distribution p. Then, the gradient of the objective lla, modified with an extra indicator as follows:
= Ex
(max..
(ilm) (x))00) E Eu-
Vo log (X, u. QT (TT w)]
(29)
satisfies g (m) 0 Vm, i.e., it lies in the orthogonal complement of the constraint gradients o (m))
16

---

Published as a conference paper at ICLR 2025
Proof. For E X, define the set 2x as the column space of the gradient of the policy π at i...
x == span (u x) : u EZ).
(30)
By assumption, this implies that for x1 # X2.
01 E xx 02 Ox2 = 02 =0. 0.
(31)
Now, note that
o (m))
Vo Pxpup (CB (c) (x)
(32)
(m) V BBBxpBurraa (1x) |mx (0, Com)) (x, u))]
(33)
Exxp B BuNto
V log T x) max (0,, c. (xx wi)]
(34)
Excelle Dec
1 (x,)s0
(x,u)>0} Ve Tu x) (x, w)]
(35)
(mm)
x E X ess sup Com)
(x, u) 0).
(36)
uell
(im(mn)
where we have used the score function gradient estimator of Co (Appendix A.4 Similarly,
g Ex (-)b)()))))
Ve log (u
x) fmaxm 6gth (x) Q To (x, u)]
(37)
To (x1x
V log ( x)1 [max,, ess super Cp (x)50) QTH (x, u)]
(38)
Ox
(39)
x.F
where
max ess ess CiTH (x, (xx,u) 50 F x
E X
(40)
m uel
ess sup cfT² (x, u) S 0, Vm
x XEX
(41)
uEld
Since ((m) nF= = 0 for all m, we have that o g 0 for all m
A.4 SCORE FUNCTION GRADIENT ESTIMATOR OF Co Proof. Using the log trick,
V VolFummo()) [C(x, f(x u))] [() Vo / C(x, [xx u)) To(x, u) du,
/ C(x, f u)) (Velog To (x, u)) (e(x, u) du,
(42)
En-ro(>)) Io C(x, f(x u))].
A.5 FORMAL STATEMENT AND PROOF OF INFORMAL THEOREM 4
We first formally state Informal Theorem 4 below.
Theorem A3 (Satisfying (66) during neighborhood changes). Suppose that the maximum distance an agent can travel in a time step is d. Let i be of the following form.
B(Ox(x)) =
Σ w(or)))(Og)) Ea(o²)
(43)
JEN.
where & RRI R R. & Oo - Ra, and E. O, + R encode the observations into some feature space, and w : O₂ + R is weighting function such that
w(or)) 0, for all Vi, xy such that lpp P9l 2 R - 2d.
(44)
17

---

Published as a conference paper at ICLR 2025
1) B satisfies 116 for all transitions where the neighborhood does not change, and 2) for any transition x + x with a neighborhood change Ni(x) # (xxx)),, there exists a transition + X where all agents that either enter or leave the neighborhood (i.e., the complement of (i(x) Nx(xxt) are moved outside the sensing radius, and all the remaining agents move identically in x and X, then B is DGCBE.
Proof. Let x and xt be consecutive states such that the neighborhood of agent changes, i.c. (x) # Mx(+).. Let E = \ (M(x+)) and F = =x+))) (x(x denote the set of agents that leave and enter the neighborhood of agent i, respectively. Since the maximum distance agents can travel in one timestep is d, the distance between agents timestep. Hence, all agents exiting the neighborhood are at least R - 2d away agent i, i.c.,
change by at most 2d in one
j E Ipp: pyll > R 2d.
(45)
Similarly, all agents entering the neighborhood are at least R 2d away from agent i at the xt, i.e.,
jE F = ll 13 || 2 R 2d.
(46)
Hence, by (4). we have that w(o;)) = 0 for all j E E and w( 0 for all j E F.
Now, by asumption, there exists consecutive states x and x with the same neighborhood (x) M(**) (x) n (x(*x) ), such that I, = 2j and Ej = z for j E (Nx(),, and similarly for non-agent states y V. y+ gt. By definition of the observation function O,, x and X share the same observation except for the ot for j E E, and similarly for x and X for j E F. Since Way = 0 for all j E E' U F, the form of B 43 implies that
w(on))a((on) JEN:
Σ
Σ
(((0;)))(00,))
(47)
jEN,
Σ w(0,)))2(00)) JEM
Σ
((0,))<2(0))
(48)
jEN;
Hence, we must have that B(O,(x)) ((),())), B(O,(x*)) B(O,(x*))
(49)
By assumption, B satisfies the DGCBF condition does not change, which includes x -> xt. Hence,
(16) for all transitions where the neighborhood
B(ot) B(o:) + a (B(o:) -(o))) B(o.) + a +a(Baa(a) (B(a))
(50)
<0,
(51)
and B also satisfies the DGCBF condition (6) for transitions where the neighborhood changes. Thus, B is DGCBE.
A.6 PROOF THAT A DGCBF CAN BE USED TO CONSTRUCT DCBF
Theorem A4. For N-agent MAS, define B X + R as
B(x) max B(Ox(x)).
(52)
Then, B is DCBE
Proof. Let μ O - denote the per-agent control policy corresponding to the DGCBF / in Defi-
nition and let μu denote the resulting joint control policy. Then, under μ,, (I6) implies that for any
Blo (xx) (x). a a(B(o.(x)) 0, Vx e X.
(53)
Now, for a given x E X, let x f(x, (x))) denote the next state following μ.. Let i and in denote the index that maximizes 52 at x and x respectively. i.c.,
i arg min B(O,(x))
B(x) ((O,,(x)))
(54)
' == arg min B(O,(xt))
= B(xt) B(0,,
(55)
(56)
18

---

Published as a conference paper at ICLR 2025
Then, using 53 and the fact that (1 a) is also an extended class-k function and thus is
monotonic function (Ahmadi et al. 2019):
0 2 ((O,, (x);, (x)) {(((O()())))))))))))))))))))
(57)
((0,,(xt)) -1 a) (x)), 2 (x)),(xt)))
(58)
a) B(O, (x)),
(59)
B(xx) -11 a1 B(x),
(60)
B(xx) (xx) +(((x))
(61)
and 4 holds. Thus, B is DCBE
Since B enables the construction of a DCBF, this implies that C. the zero sub-level of B, i.c.,
C = xx (x) s 0}
n {x1 B(a((x) s 00,
(62)
is forward-invariant under μ and hence control invariant.
A.7 DGCBF HAS GENERALIZABLE SAFETY GUARANTEES
In this subsection, we prove that the same DGCBF B from Definition can guarantee the safety of MAS with any number of agents N. We will do this by showing that there exists an N such that if B satisfies the DGCBF conditions (16) for N agents, then the same DGCBF B will also satisfy the DGCBF conditions (I6).
Let the maximum distance an agent can travel in a time step d. Given sensing radius R. define N as the maximum number of agents that can be located within a ball of radius 2R + 2d. For N-agent MAS, let Xx v and U v denote the joint state and control space respectively, and let fN denote the corresponding dynamics function. For generalizability to an arbitrary number of agents, we make the additional assumption that the dynamics fN are decoupled for each agent, i.e.,
k++ fo(æ", )²
(63)
for per-agent dynamics function fo-
For state N,, let X E X denote the restriction of to that of agent and its N closest neighbors. We then have the following theorem.
Theorem AS. Let B satisfy the DGCBF conditions 16 for Ñ agents, i.e., there exists class-K function a with ax-r) > -r for all r > 0 and a control policy μ + U satisfying
B(O, 00, Vj B(Ot (&)) B(O,())) a(B(O,())) < 0, V Vi, (64)
Then, B also satisfies the DGCBF conditions 16 for any N > Ñ agents, i.e.,
((),(x)) S vj = (x)+ -((.()))) a (x)) S 0, Vx Vi,
(65)
We will prove Theorem A5 by showing that (65) holds because it can be reduced to the case of 64
For convenience, define B in Theorem A4 so that
(x)) S Vj
B(x) S.
(66)
Before we prove Theorem
A5
first prove few helpful lemmas.
Lemma (Restriction leaves observations of all agents within R 2d invariant). For any X E X N such that B(x) S O, the restriction of x that of agent and its N 1 closest neighbors leaves the observation of agent i and all agent within R + 2d of invariant, i.e., for any i,
|p pall < R+ 2d
0,(x) 0,
(67)
Proof. Since B(x) < 0, by definition of N there can be no more than N agents within radius of 2R + d of agent Hence, x, will include all agents within radius of 2R + d of agent
19

---

Published as a conference paper at ICLR 2025
By definition of the observation function O₂
Ox(x) = ( o)
(68)
(x) depends only on all agents that are at most R away from j. Since x and x,,N agree on all agents up to 2R + 2d away from this implies that for all j that are R + 2d away from i,
Nz(x)= = (x,,x))
(69)
thus the observation is unchanged as well.
Next, we show that the observation of the next state for agent is also left unchanged after restriction. Lemma 2 (Restriction leaves the next observation for agent unchanged). For any x N such that (xx) S 0, let xt = fN((, u(x)) denote the next state under for x, and (xx,)))
x+ = (x the next state under μr starting from x, Then, for any i,
0,(x*) - (x**)
(70)
Proof. We will prove this by showing that the states of all neighbors (x agree. For each new neighbor E (N((x)), since each agent can travel at most d in one step, this implies that
|p, Pa - R + 2d.
(71)
Applying Lemma 1 gives that agent j's observation is equal in both cases, i.e.,
O,(x) = (x,x,))
(72)
Hence, the controls (x), and thus the new states match, i.e.,
xJ_ fo( (x)((x))) = Sulæ,, (0 (x,,)))) .5.
(73)
Since the new states for all agents (x(x)) agree, this implies that the new observation for agent i must also agree.
We are now ready to prove Theorem
A5
Proof of Theorem A5 Let X' such that B(x) S 0. Then, applying Lemma and Lemma implies that the current and next observations for agent i remain unchanged even when considering only the N closest agents, i.e.,
0,(x,,)) = 0x(x),
(74)
0 (x,,x) Ox)x)
(75)
Hence, taking ă = X E X and using (64
B(O+(x)) ((O,(x)) +
B(O +
(76)
B(O (x)) +(0,(()))+(((()))))))
(77)
<0. O.
(78)
Theorem A5 implies that finding a single DGCBF B that satisfies the DGCBF condition for N
agents enables the same DGCBF also be applied to larger numbers of agents > N.
A.8 PROOF OF COROLLARY I
Proof. Since VM (O vhlmm 14 (xx) max (m)) applying Theorem gives us that
palm "² (oot) (m ((:) + (pnlm M(:))
(79)
/. (xxt) y.tim), ((x) (V.lim) 1 (x))
(80)
< 0. Thus, mm e is DGCBE
(81)
n
20

---

Published as a conference paper at ICLR 2025
B POLICY Loss DETAILS
B. DERIVATION OF [44]
We first start from
which we repeat below for convenience.
min
uoc-po.u-tall ( (x, u)],
(82a)
s.1. Tx (u)))()))) max {0, (m)) (0,0mm((x,,)) u)}
$ 50, Vm.
(82b)
=xXm)(xxu)
5(m) (xc)
In the above, we additionally define ((m) (x) == max (0, C(m) u)} for convenience.
Borrowing the ideas of gradient projection from multi-objective optimization (Yu al., 2020 Liu
al., 202 we combine the gradient from the objective minimization (82a), and the gradient from constraint satisfaction (82b) (equivalent to constraint minimization due to the constraints
being non-negative) by projecting the gradient of (82a) such that it is orthogonal to the gradient of
all m constraints (2b). We can do this in a single backward pass by using Informal Theorem 3
Let to denote the stop gradient function, such that the gradient of e is equal to zero. Then, for state distribution p.
a = E Exxup Vg max C (fm)
(x)
(83)
Exvp (uw))())
Vo log (x, u) max (cm) (x, u)
ux.
(84)
(x) (xxp(
Vo log (x u) max ((m) (x, u)
(85)
V (o) R (
[log (x u) max ((m) (x, u)
(86)
m
where the second line uses the score function gradient (Appendix A.4 and
9 = ExxppT {max,. (6)))( E Funns (-1x) (Vo log To u) (x,
(87)
= (xx
(maxx. (6m) (um V log To x
(x, u))
(88)
= -VBBxwy(T0)
1 (maxm. (fm) (x)500 E. Eu- x)) [log To (x, u) u6 ( (x,
(89)
where the second line follows from the policy gradient theorem. Let == vo + g. where we take the p in the definition of ((m) to be equal to pTS (we discuss the implications of this in the next subsection Appendix B.2):
Theorem A6. The combined gradient ğ can be computed as the gradient of the loss function T, defined in 74). ie.,
ğ Vo).
(90)
where
L(0) = Ex Bume(ro(()))) log Tx u) v ( u, 0))],
(91)
Q (x, u),
(6 (x) < 0, Vm,
i(x, 0)
(92)
v Lumax (1m)) (x, u), m
otherwise.
1 mmax (x)50) (Q T0 (x, u)) + max (t()) (x, u).
(93)
21

---

Published as a conference paper at ICLR 2025
Proof. gives
Taking p pa for and summing the expressions for and in (86) and (89) respectively
va g = VV E Ex ((ol(t(() [og To (x, u) max ((m) (x, m)]
m
(94)
V V VoEx~u(pTO) (max... (x)(0))) (mmo()(((x))) [log nTa (x, (Q (x, w))l
Ve (p))) B (umv(to())))) [og (x u) Iymax (b)) (x, u) + (0 (x, u)))
(95)
Vo (0)) (unv((a((11))
log (x, u)yb v max ((m)
(x, u) + Q (x, u
(96)
=Q1x,u,0)
Ve ()x) (ux((ta((1))) [og T u)y/ (Q(x, u, 0)
(97)
In other words, Theorem A6 implies that can be computed using single backward pass.
B.2 IMPLICATIONS OF MINIMIZING THE CONSTRAINT VIOLATION OVER p
Note that key step that allows us to move the gradient operator between the inside (83) and outside (86 for the expression of is our use of th, because this allows this equivalent holding even when we take p = p
Without the vp and taking p p=* we would obtain that 86 is instead equivalent to
V ExEx Euw-To(-1 (x) ((m) (x, u)] = VolF N (om(
max ( (xm) ,u*)
(98)
m
m
QCiTe (xo.11)
Exup= Bus-ra(()p))
Vo log To u) Q (xx (x.u)
(99)
* Ex (xx
V log TT( u) max (mm)
(x, u
(100)
Note that the second line comes from the use of the policy gradient theorem, while the expression in (TOO) comes from the application of the score function gradient Appendix [A.4 Consequently, while 100) will minimize the maximum DCBF violation ((m)) (99) minimizes the sum of future DCBF violations as well.
Another way see this is note that (98 can be viewed as an optimal control problem with cost equal to the DCBF violation. In other words, in (98) the policy will additionally try to move to states where the DCBF violation is small as opposed to changing only the control such that satisfies the DCBF conditions in (100..
Experimental Validation. To validate our intuition above, we conduct experiments in the TRANSPORT2 environment to compare using (98) with (100), and plot the training curves in Figure 7 The results show that using (98) converges much slower cost, which matches the intuition above. Namely, (98) additionally tries to avoid states where the DCBF constraint violation is high, which leads to unnecessary conservatism.
C EXPERIMENTS
C.1 COMPUTATION RESOURCES
The experiments
13th Gen Intel(R) Core(TM) 17-1370KKF CPU with 64GB RAM and an NVIDIA GeForce RTX 4090 GPU. The training time around 12 hours for x 10 steps for DGPPO, 14 hours for Lagr and 10 hours for Penalty.
22

---

Published as a conference paper at ICLR 2025
Transport2
Transport2
1.0
1.5
Safety rale
Cost
1.0
0.5
0.5
0..00
0.5
1.0
1.5
2.0
0.0 0.0
0.5
1.0
1.5
2.0
Step
1e4
Step
le4
Score function gradient (ours)
Policy gradient
Figure 7: In comparison to DGPPO which uses (100). using (98) converge much slower in cost, which matches the intuition that (98) is optimizing for the wrong objective due unnecessarily avoiding states where the DCBF constraint violation is high.
C.2 ENVIRONMENTS
C.2.1 LIDAR ENVIRONMENTS
In the LiDAR environments, we assume that the agents have radius of r 0.05 and local sensing radius R = 0.5 such that one agent can observe other agents or obstacles only when they are within its sensing radius. Agents use LiDAR detect obstacles. For each agent, there are 32 evenly-spaced LiDAR rays originating from each agent measuring the relative location of obstacles. To reduce the size of the multi-agent graph, the 8 shortest LiDAR rays are returned.
We use directed graphs G = (V,E) to represent the LiDAR environments. V is the set of nodes containing the objects in the environments, including agents Va, goals/landmarks Vg, and the hitting points of LiDAR rays (obstacles) Vo. The edges E C {(i.)) i Va.j V} denote the information flow from a sender node j to a receiver node (agent) i. An edge (i.j) exists only if the distance between node and j smaller than the sensing radius R. The neighbor nodes of agent i defined as N₁ == {j (i,j) E E}. so that the information flow happens between the agents and their neighbors. The node features v; include the state of the node æ; and one-hot encoding of the type of the node (e.g., agent, goal/landmark, LiDAR hitting points). The edge features ei include the information passed from node node including the relative positions and velocities.
In all LiDAR environments, we include rectangle-shaped obstacles, and the agents need to avoid inter-agent collision and agent-obstacle collisions.
We consider 4 LiDAR environments: TARGET, SPREAD, LINE, and BICYCLE:
TARGET:
The agents need reach their pre-assigned goals (Figure 2a
SPREAD: The agents need to collectively cover a set of goals without having access to an assignment (Figure 2b).
LINE: The agents need to form line between two given landmarks (Figure 2c
BICYCLE: The agents follow the more difficult bicycle dynamics. The task here is the same as TARGET (Figure 2d).
The agents in the first 3 environments follow the double integrator dynamics. The state of agent i 3, [yč P., of, e.' where "pf.p/ Pe IR2 is the position of agent i, and [o] of] is its velocity. The control inputs u, = Ja a.l which are the acceleration along the x-axis and y-axis. The agents follow the dynamics
TT
i [ of of a!]T.
(101)
We limit the control inputs of the agents to be within 1, 1], and also the velocities to be within [-10, 10). In the BICYCLE environment, the state of agent is defined with
23

---

Published as a conference paper at ICLR 2025
r, [p] pK. cos 8, sin 0, u| where A is the heading and is the speed. The control inputs [6, a] including the steering angle ó and the acceleration The agents follow the bicycle dynamics given by
l0 cos 0 v usin 0 - sin O tan å v cos 0 tan a)
(102)
The control inputs are limited by [-10, 10) and [-1.47, we use a simulation time step of 0.03 seconds and a total horizon of 128 time steps. For all LiDAR environments, the edge features are defined as the relative positions and relative velocities between the nodes.
all LiDAR environments,
In LiDAR environments, the constraint function h contains two parts: agent-agent collisions and agent-obstacle collisions. The agent-agent collisions ((2) function is defined as
1 2r
min Ilpe Py1l:
(103)
JEM;
and the agent-obstacle collision (2) function is
(o.) (
min |pp JEN;
Pyll-
(104)
For the cost functions we consider two types of them. The first type is used in the TARGET and the BICYCLE environments, where the agents need to reach their pre-assigned goals. We define this type of cost function with
1(x, u) N Σ
(0.01 - - 0.001sign (ReLU((Ip - -0.01) 0.0001 (u:
(105)
where the first term penalizes the agents if they cannot reach the goal, the second term penalizes the agents if they cannot reach the goal exactly, and the third term encourages small controls. The second type of the cost functions is used in the SPREAD and the LINE environments, where the agents need cover some goals/landmarks. We define this type of cost function with
[(x, u)
1 N
Σ min (0.01 pp: D5 1 || 0.001sign [ReLU(II - -p, -1500 -
0.01))
|000001||u,11
(106)
Here, each goal finds its nearest agent and penalizes the whole team with the distance between them. In this way, the optimal policy of the agents is to cover all goals collaboratively.
C.2.2 MuJoCo ENVIRONMENTS
For the TRANSPORT environment, model the agents as double integrators and control the forces applied to each agent using the MuJoCo simulator (Todorov et al., 2012). We limit the control inputs of the agents be within [-1, The agents need collaboratively push box from the inside so that the box reaches given goal while avoiding colliding with each other.
We use similar cost function 1(x, u) as the TARGET environment but only for the box, i.e.,
1(x, u) 0.01 Poox gool + 0.001sign (ReLU(]]
-poeel 0.01))
(107)
We single constraint function (2))) for the agent-agent collision, defined h (o:) 2r min
||p. Pyll,
(108)
jeN,
where pt denotes the position of agent i, and is the radius of the agent.
C.2.3 VMAS ENVIRONMENTS
We use the REVERSETRANSPORE (which we call TRANSPORT2) and WHEEL environments from VMAS Bettini et al. 2022 2024). Note that the obstacles the VMAS environments are not represented using LiDAR but using states including their positions and sizes. In both environments, the agents are modeled as double integrators and control the forces applied to each agent. We limit the control inputs of the agents to be within 11,
24

---

Published as a conference paper at ICLR 2025
In this environment, N agents are placed in square red package and must push the package from the inside to a goal location. The red package and the goal location are randomly sampled. Unlike the original REVERSETRANSPON environment, we include the following two additional safety constraints:
TRANSPORT2:
Inter-agent collision avoidance: The agents must avoid colliding with each other.
Package collision avoidance: The center of the package must avoid colliding with three randomly placed circular obstacles.
We similar cost function (xx, u) as the TARGET environment but only for the package, i.e.,
(x. u) = 0.01 Packkae
Ppuckage
|| 0.001sign
(ReLU() Ppackage
PPackage - 0.01) (109)
We usee two constraint functions ((), (2) for the agent-agent collision and package-obstacle collisions respectively. The agent-agent collision function h!!) is defined as
(o.) = 2r min Ip: - -plll
(110)
where pj denotes the position of agent and is the radius of the agent. The package-obstacle collision function (2)) is defined as
(2)
(o.)
Tobs
min |Ppackage Ppll- (E(1.)33
(111)
In this environment, N agents must collectively rotate a line anchored to the origin with a large Unlike the original WHEEL environment, modify the goal be target angle that the line must rotate to. We also include the following two additional safety constraints:
WHEEL:
Inter-agent collision avoidance: The agents must avoid colliding with each other.
Line collision avoidance: The angle of the line must stay outside of certain range of angles.
We use the same cost function lxx, u) as the TARGET environment but on the angle of the line, i.e..
N 1(x, u)
Σ (0.01Ilp:-7) - -p:00 + 0.001sign [ReLU([pp: - p. soal - 0.01)) + /0,01//u..1)) N
(112)
We usee two constraint functions 1()), h2(2) for the agent-agent collision and line-obstacle collisions respectively. The agent-agent collision function (()) is defined as
(o.) = 2r min ||p: - -pyll- jEN.
(113)
where denotes the position of agent i, and r is the radius of the agent. The package-obstacle collision function (2) is defined as
( Tobs |0
Oabs
(114)
where the absolute value on the angle |-| is defined as the minimum angle between the two angles.
C.3 IMPLEMENT DETAILS AND HYPERPARAMETERS
In our experiment, we let all agents share the same parameters for their policies and value functions. More specifically, we parameterize the agent's 's policy with Ty - Ud cost-value function with
Vg X + I,, and constraint-valuc function V (m) O + 1 using graph transformers (Shi et al, 2020) with parameters 0, é, and v/, respectively. Since the cost-value function V is centralized, after the graph transformer, compute the average of all node features and pass that final layer (multi-layer perceptron) to obtain the global cost value for the MAS.
In Table I we provide the value of the common hyperparameters for DGPPO and the baselines. Besides these common hyperparameters, the value of the unique hyperparameters of DGPPO are provided in Table
25

---

Published as a conference paper at ICLR 2025
Table Common hyperparameters of DGPPO and the baselines.
Hyperparameter
Value
Hyperparameter
Value
policy GNN layers
2
RNN type
GRU
massage passing dimension
32
RNN data chunk length
16
GNN output dimension
64
RNN layers
number of attention heads
3
number of sampling environments
128
activation functions
ReLU
gradient clip norm
GNN head layers optimizer
(32, 32)
entropy coefficient
0.01
Adam
GAE
0.95
discount 7
0.99
clip
0.25
policy learning rate V learning rate network initialization
3e-4
PPO epoch
le-3
batch size
16384
Orthogonal 2
layer normalization
True
v GNN layers
Table 2: Unique hyperparameters of DGPPO
Hyperparameter
Value
V GNN layers
a
0.3
Scheduled. Initialized 1, then doubled at 0.5 and 0.75 of the total update steps.
C.4 IMPLEMENTATION OF THE BASELINES
We implement a JAX (Bradbury et al. 2018) version of the baselines following their original implementations:
InforMARL: htt:://githbbbbcmmmmsss/////nnhn (MIT license)
-MAPPOL:: License)
(MIT
C.5
TRAINING CURVES
Here, we provide the cost and safety rate during training for all algorithms in Figure g We can observe that DGPPO achieves stable training in all environments with only one constant set of hyperparameters.
C.6 ADDITIONAL ABLATION STUDIES
C. COMPARISON BETWEEN THE DECOUPLING METHOD AND THE COUPLED METHODS
we have introduced decoupling method performance gradient descent update following Equation (14). Here, we empirically compare the decoupling method (14) with the CRPO-style coupling method (12) and the CRPO-style coupling method with the constraint being C(r, 21) instead of max C(x, (Equation (10).. We conduct experiments in the TRANSPORT2 environment and compare the safety rate and cost of the converged policies with different methods in Figure 9 We can observe that DGPPO achieves much lower cost compared with the coupling methods. This because the coupling methods perform gradient descent to minimize TT only when the whole trajectory is safe, i.e., Ex~g [Cs (x)] < O. This is much more conservative than the safety requirement on the per-transition level used by the decoupling method, especially the multi-agent case. Therefore, the coupling method has little chance to minimize QT7 during training but focuses safety, resulting in safe policy with poor performance. On the other hand, the coupling method performed with C(r, u) instead of max(0, C(z,u) (Equation (10). the learned policy is longer safe. This matches our discussion in Section 4.3
In Section 4.3
26

---

Published as a conference paper at ICLR 2025
Target
Spread
Line
Bicycle
1.5
1.00
1.00
1.0
0.75
3 0.6
0.8
0.75
0.5
0.4
0.50
0.25
0.25
0.0
0.00
1.5
0.0
0.00
Step
Step
Stop
1
1.0
Target
1.0
Spread
1ime
Bicycle
0.8
0.8
0.9
0.9
0.6
Sajet ra
0.6
0.08
Saddy
0.4
0.4
0.7
0.7
0.2
0.2
0.5
0.6
6.0
0.0
0.5
1.0
L.5
0.5
0
Step
Step
1e5
Stop
1e5
Stop
1
Trınsport
Transpor2
0.5
0.5
1.5
0.4
0.4
DXPPO (eurs)
3 0.3
00.3
Penulty(0.02)
0.2
0.2
0.5
Penalty(O.))
0.1
0.1
Penalty(0.5)
0.0
1
6.0
Step
lef
0
Step
lef
Schodalo
1.0
Transport
1.0
Wheel
1.0
Transpon2
Lagi())
Lagr5)
0.8
0.8
Safey tile
0.6
Saitey rate
0.8
Safterote
Lagetri
0.4
0.6
0.4
0.2
0.2
0.0
0.4
0.0
Step
Step
Figure 8: Costs and safety rates of DGPPO and the baselines during training.
C.6.2 PROVIDE THE BASELINES WITH MORE DATA
In Section 4.5 we introduced that DGPPO requires sampling with both a stochastic and determin- istic policy. This means that DGPPO requires twice as much data per update step compared with the baselines, although the update steps are the same. Here, we answer the question that what if the baselines provided with double data? We choose the WHEEL environment and select the three best baselines in Figure namely Penalty(O.02)) Penalty(0.1), 1), and Schedule. To double the data, we consider two situations: doubling the batch size and keeping the training steps (similar to DGPPO), and doubling the training steps and keeping the batch size. The results are shown in Figure TO] where the semi-transparee transpanttt colors show the original performance of the baselines, and the transpaannn colors show the performance of baselines after doubling the data. We can observe that after doubling the data, is uncertain how the performance of the baselines changes. For example, Penalty (0.02) performs better than before with doubling the batch size in Figure 10a] but performs worse with dou- bling training steps in Figure 10b On the contrary, DGPPO consistently outperforms all baselines.
C.6.3 COMPARISON WITH CONSTRAINED OPTIMIZATION WITHOUT CBF
As the proposed algorithm DGPPO based on CBF, one natural question to ask is what if we do not learn the CBF but directly perform constrained optimization? To answer the question, we consider
27

---

Published as a conference paper at ICLR 2025
1.0
DGPPO
Safety
0.5
DGPPO (CRPO F[max C)))
DGPPO (CRPO E[C])
0.0
2
Cost
Figure 9: Comparison between DGPPO with the decoupling method and the CRPO-style update with E[max (0, C}} and with B[C].
1.000
1.000
Safety rate
+ DGPPO
Safety rate
+ DGPPO
0.975
0.975
Penalty (0.02)
+ Penalty (0.02)
0.950
0.950
+ Penalty (0.1)
+ Penalty (0.1)
0.925
Schedule
0.925
Schedule
0.900
0.900
0.2
0.2
0.4
Cost
Cost
(a) Doubling the batch size.
(b) Doubling the training steps.
Figure 10: Costs and safety rates of DGPPO and three best baselines in the WHEEL environment. We also plot the original performance of the basclines in cemi-transparent colors.
1.0
1.50
0.8
0.6
Cost
Safety
1.00
0.4
0.75
0.50
Step
le4
Step
le4
Learned CBF
No CBF
Figure Comparison between DGPPO with doing constrained optimization without CBE
another baseline which changes the constraint in Equation 11b to Excepro [max (0, ((0)())))))))))))))))))()))))( S while keep all other parts the same DGPPO. We compare DPPO (Learned CBF) with this baseline (No CBF) in the TRANSPORT2 environment. The results are shown in Figure []] which suggests that No CBF cannot achieve safety rate that because CBF not only constrains entering the avoid set, but also constrains the rate at which the agent can approach the safe-unsafe boundary. Therefore, it more robust to estimation errors than directly doing constrained optimization.
high as Learned CBF. Intuitively, is
C.6.4 SENSITIVITY ANALYSIS
In Section 5.3
we show that our proposed scheduling method achieves the best result compared with other choices of v. In particular, Here, we further perform experiments to demonstrate the influence v on DGPPO. We consider the SPREAD environment with 2, 4, 6, and train DGPPO until convergences. The results
2 results in slower convergence in the cumulative cost.
28

---

Published as a conference paper at ICLR 2025
1,0
0.8
0.6
Schodule
0.4
DGPPO
0.2
0.0
2
Step
le5
== 1.0
==
2.0
== 4.0
== 6.0
Figure 12: Influence of different on the convergence speed and the converged result of DGPPO. The dashed lines show the mean of DGPPO and the baseline with the best performance (Schedule), and the shades show the standard deviation.
HalfCheetah 2x3
HalfCheetah 2x3
Coupled [Cheetah 4x3
1.Soupeed HalfCheetah 4x3
1,00
600
400
0.75
Safety rate
0.75
Cost
Cost
400
Safety
0.50
0.50
200
0.25
0.25
0
5
0
5
0
5
Step
Step le3 Penalty(O.1)
Step Schedule Lagr(I)
le3
Step
le3
DGPPO (ours)
Lagr(5) Lagr(Ir)
Penalty(O.02)
Penalty(O.5)
Figure 13: Cost and safety rates of DGPPO and the baselines during training in the Safe multi-agent MuloCo environments.
are shown in Figure where the mean cost and standard deviation of DGPPO and the baseline with the best performance (Schedule) are shown in dashed lines and shades. We can observe that although the convergence DGPPO becomes slower with larger v, the converged costs are the same, and are much lower than Schedule. This phenomenon is different from the baselines, where different choices of hyperparameters directly affect the converged costs (See Figure 3
C.7 ADDITIONAL ENVIRONMENTS IN THE SAFE MULTI-AGENT MuJoCo ENVIRONMENTS
To further demonstrate the ability of DGPPO in environments with complex discrete time dynamics, here we consider another benchmark named safe multi-agent MuloCo (Gu et al. 2023). We use the Safe HAI.CCHETAA(((3) and the Safe COUPLED HALFCHEETAH(4X) tasks, where the agents control different subsets of joints of one or two cheetahs. The first number in the parenthesis denotes the number agents, while the second number shows the number of joints that each agent controls. The agents need to work collaboratively to maximize the forward velocity but avoid colliding with wall that moves forward a predefined velocity. The results are shown in Figure [3] which suggests the same results as the main experiments in Section 5.2 that DGPPO has best performance with fixed set of hyperparameters.
C.8 SCALABILITY AND GENERALIZABILITY
Here we test the scalability and generalizability of DGPPO. Following Zhang et al. (2025). define scalability as the number of agents during training, and generalizability as the ability to be deployed with more agents during test time.
Considering scalability, DGPPO has a similar performance as its based method GCBF+ (Zhang et al. 2025) is trained with in its original paper. In addition to Figure the main pages, here we also provide the comparison of the costs and safety rates of the converged policies of each algorithm in Figure
have shown in Section 5.2 that DGPPO can be trained on 7 agents, and GCBF+
29

---

Published as a conference paper at ICLR 2025
N= 5
1.0
1.0
Safety rate
0.9
0.9
Safety
0.8
0.8
0.2
0.4
0.6
0.2
0.4 Cost
Cost
+ DGPPO (ours)
Penalty(O.1) (enalty(O.5)
Schedule
Lagr(5) Lagr(Ir)
(en2lty(0.022)
+
Lagr(1)
Figure 14: Costs and safety rates of the converged policies of DGPPO and the baselines in environments with N 5,77
Table Generalizability test results of DGPPO.
Number of agents
Safety rate
Normalized cost
1.000 + 0.000
1.673 + 0.430
16
0.992 + 0.088
1.784 + 0.316
0.987 + 0.112
1.748 + 0.235
64
0.986 + 0.118
1.799 + 0.418
128
0.982 0.133
1.839 + 0.323
256
0.985 + 0.122
1.823 + 0.366
512
0.985 + 0.123
1.821 + 0.390
Considering generalizability, GCBF+ can be deployed on 512 agents without significant perfor- mance loss after training. Here, perform experiment in the TARGET environment where DGPPO is also trained with 8 agents. In Table 3 we show the test results of DGPPO deployed with larger numbers of agents. We observe that DGPPO maintains high safety rates and low costs when deployed on up to 512 agents.
However, the above results are obtained environments with the same agent density the training environment. We cannot deploy DGPPO in environments with significantly higher agent density than the training environment because RL algorithms are sensitive distribution shifts. We leave handling large distribution shifts to future work.
C.9 CODE
The code of our algorithm and the baselines provided in the 'dgppozzip' file in the supplementary materials and online at
com
D MORE DISCUSSION ABOUT DGPPO
D.1 ADVANTAGES ON NOT DEPENDING ON A NOMINAL POLICY
As discussed in Section T one of the drawbacks of the CBF-based methods (Wang et al., 2017 Zhang et al.) 2025) is that they require nominal policy that can achieve high task performance. Here we further discuss why relying on a nominal performant policy is not a good idea.
Requirement of Simple or Known Dynamics. Controller design usually requires the dynamics to be simple or requires knowledge of the dynamics. The PID controllers in Zhang et al.) (2025) are constructed for the unicycle dynamics. More generally, PID controllers are usually only used with single-input single-output systems. For more complicated systems, one could use LQR or MPC, but this requires full knowledge of the dynamics. In addition, PID controllers much more difficult to apply in environments with complex contact dynamics, for example, our TRANSPORT, WHEEL, and TRANSPORT2 environments.
30

---

Published as a conference paper at ICLR 2025
GCBE+ Zhang al. 2025
(b) DGPPO
Comparison of the converged policies learned using DGPPO and GCBF+ (Zhang et al. 2025) in TARGET environment, where the agent (blue) needs to avoid the large obstacle (red) and reach the goal (green). The GCBF+ policy is myopic and gets stuck in a deadlock, while the DGPPO policy avoids the obstacle and reaches the goal to minimize the cumulative cost.
Figure 15:
Deadlocks. Another drawback is that the CBF-QP approach of Zhang al. (2025) leads to deadlocks, as discussed in Section VIIII Zhang et al. (2025) or theoretically in e.g. Grover et al. (2021). This is because the safety filter approach of Zhang al. (2025) only minimizes deviations from the nominal policy at the current time step, even if this leads to deadlock at a future time step. contrast, minimizing the cumulative cost directly takes future behavior into account and hence will try to avoid deadlocks. Here we perform an additional experiment to demonstrate this. We apply the converged controller trained with GCBF+ and DGPPO respectively in the TARGET environment shown in Figure 15] where the agent needs to get around a large static obstacle and reach the goal. the figure, we can observe that the GCBF+ policy gets a deadlock behind the obstacle because the GCBF+ policy is myopic and only considers safety safe and minimizes the deviation from the reference controller at the current timestep. Therefore, stopping behind the obstacle is the optimal solution for GCBF+. On the contrary, the DGPPO policy successfully reaches the goal because it minimizes the long- long-horizon cumulative cost.
31


Reviews:
[{"review_id": "9Aikif2nDm", "reviewer": "Reviewer_uB5M", "summary": "The proposed DGPPO framework addresses challenges in multi-agent systems (MAS) by learning both a discrete graph control barrier function (DGCBF) and a high-performance safe policy under unknown discrete-time dynamics, changing neighborhoods, and input constraints. DGPPO combines reinforcement learning and DGCBF, achieving high task performance and safety across varied environments without needing a pre-existing nominal policy or multiple hyperparameter sets, consistently outperforming other methods in both metrics of safety rate vs cost for various simulations.", "questions": "1. Why is the dependency of the algorithm on a nominal policy a bad idea in the given settings? Since it appears easy enough to construct one (say a PID controller like in [1]) for the environments given, is this the right direction?\n2. What is the difference between the training and inference situations in terms of the number of agents? Does the algorithm need to be retrained for every new number of agents unlike in [1] where the algorithm was trained on 8 agents and deployed on up to 1024 agents (albeit being purely concerned with single goal reaching while avoiding collisions)?\n3. With regards to the sample efficiency and computation requirements, how is DGPPO w.r.t. the baselines (I noticed the training time was listed as 12 hours on the reference specifications)? On a related note, how is the benefit of a constant set of hyperparameters demonstrated? Can we confidently say the hyperparameter search for the baselines takes significantly longer (in wall clock time on a comparable machine)?\n4.What are the restrictions on the definition of the avoid set $\\mathcal{A}_i$ and the assumptions on the function $h_i^{(m)}$? Do the avoid sets primarily represent distance to $y^k$ greater than some safe radius?\n5. The LiDAR part of the observation appears less clear. From the appendix (Sec B.2.1) is it right to say that only the LiDAR environments use the 32 equally spaced ray capturing relative positions? How are the obstacles in the VMAS environments represented to the agent?\n6. The experiments with scalability to multiple agents (Fig. 5) appear quite close to the baselines. Is there a better comparison available?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method (DGPPO) is an elegant way to solve the discrete-time distributed MASOCP (multi-agent safe optimal control problem) with unknown environment dynamics. This assumption was not present in previous work which had to differentiate through the given transition functions.\n- The theorems introduced provide a solid foundation for the applicability of DGPPO in the discrete-time setting.", "weaknesses": "- Scalability appears limited (only up to 7 agents) compared to the continuous time setting of GCBF+ [1] (most likely due to the present of unknown environment dynamics and the noise introduced through the sample score function gradient).\n- The stability of DGPPO compared to the baselines does not seem appropriately explained. Is it a purely empirical observation or is there some theoretical justification available?\n- In the given setting, why is the assumption of unknown dynamics interesting? To me, the environments considered are purely the settings of [1] without using the environment dynamics directly (even though they are available). Would it not be a better idea to consider an environment where the dynamics are not as simple as the ones in [1] or some complex unknown function (for e.g., common Mujoco robots)?\n\nReferences:\n\n[1] GCBF+: A Neural Graph Control Barrier Function Framework for Distributed Safe Multi-Agent Control, Zhang et al, T-RO, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhtH3laQut", "reviewer": "Reviewer_XeoF", "summary": "The paper introduces a novel framework, Discrete Graph Control Barrier Functions Proximal Policy Optimization (DGPPO), for ensuring safe control in multi-agent systems (MAS) operating under unknown discrete-time dynamics and input constraints. Unlike prior approaches that rely on continuous-time models and known nominal policies, DGPPO incorporates discrete CBFs (DCBFs) and reinforcement learning to dynamically learn both a high-performance safe policy and a discrete graph CBF (DGCBF). Through extensive empirical validation across various simulated MAS environments, DGPPO claims to achieve both high task performance and safety without the need for hyperparameter adjustments specific to each environment.", "questions": "Q1: How does DGPPO ensure that agents achieve the global objective, rather than just meeting safety constraints?  \n\nQ2: Could DGPPO’s rollout requirements be reduced to improve sample efficiency without compromising safety?  \n\nQ3: What are the practical scalability limits of DGPPO when applied to larger MAS, particularly with the use of GNNs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors present an innovative combination of reinforcement learning and discrete-time CBFs to address the challenges of safety and task performance in MAS with unknown dynamics. The extension to the DCBF framework in discrete-time and the introduction of DGCBFs allow for neighborhood adaptability, overcoming limitations associated with continuous-time CBFs. The approach is well-motivated, tackling safety in unknown environments without requiring a predefined nominal policy—a substantial improvement for multi-agent reinforcement learning (MARL). I particularly appreciate the rigorous theoretical presentation to support the proposed approach.", "weaknesses": "While DGPPO introduces a novel safety mechanism for MAS, nonetheless I believe there are few critical concerns that could limit the effectiveness and general applicability of the approach.\n\n**Lack of clear performance metrics**: \n\nwhile DGPPO is shown to achieve high safety, the empirical results focus primarily on safety metrics and the minimization of constraints. It remains unclear if the agents are successfully accomplishing global objectives. Without metrics like mean global success rate or reward, it is difficult to assess if the agents are merely achieving safety (e.g., by staying stationary) rather than making meaningful progress toward the task goals while satisfying the safety constraints. This is especially relevant as the DGPPO framework does not incorporate a nominal policy, meaning that without these metrics, the experiments risks overlooking cases where the agents avoid unsafe states at the expense of task completion. Does the proposed framework present mechanisms to ensure that safety constraints do not excessively dominate the objective?\n\n**Limited scalability experiments**: \n\nthe authors state that DGPPO scale well w.r.t to the baseline approach tested, however testing 5 agents instead of 3 as in the original experiment, I believe it is too limited to claim scalability of the proposed approach. Crucially, as stated from the authors themselves the proposed approach requires both stochastic and deterministic rollouts to enforce DGCBFs. While this approach ensures safety in discrete-time settings, it also introduces significant sample inefficiency, which may limit the framework’s scalability to larger or more complex MAS. Hence, an extensive test with for instance 10 or 15 agents would strength the results of the paper.\nMoreover while the authors employ GNNs with attention mechanisms to handle changing agent neighborhoods, the computational complexity of GNNs in larger MAS could become important. In high-density environments with frequent neighborhood changes, maintaining an updated and accurate DGCBF through GNNs could pose significant computational challenges, possibly impacting real-time applicability. A detailed discussion on the scalability of GNN-based policies for larger agent systems would add valuable context to the method’s limitations.\n\n**Dependence on hyperparameter $\\nu$ for constraint enforcement**: \n\nthe authors claim on the fact that DGPPO is less sensitive to hyperparameters does not seem to be properly backed up. From the plot in Fig. 6b the value of $\\nu$—responsible for controlling the weight on constraint minimization steps—significantly impacts performance. Misalignment in $\\nu$ could lead to either overly conservative or unsafe policies, showing that DGPPO still requires careful tuning, contrary to its stated hyperparameter robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsqcFytiUP", "reviewer": "Reviewer_LZax", "summary": "This paper proposes a safe multi-agent reinforcement learning method based on distributed control barrier functions (CBFs) for multi-agent systems with limited perception capabilities. Simulation results on several multi-agent safe coordination tasks demonstrate the effectiveness of the proposed method.", "questions": "(1) The policy in this work only takes local observations as input, such that it is decentralized. Why do you refer to it as a distributed policy? \n\n(2) The modified constraint (11b) is too strict which cannot be satisfied by a lot of policy classes, such as the Gaussian policy.\n\n(3) In (12), please provide the explicit gradient formula when the safety condition is violated. Note that the authors provide a gradient in (41). Nevertheless, this gradient is not associated to the policy loss function (under safety violation) in (12).\n\n(4) Theorem 3 is very difficult to understand. The orthogonality assumption is impractical. The reviewers also find that the authors try to replace the stationary state distribution (which is a function associated to the policy parameter $\\theta$) with a constant state distribution in this theorem to obtain their gradient $g$ in (13). What is the purpose of giving Theorem 3? What is the relationship between (13) and (14)? \n\n(5) The reason of using discrete graph CBFs should be explained clearly. Note that we can regard the multi-agent system as a large but single agent. Then, you can directly use the discrete CBF given in Theorem 2 to learn safe policies. In this case, the distributed control nature can still be preserved as the learned observation-based policy is end-to-end.\n\n(6) Theorem 4 is hard to understand. What is the relationship between the discrete graph CBF and the discrete CBF? Similar to Theorem 1, it is important for the authors to show that the safe set is forward invariant based on the discrete graph CBF.\n\n(7) In (11b), the safety constraint is calculated using a stochastic policy $\\pi$. However, in Fig. 1, deterministic policies are used for estimating the discrete graph CBF.\n\n(8) Why do the agents have different GAEs in your algorithm? Are you suggesting that the agents are heterogeneous and that their local policies differ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The multi-agent safe optimal control problem considered in this paper is both general and challenging, as neither the system model nor a nominal policy is available in advance.\n\n(2) The learned policy is safe, which does not require additional safety filters in implementation.\n\n(3) Extensive simulations are conducted, and state-of-the-art baselines are compared.", "weaknesses": "There exist several theoretical issues in the paper. The motivation of employing the discrete graph CBF is unclear. Some implementation details should be incorporated. See Questions part for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)