Title: HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hierarchical RL

Multi-Agent RL

Safe RL

Control Barrier Functions

Safety-Critical Systems
Abstract: We address the problem of safe policy learning in multi-agent safety-critical autonomous systems.
In such systems, it is necessary for each agent to meet the safety requirements at all times while also cooperating with other agents to accomplish the task. Toward this end, we propose a safe Hierarchical Multi-Agent Reinforcement Learning (HMARL) approach based on Control Barrier Functions (CBFs). Our proposed hierarchical approach decomposes the overall reinforcement learning problem into two levels –- learning joint cooperative behavior at the higher level and learning safe individual behavior at the lower or agent
level conditioned on the high-level policy. Specifically, we propose a skill-based HMARL-CBF algorithm in which the higher-level problem involves learning a joint policy over the skills for all the agents and the lower-level problem involves
learning policies to execute the skills safely with CBFs. We validate our approach on challenging environment scenarios whereby a large number of agents have to safely navigate through conflicting road networks. Compared with existing state-of-the-art methods, our approach significantly improves the safety achieving near perfect (within $5\%$) success/safety rate while also improving performance across all the environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the safety of multi-agent reinforcement learning (MARL), while it emphasizes that agents have to meet safety requirements at all times. Based on Control Barrier Functions, it proposes a hierarchical approach where the higher-level problem involves learning a joint policy over the skills and lower-level problem involves learning policies to execute skills. In complex environments with a large number of agents, the method manages to present good performance in experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n  The paper further promotes the safety requirements and promotes corresponding solution, which demonstrates outstanding performance.\n  The proposed method owns good scalability and is able to deal with conditions with large amounts of agents.\n\nWeakness: \n  The paper is a little difficult to understand, for example, it refers to some equations that appear later or even in appendix, which makes the reading not smooth.\n  Similar to above, maybe the content is too substantial so that many processes that help understand the work is put in appendix, and the analysis of experiments is also too brief in the main body of the paper. Even the lidar environment is not mentioned in the main body. I think the structure of the article can be improved.\n  There are also some typos, such as line 140.\n\nQUESTIONS:\n1.As is mentioned in previous works (such as HATRPO and MAFOCOPS), the agents will affect with each other. Although the proposed method in this paper can deal with multiple agents, it seems to mainly consider each single agent. I’m curious whether this will lead to some instability during the training.\n2.As the method relies on the CBF and hierarchical skill learning, which I think both require some prior knowledge. From the paper, I think the authors are very familiar with the experiment environments. But if a new or very complex environment is encountered, how will the method perform?\n3.In the METADRIVE experiments, the chosen baselines do not seem to be algorithms related for safety constraints. How will other methods concerned with safety perform in the environment?\n\nLIMITATIONS:\nThe authors claim to discuss the limitations in both the conclusion and appendix sections but I don't find the discussion. I think the authors should analysis the computation complexity as the proposed method is based on a hierarchical framework, which tends to be more complex. Also, how the method performs if a new and complicated environment is met should as be considered.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HMARL-CBF, a hierarchical framework for cooperative safety-MARL. The approach decomposes the policy into a high-level policy and a low-level policy. The high-level policy is a policy that selects the skill to use, and the low-level policy is a policy that selects the low-level action with safety constraints conditioned on the skill. The high-level policy is trained by standard MARL methods like MAPPO or QMIX, with the environment reward with a safety penalty. The low-level policy is trained by a CBF-based method with a hand-crafted intrinsic reward. The authors evaluate the proposed method on MetaDrive, and the results show that the proposed method can achieve significant better performance than the baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The performance of the proposed method is impressive on MetaDrive.\n- Hierarchical policy is a good approach to handle the large and complex environment of MARL.\n\n### Weakness\n- The writing is not clear and has some typos, e.g.\n    - line 74: [46] is not a multi-agent RL method.\n    - line 138: $s^i, s^j$ seem to be redundant.\n    - line 140: \"hus\" should be \"Thus\".\n    - line 165: \"An skill\" should be \"A skill\".\n    - eq (8), line 182, eq (10)...: The condition should be put outside the reward function and inside the expectation, or the definition of the reward function should be changed.\n    - eq (9): $p_j^i$ is not defined.\n    - eq (9): \"for some $i$\" should be removed.\n    - footnote 1 on page 5: $\\mathbb {I}(x)$ is a binary function by definition, while in this footnote it is claimed to be a large number $M$, which is unclear.\n    - eq (14): Notations like $\\Omega, \\zeta$ are not defined.\n- The novelty to use CBF-based method to train the low-level policy in MARL is limited.\n- The approach seem to require a lot of artificial design, e.g. the hand-crafted intrinsic reward, the predefined skill set, the ad hoc Lyapunov function, etc.\n- The experimental results are impressive, but the chosen tasks are relatively few. Also, some baseline methods are not included in the comparison.\n\nQUESTIONS:\n- It seems that some baselines are not included in Related Work and Experiments sections. Can you provide more comparison with the different methods, including [1,2,3]?\n- Can you provide more experiments on different tasks, including [4] and https://safety-gymnasium.readthedocs.io/en/latest/environments/safe_multi_agent.html.\n- HMARL-CBF employs a lot of task-specific prior knowledge or ad hoc design, such as the known dynamics of the environment, the design of intrinsic rewards and predefined skill sets. Could you please list in detail the task-related prior knowledge used in this paper? Would this make the comparison with the baseline unfair?\n- Could you visualize or at least describe the low-level skills, like what does the skills look like, how long does the skill last, etc.? This would be very helpful for understanding the method presented in this paper.\n\n[1] Berducci L, Yang S, Mangharam R, et al. Learning adaptive safety for multi-agent systems[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 2859-2865.\n\n[2] Zhang S, So O, Black M, et al. Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control[J]. arXiv preprint arXiv:2502.03640, 2025.\n\n[3] Gu S, Kuba J G, Wen M, et al. Multi-agent constrained policy optimisation[J]. arXiv preprint arXiv:2110.02793, 2021.\n\n[4] O’Kelly, M., Zheng, H., Karthik, D. &amp; Mangharam, R.. (2020). F1TENTH: An Open-source Evaluation Environment for Continuous Control and Reinforcement Learning. <i>Proceedings of the NeurIPS 2019 Competition and Demonstration Track</i>, in <i>Proceedings of Machine Learning Research</i> 123:77-89 Available from https://proceedings.mlr.press/v123/o-kelly20a.html.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a hierarchical multi-agent reinforcement learning framework for learning safe policies based on control barrier functions. The control problem is formulated as a partially-observable multi-agent MDP with pointwise-in-time hard constraints based on barrier functions. The proposed framework learns high-level control policies that select skills, considering performance, while the low-level control policies implement skills by selecting actions, considering strict safety constraints (policy learning is based on existing multi- and single-agent algorithms, such as MAPPO and PPO). The proposed framework is evaluated in the MetaDrive simulator with various scenarios, showing that compared to multi-agent baselines the proposed framework provides better satisfaction of safety requirements.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Safety-critical multi-agent control is an important problem.\n* The proposed hierarchical decomposition into high-level and safe low-level policies is promising.\n\nWeaknesses:\n* Some aspects of the proposed framework are not clear. Where do the skills come from? The high-level policy seems to learn to select from a given set of skills, while the low-level policies seem to be learned for specific skill. Where do these skills come from in the first place?\n* The baseline approaches in the numerical experiments do not seem to be designed for safety (they seem to be generic multi-agent RL approaches). Do these really represent the SOTA in safe multi-agent RL?\n* \"While CMDPs ensure safety over trajectories, they do not consider pointwise time constraints critical for safety-critical systems.\"\nIs this really a limitation? One could define (1) a cost function that is non-zero for pointwise time-constraint violation and (2) cost constraint that enforces zero expected cost. Would that not capture pointwise constraints?\n* The description of termination strategies on lines 94 to 98 is rather informal, and it is not clear how the $\\tau_{continue}$ strategy is implemented in the model.\n* The notation on line 133 seems to be inconsistent: barrier function $b$ first takes agent-specific states as input, but then the specification of its domain includes the entire state space $\\mathcal{S}$.\n* The definition of the observation space on lines 137 and 138 is strange: the observation space is defined as a superset of the state space, but the environment is partially observable, which should be modeled as some states being indistinguishable to the agent, implying that the observation space is a subset of the state space. But this of course would prevent applying the barrier function $b$ to the observation. Can the barrier function really be evaluated on observations instead of states? This seems to contradict partial observability if the barrier function are general functions of state.\n* On lines 149 to 152, the discussion emphasizes relaxing the assumption of CMDP that policies depend on the joint state. But this does not seem significant, it is just a simple matter of definition.\n* The definition of the indicator function at the bottom of page 5 has some issues. The clause \"for some $i$\" does not make sense since $i$ is given as the running variable of the first summation of Equation (9), hence it makes not sense to refer to \"some $i$\" (also, within the definition of the indicator function the domain of $i$ would be unclear). Footnote 1 is unclear since it suggests that the indicator function would output some large value $M$, but according to the definition, it outputs 0 or 1.\n* Calling the proposed framework \"Safety Guaranteed\" might be misleading since safety is not guaranteed, it is achieved only high probability.\n* It may help readability if the distribution of random variable in the subscript of expectations is specified, e.g., in Equation (1), clarify that $z \\sim \\pi$ and $k, s' \\sim P$.\n\nQUESTIONS:\n* Where do skills come from?\n* Are the baseline methods SOTA in safe RL?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HMARL-CBF, a two-level framework for safe multi-agent reinforcement learning in partially-observable settings with hard safety constraints. At the high level a centralized-training decentralized-execution (CTDE) policy selects a joint skill for all agents. These skills are temporally-extended options that encode cooperative behaviour. At the low level each agent executes its chosen skill through a quadratic-program (QP) controller that embeds control-barrier-function (CBF) constraints, guaranteeing point-wise-in-time safety. Experiments in the MetaDrive traffic simulator show near-perfect collision-free success rates (≥ 95 %) and substantially shorter episodes than baselines like IPPO, MFPO, CoPo and a flat MAPPO variant.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n### Quality\nThe paper presents a technically sophisticated integration of hierarchical reinforcement learning with control-barrier-function (CBF) safety filtering. The methodology is sound: a centralized-training, decentralized-execution scheme selects joint skills, while individual agents execute them through CBF-constrained quadratic-program controllers. Empirical results in the continuous-control MetaDrive suite show dramatic gains in collision-free success rates and episode efficiency, and the authors include ablations that dissect the impact of hierarchy and safety filtering.\n\n### Clarity\nThe high-level idea and algorithmic pipeline are conveyed effectively through Figure 1 and Section 5, giving readers a coherent narrative from problem statement to implementation. An extensive appendix provides proofs and additional details, which helps knowledgeable readers follow the derivations and design choices.\n\n### Significance\nAddressing hard safety in cooperative multi-agent RL is an important open problem with clear relevance to autonomous driving and swarm robotics. Demonstrating near-perfect safety without sacrificing task efficiency could influence future research and practical deployments of MARL systems subject to stringent safety constraints.\n\n### Originality\nWhile action-space augmentation via options and CBF-based filtering are themselves established, the paper’s specific decomposition—first selecting a joint high-level skill and then letting agents independently apply CBF-filtered low-level control—appears novel. This bridges two previously separate literatures (hierarchical MARL and safe control) in a way that could inspire further hybrid approaches.\n\n\n# Weaknesses\n\n### Quality\nThe safety guarantees hinge on knowing affine system dynamics (Eq. 3); experiments likewise assume exact vehicle models, leaving robustness to modelling error unexplored. Pre-defined skills are hand-crafted and provided only to the proposed method, so performance gains may stem from privileged prior knowledge. Key baselines such as MAPPO augmented with identical skills and CBF filters are missing. Comparisons are averaged over only five seeds, and no statistical significance tests are reported.\n\n### Clarity\nDespite a clear high-level story, numerous notation and referencing issues impede readability. Symbols like $\\mathcal{U}$ in Eq. 4 are undefined; Eq. (17)/(18) are cited but absent; time-index notation is inconsistent; acronyms such as QP and LQR are never introduced or done after usage; and Section 3.2 offers little intuition for (HO)CBFs. Typos and un-hyperlinked citations and references further detract from polish. Numbers are also generally floating without being specific on whether they correspond to equations/sections/etc. \n\nSome of the notations in Section 4 are also confusing. E.g. The joint state space $\\mathcal{S}$ was defined before in line 87 as just being the joint agent states $\\mathcal{S}^i$. Then in line 127 it is defined as the joint agent states including the environment states $\\mathcal{S}^e$. Hence it is now unclear what $\\mathcal{S}^i \\subseteq \\mathcal{O}^i$ means (line 137) when the authors have that $\\mathcal{O}^i \\subseteq \\mathcal{S}$. Is $\\mathcal{S}^i$ now defined differently? What does $s^i \\in \\mathcal{O}^i$ mean? Are observations and states now the same? Is $b$ a function of observation, a single agent state, the agent joint states, or the joint states with the environment states (since sometimes it is $b(s)$, $b(o)$, $b(s^i,s^j)$)? In general the precise mathematical formulation in this section is confusing and unclear.\n\n### Significance\nThe evaluation is confined to MetaDrive traffic scenarios (and toy navigation tasks in the appendix). Generalisation to higher-dimensional platforms (e.g., MuJoCo robots like the Ant) or to settings with learned dynamics is not demonstrated. Until such breadth is shown (or at least argued theoretically) the broader impact on real-world applications remains uncertain.\n\n### Originality\nBecause both option-based action-space augmentation and CBF safety filtering have established precedents, the contribution’s novelty rests largely on their particular combination and training recipe. The paper could better differentiate itself from prior safe-HRL works to underscore what is truly unique.\n\nQUESTIONS:\n- Did you give baselines access to the same pre-trained skill library (e.g. by augmenting the action space with them) and CBF safety filter? If not, please justify. A MAPPO or IPPO baseline equipped with the identical skill set would isolate the contribution of the hierarchical credit assignment itself. This my main concern from this work.\n\n- Section 5.2 relies on known dynamics for the QP. What happens when only an approximate or learned model is available? Please include experiments with modelling errors if possible.\n\n- Five seeds are unusually low for high-variance MARL benchmarks. Please report 10–20 seeds with 95 % confidence intervals, otherwise this severely impacts the significance of the claims\n\n- Please give intuitive/informal explanations in Section 3.2. This will greatly improve accessibility for the broader NeurIPS audience.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}