Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a safety mechanism for multi-agent systems using Control Barrier Functions (CBF). The proposed framework trains a graph neural network to learn both the controller and a barrier certificate. The paper provides numerical simulations to assess the scalability of the proposed framework using 2D ground vehicles and 3D drones. 
review: - The paper comes across as a heuristic for safety. It builds a false notion of safety without guaranteeing any actual safety. Calling the trained neural network a Control Barrier Function is misleading since the trained network is not guaranteed to satisfy the CBF conditions (presented in equation (1)).

- The loss function depends on the empirical evaluation around the data points used for training. This raises the question of the effect of the data quality used during training on the correctness of the proposed framework.

- Which assumptions on the robot dynamics/workspace are needed to ensure the safety of the multi-agent system? When can one trust the decisions taken by GCBF to be safe?

- The numerical results section is inconclusive. In particular, the paper lacks to compare against model-based (manually designed) CBF-based methods. The dynamics of the 2D vehicle used in simulation have been widely studied, and manually designed CBFs for multi-agents of these robots have been presented in the literature.

- The limitations section claims that it is hard, if not impossible, to ensure that the proposed method ensures the system's safety. First, this claim was not supported by any evidence. Second, if the claim is true, this points out that the proposed approach may not be the right one to handle a crucial and consequential issue like safety.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In its current format, the paper proposes a heuristic to promote safe control actions during training. The paper does not provide any clear assumptions under which the proposed framework is guaranteed to provide safe control actions. Moreover, the numerical results lack comparison against the model-based design of CBFs. Given the criticality of safety, I do not recommend this paper for inclusion in CoRL 2023.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper addresses the problem of controlling robots to get to their destination without collision, by proposing graph control barrier functions to encode safety constraints and a graph neural network to learn the corresponding control. Experiments are performed in simulation using different kinematic models for the robot, comparing with other two approaches. 
review: The paper addresses a relevant problem in multirobot control and proposes an interesting method with graph neural networks to achieve it within the control barrier function framework. The paper shows a significant increase over two recent methods with a very large number of robots. Also, the limitations section discusses reasonably the main points observed during the experiments, identifying potential future work.

There are some directions of improvement for the paper:
- a popular type of approach for multi-robot collision avoidance is "velocity obstacle", which should be included in the discussion of the related work at least, and as part of the comparison. It would be also interesting to have a comparison with an optimal planning strategy.

- the experiments include idealized kinematic models; it would be good to use a realistic simulator or enrich the current model with noise.

- the statement that planning approaches "do not generalize to new unseen environments." is not clear, as those approaches, given the map of an environment, will calculate the corresponding path -- unseen is typically interpreted as not seen perhaps during the experiments. This point should be clarified and rephrased.

- the appendix could include the proof of Theorem 1.

A couple of typos: derivates -> derivatives; "LiDAR to detach obstacles" -> "... detect ..."
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the paper could be improved on the experimental side, the paper overall shows an interesting method based on GCBF and GNNs that significantly improves the performance in scenarios with a large number of robots. Please see the details in the "Review" box.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes to train a graph Neural Network (GNN) to control mobile robots. Every robot is equipped with a GNN and receives information from its neighbour environment, making the method distributed. A loss function based on Control Barrier Functions (CBF) is used to avoid inter-agent collisions between the robots and collisions of the robots with static obstacles. The GNN formulation makes the proposed method scalable with the size of the swarm and the paper shows that with its method more than 1000 agents can be controlled although the network was trained on only 16 agents.
review: ### Strengths
S1) Interesting deep learning approach combining multiple different methods like CBFs, GNNs and attention mechanism.

S2) Highly scalable.

S3) Training can be done in a relatively short amount of time (only 1 hour).

### Weaknesses
W1) The paper states that its controller is safe, although it has no safety guarantees (as stated by the paper itself in the limitations).

W2) While the paper derives a new type of CBFs called Graph CBF, the theoretical analysis is incomplete. The proof for the given theorem (if the policy satisfies CBF properties, then there are no collisions) is skipped and there is no further analysis (like existence results).

W3) It is not clear how the method in this paper compares to classical non-learning methods. 

W4) The paper mentions a policy refinement strategy in their method, but does not investigate it further in the experiments section.

### Evaluation
The proposed learning method is interesting and the experimental results are impressive. 
However, the paper uses the word safe in title, abstract and main text although the proposed method is not inherently safe, which the paper itself states in the limitation section.
The wording in the paper should be changed to account for this. 
It is also not clear how the method in this paper compares to classical non-learning methods, which should be addressed.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the paper is written well and the method is applicable for huge swarms, its methods lacks of a theoretical analysis. The statements in the paper about the method being safe should be reformulated to account for this. Furthermore, see main strengths and weaknesses as listed above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes to use Graph Neural Networks (GNNs) to learn a safety score and safe action policy. The main idea is to formulate the safety score as a control barrier function and use it to switch between a nominal and a learned-based controller. 
review: Strengths:

Overall the paper is well-written and well-motivated. The provided code seems to implement exactly the proposed method. 

Weaknesses:

1- The contribution seems incremental wrt to the reference [20] of the paper. The main difference wrt [20] is that this article proposes using the GNN to learn the safe action policy and use the learned control barrier function as a switching policy between the nominal and learned safe controller. I would recommend the authors revise their claimed contributions.

2 - There are only simulation results. Adding real robot experiments would increase this paper's contribution.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well-written and technically sound. The major weakness is the contribution which seems to be minor and incremental wrt to reference [20] in the paper. Moreover, there are no real robotics experiments that could be a major contribution wrt the literature.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a novel approach called graph control barrier functions (GCBFs) for addressing the challenges of distributed safe multi-agent control in large-scale environments with moving obstacles. The method combines control barrier functions for safe with a graph structure for scalable and generalizable decentralized control. The authors utilize graph neural networks to learn both the neural GCBF and the distributed control policy. They further extend the framework to handle LiDAR-based point cloud inputs, allowing for obstacle detection and thus making it more applicable to real-world robotics settings. The empirical results demonstrate the superiority of the presented approach over existing methods on $2$d navigation tasks, achieving significantly higher safety and goal-reaching rates when increasing numbers of agents during inference. The proposed approach is completely distributed and scalable, making it suitable for large-scale multi-agent systems.
review: ### Strengths

The paper demonstrates several notable strengths:

- Synergy of control and machine learning: The integration of control barrier functions (CBFs) with graph neural networks (GNNs) combines control theory with machine learning. This combination is both intriguing and promising for safe multi-agent control.
- Relevance of the topic: The problem of distributed safe multi-agent control in large-scale environments with moving obstacles is highly relevant in the context of multi-agent systems and has applications in a plethora of real-world problems.
- Clear and well-written: The paper stands out for its clarity and readability. The inclusion of clear figures and comprehensive descriptions aids in understanding the proposed approach and the experimental results.
- Code availability and experimental details: The authors provde code and extensive experimental details in the supplementary material. This transparency enables reproducibility and facilitates further exploration and validation of the proposed method.
- Convincing experiments against strong baselines: The paper presents persuasive experimental results by comparing the proposed approach against strong baseline methods like MAPPO and multi-agent distributed CBF (MDCBF). This empirical evidence showcases the superior performance and effectiveness of the presented approach in a variety of scenarios.

Overall, these strengths contribute to the significance and impact of the paper, establishing its merit within the field of multi-agent systems and machine learning.

### Weaknesses

The weaknesses of the approach are mostly and adequately addressed in the limitations section of the paper. Beyond that, some additional concerns are listed below

- Potential sensitivity to hyperparameters: The paper lacks sufficient justification for the choice of hyperparameters, such as the weights of the losses in the Appendix and the choice of the function $\alpha(h(x))$. Conducting experiments with different parameter choices or explaining the choices that are made would provide insights into the method's behavior and robustness.
- Inconsistent experimental setups: The third experiment, which involves obstacles, differs significantly from the first two experiments. Additionally, the third experiment does not increase the number of agents during inference. Providing a more consistent experimental setup and explaining the rationale behind these differences would improve the clarity of the results.
- The connection between the fixed-size inputs for the mathematical definition on Page 4, and the input to the GNN could be stated more clearly. While this is briefly explained in Footnote 3, it would be appreciated if the authors could clearly distinguish between theory and practical application, and provide a brief explanation for how the two differ.
- Additional visualizations: While videos are available in the supplementary material, incorporating additional "strips" of policies at critical timesteps in the main paper or appendix would enhance the visualization of the approach. Furthermore, including a visualization for the 3D drone task, such as e.g., a 2D image with depth colorization, would be insightful.
- Typos:
    - L. 275: “Detach” → “Detect”
    - L. 94 “assumed to” → “assumed to be”
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is enjoyable to read, demonstrates a strong foundation, and has a convincing experimental section. The availability of code and comprehensive experimental details in the supplementary material further contribute to its strength. A large part of the weaknesses identified can be addressed through revisions and clarifications during the rebuttal and revision process. Unless any major issues are discovered during the rebuttal, I recommend accepting the paper.

Edit: I want to thank the reviewers for the detailed discussion and for clearing up a majority of my concerns. I will remain with my original recommendation for a weak accept, but am now more certain of the contributions of the paper and their significance for CoRL. I raised the confidence from a 3 to a 4 to represent this.
review_title: null

Meta-review: The paper proposes a method for distributed control of multiple agents with graph barrier functions encoding safety constraints and a graph neural network to learn the controller. Experiments were performed in simulation for both 2D and 3D cases. 

The approach is novel and connects geometry deep learning with multiple methods such as CBFs, GNNs and attention mechanisms. It is also highly scalable and fast to train. 

Reviewers were mostly positive about the paper but there were also a few concerns, for example:
- Calling the trained neural network a Control Barrier Function is misleading since the trained network is not guaranteed to satisfy the CBF conditions (presented in equation (1)); which the authors agree with.
-  Comparison against model-based (manually designed) CBF-based methods. 
- The policy refinement strategy in the method is not discussed in the experiments section.

The authors have addressed these and other issues adequately in the rebuttal and provided an improved version of the paper with a different title, clarifying the most critical issue.
Decision: Accept (Poster)