Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper brings the recently introduced idea of control barrier functions (CBF) as safety certificate to the multi-agent land. To this end it introduces the idea of decentralized CBFs. This is followed with a framework for jointy learning the CBFs and policies with a PointNet inspired network architecture. The paper provides some generalization guarantees for the approach as well. Finally the approach is compared against various learning and planning based baselines where it significantly outperforms.

Overall, it's a very interesting extension of the CBF idea for the decentralized setting and permutation+#agents invariant setup seems quite scalable. It's good to have proven generalization guarantees as well.

Although the paper shows impressive performance on a variety of tasks, the learning setup is unclear. Sure you have the loss function in Sec 4.1 with the computational graph in Fig 1. But it's unclear how exacly learning happens? Like how is the data collected; how are the enviornments explored. Do you have to estimate policy gradients or you have exact gradients. How exactly do you compare against both MADDPG style model-free method as well as planning based MAFACTEST. Do they even have the exact same environment models? The details are unclear, so it's not possible for me to really judge what's going on in Fig 4. Sure the average rewards and safety rates are higher but are the quantities even commensurable for these very different methods? It's probably yes gven the quantified metrics are scalar quantities which can possibly be measured in a similar manner for the approaches. But still more information is necessary for it be meaningful.
review_title: Interesting approach; experiment details missing

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ## Post-rebuttal
The rebuttal was convincing (see my comment below) and addressed my concerns. I therefore increase my score to 8.

## Overview

This paper introduces MDBC, a method relying on a Control Barrier Function (CBF) learnt jointly with a policy, to obtain a safe and generalizable behavior.
It proceeds to show experiments on 2D and 3D navigation tasks, where the safety is defined as the avoidance of any collision.

The paper is globally well written, and the underlying theoretical foundations seem sound, and experiments show strong results.
However, the experiments leave out a lot of critical implementation details, raising concerns for the reproducibility of the work. Moreover, the baselines used seem inadequate.


## Method

The theory underpinning the proposed method is well exposed and sound.
The only question that I have is with respect to the choice of the neural network architecture. A PointNet is being used, what motivates this choice? Have you considered other alternatives such as Pointer network, Graph neural net, transformers?

## Experiments

Some information is missing to fully reproduce the experiments, including:
* The optimization parameters (optimizer, lr, and possible additions like weight decay and so on)
* The exact loss used for the policy training (the text mentions a "distance", but doesn't specify which
* How the policy loss and the safety loss $\mathcal{L}^c$ are combined (I would expect a weighted sum, but this is not specified as far as I can tell)
* Values of some other hyper-parameters, such as $\gamma$, $\mu$, $\lambda$. Even the non-linear activation $\sigma$ used in the network is unspecified
* The theory section assumes that each agents observe the states of other agents within a neighborhood. In the context of the experiments, this neighborhood is unspecified.


As for the comparison, I find the choice of some of the baselines inadequate. With the exception of MAFACTEST (which is, unexpectedly, not reported on all environments), all 3 others (PIC, MAMPS, and MADDPG) are reward based, while the proposed method uses policy imitation to a target policy (LQR or PID controller), which is a much simpler learning problem, as it sidestep many hard problems in RL such as exploration and credit assignment (both temporal and agent-wise). As such, it is not surprising that RL based approaches struggle more when trying to generalize/scale, especially if the training scenario always contain the same number of agents (better results could possibly obtained with some limited curriculum learning)

It is not even clear from the text how the rewards for these baseline are engineered in some environments like "Nested Rings" where the agents must additionally follow a predefined trajectory.

In addition, the safety loss used in the proposed approach to avoid collisions is enforced with some margin $\gamma$, which naturally encourages more conservative policies. By contrast, the reward provided to the RL baseline agents is negative only in the event of a collision (positive when the goal is reach, 0 otherwise). To make a fair comparison, the negative reward should be given whenever two agents are getting too close to each other, similar in spirit to the margin $\gamma$, so as to encourage more conservative policies in this case as well.

All in all, better baselines could probably devised, perhaps even as simple as well-tuned potential fields.

Aside from the baselines, the results shown are mostly convincing, and demonstrate good generalization capabilities. Some limitations:
* Even though results are averaged over 10 independent runs, now error bar are represented in any of the graphs, so we don't know to what extent the methods are brittle or not
* Section 4.3 introduces the "spontaneous online policy improvement" which is a test-time refinement of the policy to increase safety. However, no discussion of this component is made in the experimental section. In particular, what would be the performance without such refinement? The refinement requires doing some gradient descent over some cost function. How many step does that usually take? What is the impact on the run-time? What is the percentage of states/actions that need to be refined in practice (I expect this percentage to increase with the number of agents, perhaps the generalization capabilities can be credited solely to this component of the method? )
* It would be valuable to provide some sort of visualization of the $h$ function that has been learnt. How well does it correlate to the distance to the closest object?

review_title: Sound theory, improvable experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##########################################################################
Post rebuttal:

I have read authors' rebuttal as well as the newly added contents.
I appreciate the time and the details that the author goes in when addressing my comments. 
I am convinced that this method can be used even without the OPR module based on the ablation study. 

Overall, I think this is a high quality paper that deals with decentralised Multi-agent safety-awared learning. And learning the safe certificate is the shining point of this contribution. I recommend for acceptance and give my final score at 8.  


##########################################################################
Summary:
 
The work studies how, in a multi-agent environment, agents should avoid collisions and develop safety guarantee during learning. It presents a novel approach of learning safe multi-agent control via jointly learning the control barrier functions (CBF) as safety certificates. The benefit of this learning-based CBF framework is that during testing, it is decentralised, thus it can scale up to large number of agents, and is agnostic to the changing agent number, which nicely fits with the need of multi-agent problems. An extensive list of experiments against strong multi-agent learning baselines has been conducted, and results are significant. 

##########################################################################
Reasons for score: 

Extending the learning-based CBF approach to multi-agent problems is an important work in the area of multi-agent learning. The result of multi-agent safe control of robots and drones on 1024 agents using the policies trained on only 8 agents to me is impressive.  I am tending towards an acceptance. However,  I have some uncertainties over the the current version, see the below.

 
##########################################################################ProPros: 
 
1. This paper studies an interesting and an emerging topic in multi-agent learning: multi-agent safety control. They used a CBF approach, a classical idea in control domain, and empower it with the learning capability and extend it in the multi-agent domain. 

2. It gives out a clear definition of multi-agent safety control, and show that why a decentralised CBF can meet such multi-agent safety control definition. 

3. The author provides a classical high probability generalisation bound based on the Rademacher complexity. 

2. The author is considerate in that they account for the changing number of neighbours in a time-variant multi-agent setting. I believe it is an important topic but usually would be neglected by MARL people from the moment of problem definition. 


 
3. This paper provides comprehensive experiments, including 2D grid world, ground robots, and 3D drones. The results seems competitive against baselines, and outperforming when the agent number is large.
 
##########################################################################
Cons: 
 
1. I don't quite get the idea of section 4.3, the spontaneous online policy refinement. It seems to me that the learning agent can somehow “cheat” during the test time should they entered into a dangerous zone during the test time. I believe an ablation study on the performance with and without this module is critical. 

2. Can the author provide more insights why the proposed algorithm outperform the baselines when the agent number is large, based on figure 4 and 6, it is consistent that N=32 is the break-even point.  

3. The related work of <Guaranteed Obstacle Avoidance for Multi-Robot Operations With Limited Actuation: A Control Barrier Function Approach> seems to be doing a similar topic. Could the author include this baseline into the experiment?


4. the union bound in equation6, should it be a \cup rather than \cap?


5. am I right that your method is in fact a supervised learning approach to learn the policy and the certificate, rather than a RL approach, where the policy update will not influence the new data that you are going to collect. I would love to see a pseduco-code if it is not the case.

6. Can the author comment on how in general the dynamical system information of f_i can be known? 

##########################################################################
Questions during rebuttal period: 
 
Please address and clarify the cons above 
 
#########################################################################

review_title: An interesting paper that tackles multi-agent safety control via learning the control barrier by neural nets

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends the neural barrier certificate method from reinforcement learning to  the decentralized multi-agent reinforcement learning setting. In particular,  it borrows the idea of the control barrier function, which enforces the states of the dynamic system to stay in the safe set. It is known that in the single agent system if the control barrier function h satisfies the equation (1), the agent will never enter the dangerous set under policy pi. A straightforward way to extend the control barrier function in the MARL setting is to replace the state-action pair by  the joint state-action pair  in equation (1). However it may suffer from the exponentially large state and action space. Therefore, the author proposes a decentralized setting, where each agent just maintains its own control barrier function. To learn the policy and function h simultaneously, the author penalizes the violation of equation (2). At last, they test the proposed method in Navigation, Predator-Prey, ground robots and Drones compare it with several baselines.


Concern 1: The contribution of this paper is incremental. Particularly, the author extends the control barrier function from (1) to (2). It is quite straightforward to prove proposition 1 and 2. In addition, I am not sure whether the definition of equation (2) is reasonable or not.  To simplify the problem, it directly assumes h_i are independent. Is it true that the whole barrier function isthe product of the individual one?

Concern 2: As the author mentioned, the loss function in equation (7) just considers the safety instead of goal reaching. Thus how do you train the agent in the experiment?  It is initialized by a well-trained policy (but not safe ), and then rectified by the loss function. Or you train the goal-seeking policy and safe-policy simultaneously. 


Concern3: the definition of  safety of the Multi-agent system. It suggests that safety is defined on the Euclidean distance between agents (the agent does not collide with each other). Is it possible to extend this definition to a more general setting?


review_title: Comments

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper presents a learning framework for decentralized control barrier functions in multi-agent systems, applied to collision-avoidance problems. The paper reviews some key ideas from the CBF literature and extends them to the multi-agent, partially-observed case, then presents a learning framework for estimating CBFs and policies jointly, and concludes with experimental results.

Strengths:
* Good coverage of recent literature in this area and placement of the present work within it
* Adequate, though concise presentation of key ideas from CBF literature. I was able to follow this fine but (a small suggestion) it might be a good idea to spend a little more time on this for readers who have not heard of CBFs before.
* Clear extension to the decentralized, multi-agent case
* Strong experimental evidence that this approach can scale

Weaknesses:
* (main one) the time derivative of each agent's observation may not always exist - indeed the paper points out that observations can arbitrarily permute and change dimension at any time. This would seem to be a pretty big theoretical problem...
* the paper should comment explicitly upon how this formulation differs from existing work in (non-learned) multi-agent CBFs
* the interpretation of (3) is unclear. One component (though it would still be unclear even without this issue), is the parsing of \mathcal{X}_{i, 0}. Is this _all state, observation pairs for agent i_? If so, it would certainly help to just say that.
* The presentation and preparation for Prop 2 is very sparse and hard to follow/understand. For example, what does (5) really buy us except for a term in (6) which is not really explained clearly? To me, this looks a little bit "decorative" (to use a term I've seen several conferences use in instructions to reviewers).
* the mu term in (9) doesn't really make sense
* in the results, it would be good to comment on failure modes since the proposed method does not achieve perfect safety rates and in the video it is clear that some of the agents violate safety
* Is MAFACTEST published? If not, this would seem to merit further discussion. If so, it should be clearly cited as different from FACTEST.

Overall:
I like the main ideas here, but I have identified a number of directions of weakness and potential improvement. For now, I cannot recommend publication.
review_title: Interesting idea and results but several issues

Meta-review: Initially there were some shared concerns about the work being too incremental, lack of technical clarity on the algorithmic side and experiments, and lack of clear mathematical formulations. The authors did a good effort and cleared up many questions and remarks satisfactorily, and several reviewers have increased their scores as a consequence. In its current state I recommend to accept the paper.
Decision: Accept (Poster)