Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes using RL for decentralized quadrotor control. 
strengths_and_weaknesses: This paper poses using deep RL for decentralized control. While the paper is well written, it is missing quite a few advances in the field 
of large-scale decentralized robot control using RL/Imitiation learning (see citations below). I would like to expand this review on the basis of the following; novelty, quality and clarity 

Novelty: This paper proposes using a deep RL method for learning multi-agent RL policies. This is by no means new and there are many many papers that have expanded this field. The first of these works look to use off-the-shelf multi-agent RL policies by parametrizing each agent using a MLP and then using a centralized critic architecture [1]. These papers realize that these works having issues when attempting to scale to a larger number of robots due to the increase in dimensionality during training. The architectures proposed in this paper suffer from the same issues; as the number of robots is increased the number of inputs to the final "green" blocks in Figure 2 increases thus increasing the dimenisonality and as such making these methods not scalable. Further, the methods proposed in [2]-[4] and the papers citing them use graph neural networks to introduce sparsity into the problem while exploiting local structure. These papers produce state-of-the-art results for a large number of robots upto 100). The fact that this paper neither cites these papers nor compares against these methods reflects poorly. As such in my opinion the paper introduces no novelty and no significant contribution compared to the existing state-of-the-art methods for large scale decentralized robot control.

Quality: The paper is well written. However, as mentioned above the experimental section in this paper is underwhelming. The authors show no comparisons with other "non-author proposed architectures", i.e all baselines in Fig 4 consist of some modification proposed by the authors and as such there is no idea how these would compare with other state-of-the-art methods. Fig 6 states "baseline"; I am unsure which baseline this is. Similar issue with Fig 7. There are no meaningful baselines and as such, I fail to see the merits of this approach. 


[1] Khan, Arbaaz, et al. "Learning safe unlabeled multi-robot planning with motion constraints." 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.
[2] Graph Neural Networks for Decentralized Multi-Robot Path Planning Q.Li et. al 
[3] Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks Tolstaya et. al 
[4] Graph Policy Gradients for Large Scale Robot Control , Large Scale Distributed Collaborative Unlabeled Motion Planning With Graph Policy Gradients Khan et. al


summary_of_recommendation: As such, since the paper does not improve/cite upon the existing state-of-the-art methods, for me it adds very little to the existing body of work that looks to use machine learning for teams of robots. As such I cannot recommend this paper be accepted in its current form. 
issues: I would recommend the authors spend time looking into the works stated above and think more carefully about where their methods/paper would fit in.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a multi-agent end-to-end reinforcement learning approach to learning controllers for drone swarms that achieve formation control goals and collision avoidance using only local (and hence scalable) policies.  The policies are trained in a realistic simulated environment, and then deployed in both simulation and on Crazyflie 2.0 physical testbeds.  Each drone's policy learns and acts o a local embedding of the drone's state, an embedding of its neighbors' relative position and velocity -- for the neighbor encoder, Deepset and attention-based architectures are compared, and it is shown that the latter achieves better performance by by learning to suitably weight "more dangerous" neighbors (i.e., those that are closer and heading towards the ego-drone) more than "less dangerous" ones.  The policies are trained using PPO on several procedurally generated scenarios, including static and dynamic formations, as well as evader pursuit like settings, and the proposed method is also extended to accommodate obstacle avoidance.  Experiments evaluating the two neighbor embedding architectures, the attention weights, and scalability of the approach are conducted.  Finally, scaled-down versions of the Deepset architecture are trained in simulation and then deployed and evaluated on a computationally constrained physical platform (four Crazyflie 2.0 drones).
strengths_and_weaknesses: Strengths:
+ the paper is well written and easy to follow
+ the first successful demonstration of multi-agent end-to-end deep RL for drone swarm control is indeed novel
+ the use of an attention mechanism for distinguishing "important" drones from "unimportant" ones given a specific dynamic configuration is a natural and nice idea
+ the experiments covered a broad range of interesting tasks and demonstrated that the method learned successful collision free formation control policies.

Weaknesses:
- the baselines against which the method in the paper is compared seem to be quite weak.  The "blind" baseline (where the quadrotors can't see their neighbors) is too weak to be meaningful, and whereas the MLP (non-permutation-invariant) policy is sensible, what's missing are some more sophisticated baselines for comparison.  For example, in the related work section, the paper explains that [9,10] significantly constrain the configuration space of the robot to guarantee collision free behavior.  It would have been nice to see a comparison of collision frequency vs. goal tracking performance for the methods of [9,10] and those proposed in the paper.  Similarly, for permutation-invariant policies, recent lines of work have shown that Graph Neural Networks (GNNs) can be used to achieve scalable learning and control in the context of formation control.  See for example:

@inproceedings{tolstaya2020learning,
  title={Learning decentralized controllers for robot swarms with graph neural networks},
  author={Tolstaya, Ekaterina and Gama, Fernando and Paulos, James and Pappas, George and Kumar, Vijay and Ribeiro, Alejandro},
  booktitle={Conference on robot learning},
  pages={671--682},
  year={2020},
  organization={PMLR}
}

and

@inproceedings{khan2020graph,
  title={Graph policy gradients for large scale robot control},
  author={Khan, Arbaaz and Tolstaya, Ekaterina and Ribeiro, Alejandro and Kumar, Vijay},
  booktitle={Conference on robot learning},
  pages={823--834},
  year={2020},
  organization={PMLR}
}

(Disclaimer: I am not an author on either paper).

- the argument for scalability didn't strike me as convincing: if 500 million simulation steps are needed to refine policies trained with a smaller number of drones to a larger number of drones, this seems like it will run into computational bottlenecks if the number of drones starts getting large.  I may have misunderstood this point, so would like to see some clarification in the rebuttal: for example, showing wall-clock time for the fine-tuning step as a function of the number of drones would be helpful.

summary_of_recommendation: Suggesting acceptance based on: first demonstration of end-to-end multi-agent RL to collision free formation control, well written paper, interesting experiments (including sim2real validation).

Not suggesting strong accept based on: weak baselines and unconvincing argument of training scalability.
issues: Comparison to stronger baselines, and clarification of scalability of training time.  See comments above for more details.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Main Contributions
===============
- Successful distributed swarm control for drones in simulation and the real world. 
- Analysis of different network architectures on final performance.

Summary
============
The paper presents successful decentralized drone swarm control using a learned policy. A group of drones is tasked with maintaining static or dynamic formation flight while avoiding collisions with each other or external, possibly dynamic, obstacles. Each drone is controlled by a policy represented by a deep neural network, that observes the current state of the drone, the state of neighboring drones as well as the state of potential obstacles.
The control policy is trained using deep reinforcement learning (PPO).
The paper compares different model architectures with respect to the final performance of the drone swarm.
The proposed approach is successfully deployed in simulation and in the real world. 
strengths_and_weaknesses: Strengths
===========
- The paper demonstrates successful deployment of the proposed approach to a real-world setting with 4 crazyflie quadrotors. 
- The paper presents an in-depth analysis of different network architectures.
- The results are very well illustrated in the supplementary video.
- The paper is well written and easy to follow.

Weaknesses
==========
- Even though the presented results are impressive, the paper does not compare against prior work. All experiments are performed by comparing the proposed approach against different variants of itself (i.e. a 'blind' quadrotor, that has no access to the position of neighboring vehicles).
- To better understand the performance of the proposed approach, the paper would significantly gain value if it compared with prior work on drone swarm control such as [1], or even naive baselines based on reactive navigation [2] or potential fields.

[1] Hönig, W., Preiss, J.A., Kumar, T.S., Sukhatme, G.S. and Ayanian, N., 2018. Trajectory planning for quadrotor swarms. IEEE Transactions on Robotics, 34(4), pp.856-869.

[2] Florence, P., Carter, J. and Tedrake, R., 2020. Integrated perception and control at high speed: Evaluating collision avoidance maneuvers without maps. Algorithmic Foundations of Robotics XII. Springer, pp.304-319.
summary_of_recommendation: The paper shows convincing results, but lacks any comparison to even naive baselines. 
issues: The paper would get significantly stronger if the authors included a comparison against prior work (see above for potential baselines).
review_title: null

Meta-review: Summary: This paper proposes using a deep RL algorithm for decentralized formation control of aerial robots. The algorithm is able to accommodate static and dynamic formations in the presence of state or dynamic obstacles. The method is validated with simulation and real-world experiments on four off-the-shelf consumer drones. 

Clarity: The paper is exceptionally well-written and presents a clear, logical narrative through the motivation, design, and experimental validation of the proposed method. 

Quality: Reviewers noted that deploying a deep RL algorithm on board real quadrotors is a notable achievement and a clear contribution of the paper. However, reviewers also noted that the work in its current form leaves open questions about scalability and comparison to more appropriate baselines. 

Significance/Originality: Feedback on the contribution of the proposed method is slightly mixed. In particular, it seemed like reviewers were least convinced about how the proposed method stacked up against the existing state-of-the-art. 

Pros: 
Reviewers seemed aligned that deploying a deep learning algorithm on board real robots is rather impressive. The paper also clearly communicates the proposed approach, but in the text and supplementary video. 

Main opportunities for improvement: 
In its current form, the revision should address three major themes in the reviewers’ comments:

- More comprehensive literature review: Broadly speaking, this paper did not convince reviewers that it has a place in the existing literature, at least partly because all three reviewers noted specific missing citations. 

- All three reviewers commented that the baselines used in the experimental results section are inappropriate. Namely, they compare the proposed algorithm against variants of itself rather than other state-of-the-art algorithms. The revision should compare against more meaningful baselines. 

- A major consideration in swarm coordination algorithms, especially decentralized ones, is scalability. Reviewers s99C and g2nt both expressed serious concerns about the method’s ability to scale to the order of hundreds of robots. The revision should either include stronger evidence of the method’s scalability or characterize and make a case for why the team size the algorithm does scale to is still a meaningful “swarm”. 

This paper should be seriously commended for being the first to show learned policies on physical quadrotors. However, unfortunately, that accomplishment alone does not merit publication. The rebuttal should still address three key points highlighted above and in reviewers’ feedback - a more complete literature review, comparison against more rigorous baselines, and justification of the technique’s scalability (and exactly how scalability should be defined in this case). 

Thank you for considering our feedback, and we look forward to seeing the updated paper. 


====== Final Decision

I am pleased to accept this paper as part of CoRL 2021. I would like to sincerely thank the authors for the time they put into revising the paper. The authors thoughtfully addressed reviewer feedback and the "opportunities for improvement" listed in the original meta-review. Particularly impactful were the additional experiments. The simulation results for teams of 64 and 128, in my opinion, definitively addresses scalability concerns. Experiments against the buffered Voronoi cells baseline also added useful insight. I am also excited to see DNNs running on real robots; there is significant "invisible" debugging and design work to get that to work. 
Decision: Accept (Poster)