Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning
Authors: Sumeet Batra
Zhehui Huang
Aleksei Petrenko
Tushar Kumar
Artem Molchanov
Gaurav S. Sukhatme
Venue: CoRL2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Robot Learning
Multi-Agent Systems
Quadrotors

Abstract: We demonstrate the possibility of learning drone swarm controllers that are zero-shot transferable to real quadrotors via large-scale multi-agent end-to-end reinforcement learning. We train policies parameterized by neural networks that are capable of controlling individual drones in a swarm in a fully decentralized manner. Our policies, trained in simulated environments with realistic quadrotor physics, demonstrate advanced flocking behaviors, perform aggressive maneuvers in tight formations while avoiding collisions with each other, break and re-establish formations to avoid collisions with moving obstacles, and efficiently coordinate in pursuit-evasion tasks. We analyze, in simulation, how different model architectures and parameters of the training regime influence the final performance of neural swarms. We demonstrate the successful deployment of the model learned in simulation to highly resource-constrained physical quadrotors performing station keeping and goal swapping behaviors. Video demonstrations and source code are available at the project website https://sites.google.com/view/swarm-rl.

Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning
Sumeet Batra' Zhehui Huang" Aleksei Petrenko"
Tushar Kumar Artem Molchanov Gaurav S. Sukhatmel
Department of Computer Science
University of Southern California
(ssbatra, htttiira,zzhhuuhuppptttoo petrenko kumart, commart,ollcheen molnchno, gaurav) edu
Abstract: We demonstrate the possibility of learning drone swarm controllers that are zero-shot transferable real quadrotors via large-scale multi-agent end-to-end reinforcement learning. We train policies parameterized by neural networks that are capable of controlling individual drones in a swarm fully decentralized manner. Our policies, trained in simulated environments with realistic quadrotor physics, demonstrate advanced flocking behaviors, perform aggressive maneuvers in tight formations while avoiding collisions with each other, break and re-establish formations to avoid collisions with moving obstacles, and efficiently coordinate pursuit-evasion tasks. We analyze, in simulation, how different model architectures and parameters the training regime influence the final performance of neural swarms. We demonstrate the successful deployment of the model learned in simulation to highly cosuurce-constrainee physical quadrotors performing station keeping and goal swapping behaviors. Video demonstrations and source code are available at the project website https: /ttps://sites google. com/viet wview/a s/ssmwarlll...
Keywords: Swarms, Multi-robot systems, Multi-robot learning
1
Introduction
Teams of unmanned aerial vehicles are widely applicable to e.g. area coverage, reconnaissance etc. State of the art approaches for planning and control of drone teams require full state information and extensive computational resources to plan in advance
[1]. This greatly limits their applicability since existing planning algorithms are often brittle in partially-observalle environments and challenging to execute real time on embedded hardware. Many classical methods also suffer from the curse of dimensionality as the number of possible configurations grows combinatorially with team size. In addition, kinodynamic planners typically require a precise model of drone dynamics.
Here, we present fully Iearned approach that uses a small amount of computation during exccution and relies exclusively on local observations, yet results in effective controllers for large-scale, swarm- like, teams that are zero-shot transferable to real quadrotors. We take advantage of multi-agent deep reinforcement learning (DRL) train quadrotor drones on hundreds of millions of environment transitions in realistic simulated environments. We find neural network architectures and observation spaces that allow our neural controllers to achieve high performance on a diverse set of tasks. Our experiments demonstrate that drone swarms controlled by our neural network policies generate highly effective, smooth, and virtually collision-free trajectories. We show simulated dynamic obstacle avoidance and demonstrate that learned controllers successfully execute tasks physical drones in team sizes upto quadrotors. To the best our knowledge, this the first approach that a) demonstrates scalable coordinated flight of drone swarms in realistic physical simulation achieved through end-to-end reinforcement learning (RL) with direct thrust control, and b) demonstrates that the learned policies transfer to physical drone teams. We view this progress towards hardware agnostic real-world deployment quadrotor swarms with realtime replanning capabilities.
Equual contribution
Gaurra Sukhatme holds concurrent appointments as a Professor at USC and as an Amazon Scholar. This paper describes work performed at USC and is not associated with Amazon.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Set DenDo
ICOW
HIME
LEAROID (IMA XTCM
LAMANO POL CAN CRANE
Figure I: Drones are trained on formations randomly sampled from one of six pre-defined scenarios on the left. Drones employ a self- and neighbor encoder that learn a mapping from a combination of proprioceptive and extereoceptive imputs to thrust outputs. The combined intermediate embeddings enable policies perform high speed, aggressive maneuvers and learn collision avoidance behaviors e.g. formation creation, formation swaps, evader pursuit. These are shown on the right both in simulation and physical experiments.
2 Related work
Classical methods such as RRT [2] and PRM [3] can generate collision-free piece-wise linear trajecto- ries. These do not directly account for robot dynamics and scale poorly with team size, limiting their applicability for online re-planning. To account for the dynamics, smooth trajectory refinement [1] or optimization [4] is performed. Kinodynamic planning methods take advantage of the known dy- namics of individual robots and are more suitable for agile and aggressive flight [5]. It is also possible to convert a geometric route to a dynamically feasible trajectory [6] resulting in an agile system or to directly find dynamically feasible paths and refine them with gradient based optimization [7]. Prior learning- based approaches include [8] which estimates reachable states given the quadrotor dynamics. However, these methods require knowledge of the entire state space a priori and do not generalize well to other high dimensional environments. Additionally, kinodynamic planning is typically intractable in the multi-robot setting. Collision avoidance is particularly difficult where drones perform aggressive maneuvers. Traditional techniques use observations from neighboring drones to constrain the geometric free space or the action space of each drone. Prior work includes an n- n-body collision avoidance approach [9] that constrains the velocity space of a robot using velocity information from neighboring robots. Another approach utilizes pose information from neighbors to construct a Voronoi Cell, and each robot plans a collision free trajectory through its respective cell [10]. The approach in [11] minimally constrains the control space of an agent by constructing Siance-constraineet safety sets that account for measurement noise and disturbances present in real world environments. While these approaches guarantee collision free trajectories and are capable of running in realtime, with the exception of [11] (PrSBC), they significantly constrain the configuration space of the robot, preventing aggressive maneuvers and resulting occasional deadlock. While PtSBC minimally constrains the control space of any given policy, it requires model of neighboring agents order to construct caance-constrained safety sets. Our controllers rely on a reward formula- tion with high emphasis collision avoidance and our policies are not constrained, allowing for aggressive maneuvers in a variety of rich, dynamic environments.
Simulation trained neural networks have been used for single-agent and multi-agent quadrotor control to address shortcomings of kinodynamic planners. [12] and [13] both utilize Imitation Learning (IL) from global centralized planner, [13] further utilizes RL to balance the tradeoff between actions optimal to the team and individuals. [14, 15] learn controllers robust to aerodynamic interactions. Graph Neural Networks (GNNs) are promising architectures for swarm control and they have been proposed as a parameterization for imitation learning [16, 17) and RL [18, 19] algorithms.
Reinforcement learning has shown promise in learning policies for UAV flight [20]. Deep RL [21] has been used to learn minimum-time trajectory generation for quadrotors. Similar [20] (but in a multi-agent setting) we train from scratch using DRL via an end-to-end approach: the policies directly control the motor thrust. This is similar to [22] which uses Soft Actor Critic teach single quadrotor how to fly. Multi-Agent Reinforcement Learning (MARL) has recently been applied to autonomous driving, path finding, and cooperative multi- aulti- control [23, 13, 24] and UAV team control (e.g. [25] new gym environment that models drone dynamics, taking into account aerodynamic effects, and trains policies for basic hovering and leader-following tasks). Khan et al. [26] train a centralized Q-network to solve multi-agent motion planning problems. This approach has limited scalability as the input space of the Q-network increases with the number of drones. We address scaling by relying only on local neighborhood information both during training and inference.

---

3 Method
3.1 Problem formulation
We consider team of N quadrotor drones in a simulated 3D environment. Our task is to learn control policy that directly maps proprioceptive observations of an individual drone to motor thrusts with the goal of minimizing the distances to positions in the desired formation while avoiding collisions. We analyze the case of online decentralized control, i.e. instead of a centralized system solving the joint trajectory optimization problem offline, we consider policies which simultaneously (and implicitly) plan trajectories and control individual quadrotors in real-time. The decentralized approach assumes no access to the global state during evaluation and scales better with the size of the swarm. In the real world, this is analogous to individual quadrotors the swarm generating their own action sequences using only on-board computations. We demonstrate the efficacy of this approach by showing that our learned policies do indeed transfer to a physical setting. Formally, the ..... IN s{ ................................ SN Here
state of the environment at time t is described by the tuple ( = (9) (c)
9i E R are the goal locations for the quadrotors that together define the desired swarm formation at time t, and s!") are the states of individual quadrotors. We describe the state of a single quadrotor by the tuple (p, v, R, w), where E g is the position of the quadrotor relative to the goal location, E R is the linear velocity in the world frame, w IR3 is the angular velocity in the body frame, and R E (3)3) is the rotation matrix from the quadrotor's body coordinate frame to the world frame. Each quadrotor is controlled by a learned policy Ty (a()))(()) which maps observations o(9) to Gaussian distributions over continuous actions (*) To effectively maneuver and avoid collisions, the quadrotors need to be able to measure their own position and orientation in space (f) and relative positions Pij and relative velocities o/ of their neighbors. We therefore represent each quadrotor's
(() 0/1) ..... o is a tuple containing the neighborhood information. Here K < N 1 is the number of neighbors that each individual quadrotor can observe. We use K < N for larger swarms to improve scalability during both training and evaluation. Simulated quadrotors, similar to their real-world counterparts, are powered by motors that spin in one direction and generate only non-negative thrust. We thus transform actions (t) e R² sampled by the policy to control inputs f(() E [0. 1| via f(t) = (clip(a 11 + 1), where ((t)
observations as (( = (s! .n,"), where 17, (t) (P
(pff....,
P.K. ........P........
.....
corresponds to no thrust, and f corresponds to maximum thrust on motors 1...4.
3.2 Simulation and sim-to-real considerations
We train and evaluate our control policies in simulated enviromments with realistic quadrotor dynamics. We adopt a simulation engine with drone dynamics [20]. and augment to support quadrotor swarms. Virtual quadrotors are modeled on the Crazyflie 2.0 [27] our physical demonstration platform. A key feature of this engine is the model of hardware imperfections previously shown to facilitate zero-shot sim-to-real transfer of stabilizing policies for single quadrotors [20]. Non-ideal motors modeled using motor-lag lag and thrust noise and the simulator provides imperfect observations, with noise injected into position, orientation, and velocity estimations. Noise parameters are estimated from data collected on real quadrotors. While motor and sensor noise create a challenging learning environment, they are instrumental prevent overfitting to otherwise unrealistic idealized conditions of the simulator. To facilitate the emergence of collision-avoidanne, behaviors, in addition to modeling dynamics, we simulate collisions between individual quadrotors. In reality, collision outcomes depend on many factors, e.g. rigidity and materials of the quadrotor frames, whether the blades of two colliding quadrotors touch, etc. Instead of modeling these complex processes, we adopt simple randomized collision model. When collision between quadrotors is detected, we briefly apply random force and torque both quadrotors with opposite signs, preserving linear and angular momenta.
3.3 Training setup
For training we use a policy gradient RL algorithm. In our setup, the learning algorithm updates the parameters of the policy To (a(a(t) (0) to maximize the expected discounted sum of rewards Em ( ). For our experiments we choose Proximal Policy Optimization (PPO) [28]. In particular, we use the implementation from the high-throughput asynchronous
3

---

RL framework "SampleFactory" [29] which supports multi ageett learning and enables large scale experiments. In each training episode, the goal of every quadrotor in the swarm is reach its designated position in the formation while avoiding collisions with the ground and with other quadrotors. In order to provide rich and diverse training experience, we train our policies in a mixture of scenarios featuring a variety of geometric 2D and 3D formations. Further training details are in Section 4. The reward function optimized by the RL algorithm for quadrotor consists of three
major components: ,(()
(0) ppus + col Taux- Here, (0)
(t)
Tpos Opos
p; rewards the quadrotors for
minimizing the distance to their target locations. rool is responsible for penalizing collisions between (8) Ocol col Oprox Σ max
quadrotors and is defined as: real
(1 (1- |0 /dprox 0) The
collision involving the i-th quadrotor is detected. The second term represents the smooth proximity penalty with linear falloff distance is the distance between centers of mass of i-th and j-th quadrotors, and dprox in experiments is double the size of the quadrotor frame, which encourages them to keep minimal (t)
indicator function (o) is equal to 1 for timesteps where
dpeox. Here ((() 2
distance tight formations. addition to the first two terms Tpos and ((() that convey our main objective we also adopt an auxiliary reward function similar to [20] to facilitate the initial learning of stabilizing controllers: (t) = ma W
2 as | + ama penalizing high angular
velocity, high motor thrusts, and large rotations about horizontal (x- and y-) axes respectively.
3.4 Model architectures
Deppi Droder Ne glbrr
leghen
Archilcturr Abtrkor
Arch Mecture 2- Phyelcalyy Dedloyec Deepet
Figure Detailed neural network architectures. Left: Deepsets architecture used in simulation. Middle: Attention architecture used simulation. Right: Smaller deepsets architecture deployed on Crazyflic2.O.
During training and evaluation, the quadrotors' actions a are sampled from a parametric stochastic policy (a(o))) We omit time indices and quadrotor identity for simplicity where possible. We
represent with Gaussian distribution, function of the quadrotor's s observation at time t parameterized by feed-forward neural network, and the variance o² E R is single learned parameter independent of the state. To compute the distribution means we embed the state of the quadrotor and its neighborhood before regressing: Ha c, (s(s,, ey = [s n); fh Φa (e., en,). Here e, and en are the embedding vectors that encode each quadrotor's own state and the state of its neighborhood respectively, Φe and Φa are collecconeette neural networks, and fn is the neighborhood encoder. We analyze two types of neighborhood encoders: deep sets and attention- based (Section 3.5). The value function V₂ uses the same architecture as the policy, except that it regresses single deterministic value estimate instead of the parameters of the action distribution. Weights are not shared between π and V,, models.
(o(a)) & [N((a, where the mean Hı R is
3.5
Neighborhood encoder
Deep sets. The task of the encoder f,, is to generate compact and expressive representation e,, of each quadrotor's 's local neighborhood. Since the individual quadrotors are agnostic to the identity of their neighbors, this representation must be permutation invariant. In addition, scale invariance is a desirable property since the size K of the observable neighborhood fluctuates over time, i.e. when a sufficient number of quadrotors in the formation move beyond the sensor range. The deep sets architecture proposed by Zaheer et al. [30] has these required properties. We apply the same learned transformation v to the observed features of each quadrotor in the neighborhood, after which permutation- iaanutaaionnivvariant aggregation function is applied to neighbor embeddings ej. We calculate the

---

Figure (Left) Evader pursuit, (Middle) . 16 quadrotors in dense formation after fine-tuning (see Section 4.3), and (Right) A swarm breaking formation avoid a collision with an obstacle. Videos of the learned swarm behaviors in different scenarios are at: attps://sites
com/viet wtt.//iee///arrr//eeee// /wwrmm-rl.
mean neighbor embeddings to achieve scale invariance: e, Vyy Dty), C. Σラ= Cz. Not all neighhors equally important for trajectory planning and decision making. For example, distant and stationary drones are less likely to influence drone's behavior compared to closely located and fast-moving neighbors. The mean operation in the deep sets encoder does not allow convey the relative importance of different neighbors- - this motivates a more sophisticated encoder architecture.
Attention-based. The attention mechanism [31] provides natural way to express the relative importance of individual neighbors. The iitention-based neighborhood encoder is based on [32], adapted for quadrotors in 3D. The current quadrotor's 's state s; and the neighbor observations (piy. 0ty) for the j-th neighbor are used to compute the attention weights: ej = ( Pij, 0iy), cm
R Eド= Ej, 0j t (ej; e.). Here e, are the embedding vectors of individual neighbors, 6m is the summary of the whole neighborhood, and the and a/a fully-connected neural networks. We use the softmax operation over a, to compute the attention scores which sum up to 1. The neighborhood embedding is thus produced as 6n Σラ= Softmax((x) (o (ez), where tha, represents an additional hidden layer. Both in deep sets and attention encoders, we used multi-layer perceptrons (MLPs) with 256 neurons and tanh activations. For additional details see [33].
4 Experiments and results
We train our control policies in episodes, in diverse randomized scenarios with static and dynamic formations. Our virtual experimental arena 10 x 10 x 10 m room. At the beginning of each episode. randomly spawn the quadrotors in 3 m radius around the central axis of the room, at a height between 0.25 and m. We randomly initialize their orientation, linear and angular velocities to facilitate learning robust recovery behaviors. To provide diverse and challenging training environment, we procedurally generate scenarios different types, listed below.
Static formations. The target formation is fixed throughout the episode. The formation takes various geometric shapes e.g. 2D grid, circle, cylinder, and cube (Fig. 1). The separation r between goals in the formation is chosen randomly. A special case = 0 where the goal locations for all quadrotors coincide demands very dense configurations with high probabilities collisions. We refer to this task as the same goal scenario.
Dynamic formations. We modify the separation between the quadrotors, and the position of the formation origin. We explore gradually shrinking the inter-quadrotor separation over time, and randomly teleporting the formation around the room. To train policies avoid head-on collisions at high speed, we include scenarios where the swarm is split into two groups. We swap the target formations of quadrotors in these two groups several times per episode, which requires two teams of quadrotors to fly 'through' each other. We refer to this as the swarm-vs vs-swarm scenario.
Evader pursuit. For the evader pursuit task, the team given a shared goal that moves according to some policy (simulating an evader that the team must pursue). We use two evader trajectory parameterizations: 3D Lissajous curve and randomly sampled Bezier curve.
4.1 Model architecture study
We compare training performance with different neighborhood encoders with quadrotors (Fig. 4) and fixed number K 6 of visible neighbors. In addition to the architectures described
5

---

Arp. larget
performacce
5.00
-02
2,00
6:00
-0.
Eis
0.12
0. 2006 1000 ECCM
2009 ROGM 6COM 2934
200M KOOM 2009
2024 92N G0R
Figure 4: Comparison of different model architectures for N = 8 quadrotors with K = 6 neighbors visible to each. For cach architecture we show mean and standard deviation of four independent training runs.
in Section 3.5 we train two baselines. The first is a blind quadrotor, for which we remove the neighborhood encoder J, entirely. While blind quadrotors get close to their targets, they are not able to avoid collisions with each other. The second uses plain multi layer perceptron, which concatenates neighbor observations as input. This policy, which is not permutation invariant, fails to avoid collisions in most scenarios, suggesting that cermutaaion-invariaa The difference between the attention and deep sets encoders is most prominent in tasks that require dense swarm configurations, such the same goal scenario. In addition, quadrotors with the deep sets encoder do not get as close to their target locations, sacrificing formation density to minimize collisions. Since the attention- based architecture demonstrated the best performance in both goal reaching and collision avoidance, we use it in all further simulated experiments.
architecture is needed.
4.2 Attention weights study
We investigate the results of training attention-baset architecture for encoding relative scores of neighboring drones. We ask whether the attention mechanism learns to assign higher attention scores to neighbors that are closer and whose velocity vector points towards the current agent. In addition, we investigate to what degree distance and velocity individually affect the scores. We modify the Swarm-vs-swarme scenario to contain two teams of two drones whose goals are 1 m apart and situated in the same horizontal plane. The drones are allowed to settle at their respective goals following which the goals are swapped. We take snapshot of the experiment and record the softmax attention weights for each drone. We manually set the relative velocities of all neighbors to (0, 0, 0) for each drone and pass the modified observations to the attention encoder. The results (Fig. 5) show that the red quadrotor assigns the highest attention weight of 0.61 to the blue quadrotor, which on collision course with it. Similarly, the blue quadrotor assigns the highest weight of 0.57 to its red neighbor. For the gray and green quadrotors, all neighbors are assigned roughly equal weighting, with neighbors closer in distance having slightly higher weights. When the relative velocity observations are manually set to 0 and fed to the attention encoder, we observe a drastically different, much more uniform distribution of attention weights. We conclude that neighboring quadrotors with small relative distance and high relative velocity vectors in the direction of the viewer are prioritized over drones further away with velocities in other directions. Drones with relative velocities pointing towards the viewer seem to be prioritized higher than drones that are closer but with velocity vectors away from the viewer, implying that velocity is considered more important than distance.
4.3 Scaling
We investigate the ability of our policies to scale to larger swarm sizes without re-training from scratch. We introduce second, fixed-cost training step in which policies trained with N 8, K =
Collisions per minute
Collisions
Distance target,
Distance to target,
agents per minute
per drone
200M training
per drone
+200M training
0.02
0.42
0.00
0.41
Adbrt on wellotrs
Alnette wiights vilchhe
16
0.09
0.57
0.08
0.63
32
0.90
0.86
0.16
0.81
48
2.43
0.23
0.92
64
5,36
1.70
0.29
1.12
128
8.63
4.32
1.37
3,05
Figure 5: Attention weights between drones. Left: original velocities, right: velocities set to O.
Table I: Scaling up the attention policy with and without additional training. The size of the visible local neighborhood fixed at K = 6 drones. The metrics are averaged over 20 episodes.

---

are trained for an additional 2 x 10 steps with the new target number of quadrotors N and the same K 6, i.e. the baseline policies are copied and tuned separately in the environment with larger swarm size. The results (Table 1) show that without additional training, the number of collisions increases with the number of quadrotors because the state distribution changes significantly compared to 8-drone case. Additional tuning has a significant positive effect. Even with 128 drones, the quadrotors can avoid collisions in dynamic environments, and the higher number of collisions is largely explained by cascade effects: when a collision happens, it affects multiple drones. The additional tuning amounts to only 20% of the original training session (~~ hours).
4.4 Obstacle avoidance
We experiment with harder version of the environment by introducing spherical obstacle moving through the formations at random angles multiple times throughout the episode, At the beginning of each episode, randomly sample the obstacle size, as well as its velocity and the parameters of its trajectory. To incorporate obstacles into our training protocol, we augment the quadrotor observations o to contain the information about obstacle state ç. This information includes the radius i of the obstacle, and its position (b))) and velocity of!)) relative to the i-th quadrotor. We process the obstacle observations with an additional MLP 6o to produce the obstacle embedding cg. which is used in conjunction with the neighborhood encoder to generate the action distributions (see Section 3.4 for details): 6,- Φo(C), f.e da(es, in the same way as for
(e., ey. e.). The collision physics and the penalties are modeled collisions (Sections 3.2 and 3.3). Fig. 7 shows the training performance in obstacle avoidance scenarios. Despite increased complexity, we achieve performance comparable to the baseline, keeping dense formations close to the target locations.
Avg. disance target
Ang. collitera belwean the stutadee dovesspe
1.00
0.50
0.25
Figure 6: Time lapse of eight drones swapping goals (left) and performing a formation change (right).
sinedatc cops
Figure 7: Learning to maintain dense formations while avoiding collisions with moving external obstacles.
4.5 Additional baselines
Classical trajectory optimization and control methods have been proposed for tasks similar to ours. [34] uses graph search in discretized space followed by trajectory smoothing to switch formations with large teams of quadrotors. relies on full prior information about the (static) environment and an offline optimization process taking up to tens of seconds. We test our controller in randomized dynamic environments without access to global information, which makes it hard to make a direct comparison between our method and [34]. Graph Neural Networks [16, 17] and RL [18, 19] are a prime candidate for an alternative model architecture. GNNs can capture global information about the swarm while relying only on local communication by performing multiple consecutive graph convolution operations and communicating intermediate representations between adjacent robots. While this enables decentralized operation, the reliance on multiple message exchanges between UAVs on each step is prohibitively expensive for nano-quadrotor platforms such as Crazy flie. Our controllers run 500Hz on the real drones, which makes the communication protocol latency requirements exceptionally tight for multi-layer GNNs. Nonetheless, GNNs remain an attractive option for more powerful platforms or tasks that do not require high-frequency reactive control. [10] uses Buffered Voronoi Cells to compute safe regions around quadrotors (with margins between cells to account for kinodynamic constraints) and a PID controller to achieve positions within safe regions that are the closest to the target. This method relies only on local neighborhood information. We implement it on our hardware platform and compare its performance to our method (Section 5).
5 Physical deployment
We deployed our policies on the Crazyflie2.0, a small, lightweight, open-source quadrotor platform. We tested the neural controller in several scenarios: hovering in a close proximity to a shared goal
7

---

(same goal), following a moving target, maintaining dynamic geometric formations, and flying through a team of moving drones (swarm-vs-swarm), with up 8 quadrotors in the latter experiments wwwessoooooooooiiiiiieoooooeeooe
Crazyflie2.0 a low-power nano-quadrotor platform where on board computation is provided by microcontroller with 168MHz and 192Kb of RAM. To run team-aware neural policy on such limited hardware we trained a much smaller version of our deep sets model with only 16 and 8 neurons in the hidden layers of self- and neighbor encoders respectively. Surprisingly, even such tiny policies with 10 parameters performed well on real quadrotors. We use a Vicon system to provide position and velocity updates at 100 Hz with an added low low-pass exponential filter for neighbor positions to reduce noise. Each drone's ' controller runs at 500 Hz incorporating the IMU measurements atop the latest available Vicon data. Despite the fact that downwash was not explicitly modeled in the simulator, policies recover from the aerodynamic disturbances caused by the proximity of other drones. We observed recovery from non-destructive collisions with each other or with the ground wherein drones re-stabilize after collisions with teammates, and "bounce back" from the ground and resume flight.
(Fig. 6). Video demonstrations are available at
Swap Goals (NN)
Dynamic Same Goal
Swap Goals (Voranai Cells + PI))
1.75
102
0.2
530
6.5
0.
0
stm)
x(m)
1.0
Figure 8: (Left) Two teams of four quadrotors swap goals twice and then land at their original starting positions. (Middle) Four quadrotors follow a moving goal, starting from the bottom right going counter-clockwise. The goal positions are marked with red X's. Both plots show 3D positions over time of the four quadrotors. (Right) two teams of four quadrotors swap goals using PID controllers and buffered Voronoi cells for collision avoidance.
For comparison, we implemented an online planning and collision avoidance algorithm [10], based on buffered Voronoi cells and PID control. to understand how our controllers behave compared to a traditional approach. We tested with 8 drones on the swarm-vs-swes generated by the classical algorithm are longer than the ones produced by our neural network controller. Although there were no restrictions on the configuration space, we observed that the classical method generates trajectories predominately in single 2D xy xy-plaane, whereas the neural controllers utilize all 3 spatial dimensions (Fig. 8). Finally, we observe that our controllers execute more aggressive maneuvers, reaching goals faster with max velocity (acceleration) of up m/s (7 m/)²) respectively, compared to a max of m/s / (3 m/²²) for the classical method. For additional details see [33].
scenario. The trajectories
6
Conclusion
Our results demonstrate that drones trained with deep reinforcement learning can achieve strong goal-reaching and collision avoidance performance across diverse range of training scenarios with realistic quadrotor dynamics. We present evidence of successful swarm control in simulation and demonstrate the zero-shot transfer of policies learned in simulation to the Crazyfiie platform, on which we are able to perform successful trials on multiple tasks. Our policies learn to fly from scratch, without the use of tuned PID controllers. Our method is thus model-agnostic, i.e. we can learn policies for quadrotors with different physical parameters (e.g. mass, size, inertia matrix, thrust) by simply re-running the training in the updated simulator. In contrast with classical planning methods, we do not introduce any constraints on the velocity or acceleration and allows the controller to take advantage of full capabilities of the quadrotor. This enables agile flight with aggressive maneuvers. Our pursuit-evasion experiments are the most representative of this. In order to stay close to the fast evader, simulated quadrotons exceed speeds of m/s and reach accelerations up to 1.7 g. Permutation and scale-agnostic model architectures used in our method allow us to switch between different team sizes. We found that after additional training to adjust to the larger team size, our policies can control swarms of up to 128 members without significant increase in computation burden per quadrotor.
8

---

Acknowledgments
This research is supported in part by USC Graduate Fellowship. We thank our colleagues from the Robotic Embedded Systems Laboratory (RESL) for discussions and insights. We are grateful to Baskin Şenbaşlar, Tao Chen, and Wolfgang Hönig for their assistance and support. We particularly thank James Preiss for his help with the experimental setup.
References
[1] W. Hönig, J. A. Preiss, T. K. S. Kumar, G. S. Sukhatme, and N. Ayanian. Trajectory planning for quadrotor swarms. IEEE Trans. Robotics, 34(4):856-869) 2018
[2] K. Solovey, O. Salzman, and D. Halperin. Finding needle in an exponential haystack: Discrete RRT for exploration of implicit roadmaps in multi-robot motion planning. Int. J. Robotics Res., 35(5):501-51) 2016.
[3] Karaman and E. Frazzoli. Sampling- algorithms for optimal motion planning. Int. Robotics Res., 30(7):846-899)) 894, 2011. doi: do::10.177/0077864991100707 URL https://dot org/10. 117700278364911466775
[4] J. Tordesillas and P. How. MADER: trajectory planner in multi-agent and dynamic environ- ments. CoRR, abs/2010. 1061, 2020. URL https:///rxiiv weexxivvorggg .ogg abs/abss//1. 11061.
[5] D. Mellinger and V. Kumar. Minimum snap trajectory generation and control for quadrotors. In IEEE International Conference on Robotics and Automation, ICRA 20 Shanghai, China, 9-13 May 2011, pages 2520-2525. IEEE, 2011. doi: 1001001110/ICRA..200
URL
https://dol: org/ 10. 1109/ 1109/ICRA. ICRA, 2.w...0.19000..0... 5980409.
[6] C. Richter, Bry, and N. Roy. Polynomial trajectory planning for aggressive quadrotor flight in dense indoor environments. In Robotics research, pages 649-666. Springer, 2016.
[7] B. Zhou, Gao, L. Wang, C. Liu, and Shen. Robust and efficient quadrotor trajectory generation for fast autonomous flight. IEEE Robotics and Automation Letters, 4(4):3529-3553 3536,, 2019.
[8] R. Allen and M. Pavone. A real-time framework for kinodynamic planning with application to quadrotor obstacle avoidance. In AIAA Guidance, Navigation, and Control Conference, page 1374, 2016.
[9] van den Berg, S. Guy, M. C. Lin, and D. Manocha. Reciprocal n-body collision avoid- ance. In C. Pradalier, R. Siegwart, and G. Hirzinger, editors, Robotics Research The 14th International Symposium, ISRR 2009, August 31 September 3, 2009, Lucerne. Switzerland, volume 70 of Springer Tracts in Advanced Robotics, pages 3-19. Springer, 2009. doi: 978-3-642- 19457-3 I. URL https: //doi org/10./////// 1007 642 19457-3_1.
[10] D. Zhou, 7. Wang, S. Bandyopadhyay, and M. Schwager. Fast, on-line collision avoidance for dynamic vehicles using buffered voronoi cells. IEEE Robotics Autom. Lett, 202) 11..09///A.000
1054, 2017. doi.10.1109/IRA. R01.119...01...65560.00 htt.httss://ao..ooo 2656241.
[11] W. Luo, W. Sun, and A. Kapoor. Multi-robot collision avoidance under uncertainty with probabilistic safety barrier certificates. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: https::/prooeeddngss meurips.cc/papeeee 107880f7000fd3344aaeeoooo
Abstract.
[12] B. Rivière, W. Hönig, Y. Yue, and Chung. GLAS: global-to-local safe autonomy synthesis for multi-robot motion planning with end-to-end learning. IEEE Roboties Autom. Lett., 5(3): 4256, 2020. doi:
hRt
http:///oooogg////////// org. 1109/LAA..
2020. 2994035.

---

[13] G. Sartoretti, J. Kerr, Y. Shi, G. Wagner, T. K. S. Kumar, S. Koenig, and H. Choset. PRIMAL: pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics Autom. Lett.. 4(3):2378 2385, 2019. doi:10.1 LRA. 2019.001222 2903261.
URI. https://ddi. otg/
[14] G. Shi, W. Hönig, Y. Yue, and S. Chung. Neural-swarm: Decentralized dloseproxximit multirotor control using learned interactions. pages 3241-3247, 2020. 101.1.111///RA000000 2020.9196800. URL https: /tw.. org/10. 1109/ ICRA40945 2020. 9196800.
[15] G. Shi, W. Hönig, X. Shi, Y. Yue, and Chung. Neural-swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. CoRR, abs/2012.05457, 2020. URL htttt:ttt/axxiiii org/ abs/2bs2 2012/011....0.. 05457..
[16] Q. Li, Gama, A. Ribeiro, and A. Prorok. Graph neural networks for decentralized multi- robot path planning. In IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 January 24, 2021, pages 1 11785-11792. IEEE, 2020. doi: 100: 1ww.11///R0455443.22222020 000000009.99686.... URL https: /tt////o..... www.i.oogg/i wrw.o.g...... /trg/10.109/ IROS45743 2020 9341668.
[17] I. Tolstaya, F. Gama, J. Paulos, G. Pappas, V. Kumar, and A. Ribeiro. Learning decentralized controllers for robot swarms with graph neural networks. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 671-682 PMLR, 2019. URL http://proceedings mlr press/v100/ tolstaya20a. html.
[18] A. Khan, E. Tolstaya, A. Ribeiro, and V. Kumar. Graph policy gradients for large scale robot control. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019. Osaka, Japan. October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 823-834, PMLR, 2019, URL http:://rooceedings mlr. wtwp://iin........eee/e/o.ooo. 00..0000/aaaaaa...m. html.
[19] A. Khan, V. Kumar, and A. Ribeiro. Large scale distributed collaborative unlabeled motion planning with graph policy gradients. JEEE Robotics Autom. Lett. 60)):5340-5347, 2021. 1o.100.110/RAA.2200 2021...22..004450 3074885. URL https://doi http::://doi..r//1/// 10. 1109/LRA . 3074885.
[20] A. Molchanov, T. Chen, W. Hönig. J. A. Preiss, N. Ayanian, and G. S. Sukhatme, Sim-to-(mullti- real: Transfer of low-level robust control policies multiple quadrotors. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pages 59-66 66. IEEE, 2019. 1oi:10.1119//ROSAAA. 101100000009772009900000010 https://doi org/ 10. 1119/IROS408999 .0199.8966699. 8967695.
URL
[21] Y. Song, M. Steinweg, E. Kaufmann, and D. Scaramuzza. Autonomous drone racing with deep reinforcement learning. CoRR, abs/2103.08624, 2021. URL https://arxiv.. wwweeivivorr/ae www..orgggaa.... /abs/ 2103. 08624..
[22] G. Moraes Barros and E. Luna Colombini. Using soft actor-critic for low-level uav control. arXiv e-prints, pages arXiv-2010, 2020.
[23] S. Shalev-Shwartz, S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. CoRR, abs/1610.03295, 2016. URL //t::////xxvve/// wwaeiii.orr//eo org/ wtrggabb/16100 03295.
[24] K. Gupta, M. Egorov, and M. Kochenderfer. Cooperative multi-agent control us- ing deep reinforcement learning. In G. Sukthankar and J. A. Rodríguez-Aguiin editors, Autonomous Agents and Multiagent Systems AAMAS 2017 Workshops. Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers, volume 10642 of Lecture Notes in Computer Science pages 66-83, Springer, 2017. https://doi. org/10. 100.000007970---3000 3-178----11--7772----- 71682-4_5
URL
[25] J. Panerati, H. Zheng, S. Zhou, J. Xu, A. Prorok, and A. P. Schoellig. Learning to fly a gym environment with pybullet physies for reinforcement learning of multi-agent quadcopter control. 02142.
CoRR. abs/2103.07 10..02142. 142, 2021. URL https: /ariiviiii.
10

---

[26] A. Khan, C. Zhang, S. J.
J. Wu, B. Schlotfeldt, S. Y. Tang, A. Ribeiro, O. Bastani, and V. Kumar. Learning safe unlabeled multi-robot planning with motion constraints. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pages
IEEE, 2019. 1000.11100//005499722001199 8968499.. URL. htt.h:tts//// //to::///do... org/ 10. 1109/ IROS40897 2019. 88688483.
[27] J. Förster. System identification of the crazyflie 2.0 nano quadrocopter. BA Thesis, ETH Zurich, 2015.
[28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. CoRR, abs/1707.0633 2017.
[29] A. Petrenko, Huang, T. Kumar, G. Sukhatme, and V. Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In ICML, 2020.
[30] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, and A. J. Smola. Deep sets. In Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3391-3401, 2017. URL https: pttppe:://roceeeeinggg neurips. Abstract. html.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, pages 5998 6008, 2017.
[32] C. Chen, Y. Liu, S. Kreiss, and A. Alahi. Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 6015-6022. IEEE, 2019.
[33] Batra, Huang, A. Petrenko, Kumar, Molchanov, and Sukhatme. Decen- tralized control of quadrotor swarms with end-to-end deep reinforcement learning. CoRR. abs/2109.07733 2021. URL htt.:: bitapp:://rrivv org/abs/2109 07735.
[34] A. Preiss, W. Hönig, N. Ayanian, and G. S. Sukhatme. Downwash-awe aware trajectory planning for large quadrotor teams. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, September 24-28, 2017, pages 250-257. IEEE, 2017. 1oi: 109/IROS 2017.802165. 2017.82022 8202165.
URL. https: //doi/ org 10. 11199999/SS


Reviews:
[{"review_id": "vzaJZnGzVeI", "reviewer": "Reviewer_YbEi", "summary": "Main Contributions\n===============\n- Successful distributed swarm control for drones in simulation and the real world. \n- Analysis of different network architectures on final performance.\n\nSummary\n============\nThe paper presents successful decentralized drone swarm control using a learned policy. A group of drones is tasked with maintaining static or dynamic formation flight while avoiding collisions with each other or external, possibly dynamic, obstacles. Each drone is controlled by a policy represented by a deep neural network, that observes the current state of the drone, the state of neighboring drones as well as the state of potential obstacles.\nThe control policy is trained using deep reinforcement learning (PPO).\nThe paper compares different model architectures with respect to the final performance of the drone swarm.\nThe proposed approach is successfully deployed in simulation and in the real world. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n===========\n- The paper demonstrates successful deployment of the proposed approach to a real-world setting with 4 crazyflie quadrotors. \n- The paper presents an in-depth analysis of different network architectures.\n- The results are very well illustrated in the supplementary video.\n- The paper is well written and easy to follow.\n\nWeaknesses\n==========\n- Even though the presented results are impressive, the paper does not compare against prior work. All experiments are performed by comparing the proposed approach against different variants of itself (i.e. a 'blind' quadrotor, that has no access to the position of neighboring vehicles).\n- To better understand the performance of the proposed approach, the paper would significantly gain value if it compared with prior work on drone swarm control such as [1], or even naive baselines based on reactive navigation [2] or potential fields.\n\n[1] Hönig, W., Preiss, J.A., Kumar, T.S., Sukhatme, G.S. and Ayanian, N., 2018. Trajectory planning for quadrotor swarms. IEEE Transactions on Robotics, 34(4), pp.856-869.\n\n[2] Florence, P., Carter, J. and Tedrake, R., 2020. Integrated perception and control at high speed: Evaluating collision avoidance maneuvers without maps. Algorithmic Foundations of Robotics XII. Springer, pp.304-319.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper shows convincing results, but lacks any comparison to even naive baselines. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The paper would get significantly stronger if the authors included a comparison against prior work (see above for potential baselines).", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "a3RgzOvQiuZ", "reviewer": "Reviewer_s99C", "summary": "This paper proposes using RL for decentralized quadrotor control. ", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper poses using deep RL for decentralized control. While the paper is well written, it is missing quite a few advances in the field \nof large-scale decentralized robot control using RL/Imitiation learning (see citations below). I would like to expand this review on the basis of the following; novelty, quality and clarity \n\nNovelty: This paper proposes using a deep RL method for learning multi-agent RL policies. This is by no means new and there are many many papers that have expanded this field. The first of these works look to use off-the-shelf multi-agent RL policies by parametrizing each agent using a MLP and then using a centralized critic architecture [1]. These papers realize that these works having issues when attempting to scale to a larger number of robots due to the increase in dimensionality during training. The architectures proposed in this paper suffer from the same issues; as the number of robots is increased the number of inputs to the final \"green\" blocks in Figure 2 increases thus increasing the dimenisonality and as such making these methods not scalable. Further, the methods proposed in [2]-[4] and the papers citing them use graph neural networks to introduce sparsity into the problem while exploiting local structure. These papers produce state-of-the-art results for a large number of robots upto 100). The fact that this paper neither cites these papers nor compares against these methods reflects poorly. As such in my opinion the paper introduces no novelty and no significant contribution compared to the existing state-of-the-art methods for large scale decentralized robot control.\n\nQuality: The paper is well written. However, as mentioned above the experimental section in this paper is underwhelming. The authors show no comparisons with other \"non-author proposed architectures\", i.e all baselines in Fig 4 consist of some modification proposed by the authors and as such there is no idea how these would compare with other state-of-the-art methods. Fig 6 states \"baseline\"; I am unsure which baseline this is. Similar issue with Fig 7. There are no meaningful baselines and as such, I fail to see the merits of this approach. \n\n\n[1] Khan, Arbaaz, et al. \"Learning safe unlabeled multi-robot planning with motion constraints.\" 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.\n[2] Graph Neural Networks for Decentralized Multi-Robot Path Planning Q.Li et. al \n[3] Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks Tolstaya et. al \n[4] Graph Policy Gradients for Large Scale Robot Control , Large Scale Distributed Collaborative Unlabeled Motion Planning With Graph Policy Gradients Khan et. al\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As such, since the paper does not improve/cite upon the existing state-of-the-art methods, for me it adds very little to the existing body of work that looks to use machine learning for teams of robots. As such I cannot recommend this paper be accepted in its current form. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I would recommend the authors spend time looking into the works stated above and think more carefully about where their methods/paper would fit in. \n", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "F1IUqKrx2Ty", "reviewer": "Reviewer_g2nt", "summary": "This paper proposes a multi-agent end-to-end reinforcement learning approach to learning controllers for drone swarms that achieve formation control goals and collision avoidance using only local (and hence scalable) policies.  The policies are trained in a realistic simulated environment, and then deployed in both simulation and on Crazyflie 2.0 physical testbeds.  Each drone's policy learns and acts o a local embedding of the drone's state, an embedding of its neighbors' relative position and velocity -- for the neighbor encoder, Deepset and attention-based architectures are compared, and it is shown that the latter achieves better performance by by learning to suitably weight \"more dangerous\" neighbors (i.e., those that are closer and heading towards the ego-drone) more than \"less dangerous\" ones.  The policies are trained using PPO on several procedurally generated scenarios, including static and dynamic formations, as well as evader pursuit like settings, and the proposed method is also extended to accommodate obstacle avoidance.  Experiments evaluating the two neighbor embedding architectures, the attention weights, and scalability of the approach are conducted.  Finally, scaled-down versions of the Deepset architecture are trained in simulation and then deployed and evaluated on a computationally constrained physical platform (four Crazyflie 2.0 drones).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ the paper is well written and easy to follow\n+ the first successful demonstration of multi-agent end-to-end deep RL for drone swarm control is indeed novel\n+ the use of an attention mechanism for distinguishing \"important\" drones from \"unimportant\" ones given a specific dynamic configuration is a natural and nice idea\n+ the experiments covered a broad range of interesting tasks and demonstrated that the method learned successful collision free formation control policies.\n\nWeaknesses:\n- the baselines against which the method in the paper is compared seem to be quite weak.  The \"blind\" baseline (where the quadrotors can't see their neighbors) is too weak to be meaningful, and whereas the MLP (non-permutation-invariant) policy is sensible, what's missing are some more sophisticated baselines for comparison.  For example, in the related work section, the paper explains that [9,10] significantly constrain the configuration space of the robot to guarantee collision free behavior.  It would have been nice to see a comparison of collision frequency vs. goal tracking performance for the methods of [9,10] and those proposed in the paper.  Similarly, for permutation-invariant policies, recent lines of work have shown that Graph Neural Networks (GNNs) can be used to achieve scalable learning and control in the context of formation control.  See for example:\n\n@inproceedings{tolstaya2020learning,\n  title={Learning decentralized controllers for robot swarms with graph neural networks},\n  author={Tolstaya, Ekaterina and Gama, Fernando and Paulos, James and Pappas, George and Kumar, Vijay and Ribeiro, Alejandro},\n  booktitle={Conference on robot learning},\n  pages={671--682},\n  year={2020},\n  organization={PMLR}\n}\n\nand\n\n@inproceedings{khan2020graph,\n  title={Graph policy gradients for large scale robot control},\n  author={Khan, Arbaaz and Tolstaya, Ekaterina and Ribeiro, Alejandro and Kumar, Vijay},\n  booktitle={Conference on robot learning},\n  pages={823--834},\n  year={2020},\n  organization={PMLR}\n}\n\n(Disclaimer: I am not an author on either paper).\n\n- the argument for scalability didn't strike me as convincing: if 500 million simulation steps are needed to refine policies trained with a smaller number of drones to a larger number of drones, this seems like it will run into computational bottlenecks if the number of drones starts getting large.  I may have misunderstood this point, so would like to see some clarification in the rebuttal: for example, showing wall-clock time for the fine-tuning step as a function of the number of drones would be helpful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Suggesting acceptance based on: first demonstration of end-to-end multi-agent RL to collision free formation control, well written paper, interesting experiments (including sim2real validation).\n\nNot suggesting strong accept based on: weak baselines and unconvincing argument of training scalability.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Comparison to stronger baselines, and clarification of scalability of training time.  See comments above for more details.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: This paper proposes using a deep RL algorithm for decentralized formation control of aerial robots. The algorithm is able to accommodate static and dynamic formations in the presence of state or dynamic obstacles. The method is validated with simulation and real-world experiments on four off-the-shelf consumer drones. 

Clarity: The paper is exceptionally well-written and presents a clear, logical narrative through the motivation, design, and experimental validation of the proposed method. 

Quality: Reviewers noted that deploying a deep RL algorithm on board real quadrotors is a notable achievement and a clear contribution of the paper. However, reviewers also noted that the work in its current form leaves open questions about scalability and comparison to more appropriate baselines. 

Significance/Originality: Feedback on the contribution of the proposed method is slightly mixed. In particular, it seemed like reviewers were least convinced about how the proposed method stacked up against the existing state-of-the-art. 

Pros: 
Reviewers seemed aligned that deploying a deep learning algorithm on board real robots is rather impressive. The paper also clearly communicates the proposed approach, but in the text and supplementary video. 

Main opportunities for improvement: 
In its current form, the revision should address three major themes in the reviewers’ comments:

- More comprehensive literature review: Broadly speaking, this paper did not convince reviewers that it has a place in the existing literature, at least partly because all three reviewers noted specific missing citations. 

- All three reviewers commented that the baselines used in the experimental results section are inappropriate. Namely, they compare the proposed algorithm against variants of itself rather than other state-of-the-art algorithms. The revision should compare against more meaningful baselines. 

- A major consideration in swarm coordination algorithms, especially decentralized ones, is scalability. Reviewers s99C and g2nt both expressed serious concerns about the method’s ability to scale to the order of hundreds of robots. The revision should either include stronger evidence of the method’s scalability or characterize and make a case for why the team size the algorithm does scale to is still a meaningful “swarm”. 

This paper should be seriously commended for being the first to show learned policies on physical quadrotors. However, unfortunately, that accomplishment alone does not merit publication. The rebuttal should still address three key points highlighted above and in reviewers’ feedback - a more complete literature review, comparison against more rigorous baselines, and justification of the technique’s scalability (and exactly how scalability should be defined in this case). 

Thank you for considering our feedback, and we look forward to seeing the updated paper. 


====== Final Decision

I am pleased to accept this paper as part of CoRL 2021. I would like to sincerely thank the authors for the time they put into revising the paper. The authors thoughtfully addressed reviewer feedback and the "opportunities for improvement" listed in the original meta-review. Particularly impactful were the additional experiments. The simulation results for teams of 64 and 128, in my opinion, definitively addresses scalability concerns. Experiments against the buffered Voronoi cells baseline also added useful insight. I am also excited to see DNNs running on real robots; there is significant "invisible" debugging and design work to get that to work. 
Decision: Accept (Poster)