Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor
Authors: Anish Bhattacharya
Marco Cannici
Nishanth Rao
Yuezhan Tao
Vijay Kumar
Nikolai Matni
Davide Scaramuzza
Venue: CoRL 2024 (2024)
Keywords: event-based vision
learning for control
simulation-to-real transfer
aerial robotics
TL;DR: We demonstrate the avoidance of trees with an event camera onboard a fast-flying quadrotor.

Abstract: We present the first static-obstacle avoidance method for quadrotors using just an onboard, monocular event camera. Quadrotors are capable of fast and agile flight in cluttered environments when piloted manually, but vision-based autonomous flight in unknown environments is difficult in part due to the sensor limitations of traditional onboard cameras. Event cameras, however, promise nearly zero motion blur and high dynamic range, but produce a very large volume of events under significant ego-motion and further lack a continuous-time sensor model in simulation, making direct sim-to-real transfer not possible. By leveraging depth prediction as a pretext task in our learning framework, we can pre-train a reactive obstacle avoidance events-to-control policy with approximated, simulated events and then fine-tune the perception component with limited events-and-depth real-world data to achieve obstacle avoidance in indoor and outdoor settings. We demonstrate this across two quadrotor-event camera platforms in multiple settings and find, contrary to traditional vision-based works, that low speeds (1m/s) make the task harder and more prone to collisions, while high speeds (5m/s) result in better event-based depth estimation and avoidance. We also find that success rates in outdoor scenes can be significantly higher than in certain indoor scenes.

Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor
Anish Bhattacharya , Marco Cannici', Nishanth Rao', Yuezhan Tao',
Vijay Kumar!, Nikolai Matni', Davide Scaramuzza
GRASP, University of Pennsylvania Robotics and Perception Group, University of Zurich
Abstract: We present the first static-obstacle avoidance method for quadrotors using just onboard, monocular event camera. Quadrotors capable of fast and agile flight in cluttered environments when piloted manually, but vision-based autonomous flight in unknown environments is difficult in part due to the sen- sor limitations of traditional onboard cameras. Event cameras, however, promise nearly zero motion blur and high dynamic range, but produce a very large volume of events under significant ego-motion and further lack a continuous-tiin sensor model in simulation, making direct sim-to-real transfer not possible. By leverag- ing depth prediction pretext task in learning framework, we can pre-train reactive obstacle avoidance events-to-contry policy with approximated, simulated events and then fine-tune the perception component with limited events-and-depll real-world data to achieve obstacle avoidance in indoor and outdoor settings. We demonstrate this across two quadrotor-eve eeventttor-vvet camera platforms multiple settings and find, contrary to traditional vision-based works, that low speeds (1m/s) make the task harder and more prone collisions, while high speeds (Sm/s) result in better event-based depth estimation and avoidance. We also find that success rates in outdoor scenes can be significantly higher than in certain indoor scenes.
Figure Our event camera-equipped quadrotor avoids static obstacles under considerable ego- motion. Our simulation pre-trained, events-to-contro policy is fine-tuned with real-world perception data, such as that from a forest. Red lines indicate avoidance maneuvers during the flight. We demonstrate obstacle avoidance in indoor, outdoor, and dark environments.
Conference on Robot Learning (CoRL 2024), Munich. Germany.

---

1.0
((s)
0.0
D(O)
Sample batches
D(0)
Binary event masks
Predicted depth and avoidance command
Figure 2: A portion of the continuous event stream is shown from the forest trial in Figure I (showing 5% of the true event stream density for viewability),, with sample mss-batches and the correspond- ing model predictions.
1 Introduction
Quadrotors be flown very fast and with high agility by human pilots. More recently, tonomous, vision-based quadrotors demonstrated champion-level drone racing on fixed tracks [1]. However, achieving high-speed vision-basee autonomous flight through unknown environments re- mains challenging due to the inherent limitations of traditional cameras. Their limited frame rate, low dynamic range, and motion blur, particularly in low-lit scenes such as under forest canopy, can increase uncertainty in traditional model-based mapping mapping-planning out-of-distribution inputs for an end-to-end vision-to-control learning framework.
pipelines or can cause
In contrast, event cameras (also called dynamic vision sensors or neuromorphic cameras) [2] have low latency and low bandwidth and feature pixels that individually detect changes at microsecond- level temporal resolution, effectively alleviating motion blur. Their output event stream consists of positive or negative events that correspond to local, per-pixel increases or decreases in brightness. Therefore, unlike traditional cameras, which output Trame-based data at constant time intervals, event cameras generate continuous stream of events.
This rapid and adaptive perception capability, particularly under variable lighting conditions, sug- gests that an events-driven approach could enable vision-based agile flight at significantly faster speeds than traditional cameras. Most related works using event cameras have focused on dodg- ing dynamic objects thrown flying quadrotor [3, 4, 5]. These works focused on detecting the dynamic object by eliminating events caused by the ego-motion of the robot with respect the stationary background. To the best of knowledge, there obstacle avoidance due to the challenge of significant event-stream bandwidth under considerable ego-motion, which makes the obstacle identification task in the event stream more difficult.
published work for static-
We present a simulation pre-trained, learning-based approach to predict static-obstacll avoidance velocity commands from an incoming event stream, by leveraging depth prediction as pretext task to guide the network in extracting robust, texture-invariant latent representations. Focusing on geometry rather than appearance, depth offers more reasons, has been employed in various robotics tasks. It can easily be gathered in the real world for model fine-tuning, offering an inexpensive alternative compared to expert drone pilot commands.
representation and. for these

---

We focus on tree-like objects and demonstrate that jointly pre-training velocity and depth prediction in simulation allows us to fine-tune the perception backbone with real-world data, leading to the suc- cessful avoidance trees in a forest. We tackle the lack of a continuous-time event camera model in simulation by using a binarized brightness difference event representation, which is less sensi- tive con-continuous event streams, and employ a teacher-student approach distill privileged knowledge into vision-only student network. We demonstrate simulated, real indoor, and real out- door avoidance of trees and tree-like objects with single event camera onboard flying quadrotor, deployed on two different hardware platforms, showcasing the adaptability of the proposed system.
Our contributions are as follows:
The first events-driven static-obstacle avoidance method for mobile robot.
Successful sim-to-real, few-shot transfer of an events-based policy by leveraging depth prediction a pretext, supervisable task.
High-speed (5m/s) real-world avoidance of trees with purely onboard computation, and demonstrating that avoidance using event-based vision improves as speed increases.
Open-source code and data for simulation, data collection, training, and testing.
2
Related Work
Vision-based obstacle avoidance with drones. Various approaches have been developed to ad- dress vision-based obstacle avoidance with quadrotors cluttered and complex environments. Pre- vious literature has explored reinforcement learning with high-fidelity renderers and robust latent representations [6, 7, 1], unsupervised learning strategies interacting directly with the environment [8] or using weakly supervised data [9, 10], well as supervised methods using annotated data [1I] and expert policy distillation [12]. While neural network-based policies directly regressing action commands from images [9, 13, 11] have been proposed, depth has recently emerged as more robust modality for deployment, providing essential obstacle information while discarding unnecessary texture details. Leveraging depth as a primary sensory input, Loquercio et al. [12] demonstrate zero-shot transfer capabilities in the real world while only training in simulation using student-teacher approach. A similar approach is used by Bhattacharya et al. [14], who designed transformer- and recurrent- based policies that directly regress control commands from depth. In- spired by these works, we propose to also exploit depth representation for training; however, since we only use events as policy input, we propose to use depth as an auxiliary task to leam robust latent representation and promote the learning of texture-invariant features for optimal real-world transfer.
Control from event-based vision. Event cameras have motivated rapidly growing body of work in vision-based robotics, specifically in learning reactive policies that directly map sparse event data into control commands. Fast, reactive policies have been demonstrated in the task of catching fast-moving objects with quadruped robots using reinforcement learning [15], and in drones using model- based algorithms to avoid fast dynamic obstacles from hovering conditions [4]. In the domain of drone flight, a neuromorphic pipeline was first demonstrated by Vitale et al. [16], using spiking neural network (SNN) for line-tracking with constrained 1-DoF dualcopter. Similarly, Paredes- Vallés al. [17] employed a neuromorphic vision-to-control approach for real-time ego-motion estimation and control of an unconstrained drone, achieving successful sim-to-real transfer for var- ious flight tasks. Additionally, event-based object detection has been explored as an intermediate task for obstacle avoidance and navigation. Zhang et al. utilized an SNN combined with depth camera for object detection [18], while Andersen et al. [19] trained a convolutional and recurrent network to detect gates under fast motion in drone racing. Similar to these works, we propose to learn vision-to-control policy with event cameras; however, we follow an approach inspired by that proposed in Loquercio al. [12], and use teacher-student training approach where model-based teacher with privileged information is distilled in vision- based student network in simulation. By using depth estimation pretext task and limited real-world annotated data for fine-tuning,
3

---

Flightmare
Unity
Quzdrosor simnlatioo
Divtacle field
Expect policy
decindd
Loss Civeel vel
event froms,
bomn
degath aver
batchity
depet:
[) (depphh depch pnd
Figure 3: Data generation and learning framework. Grayscale images and depths are collected from quadrotor flights in Flightmare [20]. Vid2E [21] processes the images to generate an event stream, which is batched and converted to binary event masks (BEMs) representations for input to the depth predictor D()). The perception loss Cp is computed relative to the ground truth depth. Ground truth obstacle states inform the expert policy, which logs desired velocities for supervising the velocity predictor (o)). Multi-dimensionll quantities are in bold.
demonstrate effective sim-to-real transfer, enabling reliable drone flight and obstacle avoidance in multiple real-world scenarios.
3
Methodology
Our approach addresses the challenge posed by the lack of an effective continuous-time event camera simulator despite the need to transfer our model to continuous-time event streams in the real world.
3.1 Simulation setup and data gathering
We utilize the Flightmare simulator [20] while collecting perception (grayscale and depth images) and command (velocity) data (Figure 3). As there no native, real-time event simulator available, we leverage the photorealistic and high- fidelity images from Unity to generate event-based data (Section 3.2). We use modified version of the privileged expert policy described in Bhattacharya et al. [14], which uses receding-horizon planning to execute velocity commands towards free-space waypoints. We alter the waypoint search to be along a horizon-aligned direction for avoiding trees (see Figure 3, Expert policy). The student velocity predictor (d) predicts scalar velocity magnitude Ug [-1, 1] in the lateral direction, and since it is assumed U, 0, the forward velocity command component u, is computed via ||u, + By 1 1. This vector is then scaled up to velocity command based on a desired speed.
privileged expert policy flying quadrotor
3.2 Event data generation and representation
single event represented as e₁ = (fk: Zk. yk Ps)). denoting brightness change registered by an event camera at time la- pixel location (ær,, yr) in the event pixel array, with polarity ps {-1,+1), +1)- The polarity of event indicates positive negative change in logarithmic illumination, quan- tized by positive and negative thresholds C+.
We use Vid2E [21] to process grayscale images and generate events. While Vid2E provides an op- tion of up-sampling the input video via deep learning- methods (such as Super SloMo [22]). our learning framework requires hundreds of trajectories of data for training, such that the adaptive up-sampe up-sampling impossible due to both long processing time and disk storage limits. Therefore, must use Vid2E without up-sampling, which results in more discontinuous event stream. This

---

necessitates the batching of simulation simulation-gennratee generated events into time windows discretized by the fram- erate of the camera, minimizing the harmful impact of the discontinuities in selecting batches. This eliminates the possibility of using a voxel grid representation of events [23], which may take better advantage of the high temporal resolution of the neuromorphic sensor.
The number of positive and negative events at each pixel location is summed according to their polarity, such that equal numbers of positive and negative events at pixel within the time window At = 1/FPS cam results in a zero count. Edges of nearby objects are accentuated with this way of counting events (generally with leading/trailing edges of opposite polarities for passing objects, and the same polarity for head-on objects) relative to the object's interior or other ground or complex textures. We then convert this to binary event mask (BEM), which we found resulted in improved training over the histogram array of event counts. Given event stream E, element of the BEM given pixel location (2;, y) is follows.
((Σ 1(e
+1) -1)- - 1le, -1) $ #0)
BEM
(1)
3.3 Learning framework
The perception backbone ((0) of the learning framework consists of U-Net architecture [24] to predict depth from the input BEM, inspired by works that convert between images and events [21] and predict dense depth from monocular events in particular settings [25]. The U-Net model is popular for segmentation tasks in medical imaging and modern computer vision works; this lends well to obstacle avoidance since obstacles must be segmented from the background. A formulation close to the original U-Net was used, except that interpolations were used for skip connections rather than crops, and a ConvLSTM [26] was used after the middle layers, prior to the transpose convolutions, to promote temporal consistency in generated depth images over time. Depth images gathered through expert rollouts provided supervision of this module via L2 perception loss Ly- weighted at each pixel by the inverse ground truth depth value at that pixel, thereby prioritizing nearby objects in each gradient step.
The velocity predictor (Ø) takes as input the depth prediction following the transposed convolu- tions and consists of convolutional neural network ending in fully connected layers, producing the single scalar avoidance velocity Vy- This velocity predictor trained jointly with the percep- tion backbone and also utilizes an L2 loss function E. with the expert's commanded velocity. The weights used on the perception C, and velocity Lo loss terms are (1)
depth.. depth 2
Cpr.
L. Gu:,,uy 1luy
depth,
3.4 Few-shot transfer to the real world and to other platforms
only gather real-world perception data for fine-tuning the perception backbone. For given scene, we collect aligned event batch and depth image pairs with hand- held rig of an event camera rigidly mounted with an Intel Realsense D435 depth camera, calibrated via E2Calib [27] (see supplementary). While simulation datasets ranged from 100-200 long tra- jectories, real datasets used for fine-tuning consisted of 10-20 shorter trajectories. Frame-batching of events enables image-based data augmentation techniques that particularly apply to the events- based quadrotor fly-and-avoid task, such as applying small rotations (simulating roll angles), left- right flipping, and adding noise. Applying these augmentations real datasets drastically improved performance during deployment. See the supplementary for details on the impact of simulation pre-training for ((0).
Without an expert pilot,
We train models with the resolutions and characteristics of the Davis346 event camera [28] but demonstrate the generalization our approach by also utilizing the Prophesee Gen3. VGA event camera [29] for indoor and outdoor experiments (Figure 5). These two sensors are very different, coming from two different manufacturers, with the Prophesee camera having sensor nearly four
5

---

Velecit:: 1 Stecmumm iateral scoilaration: 24 rajs
20
Success for simulated ablations
Vedecas sujerayy
57%
37%
27%
Figure 4: Simulation trials conducted in Flightmare, where approximated events (topleft) are used to produce predicted depth and velocity commands. Execution success rates presented across ablations of our training method (lower left) show that jointly training the perception D(f) and velocity V(Ø) modules beneficial. Note that the low observed simulation success rates result from the out- of-distribution event stream approximation (Equation 3) that must be resorted to during real-time execution due the lack of continuous-time event camera simulator.
times larger and a wider dynamic range, but also a larger latency, than the Davis. When transfer- ring simulation models to the Prophesee camera, we similarly fine-tune with calibrated Prophesee- Realsense data (and center-crop desired resolution of 260 346). Further details of deployment code and methods in the supplementary.
4 Results
4.1 Simulation
We perform 100 rollouts of the trained policy in simulated forest environment, with randomized configurations of trees unseen during training (Figure 4). However, since deploy in simulation in real-time, we are limited to approximating the live, incoming event stream via the difference of log images. Given consecutive images imo, im,, the number of events at a pixel (x, y) between the corresponding timestamps is as follows:
N eeventar.y [log((im,..)) logg:...... /C#]
(3)
This makes the input event stream achieve success rates around 60% with our model on moderate- length trajectory of 10m (Figure 4, Success rates for simulated ablations, Jointly trained). If we increase the desired travel distance to 60m, we achieve zero collisions on 15% of trials. Other collision metrics are shown in the sup- plementary.
for models trained via Vid2E generation, so
Ablations. The perception backbone and velocity predictor are jointly trained (see Section 3.3), such that the velocity predictor learns to interpret the depth predictions produced by D(A). These depth images can exhibit artifacts where there are few events, and hallucinate parts of objects that are not consistent in time (see supplementary). We ablate this choice of joint training by training the perception backbone and velocity predictor independently, where the velocity predictor given ground truth depth images. Figure (lower left) shows that over 100 trials, the independently trained model achieves less than half the success rate of joint training on longer trajectories. We also train model where depth supervision is removed entirely. This model achieves the lowest success rate, showing that depth prediction is useful step in learning velocity commands for obstacle avoidance from event vision.

---

Lights turmed off
Figure 5: (top left) The two quadrotor platforms used for indoor and outdoor experiments with dif- ferent event cameras. The Falcon250 [30] additionally has a VOXL board [31] for state estimation. (lower left) An event-volume (10% true density) produced by the continuous event stream from the in-the-dark trial, with select event batches, corresponding BEMs, and network predictions shown. Note the high density of negative (blue) events and positive (red) events when the lights turn off and on, respectively. (right) A variety of real experiments in indoor and outdoor conditions. Our event camera-equipped quadrotor can avoid obstacles in the dark, where traditional cameras would fail.
Close
100%
Success
velocity increases
Pdloetat inpplen taxed
Agln Bori:366
6756
40%
IL5-1.0
1.7-2.0m/s
Figure 6: In the real world, events-based obstacle avoidance performs better velocities increase, contrary to prior work using traditional vision. This is clear from the model's improved predicted depth when the obstacle is moving faster in the camera view due proximity (left) and during faster flight outdoors (middle), and is shown in improved success rates across both platforms, even when the event camera biases for one platform is well-tuned ("Falcon250-Prows Propheeee tuned', discussed in more detail in the supplementary). This is due to the larger volume of events seen from the obstacle of interest with increased ego-motion.
4.2 Hardware experiments
We conduct hardware experiments with two platforms using different event cameras (see Section 3.4, Figure 5). Results are presented for indoor and outdoor trials, where odometry measured from motion capture and onboard VIO, respectively. As seen in Figures 6(a) and 6(b), the depth images formed by (()) can be amorphous, with close depth on the relevant obstacles but noisy backgrounds on textured surfaces and other objects. Furthermore, the predicted depth of the obstacle may not conform to the object's 's outline and it may evolve with time (see supplementary video). While certain elements of the learning framework improve this, such as the ConvLSTM layer, we find that the simulation pre-trained convolutional network V(Φ) is not always able to avoid obstacles well with real-world depth predictions. Therefore, we use the open-source, obstacle avoidance pre-trained ViTLSTM (combination vision transformer- velocity prediction module [14] which contains SegFormer-basee lightweight depth-to-velode predictor with high generalization capabilities.

---

Indoor. For both quadrotor-event camera platforms and the corresponding environments, indoor trials were conducted in motion capture environments with tree-like obstacles that were used to fine-tune the perception backbone offline. 18 and 20 trials were conducted for the Agile-Davis346 and Srroon220-PPophhsee respectively, at varying forward speeds grouped into two bins as seen in Figure 6(c). While non-event-based obstacle avoidance methods in prior work deteriorate in performance as speeds increase, this data shows events-based avoidance notably performs better 2m/s than mm/s indoors. By observation, this is due to the increase in obstacle-related event volume resulting from increased ego-motion, which helps parse obstacles from background textures. This is particularly the case when facing an obstacle head-on, compared to passing by, and also at a far distance as in Figure 6(a) (left). At up to Im/s forward speed, not enough events are seen on the object, resulting poor depth estimate and often a collision. We also conduct trials fields of multiple obstacle types as well as in the dark (Figure 5), where traditional depth cameras fail to identify obstacles.
Outdoor. Outdoor tests were conducted with the Dalcon255--Poophesee platform on real trees in an urban environment, where the background scene includes stopped and moving cars, sidewalks, buildings, and other trees. At the times of experimentation, wind speed was 8-16kph with gusts of 24-32kph. After training the model on these trees, we deployed it with desired speeds of 1-5m/s and observed a 1/13 (85%) success rate, where the only two failures (collisions) occurred when the quadrotor tried avoid right with low battery but pushed left by wind gusts. Notably, compared to indoor testing with the Palcon250-Prophesse the outdoor trees, resulting in better avoidance capacity. This may be due to the existence of certain high-volume event emitters indoors (motion capture and industrial lighting), as well as the artifi- cial and patterned background textures in the test space that may obscure the event patterns on an obstacle until very close. Furthermore, similarly to the indoor results, as forward speed increased from Im/s to 5m/s, the quadrotor exhibited better avoidance ability in terms of magnitude and earli- ness of the avoidance maneuver, contrary prior work in high-speed vision-based avoidance where performance tends to decrease with increasing velocity [12].
we observed more consistent depth prediction on
5 Conclusion
We present simulation and real-world results demonstrating that we avoid static obstacles with only an event camera onboard a moving quadrotor. We rely on frame-based representation of the event stream, which, as we do not have a continuous-time events simulator, allows simulation pre- training of policy that can then be fine-tuned with real-world perception data. Data augmentation applied to the binary event mask representation helped overcome the large events sim-to-real gap and even the real-to-real gap (stemming from varying lighting, sensor noise, and background textures). Simulation results support that supervising depth as intermediate step in the network improves success rates both moderate and long trajectory lengths.
Our real-world experiments suggest that unlike traditional perception- obstacle avoidance, event-based avoidance performs better at high speeds (5m/s) than at low speeds (Im/s), as more events are seen on obstacles which improves depth prediction quality and thereby the avoidance action. Also notably, outdoor tests had a higher success rate than indoor tests done on the same platform, likely due to the artificial event sources and patterned background textures indoors that can obscure textured obstacles.
Limitations. This work focuses on the use of event cameras for obstacle avoidance, and demon- strates surprising benefits of this perception modality, including flying the dark. However, we do not directly compare the benefit of an event camera's low sensor latency against that of a tra- ditional vision sensor, in part due the processing latency of model inference, which is currently magnitudes larger than the event camera's 's sensor latency. Please refer to the supplementary for more detailed discussion of these latencies. Finally, though collecting real-world perception data is inexpensive compared to collecting expert pilot data, it is nevertheless required to fine-tune our model with calibrated events-depth data for novel seenes and objects.
8

---

Acknowledgments
This work was partly supported by the European Research Council (ERC) under grant agreement No. 864042 (AGILEFLIGHT), and by NSF award ICCS-2231349 SLES-2331880, and CA- REER2045834. From the Robotics and Perception Group, we thank Alex Barden, who engineered the Agile-Davis platform, Angel Romero, who assisted in initial experimentation and debugging, Nico Messikomer, who provided input in events-based learning pipelines, Daniel Gehrig and Math- ias Gehrig, who advised on general directions, and Yunlong Song, who helped on simulation tech- niques. From Kumar Lab, we thank Alex Zhou, who helped develop hardware for the Falcon250 platform and attachments, Yuwei Wu and Fernando Cladera, who provided input and hands-on sup- port when debugging experimental issues, and Alice Li and Xu Liu, who advised in outdoor experi- mentation.
References
[1] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):8229887, 2023.
[2] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, Conradt, K. Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154- 2020.
[3] D. Falanga, S. Kim, and D. Scaramuzza. How fast is too fast? the role of perception latency in high-speed sense and avoid. IEEE Robotics and Automation Letters, 2019.
[4] D. Falanga, K. Kleber, and D. Scaramuzza. Dynamic obstacle avoidance for quadrotors with event cameras. Science Robotics, 5(40))caaz9712, 2020.
[5] N. J. Sanket, C. M. Parameshwara, C. D. Singh, A. V. Kuruttukulam, C. Fermüller, D. Scara- muzza, and Y. Aloimonos. Evdodgenet: Deep dynamic obstacle dodging with event cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10651- 10657. IEFE, 2020.
[6] J. Xing, L. Bauersfeld, Y. Song, C. Xing, and D. Scaramuzza. Contrastive learning for en- hancing robust scene transfer in vision-based agile flight. IEEE International Conference on Robotics and Automation (ICRA), 2024.
[7] Sadeghi and S. Levine. Cad2rl: Real single-image flight without single real image. Robotics: Science and Systems, 2017.
[8] D. Gandhi, L. Pinto, and A. Gupta. Learning to fly by crashing. In 20/7 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 3948-3955 IEEF, 2017.
[9] A. Kouris and C.-S. Bouganis. Learning fly by myself: A self-supervised can-based ap- proach for autonomous navigation. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1-9. IEEE, 2018.
[10] K. Lamers, S. Tijmons, C. De Wagter, and G. de Croon. Self-supervised monocular distance learning on a lightweight miero air vehicle. In 2016 IEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 1779-1784. IEEE, 2016.
[11] A. Loquercio, A. I. Maqueda, C. R. Del-Blanco, and D. Scaramuzza. Dronet: Learning to fly by driving. JEEE Robotics and Automation Letters, 3(2):1088- 2018.
[12] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics. 6(59), Oct. 2021. ISSN 2470-9476. doi: 10. htll://ciioobtiisaaag888//

---

[13] A. A. Zhilenkov and I. R. Epifantsev. System of autonomous navigation of the drone in difficult conditions of the forest trails. 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pages 1036-1039. IEEE, 2018.
[14] A. Bhattacharya, N. Rao, D. Parikh, P. Kunapuli, N. Matni, and V. Kumar. Vision transformers for end-to-end vision-based quadrotor obstacle avoidance. arXiv preprint arXiv:2405. 2024.
[15] B. Forrai, T. Miki, D. Gehrig. M. Hutter, and D. Scaramuzza. Event-based agile object catching with quadrupedal robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12177-12183, IEEE, 2023.
[16] A. Vitale, A. Renner, C. Nauer, D. Scaramuzza, and Y. Sandamirskaya. Event-driven vision and control for uavs on a neuromorphic chip. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 103-109. IEEE, 2021.
[17] F. Paredes- Paredes-Vallés, J. J. Hagenaars, J. Dupeyroux, S. Stroobants, Y. Xu, and G. de Croon. Fully neuromorphic vision and control for autonomous drone flight. Science Robotics, 9(00): cadi0591, 2024.
[18] X. Zhang, J. Tie, J. Li, Y. Hu, S. Liu, X. Li, Z. Li, X. Yu, J. Zhao, Z. Wan, et al. Dynamic obstacle avoidance for unmanned aerial vehicle using dynamic vision sensor. In International Conference on Artificial Neural Networks, pages 161-173. Springer, 2023.
[19] K. F. Andersen, H. X. Pham, H. I. Ugurlu, and E. Kayacan. Event-based navigation for au- tonomous drone racing with sparse gated recurrent network. In 2022 European Control Con- ference (ECC), pages 1342-1348. IEEE, 2022.
[20] Y. Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, Flightmare: A flexible quadrotór simulator. In Conference on Robot Learning, 2020.
[21] D. Gehrig, M. Gehrig, J. Hidalgo-Carrió, and D. Scaramuzza. Video to events: Recycling video datasets for event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3586-3595, 2020.
[22] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. Learmed-Milles and J. Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000-9008, 2018.
[23] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 989-997, 2019.
[24] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In Medical image computing and computer-assiste 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages
Citveettoo-MIICCA
Springer, 2015.
[25] J. Hidalgo-Carri,, D. Gehrig, and D. Scaramuzza. Learning monocular dense depth from events. In 2020 International Conference on 3D Vision (3DV), pages 534-542. IEEE, 2020.
[26] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung. W.KK. Wong, and W.-. Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural in- formation processing systems, 28, 2015.
[27] M. Muglikar, M. Gehrig, D. Gehrig, and D. Scaramuzza. How to calibrate your event camera.
In IEEE Conf. Comput. Vis. Pattern Recog. Workshops (CVPRW), June 2021.
[28] Vation. Davis346 documentation, 2019. URL http:///hiiaiiioccccmmopcccmtttt
2019/08/DAVIS346, Accessed: 2024-06-02.
10

---

[29] P. AL. Csd3-mved documentation, 2024. URL http://ssapprtrppphhissssppppptp
articles/csdmmved, Accessed: 2024-06-02.
[30] Y. Tao, Y. Wu, B. Li, F. Cladera, A. Zhou, D. Thakur, and V. Kumar. SEER: Safe efficient exploration for aerial robots using learning to predict information gain. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 1235-1241, IEEE, 2023.
[31] ModelAI ModalAI VOXL Technical Docs, 2024. URL https://does https//does.mooaaai.comooooooo
datsshertssooxl-datasheee

---

Supplementary material for: Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor
S1 Simulation details
S1.1 Simulation environment
We utilize the Flightmare simulator [1] to collect photorealistic perception data while running privileged expert [2] through a forest-like environment. This simulator package runs the Unity rendering engine, and to generate randomized forest-like environments for training we place and orient 100 trees uniform-randome, across a 40m x 50m area. A grayscale image of the scene, the corresponding events batch (computed offline via Vid2E [3]), and the depth image are shown in Figure S1.
(a) Expert policy.
(b) Corresponding event batch.
(c) Corresponding depth image.
Figure S1: One timestamp of expert policy trajectory in simulation. (I(a)) The expert chooses the closest collision-fre free waypoint (free waypoints represented as blue dots, opposed to collision waypoints in red) for planning. (1(b)) The event batch is computed offline via Vid2E, and used to train D(0) with the (I(c) depth image as supervision.
S1.2 Privileged expert
Our dataset for training both the perception backbone and velocity predictor consists of 100 trajec- tories of the privileged expert traveling through a forest-like environment. For each trajectory, the expert's 's forward speed was miiform-randoml. selected between 3-7m/s. The privileged expert has access to obstacles within 10m radius of its current position and calculates collision-free way- point relative inflated obstacles to issue velocity commands towards in receding horizon fashion. Query points are along horizon-alignee line 10m ahead of the drone, and at 5Hz the collision-fiee free waypoint closest to the center of the line (corresponding to forward flight) is chosen, then multiplied by a constant gain to calculate velocity commands. Figure S1 shows the corresponding data; ((a) depicts a white arrow with the commanded velocity.
$1.3 Additional simulation collision metrics
We present additional collision metrics from simulation rollouts for the proposed model and ab- lations described in Section 4.1 fointly-trainedd independently-trraned and seeett-supeeviiiion models) in Figure S2. Statistics are calculated from 50 trials per reported metric, across forward velocities ranging uniformly between 3-7m/s. As we allow one or two collisions in the success rate calculation (typically "success rate" refers only to zero collisions, as in the main text) we observe a

---

Success rates for ablaioon model simulation No collisions
Success rates
ablations model simulation 3 fower collsions
Success rates for ablotions our model Simulation 2 fewer ca lnsions
ILLL
(a) Zero collision rates.
(b) One or fewer collision rates.
(c) Two or fewer collision rates.
Figure S2: Rates for moderate and long trajectories in simulation of zero collisions (typically re- ferred to just as "success rates") and one and two collisions. the long trajectory there is consis- tently better performance with the jointly-trained (proposed) model.
leveling out of performance among all models on the moderate trajectory, but performance remains consistently better on the long trajectory with the jointly trained model.
S2 Event camera hardware and calibration with a depth camera
Two quadrotor-event camera platforms were used, one with a Davis346 (resolution 260 x 346) and with Prophesee Gen3.1 VGA (resolution 480 640) (Figures 5). As mentioned in the text (Section 3.4), these two sensors have very different sensor and event-pixel characteristics; fur- thermore, the biases were left as the default values for most experiments. To enable the simulation pre-trained policy to extend to a real-world Davis346 and further to the Prophesee camera, calibrated and mynhsyycchonizzed event batches and depth images were gathered to fine-tune models with real data gathered from a hand-held device with both sensors (Figure S3). ROS was used to run both sensors in parallel while recording message data that includes ROS timestamps and corresponding events messages for either event camera ViniVation AG ROS driver [4] was used for running the Davis346, and Prophesee AI ROS driver [5] 3rd-party Metavision ROS driver [6] was used for running the Prophesee) as well as ROS timestamps and corresponding depth and "infraredl" messages from the D435 depth camera. Since the "infraredl" camera is pre-aligned with the depth images, we use these images to run multi-camera calibration. E2Calib [7] was used to generate im- ages from either event camera at timestamps specified by the depth images (for both types, rosbag data needed to be converted DVS EventArray message type [8, 9, 10]). With the images from E2Calib and "infraredl". use Kalibr [11] perform multi-camera calibration.
As all perception models are sized according to the Davis346 resolution, for the Prophesee camera we center-crop a 260 x 346 sized event batch for input to the perception backbone D Note that while the simulated camera and corresponding Vid2E events output have different intrinsics than the Davis346 and Prophesee cameras, no alignment performed during the sim-to-real transfer, and we expect the fine-tuning training of the perception backbone to incorporate changes in lens geometry and distortion.
S3 Computational details for simulation and hardware
We provide computational details for the simulation computer and the Falcon250-Pros Prophesee hard- ware platform in Table $1. For an indirect comparison, [12] utilizes Jetson TX2 onboard compute for depth image-to-trajecton planning, where inference time is 38.9ms. In contrast, we use event streams directly as the perception input, predict a depth image intermediately, and predict velocity commands in an end-to-end event- event-based perception-to -onnnnrrol framework, with total onboard infer- ence time of 73ms. Note that further optimization of the computational complexity and processing latency may be possible, as this was not the focus of this work. Equation 1l of [12] supplementary material specifies how reduction in sensing time tg may be possible with event cameras over traditional cameras, may increase the theoretical maximum velocity achievable while avoiding

---

Event batch
D435 depth
Indoor Davicce
Falrondras Prophesee
Indoor
Indoar (mutt)) Falcon25
Outdeor Falcon250/Prpphi
Handheld rig for real-world data collection.
(b) Sample event batch-depth training pairs.
Figure S3: Example of rigidly-attached event camera (Prophesee Gen3.1 VGA) and depth cam- cra (Intel Realsense D435) used to gather events batches and depth images for approximate post- processing calibration and time synchronization, for use in fine-tuning perception backbone from Figure 3. Note that an IR filter is used on the event camera lens so that we may enable the infrared emitter on the D435 for better depth image precision.
Simulation
Malcon250-Prophesse
Num. parameters Inference time [ms]
Num. parameters Inference time [ms]
D(0)
9,856,673
255
9,856,673
67
CNN
VITLSTM
(Φ))
923,321
3,563,663
6
Totals
10,7799994
258
13,420,336
73
Table S1: Model size and inference time for simulation (CPU-only Intel Core 17-10710U 16GB RAM) and for hardware experiments with the Falcon250-Prophesee platform (CPU-only Intel Core 77-10710U, 32GB RAM). We provide inference times for the perception backbone D and the velocity prediction V()). Note that for simulation (e)) consists of a CNN, whereas for real-world experiments (ω)) consists of a VITLSTM model [2].
given cylindrical (i.e., tree-like) object. While the current work batches events by 33ms windows, this batch can be updated with incoming events in real-time, frequencies only upper-bounded by the pre-processing required of events. This effectively utilizes the low sensing latency (,, of an event camera. In this work, pre- pre-processing requires 650μs,, or 2.250ms with an optional event-frame align- ment step to better match training data (based on event camera and depth camera calibration). Given these pre-processing latencies, and the formulation given in [12] with the same drone parameters, we would be able to increase the theoretical maximum avoidance flight velocity from 13.5m/s to 15.83m/s 15. 15.76m/s, respectively. Considering the total sensing and processing latency t, + ty- [12] cites 66 10.3 76.3ms, and this work, specified above, would have a worst-case latency
of 2.25 73 75. 25ms.

---

Progressive D(0) fine nue-t
Traiindd
Simulation Simulation
Simulation
Real Indoor Real Indoor
Real indoor
outdoor
Evaluated
Real indoor
Real cutdoor
outdoor
Event batch
Predicted depth
GT depth
Figure S4: Progressive evaluations more challenging environments, ranging from simulation to real indoor to real outdoor. A simulation pre-trained model is evaluated in simulation, then on a real indoor environment before and after fine-tuning, and
Simulation pretraining enables faster training on real-world data
0.40
pretraining penn--rrtraining
Ep20
Event batch
Ep200
0.50
Training
0.20
0.15
3.05
50
100
150
200
250
300
350
400
Epochs
Figure S5: Simulation pre-trained (orange) events-to-deple models train faster on real data than non-pre-trained models (blue). Figure inserts contain evaluation examples of the predicted depth epochs O, 20, and 200 of training both models on real indoor data. Input event batch and ground truth depth are the left of the plot.
S4
Pre-training and fine-tuning characteristics of D(0)
S4.1
Generalization and fine-tuning
In Figure $4 we provide generalization and fine-tuning examples of the perception backbone D(0) pre-trained in simulation, then applied and fine-tuned to progressively more difficult real-world scenes. The simulation-trainee model's is evaluation on real indoor data produces a recognizable depth image, which fine-tuned to achieve appropriate relative scaling and in-filling.
S4.2 Pre-training speeds up fine-tuning on real scenes
Training loss curves during training of D(#) on real indoor event camera data for a simulation pre- trained model and non-pre-trained model are shown in Figure S5. The pre-trained model produces a recognizable depth image immediately and fine-tunes the real data in less than 20 epochs, bar some improvements in consistency, whereas by epoch 200 the non-pre- trained model has yet to learn the relative scaling and edges of the obstacles.

---

S5 State estimation for outdoor flight
As described in the main text, the Falcon250-Prophe Prophesee platform the VOXL board [13] for onboard state estimation. This board contains an independent computer, connected to the pri- mary computer via an Ethernet cable. This visual-inertial odometry (VIO) module provides six degrees-of-freedon robot poses at 30Hz. We employed an Unscented Kalman Filter (UKF) that takes high-frequency IMU data in combination with these pose estimates to provide odometry at 150Hz. Low-level attitude and thrust controllers are run on PX4 open-source [14] control stack us- ing the Pixhawk4 flight controller, with desired values provided by custom controllers running on the primary onboard computer.
S6 Tuning event camera biases for increased performance
Effect of event bias tuning indoor
1.0
Detsult bieses Aduated biesen
rate
04
0.0 0.5.10
1.500
Velocity [mis]
Figure S6: Success rates for the Malcon250-Prophese platform avoiding an obstacle indoors, with and without adjusted biases meant to reduce background textural and random photon arrival noise. This tuming results in double of the success rate while maintaining the observed upwards trend in success rate as forward speed increases.
Bias name bias diff bias diff on
Default value
Adjusted value
299
299
384
400
bias diff off
222
200
bias fo bias_hp hpf bias
1477
1590
1499
1499
1500
1400
Table S2: Default and adjusted values for Prophesee Gen3. event camera biases used for addi- tional indoor testing with the Falcon250-Prophesee platform. Changes include widening the contrast threshold gap (bias diff on, bias diff off), narrowing the sensor bandwidth (bias fo), and reducing refractory period (bias refr).
Indoor hardware results, well some outdoor results, in the main text are deliberately achieved without tuning of event camera biases for either event camera used, as we aim to show trends that appear naturally from our simulation-to-reall few-shot transfer to multiple hardware platforms. For example, we observe increasing success rates as flight speed increases, and also a higher success rate as we transition to an outdoor setting. However, as mentioned in Section 3.4, the Prophesee camera has four-times-larger sensor than the Davis346, and thus, when paired with no additional bias tuning, produces significantly higher "noise" levels due to both background textures and random photon arrival (see Figure 6(a), top row). Through observation, this results in lower success rate than that achieved by the Agile-Davis. platform.

---

This background level of events is particularly strong in the chosen indoor testing environment, where motion capture, industrial lighting, and surrounding patterned, artificial textures create large volumes events (discussion in Section 4.2). Here, we present analogous results from the same in- door conditions but with slightly tuned event camera biases (Table $2). Figure S6 compares success rates with (24 trials) and without (20 trials) tuned biases for the Falcon250- Srroo220--Propheee platform. We observe doubling of the success rate. Other tuning may improve performance further, including training with varied time window batching of events under these improved biases, so the model may events.
predict obstacle depth accurately from the fewer

---

References
[1] Y. Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, Flightmare: A flexible quadrotor simulator. In Conference on Robot Learning, 2020.
[2] A. Bhattacharya, N. Rao, D. Parikh, P. Kunapuli, N. Matni, and V. Kumar. Vision transformers
for end-to-end ussionbasees quadrotor obstacle avoidance. arXiv preprint arXiv:2405.1039) 2024.
[3] D. Gehrig, M. Gehrig, Hidalgo-Carrió, and D. Scaramuzza. Video events: Recycling video datasets for event cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3586-3595 2020.
[4] iaiion AG. Dv-ros: Robotic operating system integration for inivation cameras, 2024. URL https://gitlab.. httpa//a.....miiiaattttot/t/////
Accessed: 2024-06-10.
[5] P. AL. Prophesee ros wrapper: Ros driver for prophesee event-based sensors, 2024. URL https://gt /tgtt/////tu......./. hwt.li.cooppppphhpee http///pppeeeeeeppppppeeeee 10.
ros_ _oo__err.r.. Accessed: 2024-06-
[6] B. Pfrommer. A driver for event based cameras using the metavision sdk, 2024. URL https:
// //github, com coom/ros rros event- wcmerramatavission.coo diriiiiiiiodririirr.oieeeo Accessed: 2024-06-10 10.
[7] M. Muglikar, M. Gehrig, D. Gehrig, and D. Scaramuzza. How to calibrate your event camera.
In IEEE Conf. Comput. Vis. Pattern Recog. Workshops (CVPRW), June 2021.
[8] C. Brandli, R. Bernr, M. Yang, S.-C. Liu, and T. Delbruck. A 240x 180 130 db ps latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10): 2333-233 2333-2341, 2014.
[9] P. Lichtsteiner, C. Posch, and T. Delbruck. A 128x 128 120db 15mus latency asynchronous temporal contrast vision sensor. IEEE journal of solid-state circuits, 43(2):666-576, 2008.
[10] E. Mueggler, B. Huber, and D. Scaramuzza. Event-based, 6-dof pose tracking for high-speed maneuvers. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2761-2768. IEEE, 2014.
[11] Maye, P. Furgale, and Siegwart. Self-supervised calibration for robotic systems. In 2013 IEEE Intelligent Vehicles Symposium (IV), pages 473 473-480. IEEE, 2013.
[12] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight the wild. Science Robotics, 6(59), Oct. 2021. ISSN 2470-9476. d0i: 7226/scioobotics.abg5:53 URL http://dx. doi www.o.g.r....... 111////////oooiiaooooooo/ abg5880.
[13] ModelAI. ModalAI VOXL Technical Does, 2024. URL
[14] L. Meier, D. Honegger, and M. Pollefeys. Px4: A node-based multithreaded open source robotics framework for deeply embedded platforms. In 2015 IEEE international conference on roboties and automation (ICRA), pages 6235-6240. IEEF, 2015.


Reviews:
[{"review_id": "T54YcwRxLf", "reviewer": "Reviewer_n5tQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces the first events-only static-obstacle avoidance method for quadrotors using a monocular event camera. It leverages depth prediction to guide the network in extracting representations. The approach involves pre-training a reactive obstacle avoidance policy in simulation and fine-tuning with limited real-world data. Key contributions include demonstrating the first purely events-driven obstacle avoidance for a robot with significant ego-motion, successful sim-to-real transfer for tree dodging, and achieving purely onboard computation at 5 m/s. The method excels at higher speeds and shows better performance in outdoor settings. Event cameras enable vision-based agile flight at faster speeds due to their high dynamic range and motion blur elimination.", "summary_of_recommendation": "The paper presents a novel approach to static obstacle avoidance for quadrotors using event-based cameras, demonstrating success in both simulated and real-world environments. The work is of good quality, clearly written, and offers original contributions to the field. Its significance lies in showcasing high-speed navigation in cluttered environments and the counter-intuitive finding that performance improves at higher speeds. However, there are limitations in generalizability and full exploitation of event camera advantages due to computational constraints. The research has strengths in being the first to demonstrate this capability for quadrotors and successful sim-to-real transfer, but weaknesses in limited obstacle types and the need for fine-tuning in new environments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please provide more details on the limitations of generalizability. How much fine-tuning is required for new environments? How different is the data that the model is fine-tuned on from the data that it sees at test-time in the new environment? Can you quantify the performance drop when the system encounters entirely new obstacles or environments without fine-tuning?\n\nThe paper mentions that computational requirements limit the maximum achievable speed. Can you provide more details on the computational resources used?\n\nGiven that the low latency of event cameras isn't fully utilized due to computational constraints, how does your approach compare to traditional vision-based methods in terms of overall system latency and performance? At the speeds that your system is capable of operating at, do they outperform traditional vision-based methods? Can you add experiments in simulation that compare the performance of the two?\n\nThe focus is primarily on tree-like obstacles. Can you provide data on performance with a wider variety of obstacle types and shapes? This would help elucidate how much the approach might have overfit to this particular type of obstacle.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h41fSHSPt6", "reviewer": "Reviewer_gGzy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper the authors propose a sim-to-real transfer learning framework for vision-based autonomous flight with event cameras, and use it to demonstrate, for the first time, static obstacle avoidance with such a sensing setup. Methodologically, the authors demonstrate the value of depth prediction as an auxiliary pre-training task for learning a lateral velocity collision avoidance policy with a privileged teacher-student approach; they also propose a frame-batching approach to processing the event stream that allows the application of established neural network architectures and data augmentation techniques from the computer vision literature. Experiments demonstrate the efficacy of the sim-to-real transfer and overall policy learning framework, for indoor and outdoor flight at high speeds.", "summary_of_recommendation": "I recommend this paper's acceptance at CoRL 2024 as it brings to the robot learning community an interesting sensorimotor policy learning problem (versatile events-to-control flight) and demonstrates how recent ideas in teacher-student transfer learning may be adapted to provide a solution. However, as noted above, I believe the paper would be improved if the practical utility of this work could be more clearly established.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "From the comments/questions above, please prioritize:\n- What would the ideal application setting (e.g., what environments, what speeds) for this technology be, where it might have the potential to outperform alternative solutions?\n- How do the \"real datasets used for fine-tuning consist[ing] of 10-20 shorter trajectories\" compare to the deployment conditions (e.g., speed) of the real-world experiments, and how was the real depth data validated?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n9MRhr9PMY", "reviewer": "Reviewer_tqUw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This method proposes using event-cameras for learning quadrotor control and obstacle avoidance policies from visual observations.", "summary_of_recommendation": "The authors present a novel method of learning static obstacle avoidance for quadrotors using event cameras by leveraging depth estimation from events as an intermediate step. The novelty of the approach and qualitative and quantitative results at high velocities merit acceptance. However, the lack of sufficiently challenging scenarios and the poor performance at lower velocities, which may also be due in part to limitations of the method, hold it back from being a strong paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. When the velocity predictor is trained independently on GT data, the performance is worse. This implies that there must be some significant distribution shift between GT and generated depth maps (otherwise the performance drop from joint —> independently trained wouldn’t be so severe) indicating either 1. A lack of sufficient training data / training time for the depth network, 2. Suboptimal training procedure or 3. Both. The authors suggest one issue is temporal consistency. Did the authors try using a stack of consecutive depth maps to predict the depth map at the next timestep? Or maybe accumulating depth information through time using an RNN? Seems like this issue significantly hinders results yet seems solvable with current methods.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission proposes using event cameras to learn drone control for evading static obstacles.
This submission proposes using event-based cameras for quadrotor obstacle avoidance. The authors successfully demonstrate a novel approach for transferring a simulation-trained model to real-world conditions, achieving high-speed obstacle avoidance with onboard computation. Their experimentation across multiple platforms and environments showcases the method's robustness. Additionally, the counterintuitive finding of improved performance at higher speeds is useful.
The paper lacks a comprehensive comparison with alternative sensing solutions, making it difficult to assess the method's impact. The current approach is limited by computational constraints, hindering the full exploitation of event camera advantages like low latency and high dynamic range. The method's reliance on fine-tuning for new environments limits its generalizability. Furthermore, the focus on tree-like obstacles restricts the scope of the research.
The paper has improved during the rebuttal via detailed communication between reviewers and authors. Based on the above the paper is recommended for acceptance. Please follow up on the remaining open points for a potential camera ready version. With new results on dynamic obstacles, it will be worth reconsidering the positioning of this work slightly.
Decision: Accept