Representation Learning for Event-based Visuomotor Policies
Authors: Sai Vemprala
Sami Mian
Ashish Kapoor
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Event cameras
Representation Learning
Reinforcement Learning
Variational Autoencoder

Abstract: Event-based cameras are dynamic vision sensors that provide asynchronous measurements of changes in per-pixel brightness at a microsecond level. This makes them significantly faster than conventional frame-based cameras, and an appealing choice for high-speed robot navigation. While an interesting sensor modality, this asynchronously streamed event data poses a challenge for machine learning based computer vision techniques that are more suited for synchronous, frame-based data. In this paper, we present an event variational autoencoder through which compact representations can be learnt directly from asynchronous spatiotemporal event data. Furthermore, we show that such pretrained representations can be used for event-based reinforcement learning instead of end-to-end reward driven perception. We validate this framework of learning event-based visuomotor policies by applying it to an obstacle avoidance scenario in simulation. Compared to techniques that treat event data as images, we show that representations learnt from event streams result in faster policy training, adapt to different control capacities, and demonstrate a higher degree of robustness to environmental changes and sensor noise.

Representation Learning for Event-based Visuomotor Policies
Sai Vemprala Microsoft Research Redmond, WA 98052
Sami Mian
Ashish Kapoor Microsoft Research Redmond, WA 98052 kkaporr@microsote com
University of Pittsburgh Pittsburgh, PA 15260 sami mi anépitt edu
savempra@mmicrooote com
Abstract
Event-based cameras are dynamic vision sensors that provide asynchronous mea- surements of changes in per- per-pixel brightness at a microsecond level. This makes them significantly faster than conventional frame-based cameras, and an appealing choice for high-speed navigation. While an interesting sensor modality, this asyn- chronously streamed event data poses challenge for machine learning techniques that are more suited for Trame-basel data. In this paper, we present an event vari- ational autoencoder and show that is feasible to learn compact representations directly from asynchronous spatiotemporal event data. Furthermore, we show that such pretrained representations can be used for event-based reinforcement learning instead of end-to-end reward driven perception. We validate this framework of learning event- event-based visuomotor policies by applying it to an obstacle avoidance scenario in simulation. Compared to techniques that treat event data images, we show that representations learnt from event streams result in faster policy training, adapt different control capacities, and demonstrate higher degree of robustness.
Introduction
Autonomous navigation, which is driven by tight coupling between perception and action, is particularly challenging for fast, agile robots such unmanned micro acrial vehicles (MAV) that often deployed in cluttered and low altitude areas. For such reactive navigation applications such as obstacle avoidance, low sensor latency the key to performing agile maneuvers successfully [1]. MAVs are also limited in their size and payload capacity, which constrains onboard sensor choices to small, low-power sensors, and the computational load of the processing algorithms to be minimal.
Modern computer vision and machine learning techniques for perception and navigation typically focus on analyzing data from conventional CMOS based cameras, in various modalities such as RGB images, depth maps etc. While these cameras provide high resolution data, the main drawback of these sensors is their speed, with most averaging output at a rate of 30-60 Hz. This makes such sensors unable to scale up to the perception data rate required by agile navigation.
Inspired by biological vision, neuromorphic engineering has resulted in novel sensor known the dynamic vision sensor, or an event-based camera [2]. These cameras detect and measure changes in log-luminance per-pixel basis, and return information about 'events' with a temporal resolution on the order of microseconds. Due to the increased sampling speed of these cameras and the minimal processing needed parse the data, perception using event cameras approaches. This can allow for faster control schemes to be used, as enough relevant environmental information can be collected quickly order to make informed control choices. Moreover, the events are inherently generated by changes in brightness typically arising from motion. This makes event cameras natural motion detectors and a good fit for learning control policies.
be much faster than traditional
Work done while interning at Microsoft Research
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

0 0
UAV in Simulation
Event data
Latent Vector
Action
Figure 1: Event cameras provide fast, asynchronous measurements of per-pixel log luminance changes. We present a representation learning technique that can encode context from such spatiotemporal event bytestreams. Subsequently, show these low-dimensional representations are beneficial for learning visuomotor policies through simulated UAV obstacle avoidance task.
But the fundamentally different visual representation of event cameras poses significant challenges to quick adoption. Event cameras produce fast and asynchronous spatiotemporal data, significantly different from synchronous frame-based data expected by conventional machine learning algorithms. In addition, the quality of the data recorded by an event camera is different from traditional perception sensors; the sensors return low-level data that could vary significantly based the firing order of pixels, lighting conditions, reflections or shadows.
Previous research has approached this modality through two main classes of techniques. Some approaches [3, 4] accumulate event data over time into a two dimensional frame, and use traditional computer vision/convolutiona. neural network based techniques with these frame-based inputs. Traditional CNN approaches combined with such accumulation fail to exploit the true advantages of event cameras such as the microsecond-seal temporal resolution, and may prove to be too intensive for high-speed action generation onboard constrained platforms. Another class of techniques involves the usage of spiking neural networks (SNN) [5]. SNNs operate through spiking neurons identify tratioltemporal firings, making it a natural match for event cameras. Yet, training spiking neural networks is hard, as they do not use standard backpropagation, and often require specialized hardware to truly realize their efficiency [6, 7].
In this paper, propose learning representations directly from event camera streams using conventional (non-spiking) machine learning methods, and learning visuomotor policies over such representations (Figure 1). We present an event variational autoencoder (eVAE) framework for learning representations from event data in a way that allows for high temporal resolution as well invariance data permutations and sparsity. The eVAE is equipped with event feature computation network that can process asynchronous data from arbitrary sequence lengths, or in a recursive fashion. Inspired by the recent success of Transformer networks [8, 9, 10], the eVAE uses a temporal embedding method that preserves the event timing information when computing latent representations. Next, we show that such representations applying them as observations in reinforcement learning (RL) framework. We show how training RL policies over eVAE representations allows the control policy to generalize different data rates and even to out-of nottof-distribution environments. We define obstacle avoidance for UAVs as our task of interest and demonstrate how event camera data can be effectively utilized for avoidance high control rates. Through an event data simulator, we simulate scenarios where the UAV is assumed to be controlled up to 400 Hz, and show that the ability of the representations handle sparse data allows the policy to adapt to high control rates. The key contributions of our work are listed below.
be beneficial for reactive navigation, by
1. We present an event variational autoencoder for unsupervised representation learning from fast and asynchronous spatiotemporal event bytestream data.
2. We show that these event representations capture sufficient contextual information to be useful in learning reactive visuomotor policies.
3. We train policies over event representations using reinforcement learning for obstacle avoidance for UAVs in simulation and show that they outperform current state of the art in event-based reinforcement learning.
4. We discuss advantages of using bytestream representations for policies, such as: adaptation to different control capacities, robustness to environmental variations and noise.

---

2 Related Work
Vision-based representations and navigation: Variational autoencoders have been shown to be effective in learning well structured low-dimensional representations from complex visual data [11, 12, 13]. Leveraging such methods, recent research has focused on the decoupling of perception and planning, showing that separate networks for representation and navigation is effective [14, 15]. As the representation is expected to capture rich salient information about the world with degree of invariance, this combination allows for higher sample efficiency and smaller policy network sizes [16).
Feature learning from Event Cameras: Some of the early work conducted on processing event data resulted in computing optical flow using the asynchronous data, focusing high-speed computations with minimal bandwidth [17]. Event representations included histograms of averaged time surfaces (HATS), where temporal data is aggregated to create averaged data points capable of being used as input for traditional techniques [18] and hierarchy of event-based time surfaces (HOTS), another representation for pattern recognition [19].
Learning from Sequences and Sets: Learning from event data can be treated as a case of learning long, variable length sequences. While conventional RNNs are found to be infeasible for such lengths, approaches such as Phased LSTM [20] propose adding time gate to LSTM for long sequences. If the spatial and temporal parts were decoupled, the problem can be reformulated as connutation-invariinn learning from sets. Qi et al [21] present PointNet, which is a one such permutation invariant approach aimed at learning from 3D point cloud data. Similarly, Lee et al [22] present the Set Transformer, an attention-basee learning method for sets.
Event Cameras and Machine Learning: From machine learning perspective, Gehrig et al [23] introduced a full end-to-end pipeline for learning to represent event- beset data, which discusses several variants such event data aggregated into grid-based representation, event spike tensors, and 3D voxel grids. Asynchronous versions of convolutional neural networks are also being developed to take advantage of the sparsity in data such as that of event cameras 124, 25]. Stacked spatial LSTM networks were used with event sequences for pose relocalization in [4]. EV-FlowNet [26] is an encoder-decoder architecture for self-supervised optical flow for events, which uses frame-based inputs processed through convolutional layers. The asynchronous nature of event data was handled through intuuuaationiivvriint and recursive approach in EventNet [27]. Event camera based perception was used in other applications as well, such as self-supervised learning of optical flow [28], steering prediction for self driving cars [3]. Spiking neural networks were also used to examine event-based data [29, 30, 31, 32, 33, 34).
Sensorimotor Policies with Event Cameras: Only very recently has there been work on combining event camera data with sensorimotor policies. Event camera data was coupled with control for autonomous UAV landing in [35]. [36]. EVDodge [37] creates avoidance system for UAVs by using event data to track moving objects and infer safe avoidance maneuvers based on these measurements, combining multiple modules such as homography, segmentation, with the actions driven by a classical control policy. Event camera data was also used to power a closed-loop control scheme for UAV on fixed bench setup by tracking roll angles and angular velocities in [38]. Reinforcement learning using event camera data has only been explored very recently, using accumulated event frames fed into CNN-based policy networks for ground robots [39] and for UAV obstacle avoidance [40].
3 Representation Learning for Event Cameras
3.1 Event-based camera
An event based camera special vision sensor that measures changes in intensity levels inde- pendently at each of its pixels. Given pixel location (z,,), the fundamental working principle of an event-based camera is to measure the change in logarithmic brightness at that pixel, i.e.. Alog I({x, y). ). () where / is the photometric intensity. When this change in logarithmic brightness exceeds a set threshold, the camera generates an 'event', reporting the time and location of change, along with the 'sign' of the change. In contrast conventional cameras which output set number of frames per second, an event camera outputs events sparsely and asynchronously in time as a stream of bytes, which we refer to as event 'bytestream'. These events produced at non-uniform
3

---

rate, and the number can range from zero to millions of events per second. For example, the DAVIS 240 camera [41] has a theoretical maximum limit of 12 million events per second.
3.2 Definitions and Notations
For an event camera of resolution (H, W), an event can be defined as a tuple of four quantities (t, x. J.P) where is global timestamp which the event reported by the camera, (æ, y) the pixel coordinates, and p the polarity. A sequence of events over a time window of T can thus be represented as E, {eal(= sequence of events, we can see that the length of E, will not be constant as the number of events fired in that interval would change based on environmental or sensory considerations. The events in F., can also be accumulated and represented as a corresponding event image frame TET.
< l T). When sliding a constant time window of over longer
3.3 Event bytestream processing
Given event data as an arbitrarily long bytestream E,, the objective of representation learning is to map to a compressed vector representing the latent state the environment 2, through an encoder function qe(E,) (E The challenges here two-fold. First, due to the non-uniform and asynchronous nature of the event camera data, the same scene when imaged multiple times by an event camera could result in different permutations of the output. Hence, to handle the asynchronicity of event cameras, we require a feature computation technique that is invariant to data ordering. Secondly, while event sequences are time-based data, recurrent neural networks would prove to be infeasible due to the often long sequence lengths. Decoupling the temporal information from the spatial/polarrty information alleviates this problem. We propose backbone called the event context network (ECN) to achieve this for event data.
The ECN can be thought of as a preprocessing neural network for event streams, similar to archi- tectures aimed at learning unordered spatial data such as PointNet [21] and EventNet [27]. The ECN takes arbitrarily long list of events, and first computes feature for each event. Eventually, these features are passed through a symmetric function (similar to PointNet, we also use a max operation), resulting in global feature that is expected condense information from all the events. The symmetric nature of this function ensures that these events in a given list can be processed either single batch, or recursively with any minibatch size compute the output. We call the output of this feature network a "context vector'. The ECN consists of three dense layers which, for N input events, output N D set of features. The data passed into these dense layers is only the (1r, y, part of the events and we discuss how we handle the temporal information next.
3.3.1 Temporal embedding
Timestamps in the event data inherently encode the continuous time representation of the scene that was perceived during the given time slice, and it is important to retain them so the compressed representation is sufficiently informative of the evolution of the world state. On the other hand, incorporating the timestamp is not straightforward. Due to the asynchronicity of the data, a particular event may have any arbitrary timestamp within given sequence. Hence, including the temporal data as an input to the ECN directly would interfere with the feature computation, as the global timestamps are arbitrary values, and even the relative time difference of each event would change every time new events are is received, necessitating recomputation of the features.
Instead, we propose using 'temporal embeddings', inspired by the positional encoding principle that was first proposed for Transformer networks in [8]. For an event set E,, with n events, we first normalize the timestamps to [0, 1] such that the timestamp corresponding maps to 1. This allows the model to encode recency of events in generalizable way, thus allowing the model understand which events are more important than the others, as they represent recent activity in the scene, regardless of the length of the event sequence. The ECN computes a d-dimensional temporal feature for each normalized timestamp as follows.
the end of the window
100% 26) sin (1000//4)
) ,,tet, 2i cos
100%
(1)
10001/4
where i [0,d/ 6 E [0, 1]
4

---

1024)
mip H W)
ie
Mxx
boeaas 200
XY
shared
1024
pool
qF
40
E
0340330 11
DKt.
Lree 1g
23-23
036444022
Edc. 1024
136442013 29
23-4
Figure 2: Architecture of the event variational autoencoder (eVAE). Events from the bytestream are directly processed by PointNct-styled network to compute individual features, Temporal embeddings added to these features and the operation results in global context vector. This then projected into latent space, subsequently decoded into an 'event image'.
These embeddings are summed up with their corresponding features. The ECN passes this N x D feature set through the symmetric function max to obtain D final context vector. The ECN contains three dense layers for the feature computation along with the temporal embedding module and the max pool operator (Figure 2).
3.4 Event Variational Autoencoder
When learning representations for control, is important for an efficient dimensionality reduction technique create smooth, continuous, and consistent representation. is also desirable to have the encoded vectors' dimensions map to specific learned attributes of the perceived information, which can then be exploited by the control policies for interpretable learning. To achieve this, we extend the feature computation described in the previous section using variational autoencoders.
A variational autoencoder (VAE) [11] provides probabilistic framework for mapping observations into a latent space. A VAE thus requires its encoder to describe a probability distribution for each latent attribute, instead of mapping attributes to outputs randomly. In the current framework, the event VAE (e (AF) operates on the context vector computed by the ECN and attempts to project into compressed latent space. Our encoder is composed of two dense layers seen in Figure 2. In the decoding phase, instead of trying to reconstruct the entire input stream, we use an 'event image decoder' which attempts decode the latent vector back an approximate event image corresponding to the input sequence. This event image is a single channel image frame that is the result of accumulating all the events according to their pixel locations and polarity values, scaled by the relative timestamps. Similar to a standard VAE loss, the eVAE attempts to learn parametric latent variable model by maximizing the marginal likeliihooo of the training data, composed of the event image reconstruction loss and a KL-divereence loss. The combined loss can be expressed as follows where the first term denotes the reconstruction loss, P(=) the distribution over the latent variable and Q(=(x) is the approximated posterior by the VAF.
M
1(0) 2
Σ R. (ogpo( DRL ((((()())s))))(())))
A key thing to note here is that this deviates from the conventional definition of an autoencoder, where perfect reconstruction input sought. Instead, eVAE's encoder-decoder structure operates upon the 'context vector', the event features; and not the input stream itself. Hence, the goal is to encode the essence of the environment in a gencralizable fashion, thus utilizing the low-level nature of the data. The decoder qp is another two dense-layer network that takes the (sampled) latent vector 2, and outputs reconstructed image Í
The training is performed end-to-end, so the weights for the ECN and encoder-decoder are all learnt simultaneously. While training, the eVEE can receive input data in two ways. The data can be passed as a set of batches with a predefined number of events per batch, or can be sliced according to a predefined time window where each window has different number of events. During inference, as
5

---

in our application, the eVAE is expected to drive control commands, the length of the time window corresponds to the control frequency of the vehicle. This allows the context vectors to be computed either once at the end of the time window, or recursively at faster rate where the context is computed and updated internally, and mapped to the latent vector when the control command is needed. More details about the e eVAE training, computational effort etc. can be found in Appendices A and C.
4 Event-based Reinforcement Learning
Next, we focus on using event-based representations for navigation/planning purposes. While a straightforward approach would be to learn perceptual features together with actions, this would not scale well to event streams. As event cameras return data at a very high rate, relying on slow, sparse rewards to learn features in an end-to-end manner would be a disadvantage. Recent research has identified that generally, decoupling perception and policy networks and using intermediate representations enables faster training, higher performance and generalization ability [42]. We adapt this approach to event cameras, and propose using the e VAE representations in a reactive navigation framework. We define task collision avoidance for a quadrotor drone: where in simulation, the drone is expected to navigate from a start region to a goal region through an obstacle course, while avoiding collisions with any obstacle. Regardless of global positions of the drone the obstacle(s), the drone should move in a particular direction that allows it to continue in collision-free areas, and repeat this behavior till the drone reaches its goal state. Hence, navigation and obstacle avoidance constitute sequential decision making problem, which we address through reinforcement learning.
4.1 Background
We follow a conventional RL problem formulation for the reactive navigation task. As the quadrotor navigates in the environment and obtains event camera data, we pass the sequences output by the camera through the eVAE's encoder and consider the output latent vector to be the observation of the world state, such that 2 = (-(se) The objective of the reinforcement learning approach is to learn good policy (a)2)))
We train our policies using the Proximal Policy Optimization (PPO) [43] algorithm. PPO is an on-policy policy gradient method, class of methods that generally seek to compute estimator of the policy gradient and use a stochastic gradient ascent algorithm over the network weights. The core principle of PPO is to 'clip' the extent of policy updates in order to avoid disastrously large changes to the policy. At time t, for an advantage function A, and for a given ratio of probability under new and old policies rı, PPO solves modified objective function for the estimator that can be written
cliy (0) = T8 mmnm(::()) A., (r))
4.2 Implementation
We create obstacle avoidance scenario within the high fidelity quadrotor simulator AirSim [44], where a quadrotor drone is assumed to be equipped with a forward-facing event camera. We use an event simulator using the logarithmic image difference event generation model simulate events. To generate events across a span of time, it is necessary to first capture two images and compute the difference. Particularly when high control frequency is desired (i.e., events should be computed and processed at a high rate), this complicates real time operation of the task. Due to this limitation, we instead use steppable simulation. To emulate different control frequencies, we assume that the drone is moving at a constant predefined velocity and vary the step size of the actions dependent on the desired frequency. We assume the drone to be a simplistic model capable of moving at speed of 20 m/s; thus, for example, the step size for a 200 Hz control would be 0.1 m. Further details about the RL training procedure and the environment can be found in Appendix D. In Figure 3, we show the environments used for training and testing the policies. In the interest of furthering research, we open source our representation learning and reinforcement learning framework along with the environments ?
For the evaluation, we use total of four policies, with two policies using the eVAE representations and two baselines.
Our code and environments can be found at ht httpddt///////tttttbbboo. httpp//s///..taaaccc///////r///s

---

(a) RL Training environment
(b) RL Test environment
(c) RL Test change
Texture
(d) RL
Test
Shape
change
Figure 3: Environments used for RL training/evaluation
BRP-xy: Policy learnt over pretrained e eVAE representations encoding only XY locations from event data.
BRP-full: Policy learnt over pretrained eVEE representations encoding full event data with timestamps and polarities.
EIP: A policy with CNN trained end-to-end, using the event image as input (similar to [39, 40))
EIVAE: Policy learnt over pretrained VAE that was trained using event images and CNN backbone.
5 Results and Discussion
5.1
Representation Learning
Our first set of experiments aims to validate the learning of compressed representations encoded from the event sequences, and analyze the context-capturing ability of the eVAE To train eVEE, we simulate event data through AirSim's 's event simulator in three environments named poles, cones, and gates (drone racing gates), each indicative the object of interest in it. More details about these environments be found in Appendix B. The simulated event camera is assumed to be of 64 x 64 resolution and the data collected by navigating in 2D around the objects.
Qualitative performance: Figure 4a displays the general performance of the eVAE at learning context out of event bytestreams. From the reconstructions, we observe that the eVAE latent space is able to encode the underlying essence of the input bytestream: locations of the objects, patterns of polarities, and information regarding time of firing (brighter pixels in original/reconstruucion indicate recent firings) are captured. We note that by encoding the arrangement of polarities, the latent space implicitly captures direction of motion, which in this case is due to the egomotion of the vehicle as assume the environments be static. In Appendix F, we show through qualitative comparison that our proposed temporal embeddings result in better representation learning than the temporal coding approach proposed for event data in [27].
Invariance to sparsity: key feature of the eVAE is its generalization to varying lengths of an event sequence, as the number of events at the input to the network can vary greatly. In Figure 4b, we show a comparison of the decoded image, when the eVAE is given sequences of different lengths starting at the same timestamp. The e VAE is quickly able represent the object number of events matching that spatial arrangement are seen, and this projection into the latent space stays constant more events are accumulated. Because the eVA operates upon the context that is extracted from the stream, even short sequences are mapped to informative parts of the latent space based the locations of the events. We compare this with VAE trained on event images using CNN encoder (Figure 4c), whose reconstruction quality degrades with decreasing sequence length, indicating the CNN's difficulty handling sparse images.
'gate' once minimum
Generalization: This context capturing ability also extends to unseen appearances obstacles. highlighting the advantages of using low-level event data. In Figure 4e, we show samples of an eVAE trained on the poles data trying decode data from the cones environment, and vice versa. Main environmental features (location of object, polarities etc.) are still captured by the latent vector, while the decoded image maps to the objects the e vAE has seen during training. This creates a degree of robustness in the e VAE specifically for reactive navigation: where the goal is to avoid obstacles
7

---

(a) Comparison expected event frames (top) reconstructed (bottom). eVE encodes locations of the obstacles and motion information from input sequences.
(b) eVA reconstructonns stay similar with changing data lengths (100, 200, 500 events).
(c) EIVAE reconstructions degrade for
shorter lengths event sequences.
(d) Latent space transitions corresponding to the actions from a control policy form structured trajectory in the space.
(c) e VAE representations when applied to out- of-distribution data, decode to seen objects while preserving context.
Figure 4: Qualitative results the event variational autoencoder learning from various types of sequences.
matter what their shape/appearance is. We show in the later sections that this allows policies to work on radically different obstacle appearances without needing to retrain the policy.
Smoothness of latent space: As the eVAE combines the inherent manifold smoothness advantage of VAES with high frequency input data, we observe that the smoothness automatically arises within the latent space similar environmental factors map the same latent variable values. We show an example in Figure 4d where we take a representation trained on the gates environment, which contains set of drone racing gates, and observe the latent vectors when drone navigates through the gates while collecting event observations. As the drone executes this set of actions, we see that the eVAE-encoded representation also shows a certain amount of structure. This way, state information from event data can potentially be projected into an approximately locally linear latent space, which has been shown benefit high speed optimal control [45].
5.2 Reinforcement Learning for Obstacle Avoidance
Policy training and control performance: Next, we evaluate the results of using these pretrained representations as observations in reinforcement learning framework for collision avoidance.
Considering that the bytestream-based policies are being trained over well structured lower dimen- sional representations, we observe improved performance during training. Comparison of the training rewards over the first 500000 timesteps can be seen in Figure 5a, where the bytestream representation policy (BRP) training is seen to have lower sample complexity than the event image policy (EIP)

---

Training performance
Effect of control fisquency on navigalion performance
200
100
BRPwa
3:20
25
EIP
Forcontage
50
25
100
200000
45 Hiz
100 Hi
Hix 100
Timeslop
Control frequency (Hz)
(a) e vAE representations allow faster train- ing of policies compared to image-based poli- cics.
(b) eVAE representations benefit high speed control by being able to handle sparse se- quenees.
encirnemmartaal drangos on policy prrformancr
Effect commrn thaesheld un pwppllyyppeeeee performancr
125
125
Datana bavveel (m)
100
Disunnee Saveld (m)
100
80
Shape change
(c) VVE based policies generalize better to obstacles of unseen textures or shapes.
(d) eVAE based policies generalize to differ- ent event thresholds than seen in training.
pisels firing policy performance
priel on policy performance
125
125
100
100
75
75
30
25
06
OH: 20%
Off: SIS
BA: 0%
RA: 5%
RA: 10%
(e) Policies trained over eVAE representa- tions exhibit robustness to induced sparsity in event data.
(f) Policies trained over eVAB representa- tions exhibit robustness to additional noisy event firing.
Figure Analysis of policy learning using event bytestrcam representations, compared against end-to-end trained event image + CNN policy.
or RGB images. We also find that the EIVAE policy it results in worse performance, which we hypothesize is due to its lower generalization ability to changing sequence lengths (c.g. very sparse images), which causes the latent representations to vary greatly for sequences that are too short or too long, making it harder to learn the corresponding actions.
Given the high data rate from event cameras, is possible control the vehicles at higher frequency than with standard RGB camera images. We conduct an experiment where different control frequencies are simulated for the drone (varying the step size as discussed in section 4), and the trained BRP and EIP policies are tested. As conventional CMOS cameras often output data around 30-60 Hz, we choose 45 Hz as the minimum for the test, and 400 Hz (motor level control frequency of quadrotors) as the maximum. The results are seen in Figure 5b as success percentage over 40 trials in two environments, with success defined whether or not the drone navigates through 100m long obstacle course without collisions. We observe that all modalities suffer from low rate of success 45 Hz, demonstrating the drawbacks of slow control in densely populated obstacle courses. At higher control frequencies, the motion of the camera and subsequently the number of events tend be smaller. Even these cases, similar the observation figure Figure 4b, extracting latent representation allows the BRPs to be accurate, maintaining high policy success rate at simulated data rate of even 400 Hz. Intuitively, being able to perceive and control faster also means that agent has enough chances to recover even in case of the occasional bad action. In contrast, we notice a falloff in the accuracy of EIP at higher control frequencies, as the event images get much sparser, which could prove to be problematic for CNN.

---

Robustness to environmental changes: In the context of reactive navigation, the idea is to be able avoid any obstacle regardless of characteristies like shape, appearance, texture etc. Through the BRP, we observe a key strength of the e eVAE representations which is the generalization ability. First, we evaluate the performance when transferring policy trained on the poles environment to unseen environments: one involving a change in texture of the obstacles, and another involving a change in shape (Figure 3). From the results in Figure Sc, we see that the EIP exhibits good performance on the environment it was trained on, but fails when applied to other environments due to the radically different obstacle appearances. Whereas, as seen in section 5.1, the VAE brings a degree of invariance to the latent space projection, and hence the both BRPs perform better than the EIP with differently textured/shaped obstacles. We analyze this by running 20 trials of a policy under the test settings, and comparing the mean and standard error of the distance traveled without collision. In Appendix F, we show an additional experiment with more complex out-of ditto--distribution environment with obstacles that are of both different shape and texture containing moving textures, where the BRPs still maintain better performance.
Robustness to camera parameters: Similarly, we examine the effect of event camera sensor parameters on policy performance. For instance, in Figure 5d we examine the effect of the event threshold: which is the parameter that determines at what level of intensity change should an event be fired. A low value of threshold thus means a large number of events are fired, making the camera more sensitive to motion. When tested with different camera thresholds, which results in changing amounts of detail in the sequences, BRPs outperform the EIP. The e eVAE affords the policies a degree of invariance to this redundant/umnecessite
data, whereas the end-to-end CNN policy does not.
We also observe the bytestream representations benefiting the policy in case of induced sparsity in the event data. For this, we manually 'turn off", i.e., skip certain pixel locations in the event data. Figure 5e shows that the bytestream representation helps the policy maintain accuracy even up to the case where the event data is 50% sparser. Finally, event cameras are also prone to background activity (BA) [46], i.e., events being fired when there is no real intensity change. To simulate this, we add random events to the sequences. We observe that the BRPs still outperform the EIP (Figure 5f)- but we note that the BRPs are more sensitive to this type of noise than induced sparsity. In case of BA noise, fRP- exhibits lower performance than ERP-xy, likely due to spurious polarities.
6 Conclusions
The event-based camera, being a low-level modality with fast data generation rate, is a good choice for high speed reactive behavior. Instead of treating event data as images, we present event variational autoencoder that combines a spatiotemporal feature computation framework with the inherent advantages of variational autoencoders, enabling the learning of smooth and consistent representations directly from asynchronous event streams. By applying these representations in a reinforcement learning pipeline for navigation, we show that these representations effectively encode environmental context from fast streams, and can extract object locations, timing and motion information from polarity etc. in way that generalizes over different sequence lengths and different types of objects, outperforming event image based methods.
Limitations and Future Work. We present this work as an initial exploration towards connecting representation and reinforcement learning for event cameras. We have not tested this approach on large, diverse datasets, and some changes might be required to adapt these representations for more general representation learning with complex scenes. Our RL problem also focuses a simpler setting due to computational issues with event simulation because complex scenes create more rendering overhead in simulators. Interesting avenues for future work could be leveraging GPUs for event simulation, applying these methods to harder tasks such as drone racing, and investigating recent advances in asynchronous convolutional networks [24] for representation learning.
Broader Impact. On one hand, event cameras bring in a lot of potential for fast perception- action loops, such as for instance, integrating high speed reactive control with slow deliberative perception (thinking fast and slow [47]) for better robot intelligence. The low-level nature of event data also makes it a generally interesting candidate for vision, particularly for inducing shape bias as opposed to texture bias that is commonly seen in CNNs [48], and for privacy-preeerving vision. On the other hand, there is potential for misuse with event cameras being used for surveillance, or drones equipped with those used in unethical applications, which requires careful consideration.
10

---

References
[1] Davide Falanga, Suseong Kim, and Davide Scaramuzza. How fast is too fast? the role of perception latency in high-speed sense and avoid. TEEE Robotics and Automation Letters, 4(2):1884-189,, 2019.
[2] David Tedaldi, Guillermo Gallego, Elias Mueggler, and Davide Scaramuzza. Feature detection and tracking with the dynamic and active- pixel vision sensor (davis). In 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP), pages 1-7. IEFF, 2016.
[3] Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Gareía, and Davide Scaramuzza, Event- based vision meets deep learning steering prediction for self-driving cars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5419-5427, 2018
[4] Anh Nguyen, Thanh-Tooe Do, Darwin G Caldwell, and Nikos G Tsagarakis. Real-time 6dof pose relocalization event cameras with stacked spatial Istm networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[5] Moritz B Milde, Hermann Blum, Alexander Dietmüller, Dora Sumislawska, Jörg Conradt, Giacomo Indiveri, and Yulia Sandamirskaya. Obstacle avoidance and target acquisition for robot navigation using a mixed signal analog/digital neuromorphic processing system. Frontiers neurorobotics, 11:28, 2017.
[6] Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 121:2944307, 2020.
[7] Emre O Neftei, Charles Augustine, Somnath Paul, and Georgios Detorakis. Event-driven random back- propagation: Enabling neuromorphic deep learning machines. Fromtiers in neuroxcience, 11:324, 2017.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 19998-0039 2017.
[9] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stane-alone self-attention in vision models. arXiv preprint wew:::9900000000 2019.
[10] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamers. arXiv preprint 2rX.v:2005.44165 2020.
[11] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6144 2013.
[12] 1. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[13] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803. 10122 2018.
[14] Rogerio Bonatti, Ratnesh Madaan, Vibhav Vineet, Sehastian Scherer, and Ashish Kapoor. Learning controls using cross-modal representations: Bridging simulation and reality for drone racing. arXiv preprint arXiv: (et.it999999999 2019.
[15] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555 2565. PMLR, 2019.
[16] Giuseppe Cuccu, Julian Togelius, and Philippe Cudré-Maurond Playing atari with six neurons, arXiv preprint arXiv: / 008.00603366 01363,, 2018
[17] Ryad Benosman, Sio- Hoi leng, Charles Clereq, Chiara Bartolozzi, and Mandyam Srinivasan. Asynchronous frameless event-based optical flow. Neural Networks, 27:32-37, 2012.
[18] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1731-1740, 2018.
[19] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman. Hots: a hierarchy of event- based time-surfacee for pattern recognition. IEEE transactions on pattern analysis and machine intelligence, 39(7):1346 1359, 2016.

---

[20] Daniel Neil. Michael Pfeiffer. and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event- best- sequences. Advances in neural information processing systems, 29:3882-383 -8900, 2016.
[21] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 652-660, 2017.
[22] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Scungjin Choi, and Yee Whye Tch. Set transformer: A framework for attention basoon commutation-invariinn neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019.
[23] Daniel Gehrig, Antonio I. Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE International Conference Computer Vision. pages 5633-5643, 2019.
[24] Nico Messikommer, Daniel Gehrig. Antonio Loquercio, and Davide Scaramuzza. Exvettbbsse asyn- chronous sparse convolutional networks. arXiv preprint arXiv:2003.099 148, 2020.
[25] Marco Cannici. Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. Asynchronous convolutional networks for object detection in neuromorphic cameras. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 0-0, 2019.
[26] Alex Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniiliiss. Ev-flownet: self-supe optical flow estimation for event-hased cameras. In Proceedings of Robotics: Science and Systems, Pittshurgh, Pennsylvania, June 2018.
[27] Yusuke Sekikawa, Kosuke Hara, and Hideo Saito. Eventnet: Asynchronous recursive event processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3887-3896.
2019.
[28] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danillidis. Unsupervised event-based optical flow using motion compensation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0-0, 2018.
[29] Evangelos Stromatias, Miguel Soto, Teresa Serrano- Serrano-Gotarree Gotarredona, and Bernaéé Linares- Barranco, An event-driven classifier for spiking neural networks fed with synthetic or dynamic vision sensor data. Frontiers in neuroscience, 11:350, 2017.
[30] Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: opportunities and challenges. Frontiers in neuroscience, 12:774, 2018
[31] Yu Miao, Huajin Tang. and Gang Pan. A supervised multi-spike learning algorithm for spiking neural networks. 2018 International Joint Conference on Neural Newworks (IJCNN, pages 1-7. JEEE, 2018.
[32] Guangzhi Tang. Neelesh Kumar, and Konstantinos P Michmizos. Reinforcement co-learning of deep and spiking neural networks for preprint 27X1v:2003.0115,, 2020.
mapless navigation with neuromorphic hardware, arXiv
[33] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio- temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
[34] Peter O' O'Connor, Daniel Neil, Shih-Chii Liu, Tobi Delbruck, and Michael Pfeiffer. Real-time classification and sensor fusion with spiking deep belief network. Frontiers in neuroscience, 7:178, 2013.
[35] Bas J Pijnacker Hordijk. Kirk YW Scheper, and Guido CHE De Croon. Vertical landing for micro air vehicles using event-based optical flow. Journal of Field Robotics, 35(1):699995 2018.
[36] Jesse J Hagenaars, Federico Paredes-Vallés. Sander M Bohté, and Guido CHE de Croon, Evolved neuromorphic control for high speed divergence-based landings of mavs, arXiv preprint a3X 2020.
[37] Nitin Sanket, Chethan M Parameshwara, Chahat Deep Singh, Ashwin V Kuruttukulam, Cornelia Fermuller, Davide Scaramuzza, and Yiannis Aloimonos. Evdodge: Embodied ai for high-speed dodging on a quadrotor using event cameras. arXiv preprint (90.:/990009999 pages 31-45, 2019.
[38] Rika Sugimoto Dimitrova, Mathias Gehrig, Dario Brescianini, and Davide Scaramuzza. Towards low- latency bangwhhhd control of quadrotors using event cameras. In 2020 IEEE International Conference Robotics and Automation (ICRA). pages 4294 4300. IEEE, 2020.
12

---

[39] Riku Arakawa and Shintaro Shiba. Exploration of reinforcement learning for event camera using car-like robots. arXiv preprint arXiv:2004.0080) 2020.
[40] Nikolaus Salvatore, Sami Mian, Collin Abidi, and Alan George. A neuro-inspired approach to intelligent collision avoidance and navigation. In 2020 JEEE/AIAA 37th Digital Avionics Systems Conference (DASC). IEEE, 2020.
[41] IniVision. DAVIS 240 dynamic vision sensor. http:://lalvatllooc.ccowoooooooo uploads/ /0199/ /009////////// CB/////////A DDBDD22224000 pdf, 2019.
[42] Brady Zhou, Philipp Krätenbühh,, and Vladlen Koltun. Does computer vision matter for action? Science Robotics, 4(30), 2019.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1..X::11777.0347,, 2017.
[44] Shital Shah, Debadeepta Dey. Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation autonomous vehicles. Field and service roboticx, pages 621-635. Springer, 2018.
[45] Manuel Watter, Jost Springenherg, Joschka Boedecker, and Martin Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. Advances neural information processing systems, pages 2746-2754, 2015.
[46] Baldwin, Mohammed Almatrafi, Vijayan Asari, Keigo Hirakawa. Event probability mask (epm) and event denoising convolutional neural network (edncnn) for neuromorphic cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1701-1710, 2020.
[47] Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011.
[48] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are hiased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint urXiv:181.122311 2018.
[49] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint arXiv:151 2015.
[50] Antonin Raffin, Ashley Hill, Maximilian Ernsstus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. https: /tt::://gttttt thub. www//ow.c-e.......... DD/R/D/RRRRRDDDD st wawaabll-bseeeeessee 2019.
[51] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 533, 2015.
13


Reviews:
[{"review_id": "hPwdoCb-hlu", "reviewer": "Reviewer_ZPhQ", "summary": "The paper targets navigation and obstacle avoidance using high-frequency event cameras. A new event VAE (eVAE) is proposed to encode inputs from event cameras. An unsupervised learning scheme for eVAE is also proposed. The encoder is first evaluated qualitatively under variations of the input data. The pre-trained eVAE is then used to learn RL navigation policies. Results favorably compare to alternative encoding methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Learning efficient visual representations for RL policies is an interesting and open problem\n- Learning visiomotor policies for evet cameras should be interesting for many applications\n- The paper is well-written and is easy to follow in general. \n- The proposed eVAE encoding is evaluated and compared to baselines under variations of observed scenes and used parameters, e.g. length of event sequences \n- The eVAE-based RL policy is tested for textures and shapes unseen during training and while changing the sparsity of event data \n- Results are convincing given the comparison to baselines.\n\nCons:\n- The definition of terms in eq.(2) is missing. The exact form of the loss for learning eVAE is hence unclear. What are Q,z,x in (2)?\n- The robot seems to be always moving forward. What will happen if gets to a trap with no free passage for moving forward?\n- Results for RBP-xy are similar to RBP. Does this mean that \"e\" part in eVAE is not important? Is the contribution of eVAE validated in this case?   \n- I have not found a commitment by the authors to release the experimental setup and environments. Such a release will be crucial for future comparisons. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No specific issues other than standard for vision and robotics", "review_text": "", "extra_scores": {"ethical_concerns": "No specific issues other than standard for vision and robotics", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "dZwXs7iaRy-", "reviewer": "Reviewer_twzt", "summary": "The author proposed an event based VAE for high speed robot navigation useful for reinforcement learning. They then compare to other image based RL approaches in AirSim and show they outperform other event based RL baselines. They demonstrate that the representation is useful for training visuomotor policies. They show that it reaches SOTA in event based reinforcement learning on the given tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nThe paper proposed an event based VAE and uses it for downstream reinforcement learning. The paper focuses on using high sampling rate data streams which is an important and not very often studied problem in RL (particularly due to the amount it slows down training in simulators like Airsim. \n\n# Quality\nThe paper provides a compelling argument that their technique performs better on high sampling rates particularly through the graphs in figure 4. They demonstrate that the paper reached SOTA on the obstacle avoidance task. I also appreciated the detailed discussion section explaining the benefits of the approach in regards to generalization. I also am glad that they examine the full event based representation and the simplified xy version and found different performance between the two. I don't see any issues with the methodology, but the RL problem is rather simplistic as the authors note. Especially given the small size of dataset and latent space, I am concerned about whether it would generalize to real world environments. However, this paper demonstrates that the method is more robust than RGB inputs, and therefore this is a promising first step.\n\n# Clarity\nI appreciate figure 3 for providing good qualitative visualization of the event based autoencoder, but I would recommend making the text larger in the final version. The impact of figure 4 is reduced due to the difficult of reading it. The paper text, itself is very well written and understandable. The supplemental material aided in the understanding of the paper as well so much so that parts of it (like an example of the environment should have been included in the main paper).\n\nWould the authors clarify what they mean by: \"Our RL problem also focuses on a simpler setting due to computational issues in event simulation\" ? Is that saying that event based policies are computationally expensive? Or that the high framerate simulation required for training them is the main bottleneck? Specifying which computational issues are the constraints here, (speed, memory, gpu memory, physics simulation wall clock, graphics rendering, inference or training bottleneck in one of the network architectures etc.) is important for future authors who wish to build upon the approach. \n\nThe main issue in clarity is that the simulation environment is not shown qualitatively in the main paper which made the problem setup difficult to visualize. \n\n# Significance\nThis representation could prove useful for drone in particular due to their high speed, but also is applicable to other high speed robotics. Additionally, the representation learned seems to have potential for improving sim2real transfer due to the robustness to shape and texture within simulation as shown in the appendix as well. They also demonstrate the importance of high frequency control in how all techniques struggle at a lower frequency of samples at 45hz.\n\nI will admit that the paper does a good job proposing an interesting observation space that may generalize better for sim2real tasks as the paper demonstrates. With the changes to paper clarity and depending on the discussion of the computational bottlenecks, I would be willing to raise my score.\n\nEdit: I would also like to apologize to the authors that my original review was truncated. I have edited my review.\n\nPost Rebuttal:\nThank you for the authors detailed rebuttal. Ultimately, the rebuttal did not sufficiently address my concerns to raise my score beyond a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "MyYdikSR-H4", "reviewer": "Reviewer_sdni", "summary": "The authors present an event-based variational auto-encoder (eVAE), which learns a latent space representation that is subsequently used for reinforcement learning (RL). The RL task consists of a simulated drone that has to avoid obstacles while flying forward in AirSim. Although the authors themselves do not mention this as a contribution in the paper, I think that the Event Context Network (ECN) they introduce is actually a major contribution of this article. In the ECN they combine an EventNet-like architecture with a Transformer-like position encoding (here temporal encoding). This ECN is a core part of the eVAE and works better than the original EventNet-way of encoding temporal order (as evidenced in the appendix). Finally, the authors show that with the eVAE, a learned policy transfers better to different environments.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed ECN and eVAE will be a valuable contribution to the construction of event-based latency spaces for - eventually - use on embedded robots. In terms of originality, the proposed method is primarily a very skillful combination of techniques found in the literature, but that is fine in my opinion. \n\nThe article is (apart from a few minor points) clear and well-written. However, I would advise the authors to include the new ECN as a contribution. This would also make the comparison with the original EventNet more important, which I would then suggest to also make part of the main article. I know that due to the page limits this may be challenging, but it could be added to, e.g., Fig 3 if the other panels there show 1 less example in order to make room for the EventNet comparison. On the topic of Figures, Fig 4 has super small legends, which are impossible to read. \n\nThe main criticism I would have on the article is related to the simplicity of the dataset (also mentioned by the authors under limitations). A very small latent space is learned (8-dimensional) and the examples shown by the authors that illustrate the superiority of the method over other methods mostly show that given a few events, the latent space will interpret an image still like the most likely pose of a pole or gate. The real world is obviously more complex and it would be interesting to see the eVAE in action for a real-world event-camera dataset (potentially with a larger latent space). This is even more relevant since the authors simulate events based on image differencing. Is there a study in the literature on how good this model is compared to the real outputs of an event-based camera?\n\nThis criticism extends to the used simulator and simulated environments, which is far cry from real autonomous drone flight. However, I do think it is valid as a first step, given the goals of the article. Moreover, as the authors mention themselves, event-based camera inputs will generalize better to different settings (including possibly the real world) in comparison to RGB values. The experiments in simulation already testify to this.\n\nSmaller remarks:\n- \"Event camera data was also used to power a closed-loop control scheme for a UAV in flight by tracking roll angles and angular velocities in [38].\" --> these experiments were not in flight, but with a fixed bench setup.\n- For Eq 1 I had to go back to the Transformer paper. i and d are not (well) explained in the text, as only D was mentioned.\n- Figure 2: how does the MLP at the start compare to the one in PointNet / EventNet (PointNet's is much more extensive I have the idea)\n- Figure 3c: \"eVAE representations stay consistent for different lengths of event sequences.\" What lengths? (I see three panels, but no indication of what the lengths were)\n- \"We note that by encoding the arrangement of polarities, the latent space implicitly captures direction of motion, \" --> This is only true if you know the fore- & background color, right? Else there is ambiguity...\n- Figure 4b: at 100Hz the EIP works best. Is there any explanation for this?\n- The main article should have a picture of the simulated environment for the obstacle avoidance.\n\nPOST-REBUTTAL:\n\nAfter reading all reviews & rebuttals, and the responses to my own comments, my score has not changed. If the authors perform the modifications they mention in the rebuttal, I think this will be a valuable contribution to the field.\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed both social impact and limitations of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns that were not identified by the authors.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "9oafL4Edyhb", "reviewer": "Reviewer_guqw", "summary": "This paper proposes a method for learning representations of event-based camera percepts which can be used for reinforcement learning. The method is able to digest irregular, streaming camera information using a VAE conditioned on a context vector computed from event data using a PointNet style network. The empirical evaluation shows that using this pretrained representation for PPO is more performant and generalizable than frame-based methods on a simulated UAV obstacle avoidance task. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\nThe proposed eVAE method appears to be a novel approach to representation learning from this modality. The empirical results, particularly the generalization results, are strong compared to existing methods (of which there are not too many, as this is a relatively new direction of research). \n\nQuality:\n\nThe experimental results provide reasonable support for the author’s claims, but are limited in scope by just one evaluation environment. The comparisons are appropriate and the proposed method achieves strong results. \n\nMy concerns in this section are the following:\n\nThe results presented in Fig 4A do not have error bars, could they be trained with multiple random seeds and added?\n\nWhen training on one scene and testing on others, it seems like generalization occurs because unseen states are mapped to the closest possible latent. However, it’s not clear that this would result in a policy which actually reacts to the new obstacle shape. For example, if the cones were upside-down and had a significantly different shape as the poles, but got mapped to poles through the latent space, would the UAV be able to avoid them? This seems unlikely if trained using only one scene, but maybe if it were trained using a greater number of (or more diverse) environments, this would work?\n\nClarity:\n\nOverall, the paper is well written. The authors have provided and documented code to make this work as reproducible as possible.\nIt could help improve clarity to address the following questions:\n\nIn Fig 4a, does the eVAE pretraining lead to improved asymptotic performance compared to the EIP policy? If the EIP policy is allowed to train for the same amount of effective gradient steps as the VAE training + RL training, will it achieve the same performance (or better, since it can learn features with the RL objective directly)? \n\nIt doesn’t appear that including the temporal or polarity information improves the performance of the learned policies. Do you have any explanation regarding why this information appears discardable for this task? Are there other tasks where it might not be? \n\nAdditionally, it could help to add some justification for the timestamps of a particular event window to be normalized to 1. Wouldn’t this cause event windows of differing lengths (in seconds) to become encoded to the same vector?\n\nWhile there is some runtime analysis in the appendix, It would be interesting to compare the computational effort of the eVAE-based policy to the EIP-based policy in wall-clock time.\n\nA brief description about how the event image is formed from the events would be helpful for understanding the VAE training process. How is the polarity involved in the computation and how is the relative time scaling performed?\n\nNitpicks:\n\nThe dotted line in Fig 4a is unlabeled. The actual values of rewards in the environment have been omitted from the main text, so it is unclear how to interpret them.\n\nSignificance:  \n\nAs event based cameras could lead to the development of higher frequency control policies which are critical for certain types of tasks, the research question of how to best utilize this variable length asynchronous data stream is quite important.\n\nRecommendation: Although the empirical evaluation could be more thorough in terms of tasks, environments, and more experimental trials, the eVAE method proposed is novel and appears effective, and seems like it will be quite impactful for future work using RL for control from event-based cameras. So, I think this is a good contribution and vote to accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The method is tested only in one domain (simulated quadrotor flying, using the poles data). While the authors explain in Section 6 that this work is an initial exploration, it could make the paper much stronger to be able to demonstrate it with another task or more visually diverse scenes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new type of variational auto-encoders that can handle streams of high-frequency event-based spatiotemporal data (t, x, y, p) in order to decode dense images from a small set of N events, then combines a pre-trained eVAE with traditional reinforcement learning (PPO) to learn visuomotor policies to control (and fly while avoiding obstacles) a quadcopter in the AirSim simulator.
Reviewers have praised the idea, the application of RL to event-based data streams, the writing of the paper and the extent of experiment and comparison to non-event-based VAE + RL. Reviewers had some questions about equations, about specifics of some ablations, about rendering time for event-based simulation, about showing the simulator, etc., that were all satisfactorily answered by the authors.
After careful consideration of the paper and given review scores of (6, 6, 7, 7, mean 6.5), it appears that the paper should be accepted. The authors are invited to carry out all the changes they promised to the reviewers, including open-sourcing their code.

Decision: Accept (Spotlight)