Representation Learning for Event-based Visuomotor Policies

1 Introduction . Autonomous navigation , which is driven by a tight coupling between perception and action , is particularly challenging for fast , agile robots such as unmanned micro aerial vehicles ( MAV ) that are often deployed in cluttered and low altitude areas . For such reactive navigation applications such as obstacle avoidance , low sensor latency is the key to performing agile maneuvers successfully [ 1 ] . MAVs are also limited in their size and payload capacity , which constrains onboard sensor choices to small , low-power sensors , and the computational load of the processing algorithms to be minimal . Modern computer vision and machine learning techniques for perception and navigation typically focus on analyzing data from conventional CMOS based cameras , in various modalities such as RGB images , depth maps etc . While these cameras provide high resolution data , the main drawback of these sensors is their speed , with most averaging output at a rate of 30-60 Hz . This makes such sensors unable to scale up to the perception data rate required by agile navigation . Inspired by biological vision , neuromorphic engineering has resulted in a novel sensor known as the dynamic vision sensor , or an event-based camera [ 2 ] . These cameras detect and measure changes in log-luminance on a per-pixel basis , and return information about ‘ events ’ with a temporal resolution on the order of microseconds . Due to the increased sampling speed of these cameras and the minimal processing needed to parse the data , perception using event cameras can be much faster than traditional approaches . This can allow for faster control schemes to be used , as enough relevant environmental information can be collected quickly in order to make informed control choices . Moreover , the events are inherently generated by changes in brightness typically arising from motion . This makes event cameras natural motion detectors and a good fit for learning control policies . ∗Work done while interning at Microsoft Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . But the fundamentally different visual representation of event cameras poses significant challenges to quick adoption . Event cameras produce fast and asynchronous spatiotemporal data , significantly different from synchronous frame-based data expected by conventional machine learning algorithms . In addition , the quality of the data recorded by an event camera is different from traditional perception sensors ; the sensors return low-level data that could vary significantly based on the firing order of pixels , lighting conditions , reflections or shadows . Previous research has approached this modality through two main classes of techniques . Some approaches [ 3 , 4 ] accumulate event data over time into a two dimensional frame , and use traditional computer vision/convolutional neural network based techniques with these frame-based inputs . Traditional CNN approaches combined with such accumulation fail to exploit the true advantages of event cameras such as the microsecond-scale temporal resolution , and may prove to be too intensive for high-speed action generation onboard constrained platforms . Another class of techniques involves the usage of spiking neural networks ( SNN ) [ 5 ] . SNNs operate through spiking neurons to identify spatio-temporal firings , making it a natural match for event cameras . Yet , training spiking neural networks is hard , as they do not use standard backpropagation , and often require specialized hardware to truly realize their efficiency [ 6 , 7 ] . In this paper , we propose learning representations directly from raw event camera streams using conventional ( non-spiking ) machine learning methods , and learning visuomotor policies over such representations ( Figure 1 ) . We present an event variational autoencoder ( eVAE ) framework for learning representations from event data in a way that allows for high temporal resolution as well as invariance to data permutations and sparsity . The eVAE is equipped with an event feature computation network that can process asynchronous data from arbitrary sequence lengths , or in a recursive fashion . Inspired by the recent success of Transformer networks [ 8 , 9 , 10 ] , the eVAE uses a temporal embedding method that preserves the event timing information when computing latent representations . Next , we show that such representations can be beneficial for reactive navigation , by applying them as observations in a reinforcement learning ( RL ) framework . We show how training RL policies over eVAE representations allows the control policy to generalize to different data rates and even to out-of-distribution environments . We define obstacle avoidance for UAVs as our task of interest and demonstrate how event camera data can be effectively utilized for avoidance at high control rates . Through an event data simulator , we simulate scenarios where the UAV is assumed to be controlled at up to 400 Hz , and show that the ability of the representations to handle sparse data allows the policy to adapt to high control rates . The key contributions of our work are listed below . 1 . We present an event variational autoencoder for unsupervised representation learning from fast and asynchronous spatiotemporal event bytestream data . 2 . We show that these event representations capture sufficient contextual information to be useful in learning reactive visuomotor policies . 3 . We train policies over event representations using reinforcement learning for obstacle avoidance for UAVs in simulation and show that they outperform current state of the art in event-based reinforcement learning . 4 . We discuss advantages of using bytestream representations for policies , such as : adaptation to different control capacities , robustness to environmental variations and noise . 2 Related Work . Vision-based representations and navigation : Variational autoencoders have been shown to be effective in learning well structured low-dimensional representations from complex visual data [ 11 , 12 , 13 ] . Leveraging such methods , recent research has focused on the decoupling of perception and planning , showing that separate networks for representation and navigation is effective [ 14 , 15 ] . As the representation is expected to capture rich salient information about the world with a degree of invariance , this combination allows for higher sample efficiency and smaller policy network sizes [ 16 ] . Feature learning from Event Cameras : Some of the early work conducted on processing event data resulted in computing optical flow using the asynchronous data , focusing on high-speed computations with minimal bandwidth [ 17 ] . Event representations included histograms of averaged time surfaces ( HATS ) , where temporal data is aggregated to create averaged data points capable of being used as input for traditional techniques [ 18 ] and hierarchy of event-based time surfaces ( HOTS ) , another representation for pattern recognition [ 19 ] . Learning from Sequences and Sets : Learning from event data can be treated as a case of learning long , variable length sequences . While conventional RNNs are found to be infeasible for such lengths , approaches such as Phased LSTM [ 20 ] propose adding a time gate to LSTM for long sequences . If the spatial and temporal parts were decoupled , the problem can be reformulated as permutation-invariant learning from sets . Qi et al [ 21 ] present PointNet , which is a one such permutation invariant approach aimed at learning from 3D point cloud data . Similarly , Lee et al [ 22 ] present the Set Transformer , an attention-based learning method for sets . Event Cameras and Machine Learning : From a machine learning perspective , Gehrig et al [ 23 ] introduced a full end-to-end pipeline for learning to represent event-based data , which discusses several variants such as event data aggregated into a grid-based representation , event spike tensors , and 3D voxel grids . Asynchronous versions of convolutional neural networks are also being developed to take advantage of the sparsity in data such as that of event cameras [ 24 , 25 ] . Stacked spatial LSTM networks were used with event sequences for pose relocalization in [ 4 ] . EV-FlowNet [ 26 ] is an encoder-decoder architecture for self-supervised optical flow for events , which uses frame-based inputs processed through convolutional layers . The asynchronous nature of event data was handled through a permutation-invariant and recursive approach in EventNet [ 27 ] . Event camera based perception was used in other applications as well , such as self-supervised learning of optical flow [ 28 ] , steering prediction for self driving cars [ 3 ] . Spiking neural networks were also used to examine event-based data [ 29 , 30 , 31 , 32 , 33 , 34 ] . Sensorimotor Policies with Event Cameras : Only very recently has there been work on combining event camera data with sensorimotor policies . Event camera data was coupled with control for autonomous UAV landing in [ 35 ] , [ 36 ] . EVDodge [ 37 ] creates an avoidance system for UAVs by using event data to track moving objects and infer safe avoidance maneuvers based on these measurements , combining multiple modules such as homography , segmentation , with the actions driven by a classical control policy . Event camera data was also used to power a closed-loop control scheme for a UAV on a fixed bench setup by tracking roll angles and angular velocities in [ 38 ] . Reinforcement learning using event camera data has only been explored very recently , using accumulated event frames fed into CNN-based policy networks for ground robots [ 39 ] and for UAV obstacle avoidance [ 40 ] . 3 Representation Learning for Event Cameras . 3.1 Event-based camera . An event based camera is a special vision sensor that measures changes in intensity levels independently at each of its pixels . Given a pixel location ( x , y ) , the fundamental working principle of an event-based camera is to measure the change in logarithmic brightness at that pixel , i.e. , ∆log I ( { x , y } , t ) where I is the photometric intensity . When this change in logarithmic brightness exceeds a set threshold , the camera generates an ‘ event ’ , reporting the time and location of change , along with the ‘ sign ’ of the change . In contrast to conventional cameras which output a set number of frames per second , an event camera outputs events sparsely and asynchronously in time as a stream of bytes , which we refer to as an event ‘ bytestream ’ . These events are produced at a non-uniform rate , and the number can range from zero to millions of events per second . For example , the DAVIS 240 camera [ 41 ] has a theoretical maximum limit of 12 million events per second . 3.2 Definitions and Notations . For an event camera of resolution ( H , W ) , an event can be defined as a tuple of four quantities e = ( t , x , y , p ) where t is a global timestamp at which the event was reported by the camera , ( x , y ) the pixel coordinates , and p the polarity . A sequence of events over a time window of τ can thus be represented as Eτ = { ei|t < i < t+ τ } . When sliding a constant time window of τ over a longer sequence of events , we can see that the length of Eτ will not be constant as the number of events fired in that interval would change based on environmental or sensory considerations . The events in Eτ can also be accumulated and represented as a corresponding event image frame IEτ . 3.3 Event bytestream processing . Given event data as an arbitrarily long bytestream Eτ , the objective of representation learning is to map it to a compressed vector representing the latent state of the environment zτ through an encoder function qe ( Eτ ) . The challenges here are two-fold . First , due to the non-uniform and asynchronous nature of the event camera data , the same scene when imaged multiple times by an event camera could result in different permutations of the output . Hence , to handle the asynchronicity of event cameras , we require a feature computation technique that is invariant to data ordering . Secondly , while event sequences are time-based data , recurrent neural networks would prove to be infeasible due to the often long sequence lengths . Decoupling the temporal information from the spatial/polarity information alleviates this problem . We propose a backbone called the event context network ( ECN ) to achieve this for event data . The ECN can be thought of as a preprocessing neural network for event streams , similar to architectures aimed at learning unordered spatial data such as PointNet [ 21 ] and EventNet [ 27 ] . The ECN takes an arbitrarily long list of events , and first computes a feature for each event . Eventually , these features are passed through a symmetric function ( similar to PointNet , we also use a max operation ) , resulting in a global feature that is expected to condense information from all the events . The symmetric nature of this function ensures that these events in a given list can be processed either as a single batch , or recursively with any minibatch size to compute the output . We call the output of this feature network a ‘ context vector ’ . The ECN consists of three dense layers which , for N input events , output an N ×D set of features . The data passed into these dense layers is only the ( x , y , p ) part of the events - and we discuss how we handle the temporal information next .

Summary: This paper proposes a method for learning representations of event-based camera percepts which can be used for reinforcement learning. The method is able to digest irregular, streaming camera information using a VAE conditioned on a context vector computed from event data using a PointNet style network. The empirical evaluation shows that using this pretrained representation for PPO is more performant and generalizable than frame-based methods on a simulated UAV obstacle avoidance task.