Representation Learning for Event-based Visuomotor Policies

1 Introduction . Autonomous navigation , which is driven by a tight coupling between perception and action , is particularly challenging for fast , agile robots such as unmanned micro aerial vehicles ( MAV ) that are often deployed in cluttered and low altitude areas . For such reactive navigation applications such as obstacle avoidance , low sensor latency is the key to performing agile maneuvers successfully [ 1 ] . MAVs are also limited in their size and payload capacity , which constrains onboard sensor choices to small , low-power sensors , and the computational load of the processing algorithms to be minimal . Modern computer vision and machine learning techniques for perception and navigation typically focus on analyzing data from conventional CMOS based cameras , in various modalities such as RGB images , depth maps etc . While these cameras provide high resolution data , the main drawback of these sensors is their speed , with most averaging output at a rate of 30-60 Hz . This makes such sensors unable to scale up to the perception data rate required by agile navigation . Inspired by biological vision , neuromorphic engineering has resulted in a novel sensor known as the dynamic vision sensor , or an event-based camera [ 2 ] . These cameras detect and measure changes in log-luminance on a per-pixel basis , and return information about ‘ events ’ with a temporal resolution on the order of microseconds . Due to the increased sampling speed of these cameras and the minimal processing needed to parse the data , perception using event cameras can be much faster than traditional approaches . This can allow for faster control schemes to be used , as enough relevant environmental information can be collected quickly in order to make informed control choices . Moreover , the events are inherently generated by changes in brightness typically arising from motion . This makes event cameras natural motion detectors and a good fit for learning control policies . ∗Work done while interning at Microsoft Research 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . But the fundamentally different visual representation of event cameras poses significant challenges to quick adoption . Event cameras produce fast and asynchronous spatiotemporal data , significantly different from synchronous frame-based data expected by conventional machine learning algorithms . In addition , the quality of the data recorded by an event camera is different from traditional perception sensors ; the sensors return low-level data that could vary significantly based on the firing order of pixels , lighting conditions , reflections or shadows . Previous research has approached this modality through two main classes of techniques . Some approaches [ 3 , 4 ] accumulate event data over time into a two dimensional frame , and use traditional computer vision/convolutional neural network based techniques with these frame-based inputs . Traditional CNN approaches combined with such accumulation fail to exploit the true advantages of event cameras such as the microsecond-scale temporal resolution , and may prove to be too intensive for high-speed action generation onboard constrained platforms . Another class of techniques involves the usage of spiking neural networks ( SNN ) [ 5 ] . SNNs operate through spiking neurons to identify spatio-temporal firings , making it a natural match for event cameras . Yet , training spiking neural networks is hard , as they do not use standard backpropagation , and often require specialized hardware to truly realize their efficiency [ 6 , 7 ] . In this paper , we propose learning representations directly from raw event camera streams using conventional ( non-spiking ) machine learning methods , and learning visuomotor policies over such representations ( Figure 1 ) . We present an event variational autoencoder ( eVAE ) framework for learning representations from event data in a way that allows for high temporal resolution as well as invariance to data permutations and sparsity . The eVAE is equipped with an event feature computation network that can process asynchronous data from arbitrary sequence lengths , or in a recursive fashion . Inspired by the recent success of Transformer networks [ 8 , 9 , 10 ] , the eVAE uses a temporal embedding method that preserves the event timing information when computing latent representations . Next , we show that such representations can be beneficial for reactive navigation , by applying them as observations in a reinforcement learning ( RL ) framework . We show how training RL policies over eVAE representations allows the control policy to generalize to different data rates and even to out-of-distribution environments . We define obstacle avoidance for UAVs as our task of interest and demonstrate how event camera data can be effectively utilized for avoidance at high control rates . Through an event data simulator , we simulate scenarios where the UAV is assumed to be controlled at up to 400 Hz , and show that the ability of the representations to handle sparse data allows the policy to adapt to high control rates . The key contributions of our work are listed below . 1 . We present an event variational autoencoder for unsupervised representation learning from fast and asynchronous spatiotemporal event bytestream data . 2 . We show that these event representations capture sufficient contextual information to be useful in learning reactive visuomotor policies . 3 . We train policies over event representations using reinforcement learning for obstacle avoidance for UAVs in simulation and show that they outperform current state of the art in event-based reinforcement learning . 4 . We discuss advantages of using bytestream representations for policies , such as : adaptation to different control capacities , robustness to environmental variations and noise . 2 Related Work . Vision-based representations and navigation : Variational autoencoders have been shown to be effective in learning well structured low-dimensional representations from complex visual data [ 11 , 12 , 13 ] . Leveraging such methods , recent research has focused on the decoupling of perception and planning , showing that separate networks for representation and navigation is effective [ 14 , 15 ] . As the representation is expected to capture rich salient information about the world with a degree of invariance , this combination allows for higher sample efficiency and smaller policy network sizes [ 16 ] . Feature learning from Event Cameras : Some of the early work conducted on processing event data resulted in computing optical flow using the asynchronous data , focusing on high-speed computations with minimal bandwidth [ 17 ] . Event representations included histograms of averaged time surfaces ( HATS ) , where temporal data is aggregated to create averaged data points capable of being used as input for traditional techniques [ 18 ] and hierarchy of event-based time surfaces ( HOTS ) , another representation for pattern recognition [ 19 ] . Learning from Sequences and Sets : Learning from event data can be treated as a case of learning long , variable length sequences . While conventional RNNs are found to be infeasible for such lengths , approaches such as Phased LSTM [ 20 ] propose adding a time gate to LSTM for long sequences . If the spatial and temporal parts were decoupled , the problem can be reformulated as permutation-invariant learning from sets . Qi et al [ 21 ] present PointNet , which is a one such permutation invariant approach aimed at learning from 3D point cloud data . Similarly , Lee et al [ 22 ] present the Set Transformer , an attention-based learning method for sets . Event Cameras and Machine Learning : From a machine learning perspective , Gehrig et al [ 23 ] introduced a full end-to-end pipeline for learning to represent event-based data , which discusses several variants such as event data aggregated into a grid-based representation , event spike tensors , and 3D voxel grids . Asynchronous versions of convolutional neural networks are also being developed to take advantage of the sparsity in data such as that of event cameras [ 24 , 25 ] . Stacked spatial LSTM networks were used with event sequences for pose relocalization in [ 4 ] . EV-FlowNet [ 26 ] is an encoder-decoder architecture for self-supervised optical flow for events , which uses frame-based inputs processed through convolutional layers . The asynchronous nature of event data was handled through a permutation-invariant and recursive approach in EventNet [ 27 ] . Event camera based perception was used in other applications as well , such as self-supervised learning of optical flow [ 28 ] , steering prediction for self driving cars [ 3 ] . Spiking neural networks were also used to examine event-based data [ 29 , 30 , 31 , 32 , 33 , 34 ] . Sensorimotor Policies with Event Cameras : Only very recently has there been work on combining event camera data with sensorimotor policies . Event camera data was coupled with control for autonomous UAV landing in [ 35 ] , [ 36 ] . EVDodge [ 37 ] creates an avoidance system for UAVs by using event data to track moving objects and infer safe avoidance maneuvers based on these measurements , combining multiple modules such as homography , segmentation , with the actions driven by a classical control policy . Event camera data was also used to power a closed-loop control scheme for a UAV on a fixed bench setup by tracking roll angles and angular velocities in [ 38 ] . Reinforcement learning using event camera data has only been explored very recently , using accumulated event frames fed into CNN-based policy networks for ground robots [ 39 ] and for UAV obstacle avoidance [ 40 ] . 3 Representation Learning for Event Cameras . 3.1 Event-based camera . An event based camera is a special vision sensor that measures changes in intensity levels independently at each of its pixels . Given a pixel location ( x , y ) , the fundamental working principle of an event-based camera is to measure the change in logarithmic brightness at that pixel , i.e. , ∆log I ( { x , y } , t ) where I is the photometric intensity . When this change in logarithmic brightness exceeds a set threshold , the camera generates an ‘ event ’ , reporting the time and location of change , along with the ‘ sign ’ of the change . In contrast to conventional cameras which output a set number of frames per second , an event camera outputs events sparsely and asynchronously in time as a stream of bytes , which we refer to as an event ‘ bytestream ’ . These events are produced at a non-uniform rate , and the number can range from zero to millions of events per second . For example , the DAVIS 240 camera [ 41 ] has a theoretical maximum limit of 12 million events per second . 3.2 Definitions and Notations . For an event camera of resolution ( H , W ) , an event can be defined as a tuple of four quantities e = ( t , x , y , p ) where t is a global timestamp at which the event was reported by the camera , ( x , y ) the pixel coordinates , and p the polarity . A sequence of events over a time window of τ can thus be represented as Eτ = { ei|t < i < t+ τ } . When sliding a constant time window of τ over a longer sequence of events , we can see that the length of Eτ will not be constant as the number of events fired in that interval would change based on environmental or sensory considerations . The events in Eτ can also be accumulated and represented as a corresponding event image frame IEτ . 3.3 Event bytestream processing . Given event data as an arbitrarily long bytestream Eτ , the objective of representation learning is to map it to a compressed vector representing the latent state of the environment zτ through an encoder function qe ( Eτ ) . The challenges here are two-fold . First , due to the non-uniform and asynchronous nature of the event camera data , the same scene when imaged multiple times by an event camera could result in different permutations of the output . Hence , to handle the asynchronicity of event cameras , we require a feature computation technique that is invariant to data ordering . Secondly , while event sequences are time-based data , recurrent neural networks would prove to be infeasible due to the often long sequence lengths . Decoupling the temporal information from the spatial/polarity information alleviates this problem . We propose a backbone called the event context network ( ECN ) to achieve this for event data . The ECN can be thought of as a preprocessing neural network for event streams , similar to architectures aimed at learning unordered spatial data such as PointNet [ 21 ] and EventNet [ 27 ] . The ECN takes an arbitrarily long list of events , and first computes a feature for each event . Eventually , these features are passed through a symmetric function ( similar to PointNet , we also use a max operation ) , resulting in a global feature that is expected to condense information from all the events . The symmetric nature of this function ensures that these events in a given list can be processed either as a single batch , or recursively with any minibatch size to compute the output . We call the output of this feature network a ‘ context vector ’ . The ECN consists of three dense layers which , for N input events , output an N ×D set of features . The data passed into these dense layers is only the ( x , y , p ) part of the events - and we discuss how we handle the temporal information next .

Summary: The author proposed an event based VAE for high speed robot navigation useful for reinforcement learning. They then compare to other image based RL approaches in AirSim and show they outperform other event based RL baselines. They demonstrate that the representation is useful for training visuomotor policies. They show that it reaches SOTA in event based reinforcement learning on the given tasks.