Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

1 Introduction . Neuromorphic hardware promises highly energy-efficient and low-latency sensing and processing thanks to its sparse and asynchronous nature . Event cameras capture brightness changes at microsecond resolution [ 17 ] , while neuromorphic processors have demonstrated orders of magnitude lower energy consumption and latency compared to von Neumann architectures [ 11 , 29 ] . To realize the full potential of such neuromorphic pipelines , we have to move towards an event-based communication and processing paradigm , where single events are passed as-is between the event-based sensor/camera and the neuromorphic processor running a spiking neural network ( SNN ) , without processing or ∗Equal contribution , with alphabetical ordering . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . accumulation of any kind in between . Because of this , all temporal integration of information needs to happen inside the network itself . Most work on employing SNNs to event-based computer vision follows this approach [ 10 , 14 ] , but is limited to problems of limited temporal complexity ( like classification ) . On the other hand , most state-of-the-art artificial neural network ( ANN ) pipelines for event-based computer vision combine a stateless feedforward architecture with encoding temporal information in the input [ 54 , 56 ] . Apart from incompatible pipelines , one of the larger impediments to widespread neuromorphic adoption is the fact that learning algorithms designed for traditional ANNs do not transfer one-to-one to SNNs , which exhibit sparse , binary activity and more complex neuronal dynamics . On the one hand , this has driven research into the conversion of ANNs to SNNs without loss of accuracy , but with the promised efficiency gains [ 42 ] . On the other hand , it has limited the application of directly-trained SNNs in the computer vision domain to less complicated and often discrete problems like image classification [ 10 , 14 ] on constrained datasets such as N-MNIST [ 33 ] or DVS128 Gesture [ 2 ] . Still , many ongoing developments in the area of direct SNN training are promising and may form building blocks for tackling more complex tasks . Surrogate gradients [ 32 , 41 , 45 , 51 ] , which act as stand-in for the non-differentiable spiking function in the backward pass , enable traditional backpropagation with few adjustments . Similarly , the inclusion of parameters governing the neurons ’ internal dynamics in the optimization was demonstrated to be beneficial [ 14 , 38 ] . Many works also include some form of activity regularization to keep neurons from excessive spiking [ 5 , 51 ] or to balance excitability through adaptation [ 4 , 36 ] . Kickstarting initial activity ( hence gradient flow ) is often not the goal of these regularization terms , even though [ 51 ] shows that there is a narrow activity band in which learning is optimal . This ties in with the initialization of parameters , which has not been rigorously covered for SNNs with sparse inputs yet , leaving room for improvement . Our goal is to demonstrate the potential of neuromorphic sensing and processing on a complex task . To this end , we tackle a real-world large-scale problem by learning , in a self-supervised fashion and using SNNs , to estimate the optical flow encoded in a continuous stream of events ; a task that is usually tackled with deep , fully convolutional ANNs [ 54 , 56 ] . By focusing on such a problem , we aim to identify and tackle emerging knowledge gaps regarding SNN training , while approximating a truly asynchronous pipeline . In summary , the main contribution of this article is two-fold . First , we propose a novel self-supervised learning ( SSL ) framework for event-based optical flow estimation that puts emphasis on the networks ’ capacity to integrate temporal information from small , subsequent slices of events . This training pipeline , illustrated in Fig . 1 , is built around a reformulation of the self-supervised loss function from [ 56 ] that improves its convexity . Second , through this framework , we train the first set of deep SNNs that successfully solve the problem at hand . We validate our proposals through extensive quantitative and qualitative evaluations on multiple datasets1 . Additionally , for the SNNs , we investigate the effects of elements such as parameter initialization and optimization , surrogate gradient shape , and adaptive neuronal mechanisms . 2 Related Work . Due to the potential of event cameras to enable low-latency optical flow estimation , extensive research has been conducted on this topic since these sensors were introduced [ 1 , 6 , 8 , 15 ] . Regarding learningbased approaches , in [ 54 ] , Zhu et al . proposed the first convolutional ANN for this task , which was trained in an SSL fashion with the supervisory signal coming from the photometric error between subsequent grayscale frames captured with the active pixel sensor ( APS ) of the DAVIS240C [ 7 ] . Alongside this network , the authors released the Multi-Vehicle Stereo Event Camera ( MVSEC ) dataset [ 55 ] , the first event camera dataset with ground-truth optical flow estimated from depth and ego-motion sensors . A similar SSL approach was introduced in [ 48 ] , but here optical flow was obtained through an ANN estimating depth and camera pose . Later , Zhu et al . refined their pipeline and , in [ 56 ] , proposed an SSL framework around the contrast maximization for motion compensation idea from [ 15 , 16 ] ; with which , as explained in Section 3.2 , the supervisory signal comes directly from the events and there is no need for additional sensors . More recently , Stoffregen and Scheerlinck et al . showed in [ 44 ] that , if trained with synthetic event sequences ( from an event camera simulator [ 39 ] ) and ground-truth data in a pure supervised fashion , the ANN from [ 54 , 56 ] reaches higher accuracy levels when evaluated on MVSEC . Lastly , to hold up to the promise of high-speed optical flow , there has been a significant effort toward the miniaturization of optical flow ANNs [ 28 , 35 ] . With respect to learning-based SNNs for optical flow estimation , only the works of Paredes-Vallés et al . [ 36 ] and Lee et al . [ 26 , 27 ] are to be highlighted . In [ 36 ] , the authors presented the first convolutional SNN in which motion selectivity emerges in an unsupervised fashion through Hebbian learning [ 22 ] and thanks to synaptic connections with multiple delays . However , this learning method limits the deployability of this architecture to event sequences with similar statistics to those used during training . On the other hand , in [ 26 ] , the authors proposed a hybrid network , integrating spiking neurons in the encoder with ANN layers in the decoder , trained through the SSL pipeline from [ 54 ] . This architecture was later expanded in [ 27 ] with a secondary ANN-based encoder used to retrieve information from the APS frames . Lastly , SNNs have also been implemented in neuromorphic hardware for optical flow estimation [ 19 ] , although this did not involve learning . Hence , until now , no one has yet attempted the SSL of optical flow with a pure SNN approach . Most of the SNN work in other computer vision domains has so far been focused on discrete problems like classification [ 10 , 14 , 46 , 53 ] and binary motion-based segmentation [ 34 ] . A notable exception is the work from Gehrig et al . [ 18 ] , who propose a convolutional spiking encoder to continuously predict angular velocities from event data . However , until now , no one has yet attempted a dense ( i.e. , with per-pixel estimates ) regression problem with deep SNNs that requires recurrency . 3 Method . 3.1 Input event representation . An event camera consists of a pixel array that responds , in a sparse and asynchronous fashion , to changes in brightness through streams of events [ 17 ] . For an ideal camera , an event ei = ( xi , ti , pi ) of polarity pi ∈ { + , − } is triggered at pixel xi = ( xi , yi ) T and time ti whenever the brightness change since the last event at that pixel reaches the contrast sensitivity threshold for that polarity . The great majority of learning-based models proposed to date for the problem of event-based optical flow estimation encode , in one form or another , spatiotemporal information into the input event representation before passing it to the neural architectures . This allows stateless ( i.e. , non-recurrent ) ANNs to accurately estimate optical flow at the cost of having to accumulate events over relatively long time windows for their apparent motion to be perceivable . The most commonly used representations 1The project ’ s code and additional material can be found at https : //mavlab.tudelft.nl/event_flow/ . make use of multiple discretized frames of event counts [ 26 , 27 , 35 , 44 , 56 ] and/or the per-pixel average or the most recent event timestamps [ 28 , 48 , 54 ] . Ideally , SNNs would immediately receive spikes at event locations , which implies that temporal information should not be encoded in the input representation , but should be extracted by the network . To enforce this , we use a representation consisting only of per-pixel and per-polarity event counts , as in Fig . 1 . This representation gets populated with consecutive , non-overlapping partitions of the event stream εinpk . = { ei } N−1i=0 ( referred to as input partition ) each containing a fixed number of events , N . 3.2 Self-supervised learning of optical flow via contrast maximization . We use the contrast maximization proxy loss for motion compensation [ 16 ] to learn to estimate optical flow from the continuous event stream in a self-supervised fashion . The idea behind this optimization framework is that accurate optical flow information is encoded in the spatiotemporal misalignments among the events triggered by the same portion of a moving edge ( i.e. , blur ) and that , to retrieve it , one has to compensate for this motion ( i.e. , deblur the event partition ) . Knowing the per-pixel optical flow u ( x ) = ( u ( x ) , v ( x ) ) T , the events can be propagated to a reference time tref through : x′i = xi + ( tref − ti ) u ( xi ) ( 1 ) In this work , we reformulate the deblurring quality measure proposed by Mitrokhin et al . [ 30 ] and Zhu et al . [ 56 ] : the per-pixel and per-polarity average timestamp of the image of warped events ( IWE ) . The lower this metric , the better the event deblurring and the more accurate the optical flow estimation . We generate an image of the average timestamp at each pixel for each polarity p′ via bilinear interpolation : Tp′ ( x ; u|tref ) = ∑ j κ ( x− x′j ) κ ( y − y′j ) tj∑ j κ ( x− x′j ) κ ( y − y′j ) + κ ( a ) = max ( 0 , 1− |a| ) j = { i | pi = p′ } , p′ ∈ { + , − } , ≈ 0 ( 2 ) Previous works minimize the sum of the squared temporal images resulting from the warping process [ 35 , 56 ] . However , we scale this sum prior to the minimization with the number of pixels with at least one warped event in order for the loss function to be convex : Lcontrast ( tref ) = ∑ x T+ ( x ; u|tref ) 2 + T− ( x ; u|tref ) 2∑ x [ n ( x ′ ) > 0 ] + ( 3 ) where n ( x′ ) denotes a per-pixel event count of the IWE . As shown in Appendix A , without the scaling , the loss function is not well-defined as the optimal solution is to always warp events with large timestamps out of the image space so they do not contribute to Eq . 2 . Previous works circumvented this issue by limiting the maximum magnitude of the optical flow vectors that could be estimated through scaled TanH activations in the prediction layers [ 35 , 56 ] . As in [ 56 ] , we perform the warping process both in a forward ( tfwref ) and in a backward fashion ( t bw ref ) to prevent temporal scaling issues during backpropagation . The total loss used to train our event-based optical flow networks is then given by : Lcontrast = Lcontrast ( tfwref ) + Lcontrast ( tbwref ) ( 4 ) Lflow = Lcontrast + λLsmooth ( 5 ) where λ is a scalar balancing the effect of the two losses and Lsmooth is a Charbonnier smoothness prior [ 9 ] , as in [ 54 , 56 ] . Since Lcontrast does not propagate the error back to pixels without input events , we mask the output of our networks so that null optical flow vectors are returned at these pixel locations . Furthermore , we mask the computation of Lsmooth so that this regularization mechanism only considers optical flow estimates from neighboring pixels with at least one event . As hinted by the motion model in Eq . 1 and discussed in [ 16 , 43 ] , there has to be enough linear blur in the input event partition for Lcontrast to be a robust supervisory signal . This is usually not the case in our training pipeline due to the small number of input events N that we pass to the networks at each forward pass . For this reason , we define a secondary event partition , the so-called training partition εtraink→k+K . = { ( εinpi , ûi ) } Ki=k , which is a buffer that gets populated every forward pass with an input event partition and its corresponding optical flow estimates . At training time , we perform a backward pass with the content of the buffer using backpropagation through time once it contains K successive event-flow tuples , after which we detach the state of the networks from the computational graph and clear the buffer . Note that Lsmooth is also applied in the temporal dimension by smoothing optical flow estimates at the same pixel location from adjacent tuples .

Summary: The paper investigates optical-flow estimation from event-cameras, and in particular provides the first deep SNN solution for real-world applications of this concept. The approach filters event streams from the input into an event-representation where every partition has the same number of events, which is then fed to the SNN. A backward pass is also possible using a contrast maximization loss. A new self-supervised learning framework is introduced.  The results show the new method to be on par with the state-of-the-art.