Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks
Authors: Jesse Hagenaars
Federico Paredes Valles
Guido De Croon
Venue: NeurIPS 2021 Poster (2021)
Keywords: spiking neural networks
neuromorphic computing
event camera
self-supervised learning
optical flow

Abstract: The field of neuromorphic computing promises extremely low-power and low-latency sensing and processing. Challenges in transferring learning algorithms from traditional artificial neural networks (ANNs) to spiking neural networks (SNNs) have so far prevented their application to large-scale, complex regression tasks. Furthermore, realizing a truly asynchronous and fully neuromorphic pipeline that maximally attains the abovementioned benefits involves rethinking the way in which this pipeline takes in and accumulates information. In the case of perception, spikes would be passed as-is and one-by-one between an event camera and an SNN, meaning all temporal integration of information must happen inside the network. In this article, we tackle these two problems. We focus on the complex task of learning to estimate optical flow from event-based camera inputs in a self-supervised manner, and modify the state-of-the-art ANN training pipeline to encode minimal temporal information in its inputs. Moreover, we reformulate the self-supervised loss function for event-based optical flow to improve its convexity. We perform experiments with various types of recurrent ANNs and SNNs using the proposed pipeline. Concerning SNNs, we investigate the effects of elements such as parameter initialization and optimization, surrogate gradient shape, and adaptive neuronal mechanisms. We find that initialization and surrogate gradient width play a crucial part in enabling learning with sparse inputs, while the inclusion of adaptivity and learnable neuronal parameters can improve performance. We show that the performance of the proposed ANNs and SNNs are on par with that of the current state-of-the-art ANNs trained in a self-supervised manner.

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks
Jesse Hagenaars" j.j hagenaars@tudellf.. ml
Federico Paredes-Vallés f parrodssallessttuelff...t nl
Guido C. H. E. de Croon g g.c.h. c.h.e decroon@tuuelltt.tm
Micro Air Vehicle Laboratory Delft University of Technology, The Netherlands
Abstract
The field of neuromorphic computing promises extremely low-power and low- latency sensing and processing. Challenges in transferring learning algorithms from traditional artificial neural networks (ANNs) to spiking neural networks (SNNs) have so far prevented their application to large-scale, complex regression tasks. Furthermore, realizing truly asynchronous and fully neuromorphic pipeline that maximally attains the abovementioned benefits involves rethinking the way which this pipeline takes in and accumulates information. In the case of perception, spikes would be passed as-is and one-by-one between an event camera and an SNN, meaning all temporal integration of information must happen inside the network. In this article, we tackle these two problems. We focus on the complex task of learning estimate optical flow from event-based camera inputs in a self-supervised manner, and modify the statc-of-the-arr ANN training pipeline to encode minimal temporal information in inputs. Moreover, we reformulate the self-supervised loss function for event-based optical flow to improve its convexity. We perform experiments with various types of recurrent ANNs and SNNs using the proposed pipeline. Concerning SNNs, we investigate the effects of elements such as parameter initialization and optimization, surrogate gradient shape, and adaptive neuronal mechanisms. We find that initialization and surrogate gradient width play a crucial part in enabling learning with sparse inputs, while the inclusion of adaptivity and learnable neuronal parameters show that the performance of the proposed ANNs and SNNs are on par with that of the current state-of- state-of-the-art the-art ANNs trained in self-supervised manner.
improve performance. We
Introduction
Neuromorphic hardware promises highly energy-efficient and low-latency sensing and processing thanks to its sparse and asynchronous nature. Event cameras capture brightness changes at microsec- ond resolution [17]. while neuromorphic processors have demonstrated orders of magnitude lower energy consumption and latency compared to von Neumann architectures [11, 29J. To realize the full potential of such neuromorphic pipelines, we have move towards an event-based communication and processing paradigm, where single events are passed as-is between the event- based sensor/camera and the neuromorphic processor running a spiking neural network (SNN), without processing
Equal contribution, with alphabetical ordering.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Event stream
Optical flw
HH
SNN
22 Ma
Contrast maximization loss via motion compensation
representation
Figure I: Self-supervised event-based optical flow pipeline for deep SNNs. In order of processing, the event stream is split into small partitions with the same number of events, which are formatted and then fed to the network in sequential fashion. An optical flow map is predicted for each partition, associating every input event with a motion vector. Once a sufficient number of events has been processed, we perform a backward pass using our contrast maximization loss [16].
accumulation of any kind in between. Because this, all temporal integration of information needs happen inside the network itself. Most work on employing SNNs event-based computer vision follows this approach [10, 14), but is limited to problems of limited temporal complexity (like classification). On the other hand, most state-of-the-art artificial neural network (ANN) pipelines for event-based computer vision combine a stateless feedforward architecture with encoding temporal information in the input [54, 56].
Apart from incompatible pipelines, one of the larger impediments to widespread neuromorphic adoption is the fact that learning algorithms designed for traditional ANNs do not transfer one-to-one to SNNs, which exhibit sparse, binary activity and more complex neuronal dynamics. On the one hand, this has driven research into the conversion of ANNs to SNNs without loss of accuracy, but with the promised efficiency gains [42]. On the other hand, it has limited the application of directly- -rraine SNNs in the computer vision domain to less complicated and often discrete problems like image classification [10, 14] on constrained datasets such as N-MNIST [33] or DVS128 Gesture [2].
Still, many ongoing developments in the area of direct SNN training are promising and may form building blocks for tackling more complex tasks. Surrogate gradients [32, 41, 45, 51], which act stand-in for the non-diffcrentiable spiking function in the backward pass, enable traditional backpropagation with few adjustments. Similarly, the inclusion of parameters governing the neurons' internal dynamics in the optimization was demonstrated to be beneficial [14, 38). Many works also include some form of activity regularization to keep neurons from excessive spiking [5, 51] or to balance excitability through adaptation [4, 36). Kickstarting initial activity (hence gradient flow) is often not the goal of these regularization terms, even though [51] shows that there is a narrow activity band in which learning optimal. This ties in with the initialization of parameters, which has not been rigorously covered for SNNs with sparse inputs yet, leaving room for improvement.
Our goal is to demonstrate the potential of neuromorphic sensing and processing To this end, we tackle a real-world large-scale problem by learning, in a self-supervised fashion and using SNNs, to estimate the optical flow encoded in continuous stream of events; task that is usually tackled with deep. fully convolutional ANNs [54, 56]. By focusing on such problem, we aim to identify and tackle emerging knowledge gaps regarding SNN training, while approximating truly asynchronous pipeline.
complex task.
In summary. the main contribution of this article is two-fold. First, we propose novel self self-supervised learning (SSL) framework for event-based optical flow estimation that puts emphasis on the networks' capacity integrate temporal information from small, subsequent slices of events. This training

---

pipeline, illustrated in Fig. 1, is built around reformulation of the self superviid loss function from [56] that improves its convexity. Second, through this framework, we train the first set of deep SNNs that successfully solve the problem at hand. We validate our proposals through extensive quantitative and qualitative evaluations on multiple datasets'. Additionally, for the SNNs, we investigate the effects of elements such as parameter initialization and optimization, surrogate gradient shape, and adaptive neuronal mechanisms.
2 Related Work
Due to the potential of event cameras to enable low-latency optical flow estimation, extensive research has been conducted on this topic since these sensors were introduced [1, 6, 8. 15]. Regarding learning- based approaches, in [54], Zhu et al. proposed the first convolutional ANN for this task, which was trained in an SSL fashion with the supervisory signal coming from the photometric error between subsequent grayscale frames captured with the active pixel sensor (APS) of the DAVIS240C [7]. Alongside this network, the authors released the Multi-Vehicle Stereo Event Camera (MVSEC) dataset [55], the first event camera dataset with ground-truth optical flow estimated from depth and ego-motion sensors. A similar SSL approach was introduced in [48], but here optical flow was obtained through an ANN estimating depth and camera pose. Later, Zhu al. refined their pipeline and, in [56], proposed an SSL framework around the contrast maximization for motion compensation idea from [15, 16]; with which, as explained in Section 3.2, the supervisory signal comes directly from the events and there is no need for additional sensors. More recently, Stoffregen and Scheerlinck et al. showed in [44] that, if trained with synthetic event sequences (from an event camera simulator [39]) and ground-trum -runt data in a pure supervised fashion, the ANN from [54, 56] reaches higher accuracy levels when evaluated on MVSEC. Lastly, to hold up to the promise of high-speed optical flow, there has been significant effort toward the miniaturization of optical flow ANNs [28, 35).
With respect to bease SNNs for optical flow estimation, only the works of Paredes- et al. [36] and Lee et al. [26, 27] are to be highlighted. In [36], the authors presented the first convolutional SNN in which motion selectivity emerges an unsupervised fashion through Hebbian learning [22] and thanks to synaptic connections with multiple delays. However, this learning method limits the deployability of this architecture event sequences with similar statistics those used during training. On the other hand, in [26]. the authors proposed hybrid network, integrating spiking neurons in the encoder with ANN layers the decoder, trained through the SSL pipeline from [54]. This architecture was later expanded in [27] with secondary ANN-based encoder used to retrieve information from the APS frames. Lastly, SNNs have also been implemented in neuromorphic hardware for optical flow estimation [19]. although this did not involve learning. Hence, until now, no one has yet attempted the SSL of optical flow with a pure SNN approach.
Most of the SNN work in other computer vision domains has so far been focused on discrete problems like classification [10, 14, 46, 53] and binary motion-based segmentation [34]. A notable exception is the work from Gehrig et al. [18], who propose a convolutional spiking encoder to continuously predict angular velocities from event data. However, until now, no one has yet attempted a dense (i.e.. with pixel estimates) regression problem with deep SNNs that requires recurrency.
3 Method
3.1 Input event representation
An event camera consists of a pixel array that responds, in a sparse and asynchronous fashion, to changes in brightness through streams of events [17]. For an ideal camera, an event e; = (æ,, Pt) of polarity pt {+,- }} is triggered at pixel = (æi, y² and time L; whenever the brightness change since the last event at that pixel reaches the contrast sensitivity threshold for that polarity.
The great majority of learning-bass based models proposed to date for the problem of event- based optical flow estimation encode, in one form or another, spatiotemporal information into the input event representation before passing it to the neural architectures. This allows stateless (i.e., non-recurrent) ANNs to accurately estimate optical flow at the cost of having to accumulate events over relatively long time windows for their apparent motion to be perceivable. The most commonly used representations
The project's code and additional material can be found at https://mavlabbbbb. www.eef....eeeeeett.ttt.mmmm.ee.
3

---

make use of multiple discretized frames of event counts [26, 27, 35, 44, 56] and/or the per per-pixel average or the most recent event timestamps [28, 48, 54].
Ideally, SNNs would immediately receive spikes at event locations, which implies that temporal information should not be encoded in the input representation, but should be extracted by the network. To enforce this, we use a representation consisting only of per-pixel and per-polarity event counts, as in Fig. This representation gets populated with consecutive, non-overlapping partitions of the event inp stream & [e - (referred to as input partition) each containing fixed number of events, N.
3.2 Self-supervised learning of optical flow via contrast maximization
We use the contrast maximization proxy loss for motion compensation [16] learn estimate optical flow from the continuous event stream in a self-supervised fashion. The idea behind this optimization framework is that accurate optical flow information is encoded in the spatiotemporal misalignments among the events triggered by the same portion of a moving edge (i.e., blur) and that, to retrieve it, one has to compensate for this motion (i.e., deblur the event partition). Knowing the per-pixel optical flow u<x) = (u( v(x)) the events can be propagated to a reference time bee through:
a 27 + (Tref 14) (a;))
In this work, we reformulate the deblurring quality measure proposed by Mitrokhin et al. [30] and Zhu et al. [56]: the per-pixel and per-polarity average timestamp of the image of warped events (IWE). The lower this metric, the better the event deblurring and the more accurate the optical flow estimation. We generate an image of the average timestamp at each pixel for each polarity p' via bilinear interpolation:
E, kl( 2( x2) 8,Y4,
Tp (2; Σ, k((x 10(T-xj)1 x2)ri(y - Vn)) +
<(a) = max(0, (al)
= p: p=)),, p' E -+. 6
Previous works minimize the sum of the squared temporal images resulting from the warping process [35, 56). However, we scale this sum prior to the minimization with the number of pixels with at least one warped event in order for the loss function be convex:
Leontrase (Iret) Ex T4 (x: 2 + T- (x;u)ttirr²
En [n((a))) 0) +
where n(x') denotes a per-pixel event count of the IWE. As shown in Appendix A, without the scaling, the loss function is not well-defined as the optimal solution is to always warp events with large timestamps out of the image space so they do not contribute Eq. 2. Previous works cireumvented this issue by limiting the maximum magnitude of the optical flow vectors that could be estimated through scaled TanH activations in the prediction layers [35, 56).
As in [56). we perform the warping process both in a forward (f and in a backward fashion (1b7) to prevent temporal scaling issues during backpropagation. The total loss used train our event-based optical flow networks is then given by:
Looutrast = Leontrast (f()r + Lcontrast (t
(4)
Lnow Leontrast + X.
(5)
where À is scalar balancing the effect of the two losses and Lemooth is Charbonnier smoothness prior [9], as in [54, 56]. Since Leontass does not propagate the error back to pixels without input events, we mask the output of our networks so that null optical flow vectors are returned at these pixel locations. Furthermore, we mask the computation of Lamooth so that this regularization mechanism only considers optical flow estimates from neighboring pixels with at least one event.
As hinted by the motion model in Eq. and discussed in [16, 43), there has to be enough linear blur in the input event partition for Leomtass to be robust supervisory signal. This is usually not the case in our training pipeline due to the small number of input events N that we pass to the networks at each forward pass. For this reason, define secondary event partition, the so-called training partition

---

& = [[e]"r ú which is a buffer that gets populated every forward pass with an input event partition and its corresponding optical flow estimates. At training time, we perform backward pass with the content of the buffer using backpropagation through time once it contains K successive event-flow tuples, after which we detach the state of the networks from the computational graph and clear the buffer. Note that smoohh is also applied in the temporal dimension by smoothing optical flow estimates at the same pixel location from adjacent tuples.
stain
3.3 Spiking neuron models
We compare various spiking neuron models from literature on the task of event-based optical flow estimation. All models are based on the leaky-integrate-and and-fire (LIF) neuron, whose membrane potential U and synaptic input current / at timestep k can be written as:
U = (1 S )ao)) + (1- a
(6)
,. = Σ W S + Σ WK
Sk
(7)
where j and r denote presynaptic neurons while i is for postsynaptic, a is the membrane decay or leak, S {0, 1} a neuron spike, and Wiff and Whee feedforward and recurrent connections, respectively. Membrane decays can either be fixed or learned. A neuron fires an output spike S if the membrane potential exceeds threshold 0, which can either be fixed, learned, adaptive (see below). Firing also triggers a reset of U, which is either hard, as in Eq. 6, or soft, as in [4]. The former is said to be more suitable for deeper networks, it gets rid of errors accumulated by the surrogate gradient [25].
Following [4], we introduce an adaptive threshold to make up the adaptive LIF (ALIF) model. A second state variable T' acts low-pass filter over the output spikes, adapting the firing threshold based on the neuron's 's activity:
0, = 3 + 311 T - - + (1 - -
(8)
(9)
where Btu,1) are (learnable) constants, and 17 is the (learnable) threshold decay/leak. ALIF's equations for U and are identical to the LIF formulation. By decaying the threshold very slowly, T can act as a longer-term memory of the neuron [5].
Instead of postsynaptic adaptivity, we can keep trace of presynaptic activity and use that to regularize neuron firing, giving the presynaptic LIF (PLIF) model. The authors of [36] implement this kind of adaptation mechanism by subtracting a presynaptic trace P from the input current
#=Σ
W
ck +
Σ W S
Po.
(10)
P P1 Σ s
(11)
R
JER.
where P)o.1) are (learnable) addition and decay constants, and R² is the set of receptive fields of neuron é over all channels (i.e., the second term in Eq. 11 is an average pooling averaged over all channels). Adaptation based on presynaptic instead of postsynaptic activity minimizes adaptation delay, making it especially suited to the fast-changing nature of event data [36]. In this spirit, we also propose the XLIF model, crossover between ALIF and PLIF, which adapts its threshold based on presynaptic activity:
0: = 8o + P.P.
(12)
As surrogate gradient for the spiking function o, we opt for the derivative of the inverse tangent d'xx) = aan'' = 1/(1 + ?x² [14] because it is computationally cheap, with y being the surrogate width and x U - 0. In order to ensure gradient flow (hence learning) in the absence of neuron firing, the width should be sufficient to cover at least range of subthreshold membrane potentials, while the height should be properly scaled (i.e., S 1) for stable learning [51]. Exact shape is of less importance for final accuracy. Further details on all hyperparameters can be found in Appendix G.
5

---

EV-FlowNet
ML
Conv
91
R1
R2
P:
ConvRecurr
Residual block
FireNet
Upsampling
91
8/
Eg
&
&
P
Me
Strided Conv + ConvRecurr
Schematic of the base neural networks used in this work. All evaluated variants inherit from these architectures and only vary the neuron model and/or the convolutional recurrent layer.
Figure 2:
3.4 Network architectures
We evaluate the two trends neural network design for event cameras through (spiking) recurrent variants of EV-FlowNet [54] (encoder-decoder) and FireNet [40] (lightweight, no downsampling). An overview of the evaluated architectures can be found in Fig. 2. The use of explicit recurrent connections in our ANNs and SNNs is justified through the ablation study in Appendix D.
The base architecture referred to as EV-FlowNet is recurrent version of the network proposed in [54]. Once represented in Section 3.1. the input event partition passed through four recurrent encoders performing strided convolution followed by ConvGRU [3] with output channels doubling after each encoder (starting from 32), two residual blocks [13, 20]. and four decoder layers that perform bilinear upsampling followed by convolution. After each decoder, there is (concatenated) skip connection from corresponding encoder, as well as a depthwise (i.e., x 1) convolution to produce lower scale flow estimate, which is then concatenated with the activations of the previous decoder. The Laow loss (see Eq. 5) is applied to each intermediate optical flow estimate via upsampling. All layers use 3 x 3 kernels and ReLU activations except for the prediction layers, which use TanH activations.
The FireNet architecture Fig. 2 is an adaptation of the lightweight network proposed in [40], which was originally designed for event-based image reconstruction. However, as shown in [35], this architecture is also suitable for fast optical flow estimation. The base architecture consists of five encoder layers that perform Lingle-strieed convolution, two ConvGRUs, and a final prediction layer that performs depthwise convolution. All layers have output channels and use 3 kernels and ReLU activations except for the final layer, which uses TanH activation.
Based on these architectures, have designed several variants: (i) RNN-EV-FlowNet and RNN- FireNet, which use vanilla ConvRNNs (see Appendix B) instead of ConvGRUs; (ii) Leaky-EV- FlowNet and Leaky-FireNet, which use ConvRNNs and whose neurons are stateful cells with leaks (for a more direct comparison with the SNNs, see Appendix B); and (iii) SNN-EV-FlowNe and SNN-FireNet (with SNN being LIF, ALIF, PLIF XLIF), the SNN variants that use ConvRNNs and whose neurons are spiking and stateful according to the neuron models in Section 3.3.
The prediction layers of all SNN variants are kept real-valued with TanH activation, acting learned decoder from binary spikes to dense optical flow estimate. The first layer of the SNNs can likewise be viewed as learned spike encoder, receiving integer event counts and emitting spikes.
Experiments
To highlight the robustness of our SSL pipeline, we train our networks on the indoor forward-facing sequences from the UZH-FPV Drone Racing Dataset [12], which is characterized by much wider distribution of optical flow vectors than the datasets that use for evaluation, i.c. MVSEC [54], High Quality Frames (HQF) [44], and the Event-Camen Dataset (ECD) [31]. The selected training sequences consist of approximately 15 minutes of event data that we split into 140 128 128 (randomly cropped) sequences with 500k events each. We further augment this data using random horizontal, vertical, and polarity flips.
Our framework is implemented in PyTorch. We use the Adam optimizer [24] and a learning rate of 0.0002, and train with batch size of 8 for 100 epochs. We clip gradients based on global norm of 100. We fix the number of events for each input partition N 1k, while we use 10k events for each training event partition. This is equivalent to K 10 forward passes per backward pass (i.c.,

---

indoor_ Flying
indoor, flyingg
Ground truth
EV-FlowNet
UFFEEVFlowNet
FireNet
PLIF-FireNet
Figure 3: Qualitative evaluation of our best performing ANNs and SNNs on sequences from the MVSEC dataset [55]. The optical flow color-coding scheme can be found in Appendix C.
the network's s unrolling), as described in Section 3.2 and illustrated in Fig. 1. Lastly, we empirically set the scaling weight for Lumooth À 0.001.
We evaluated our architectures on the MVSEC dataset [55] with the ground-trun truth optical flow data provided by Zhu et al. in [54], which was generated at each APS frame timestamp. and scaled to be the displacement for the duration of one (dt = 1) and four (dt = 4) APS frames. Optical flow predictions were also generated at each frame timestamp by using all the events in the time window as input for dt = 1, or 25% of the window events at a time for dt = 4 (due to the larger displacements). For comparison against the ground truth, the predicted optical flow is converted from units of pixels/partition to units of pixel displacement by multiplying it with dig//diipput We compare our recurrent ANNs and SNNs against the state-of-the-ee
self-supervised event-based optical flow estimation: the original (non-recurrent) EV-FlowNet [54] trained with either photometric error as in [54] or contrast maximization [56]. and the hybrid SNN-ANN network from [26]. Quantitative results of this evaluation are presented in Table 1. We report the average endpoint eтrоr (AEE) and the percentage of points with AEE greater than pixels and 5% of the magnitude of the optical flow vector, denoted by Koullier over pixels with valid ground-truth data and at least one input event. Qualitative results of our best performing networks on this dataset are shown Fig.
For the sake of completeness, as in [35, 44] we also evaluate our architectures on the ECD [31] and HQF [44] datasets. The details and results of this evaluation can be found in Appendix C. Due the lack of ground-truth data in these datasets, we assess the quality of the estimated optical flow based on metries derived from the contrast maximization framework [15, 16].
Regarding the SNN variants, the results in Table 1, Fig. 3 and Appendix C correspond to networks whose neuronal parameters (i.c., leaks, thresholds, adaptive mechanisms) were also optimized when applicable. See Appendix for an ablation study on the learnable parameters; further details regarding parameter settings and initialization given in Appendix G.
4.1 Evaluation of the ANN and SNN architectures
Firstly, the quantitative results in Table confirm the validity of the proposed SSL framework for event-based optical flow estimation with recurrent networks. As shown, our base architectures EV- FlowNet and FireNet perform on par with the current state-of-the-art, even though these non-recurrent networks from literature encode explicit temporal information in their input event representations, and were trained on other very similar sequences from MVSEC [55] to prevent the input statistics from deviating from the training distribution during inference [26. 54, 56]. Since we train on very different dataset [12], this on-par performance also confirm the generalizability of our ANNs and SNNs distinctly different scenes and distributions of optical flow vectors. This claim is further supported by qualitative results in Fig. and additional results in Appendix C.
Secondly, from the comparison between our base ANN architectures and their spiking counterparts without adaptation mechanisms (i.e., L.F-EEVFllwNNet and Lire-FirNeet, we can conclude that, although there is general increase in the AEE and the percentage of outliers when going spiking,
7

---

Table 1: Quantitative evaluation MVSEC [55]. For each sequence, report the AEE (lower is
better, 4) in pixels and the percentage of outliers, Vomlier (4). Best bold, runner up underlined.
outdoor dayl
indoor flyingl
indoor
flying2
indoor flying3
dt
AEE
%ooulier
AEE
AEE
Aouller
AEE
%oOulier
EV-PlowNet" [54]
0,49
0.20
1.03
2.20
1.72
15.10
1.53
11.90
EV-FlowNet" [56]
0.32
0.00
0.58
0.00
1.02
4.00
0.87
3.00
Eeloon-E--FloWeet [26]
0.49
0.84
1.28
1.11
EV-FlowNet
0.47
0.25
0.60
0.51
1.17
8.06
0.93
5.64
RNN-EV-FlowNel
0.56
1.09
0.62
0.97
1.20
8.82
0.93
5.51
Leaky-EV-FllwNNe.
0.53
0.28
0.60
11.37
LIF-EV-FlowNet
0,53
0,33
0.71
1.41
1.44
12.75
1.16
9.11
ALIF-EV-Flowwel
0.57
0.42
1.00
2.46
1.78
17.69
1.55
15.24
PLIF-EV-FlowNel
0.60
0.52
0.75
0.85
1.52
13.38
1.23
9.48
Xl.I--EV-llowNe
0.45
0.16
0.73
0.92
1.45
12.18
1.17
8.35
FireNet
0.55
0.35
0.89
1,93
1.62
14.65
1.35
T0.64
RNN-FireNet
0.62
0.52
0.96
2.60
1.77
17.55
1.48
13.60
Leaky-Fire
0.52
0.41
0.90
2.66
16.09
1.43
13.16
LIF-PireNet
0.57
0.40
0.98
2.48
1.77
16.40
1.50
12.81
ALIF-FireNet
0.62
0.45
1.04
3.02
1.85
18.88
1.58
15,00
PLIF-FireNet
0.56
0.38
0.90
1.93
1.67
14.47
1.41
11.17
XIIF-FireNet
0.54
0.34
0.98
2.75
1.82
18.19
1.54
14.57
dt
EV-FlowNet" [54]
1.23
7.30
2.25
24.70
4.05
45.30
3.45
39.70
EV-FlowNet" [56]
1.30
9.70
2.18
24.20
3.85
46.80
3.18
47.80
Hybrid-EV-FlowNeet [26]
1.09
2.24
3.83
3.18
EV-FlowNei
I.69
12.50
2.16
21.51
3.90
40.72
3,00
29.60
RNN-EV-FlowNet
1.91
16.39
22.10
4.01
41.74
3.07
30.87
Leaky--VV-FlwNet
1.99
17.86
2.59
30.71
4.94
54.74
3.84
42.33
LIF-EV-FlowNet
2.02
18.91
2.63
29.55
4.93
$1.10
3,88
41.49
ALIF-EV-FlowNet
2.13
20.96
3.81
50.36
6.40
66.03
5.53
61.07
PL.--EV-HoowNet
2.24
23.76
2.80
34.34
5.21
52.98
4.12
45.31
XLIF-EV-FlowNet
1.67
12.69
2.72
31.69
51.36
3.91
42.52
FireNet
2.04
20.93
3.35
42.50
5.71
61.03
4.68
53.42
RNN-FireNet
2.35
24.31
3.64
46.54
6.33
63.89
5.20
Leaky-FireNet
1.96
18.26
3.42
42.03
5.92
58.80
4.98
52.57
LIF-FireNet
2.12
21.00
3.72
48.27
6.27
64.16
5.23
58.43
ALIF-FireNet
2.36
25.82
3.94
52.35
6.65
67.61
5.60
61.93
PLIF-FireNet
2.11
20.64
3.44
44.02
5.94
64.02
4.98
57.53
XLIF-FireNet
2.07
18.83
47.89
6.51
67.25
5,43
60.59
Non-recurrent ANNs with input event representations encoding spatiotemporal information, as described [26, 54, 56].
the proposed SNNs are still able to produce high quality event-based optical flow estimates. In fact, according Table 1, the main drop in accuracy does not come from the incorporation of the spiking function (and the selection of a'an' as surrogate gradient), but mainly from the use of vanilla convolutional recurrent layers instead of gated recurrent units. As shown, our spiking LIF architectures perform very close to their RNN and leaky counterparts, despite the latter being ANNs. This highlights the important need for more powerful convolutional recurrent units for SNNs, similar to ConvLSTMs [47] and ConvGRUs [3] for ANNs, as this would narrow the performance gap between these two processing modalities according to our observations. Interestingly, a previous comparison of the performance of recurrent ANNs and SNNs for event-based classification [21] suggested similar improvements to SNN units.
4.2 Impact of adaptive mechanisms for spiking neurons
Table and Appendix C also allow us draw conclusions about the effectiveness of the adaptive mechanisms for spiking neurons introduced Section 3.3. For both EV-FlowNet and FireNet, we observe that threshold adaptation based postsynaptic activity (i.c., the ALIF model) performs

---

worse compared to other models. The loss curves in Appendix F support this observation. While the ALIF model was shown to be effective for learning long temporal dependencies from relatively low-dimensional data as in [4, 5, 49], the adaptation delay introduced by relying on postsynaptic signal seems detrimental when working with fast-changing. high-dimensional event data. This is in line with suggestions by Paredes- Vallés et al. in [36], who use presynaptic adaptation for this reason. Our own results with presynaptic adaptation (i.e., PLIF and XLIF models) are somewhat inconclusive. While PLIF performs better in the case of FireNet, this is not the case for EV-FlowNet On the other hand, XLIF's performance is very similar to the LIF model for both FireNet and EV-FlowNet architectures. Based on these observations, we think that adaptivity based on presynaptic activity should be considered for further development. In this regard, the XLIF model has the advantage that it is able to generate activity (leading to gradient flow, and thus learning) even for very small inputs, whereas PLIF is incapable of this for a given threshold (because P is always positive). A more detailed comparison of activity levels for the different variants is given in Appendix I, along with an approximation of the energy efficiency gains of SNNs compared to ANNs.
4.3 Further lessons on training deep SNNs
Multiple problems arise when training deep SNNs for a regression task that involves sparse data, as is done here. Regarding learning, we find that gradient vanishing poses the main issue. Even considering dense inputs/loss and a shallow (in timesteps or in layers) SNN, sufficient gradient flow is a result of wide enough (in our case, covering at least J - 0 < f) and properly scaled (i.e., S 1) surrogate gradients [25, 49, 51], and parameter initializations that lead to non-negligible amounts of spiking activity [51]. Sparse data and deep networks make finding the proper settings more difficult, and for this reason, we have tried to increase the robustness of various of these hyperparameter settings. First, we looked at the learning performance and gradient flow of networks with various surrogate gradient shapes and widths. Compared to the a'Ta'' surrogate specified in Section 3.3, SuperSpike [50] with y 10 and y 100 (both narrower) show little learning due to negligible gradient flow (see Appendix H for more details). One way of reducing the effeet of too narrow surrogate gradient would be to trigger spiking activity through regularization terms in the loss function, as done in, e.g.. [5, 51]. These form a direct connection between loss and the neuron in question, bypassing most of the gradient vanishing that would happen in later layers. We tried the variant proposed in [51], which is aimed at achieving at least a certain fraction of neurons to be active at any given time. With this fraction to 5%, we saw that for SuperSpike with 7 = 10 there was some learning happening, while for = 100 there was no effect. Plots of the loss curves and gradient magnitudes are available in Appendix H. Of course, more research into these and other regularization methods is necessary. Alternatively, as done in [25], batch normalization (or other presynaptic normalization mechanisms) could be used to ensure proper activity and gradient flow.
Regarding the network output, there seems to be an intuitive gap between classification and regression tasks, with the latter requiring higher resolution be solved successfully. In our view, there two aspects to this that might pose an issue to SNNs. First, given the single prediction layer that the here-presented SNNs have go from binary real-valued activations, one could expect loss in output resolution compared to equivalent ANN architectures. Second, even for moderate activity levels, the outputs of spiking layer can be much larger in magnitude than an equivalent ANN layer, even for comparable parameter initializations. Intuitive solutions to these shortcomings are (i) increase the number of channels to increase the resolution, and (ii) to initialize the weights of the non-spiking prediction layer as to have a smaller magnitude. While increasing the number of output channels in Es (LIF-FireNet, see Fig. 2) did not lead to significantly improved performance or learning speed, decreasing the initialization magnitude of the weights in layer P did. As the loss curves Appendix H show, the improved initialization leads to faster convergence and less variability across neuron models and the selection of learnable parameters.
5 Conclusion
In this article, we presented the first set of deep SNNs to successfully solve the real-world large- scale problem of event-based optical flow estimation. To achieve this, we first reformulated the state-of-the-art training pipeline for ANNs to considerably shorten the time windows presented to the networks, approximating the way in which SNNs would receive spikes directly from the event camera. Additionally, we reformulated the tfe self-super loss function to

---

improve its convexity. Prior to their training with this framework, we augmented several ANN architectures from literature with explicit and/or implicit recurrency. besides the addition of the spiking behavior. Extensive quantitative and qualitative evaluations were conducted on multiple datasets. Results confirm not only the validity of our training pipeline, but also the on-par performance of the proposed set of recurrent ANNs and SNNs with the self-supervvsed state-of-the-art To the best of our knowledge, and especially due to the addition of explicit recurrent connections, the proposed SNNs correspond to the most complex spiking networks in the computer vision literature, architecturally speaking. For the SNNs, we also conducted several additional studies and (i) concluded that parameter initialization and the width of the surrogate gradient have significant impact learning: smaller weights in the prediction layer speed up convergence, while too narrow surrogate gradient prevents learning altogether; and (ii) observed that adaptive mechanisms based presynaptic activity outperform those based on postsynaptic activity, and perform similarly or hetter than the baseline without adaptation. Overall, believe this article sets the groundwork for future research on neuromorphic processing for not only the wvent-hased structure-froms from-motion problem, but also for other, similarly complex computer vision applications. For example, results suggest the need for more powerful recurrent units for SNNs. On another note, future work should also focus on the implementation of these deep SNNs on neuromorphic hardware, as is there where these architectures excel due to the power efficiency that their sparse and asynchronous nature brings.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their constructive feedback and suggestions, and Kirk Y. W. Scheper for his insights self-supervised learning of event-based optical flow. This work was supported by funding from NWO (grants 199...99.992999 and 612.001.701).
References
[1] Mohammed Almatrafi, Raymond Baldwin, Kiyoharu Aizawa, and Keigo Hirakawa. Distance surface for event-based optical flow. IEEE Trans. Pattern Anal. and Mach. Intell., 42(7))1547-1558) 2020.
[2] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garrea,, Marcela Meudoza, et al. A low power. fully event-based gesture recognition system. In IEEE Conf. Comput. Vis. Pattern Recog, pages 7243-7252. 2017.
[3] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. Int. Conf. Learn. Representations, 2015.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short- term memory and learning-to-leant in networks of spiking neurons. Advances Neural Information Process. Syst. volume 31, pages 787-797, 2018.
[5] Guillaume Bellec, Franz Scherr, Anand Subramoney, Flias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Commun., 11():33225, 2020. 2020.
[6] Ryad Benosman, Charles Clereq, Xavier Lagorce, Sio-Hoi leng, and Chiara Bartolozzi, Event- banttbased visual flow, IEEE Trans. Neural Netw Learn. Syst., 25(2):407 417, 2013.
[7] Christian Brandli, Raphacl Berner, Minhao Yang, Shih-Chii Liu, and Tohi Delbruck. A 240 180 130 dB 3ys latency global shutter spatiotemporal vision sensor, IEEE Solid-State Circuits, 49110)1333,, 2014.
[8] Tobias Brosch, Stephan Tschechne, and Heiko Neumann. On event- based optical flow detection, Frontiers in Neuroscience, 9:137, 2015.
[9] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half- quadratic regularization algorithms for computed imaging. In IEEE Int. Conf. Image Procexs. volume 2, pages 168-172, 1994.
[10] Loic Cordone, Benoît Miramond, and Sonia Ferrante. Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks. In Int. Joint Conf. Neural Networks (LJCNN), pages 1-8, July 2021.
10

---

[11] Mike Davies, Andreas Wild, Garrick Orchard. Yulia Sandamirskaya, Gabriel A. Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R. Risbud. Advancing neuromorphic computing with Loihi: A survey of results and outlook. IEEE Proc. pages 1-24, 2021.
[12] Jeffrey Delmerico, Titus Cieslewski. Henri Rebecq, Matthias Faessler, and Davide Scaramuzza. Are we ready for autonomous drone racing? The UZH-FPV drone racing dataset. In IEEE hnt. Conf. Robot, Autom., pages 67 13-6719, 2019.
[13] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep Residual Learning in Spiking Neural Networks. arXiv:2102.0413 04919 [cs]. October 2021.
[14] Wei Fang, Zhaofei Yu, Yangi Chen, Timothee Masquelier, Tiejun Huang. and Yonghong Tian. Incorporating Learnable Membrane Time Constant To Enhance Learning of Spiking Neural Networks. In JEEF/CVF Int. Conf. Compur. Vis, pages 2661-2671, 2021.
[15] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. unifying contrast maximization framework event cameras, with applications to motion, depth, and optical flow estimation. IEEE Conf. Comput. Vis. Pattern Recog., pages 3867 3876, 2018.
[16] Guillermo Gallego, Mathias Gehrig, and Davide Scaramuzza, Focus is all you need: Loss functions for event-based vis. In IEEE Conf. Comput. Vis. Pattern Recog., pages 12280-12289, 2019.
[17] Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Danilidiis, et al. Event-based vision: A survey. IEEE Trans. Pottern Anal, and Mach. Intell., 2020.
[18] M. Gehrig. S. B. Shrestha, D. Mouritzen, and D. Scaramuzza. Event-based angular velocity regression with spiking networks. IEEE Int. Conf. Robot. Autom., pages 4195-42000 2020.
[19] Germain Haessig, Andrew Cassidy, Rodrigo Alvarez, Ryad Benosman, and Garrick Orchard. Spiking optical flow event-based sensors using ibm's 's truenorth neurosynaptic system. IEFE Trans. Biomed. Circuits Syst. 12(4):860-870, 2018.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Pattern Recog., pages 770-778 2016
[21] Weihua He, YuJie Wu, Lei Deng, Guoqi Li, Haoyu Wang, Yang Tian, Wei Ding, Wenhui Wang, and Yuan Xie. Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences. Neural Networks, 132: 108-120, 2020.
[22] Donald O. Hebb. The organization of behavior: A neuropsychological theory. Wiley, 1952.
[23] Mark Horowitz. 1.1 Computing's energy problem (and what we do about it). In IEEE Int. Solid-State Cire. Conf. Digest of Tech. Papers (ISSCC), pages 10-14, Fehruary 2014.
[24] Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. Int. Conf. Learn. Representations, 2014.
[25] Eimantas Ledinauskas, Julius Ruseckas, Alfonsas Juršenas, and Giedrius Buračas. Training deep spiking neural networks. arXiv:2006.04430 2020.
[26] Chankyu Lec, Adarsh Kumar Kosta, Alex Z. Zhu, Kenneth Chancy, Kostas Daniilidis, and Kaushik Roy. Spike-FlowNet: Event-based optical flow estimation with energy-efficient hybrid neural networks. In European Conf. Comput. Vis., pages 366-382, 2020.
[27] Chankyu Lee, Adarsh Kumar Kosta, and Kaushik Roy. estimation using sensor fusion and deep fused spiking- analog network architectures. 1rXiv:2103.10592 2021.
Energy-efficient optical flow
[28] Zhuoyan Li, Jiawei Shen, and Ruitao Liu. A lightweight network to leam optical flow from event data, In Int. Conf. on Pattern Recog-, pages 1-7, 2021.
[29] Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-lczza Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with scalable communication network and interface. Science, 345(6197):668-6)) 2014.
[30] Anton Mitrokhin, Comelia Fermüller, Chethan Parameshwara, and Yiannis Aloimonos. Event-basee moving object detection and tracking. In IEEE/RSJ Int. Conf. Intell. Robots and Syst., 2018
11

---

[31] Elias Mueggler, Henri Rebecq. Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event- camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. Int. J. Robot. Research,
2017.
[32] Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power gradient-based optimization spiking neural networks. IEEE Signal Process, Magazine, 36(6):51 2019.
[33] Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 9:437, 2015.
[34] Chethan M. Parameshwara, Simin Li, Comelia Fermüller, Nitin J. Sanket, Matthew S. Evanusa, and Yiannis Aloimonos. SpikeMS: Deep spiking neural network for motion segmentation. arXi:22005.0666, 2021.
[35] Federico Paredes- Paredes-Vallés and Guido C. E. de Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In IEEE Conf. Comput. Vis. Pattern Recog,, 2021.
[36] Federico Paredes- Parédes-Vallés, Kirk Y. W. Scheper, and Guido C. H. E. De Croon. Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: From events to global motion perception. IEEE Trans. Pattern Anal. and Mach. Intell., +2(8):055122066 2020.
[37] Nicolas Perez-Nieves and Dan F. M. Goodman. Sparse Spiking Gradient Descent. arXiv:2105.08810. 2021.
[38] Nicolas Perez-Nieves, Vincent C. H. Leung, Pier Luigi Dragotti, and Dan F. M. Goodman. Neural heterogeneity promotes robust learning. Nature Commun., 12(1))5791, October 2021.
[39] Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza. ESIM: An open event camera simulator. In Conf. Robot Learn., pages 969 982, 2018
[40] Cedric Scheerlinck, Henri Rebecq. Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza. Fast image reconstruction with an event camera. In IEEE Winter Conf. Appl. Comput. Vis., pages 156-163, 2020.
[41] Sumit B. Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Process. Syst., 2018.
[42] Christoph Stöckl and Wolfgang Maass, Optimized spiking neurons can classity images with high accuracy through temporal coding with two spikes. Nature Mach. Intell., pages 1-9, 2021.
[43] Timo Stoffregen and Lindsay Kleeman. Event cameras, contrast maximization and reward functions: An analysis. In JEEE Conf. Comput. Vis. Pattern Recog., pages 12300-12308 2019.
[44] Timo Stoffregen, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Klee- man, and Robert Mahony. Reducing the sim-to-real gap for event cameras. In European Conf. Comput. Vis., 2020.
[45] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers Neuroscience, 12, 2018
[46] Yannan Xing, Gactano Di Caterina, and John Soraghan. new spiking convolutional recurrent neural network (SCRNN) with applications to event-based hand gesture recognition. Fruntiers Neuroscience. 14, 2020.
[47] Shi Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-Chun Woo. Con- volutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in Newral Information Process. Syst. pages 802-810 2015.
[48] C. Ye, A. Mitrokhin, C. Fermüller, J. A. Yorke, and Y. Aloimonos. Unsupervised Learning of Dense Optical Flow, Depth and Egomotion with Event-Based Sensors. In IEEE/RS] Int. Conf. Intell. Robots and Syst., pages 5831-5838, October 2020.
[49] Bojian Yia. Federico Corradi, and Sander M. Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Mach. Intell., 3(10)905-99 October 2021.
[50] Friedemann Zenke and Surya Ganguli. SuperSpike: Supervised learning in multilayer spiking neural networks. Neural Computation, 1046):111-1541, 2018
12

---

[51] Friedemann Zeake and Tim P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, pages 1-27, 2021.
[52] Friedemann Zenke, Sander M. Bohté, Claudia Clopath, Tulia M. Comşa, Julian Göltz, Wolfgang Maass, Timothée Masquelier, Richard Naud, Emre O. Neftci, Mihai A. Petrovici. Franz Scherr. and Dan M. Goodman. Visualizing a joint futtrre neuroscience and neuromorphic engineering, Neuron, 109(4):
571-575. 2021.
[53] Hanle Zheng, Yujie Wu, Deng. Yifan Hu, and Guoqi Going Deeper With Directly-Trained Trained Larger Spiking Neural Networks. Proc. AAAI Conf. on Artificial Intell., 35(12): 1062 11070, May 2021.
[54] Alex Zhu and Liangzhe Yuan. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In Robot.: Science and Syst., 2018.
[55] Alex Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Danillidis. The multivehicle stereo event camera dataset: An event camera dataset for 3D perception. IEEE Robot. and Autom. Lett. 333)20032 2039, 2018.
[56] Alex Z. Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In IEEE Conf. Comput. Vis. Pattern Recog., pages 989 989-997, 2019.
13


Reviews:
[{"review_id": "bgZoF0gTbke", "reviewer": "Reviewer_JxQc", "summary": "The paper investigates optical-flow estimation from event-cameras, and in particular provides the first deep SNN solution for real-world applications of this concept. The approach filters event streams from the input into an event-representation where every partition has the same number of events, which is then fed to the SNN. A backward pass is also possible using a contrast maximization loss. A new self-supervised learning framework is introduced.  The results show the new method to be on par with the state-of-the-art.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present the idea behind their approach, as well as the implementation details and experimental evaluation in great detail. As someone who is not fully into the related work on event-based optical flow, I found the argumentation hard to follow, but greater experts in the domain will understand this. Overall I have the impression that this advances the field of optical flow for event-based cameras, and the descriptions are fairly complete. As a suggestion I would therefore start with a shortened, more high-level explanation of the main ideas behind the approach, before going into more details in section 3.\nThe paper explores various variants of the approach, e.g. different network variants.\n\nOverall I would support accepting this paper, if other more qualified reviewers agree. I my self have a hard time seeing the additional benefits over existing event-based optical flow methods. The limitations compared to standard optical flow should be more addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address limitations in their experimental evaluation (section 4.1)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "VN1Z3GlGmRw", "reviewer": "Reviewer_Dkce", "summary": "The paper transforms two models for optical flow prediction into spiking neural networks (SNNs) with event-based inputs and an adapted loss function. Many of the different SNN variants largely perform relatively on par with their rate-based counterparts.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* applying spiking neural networks in more challenging tasks is a timely question\n* the proposed models seem to perform well on a real-world dataset\n\nCriticisms:\n* lack of demonstrated improvements: As the paper points out, the premise of neuromorphic computing is more energy- or bandwidth-efficient processing. While this paper shows successful training of SNNs on a challenging task while ~maintaining performance relative to rate-based models, it falls short of demonstrating any of these improvements. I realize actually running experiments on hardware (e.g. Intel Loihi) is challenging, but approximating variables of interest is certainly doable: for instance, the required bandwidth or energy usage of each model could be approximated based on the number of events. For this work in particular, since the focus is on training, that aspect should be particularly emphasized. Otherwise, the current framing sounds more like building more complex models for the sake of it but lacks demonstrations of the motivated benefits.\n* unclear presentation of specific model proposal: The analyses as well as the text outline many different variants of SNNs (RNN, Leaky, LIF, ALIF, PLIF, XLIF) on top of the two base architectures (EV-FlowNet, FireNet). Does one of these models have desirable properties over the other? The manuscript is quite hard to follow in which of the proposed models the reader should focus on. The ablation studies are unclear to me in that regard as well where stateless ANNs (\"FireFlowNet\") are \"not capable of learning\" with a 1.02 AEE compared to LIF-FireFlowNet's 0.84 AEE.\n\nIf the authors can make the specific model proposal and its benefits more (quantitatively) clear, I am willing to increase my score.\n\nMinor:\n* the AEE metric should be described in more detail. What are the units? What does a score of 1 mean? How much worse is a score of 1 compared to a score of 0.5?\n* why do the reproduced numbers differ so much from the ones reported in literature? E.g. Table 1, dt=4, line 1 lists EV-FlowNet[50] with 1.23 AEE and the reproduced EV-FlowNet with 1.69 AEE. If these kinds of deviations are expected, none of the proposed models as well as the models deemed \"not capable to learn\" in supplemental Table 3 can be distinguished from one another.\n* using the EV-FlowNet identifier for two models (from citations 50 and 52) is confusing, the differences between models should be emphasized\n* typos: line 47 is missing the \"a\" in front of task; in line 134 \"pixels locations\" should be \"pixel locations\"\n\n\nEDIT after rebuttal: I appreciate the author's arguments and have increased my score, yet still find the presentation of claims and supporting analyses supporting (see full comment).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of limitations is lacking, see above.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-5g-WId2s_j", "reviewer": "Reviewer_1JoW", "summary": "This paper modifies the basic network architecture of two mainstream self-supervised optical flow estimation methods from events in order to create a Spiking NN version of them. The paper compares several neuron models, loss functions, and gradient computation methods for training. Performance is comparable to classic CNN methods.\n\n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nA really thorough paper proposing a novel SNN for self-supervised flow computation with attention to detail in implementation and exhaustive combinations of all possible neuron models and loss functions. This paper should serve as an example par excellence for experimental papers even beyond the neuromorphic community. The paper is a delight to read.\n\nMoreover it is one of the very few papers with a full Spiking Network solution for a computer vision problem (optical flow estimation). \n\nA SNN does not keep as input the timestamp of events. If events are arriving to the network according to their timestamps the neuron itself can encode implicitly the time of arrival. At the output, two self-supervised contrast functions are tested with a modification of the former to become convex w.r.t the flow.\nLIF neurons are used for convolutions and two mainstream methods, EVflownet and Firenet, are first converted to recurrent methods and then to their SNN counterparts.\n\n\nExperiments show that both SNN versions of EVflownet and Firenet are competitive with their ANNs prototypes when tested on standard benchmarks like MVSEC.\nThe comparison of all possible combinations/ablations in Table 1 is amazing!\nThe authors observe that the adaptive LIF neuron model seems detrimental for performance. \nThe paper identifies correctly the vanishing of gradients as a the main challenge in SNNs and proposes several heuristics. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Thoroughly presented.", "review_text": "", "extra_scores": {"ethical_concerns": "none.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear authors, 

congratulations on your paper being accepted at Neurips. The reviewers appreciated the demonstration that SNN can achieve impressive performance on a challenging and 'dense' prediction task. Please incorporate the feedback by the reviewers into the final version of the manuscript. In addition, it will be important to make the code and results publicly available, as promised in the manuscript. Your AC
Decision: Accept (Poster)