Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper modifies the basic network architecture of two mainstream self-supervised optical flow estimation methods from events in order to create a Spiking NN version of them. The paper compares several neuron models, loss functions, and gradient computation methods for training. Performance is comparable to classic CNN methods.


main_review: 
A really thorough paper proposing a novel SNN for self-supervised flow computation with attention to detail in implementation and exhaustive combinations of all possible neuron models and loss functions. This paper should serve as an example par excellence for experimental papers even beyond the neuromorphic community. The paper is a delight to read.

Moreover it is one of the very few papers with a full Spiking Network solution for a computer vision problem (optical flow estimation). 

A SNN does not keep as input the timestamp of events. If events are arriving to the network according to their timestamps the neuron itself can encode implicitly the time of arrival. At the output, two self-supervised contrast functions are tested with a modification of the former to become convex w.r.t the flow.
LIF neurons are used for convolutions and two mainstream methods, EVflownet and Firenet, are first converted to recurrent methods and then to their SNN counterparts.


Experiments show that both SNN versions of EVflownet and Firenet are competitive with their ANNs prototypes when tested on standard benchmarks like MVSEC.
The comparison of all possible combinations/ablations in Table 1 is amazing!
The authors observe that the adaptive LIF neuron model seems detrimental for performance. 
The paper identifies correctly the vanishing of gradients as a the main challenge in SNNs and proposes several heuristics. 
limitations_and_societal_impact: Thoroughly presented.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper transforms two models for optical flow prediction into spiking neural networks (SNNs) with event-based inputs and an adapted loss function. Many of the different SNN variants largely perform relatively on par with their rate-based counterparts.
main_review: Strengths:
* applying spiking neural networks in more challenging tasks is a timely question
* the proposed models seem to perform well on a real-world dataset

Criticisms:
* lack of demonstrated improvements: As the paper points out, the premise of neuromorphic computing is more energy- or bandwidth-efficient processing. While this paper shows successful training of SNNs on a challenging task while ~maintaining performance relative to rate-based models, it falls short of demonstrating any of these improvements. I realize actually running experiments on hardware (e.g. Intel Loihi) is challenging, but approximating variables of interest is certainly doable: for instance, the required bandwidth or energy usage of each model could be approximated based on the number of events. For this work in particular, since the focus is on training, that aspect should be particularly emphasized. Otherwise, the current framing sounds more like building more complex models for the sake of it but lacks demonstrations of the motivated benefits.
* unclear presentation of specific model proposal: The analyses as well as the text outline many different variants of SNNs (RNN, Leaky, LIF, ALIF, PLIF, XLIF) on top of the two base architectures (EV-FlowNet, FireNet). Does one of these models have desirable properties over the other? The manuscript is quite hard to follow in which of the proposed models the reader should focus on. The ablation studies are unclear to me in that regard as well where stateless ANNs ("FireFlowNet") are "not capable of learning" with a 1.02 AEE compared to LIF-FireFlowNet's 0.84 AEE.

If the authors can make the specific model proposal and its benefits more (quantitatively) clear, I am willing to increase my score.

Minor:
* the AEE metric should be described in more detail. What are the units? What does a score of 1 mean? How much worse is a score of 1 compared to a score of 0.5?
* why do the reproduced numbers differ so much from the ones reported in literature? E.g. Table 1, dt=4, line 1 lists EV-FlowNet[50] with 1.23 AEE and the reproduced EV-FlowNet with 1.69 AEE. If these kinds of deviations are expected, none of the proposed models as well as the models deemed "not capable to learn" in supplemental Table 3 can be distinguished from one another.
* using the EV-FlowNet identifier for two models (from citations 50 and 52) is confusing, the differences between models should be emphasized
* typos: line 47 is missing the "a" in front of task; in line 134 "pixels locations" should be "pixel locations"


EDIT after rebuttal: I appreciate the author's arguments and have increased my score, yet still find the presentation of claims and supporting analyses supporting (see full comment).
limitations_and_societal_impact: The discussion of limitations is lacking, see above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper investigates optical-flow estimation from event-cameras, and in particular provides the first deep SNN solution for real-world applications of this concept. The approach filters event streams from the input into an event-representation where every partition has the same number of events, which is then fed to the SNN. A backward pass is also possible using a contrast maximization loss. A new self-supervised learning framework is introduced.  The results show the new method to be on par with the state-of-the-art.
main_review: The authors present the idea behind their approach, as well as the implementation details and experimental evaluation in great detail. As someone who is not fully into the related work on event-based optical flow, I found the argumentation hard to follow, but greater experts in the domain will understand this. Overall I have the impression that this advances the field of optical flow for event-based cameras, and the descriptions are fairly complete. As a suggestion I would therefore start with a shortened, more high-level explanation of the main ideas behind the approach, before going into more details in section 3.
The paper explores various variants of the approach, e.g. different network variants.

Overall I would support accepting this paper, if other more qualified reviewers agree. I my self have a hard time seeing the additional benefits over existing event-based optical flow methods. The limitations compared to standard optical flow should be more addressed.
limitations_and_societal_impact: The authors address limitations in their experimental evaluation (section 4.1)
review_title: null

Meta-review: Dear authors, 

congratulations on your paper being accepted at Neurips. The reviewers appreciated the demonstration that SNN can achieve impressive performance on a challenging and 'dense' prediction task. Please incorporate the feedback by the reviewers into the final version of the manuscript. In addition, it will be important to make the code and results publicly available, as promised in the manuscript. Your AC
Decision: Accept (Poster)