Learning Optical Flow from Continuous Spike Streams
Authors: Rui Zhao
Ruiqin Xiong
Jing Zhao
Zhaofei Yu
Xiaopeng Fan
Tiejun Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Optical Flow
Neuromorphic Camera
Computer Vision

Abstract: Spike camera is an emerging bio-inspired vision sensor with ultra-high temporal resolution. It records scenes by accumulating photons and outputting continuous binary spike streams. Optical flow is a key task for spike cameras and their applications. A previous attempt has been made for spike-based optical flow. However, the previous work only focuses on motion between two moments, and it uses graphics-based data for training, whose generalization is limited. In this paper, we propose a tailored network,  Spike2Flow that extracts information from binary spikes with temporal-spatial representation based on the differential of spike firing time and spatial information aggregation. The network utilizes continuous motion clues through joint correlation decoding. Besides, a new dataset with real-world scenes is proposed for better generalization. Experimental results show that our approach achieves state-of-the-art performance on existing synthetic datasets and real data captured by spike cameras. The source code and dataset are available at \url{https://github.com/ruizhao26/Spike2Flow}.

Learning Optical Flow from Continuous Spike Streams
Rui Zhao 1.2 Ruiqin Xiong 1.2 Jing Zhao Zhaofei 1.244 Xiaopeng Fan Tiejun Huang 1.2
National Engineering Research Center of Visual Technology (NERCVT), Peking University
Institute of Digital Media, School of Computer Science, Peking University
National Computer Network Emergency Response Technical Team
"Institute for Artificial Intelligence, Peking University School of Computer Science and Technology, Harbin Institute of Technology ruizhaodatu. pku. edu. cn, {rqxiong, yuzf12, t-jhaan] &pku. edu. cn, zhaojing@cert org.rr..oo. en, fxpehit. edu.cn
Abstract
Spike camera is an emerging bio-inspired vision sensor with ultra-high temporal resolution. It records scenes by accumulating photons and outputting continuous binary spike streams. Optical flow is a key task for spike cameras and their applications. A previous attempt has been made for spike-based optical flow. However, the previous work only focuses on motion between two moments, and it uses graphics based data for training, whose generalization is limited. In this paper, we propose tailored network, Spike2Flow that extracts information from binary spikes with temporal-spann -patiaal representation based on the differential of spike firing time and spatial information aggregation. The network utilizes continuous motion clues through joint correlation decoding. Besides, a new dataset with real-world scenes proposed for better generalization. Experimental results show that our approach achieves state-of- -of-the-art performance on existing synthetic datasets and real data captured by spike cameras. The source code and dataset are available at https: //gi: thub. com htmmmmmiiiaa222/5kkk22220.0.
Introduction
Neuromorphic camera (NeurCam) is kind of emerging vision sensor [35; 43; 46; 18; 11; 19] inspired by retina. Using asynchronous sampling models, NeurCams can record the natural scene continuously by firing sparse spike streams with ultra-high temporal resolution. Besides, NeurCams have high dynamic range, low latency, and low energy consumption. Compared with traditional cameras with single-exposure imaging, NeurCams are more suitable for high-speed imaging in application scenarios such as unmanned acrial vehicles and chemical particle observation. One kind of NeurCams is the event camera [35; 43; 46: 18] inspired by the peripheral retina. It asynchronously fires spikes when the brightness change exceeds a certain threshold in the logarithmic domain. However, event cameras can hardly recover textures due to their differential sampling model, especially for static regions. Another kind of emerging NeurCam is the spike camera [11: 19] inspired by the fovea of retina. Spike cameras also record the scene by firing spikes asynchronously in each pixel, but different from event cameras that record the relative brightness changes, each pixel in spike cameras encodes the scene by accumulating photons and firing a spike once the accumulation exceeds certain threshold independently. With such an integral sampling model, spike cameras can record fine textures of the scene, which makes more appropriate for pixel-level tasks than event cameras. Since spike cameras
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

can record high-speed motion and recover the scenes well, they have shown great potential in tasks such as video reconstruction [73; 66; 69; 75; 67: 74], denoising [61], super-resolution [65; 59] and detection [19]. In these tasks, the relative motion between the spike camera and objects in the scene, i.e., the optical flow, fundamental and pivotal.
Optical flow estimation has been an important and challenging problem since it was proposed [16]. In recent years, there are a deal of developments in optical flow [12: 23; 52; 20; 21; 62; 56; 68; 57; 64]. A simple way to estimate optical flow for spike cameras is reconstructing images and use video-based methods. However, there are two problems. Firstly, simple reconstruction methods may introduce a lot of noise while complex methods have high computational costs [66: 69; 67: 65]. Secondly, simple reconstruction ways such as averaging the spike streams along the temporal axis would lose the precise temporal information in spike streams and introduce blur, which may mislead the flow estimation. Evvettbasee methods [71: 72; 32; 15; 14] also cannot have satisfactory performance for optical flow estimation for spike cameras due to the difference of data modalities. Thus, more efficient methods are needed for estimating optical flow from spike streams. Hu et al. [17] propose SCFlow as early exploration of this question. They design pyramidal network and two synthetic datasets based on graphics models to train and evaluate the network, respectively. However, the above-mentioned method is not robust since it omits several issues:
(1) Efficient representation of binary spikes. Each spike output by the spike camera represents not the status at the current moment but the result of the integral process of the spike. Besides, a single spike cannot express the information of the corresponding space-time time point. Using only convolution to extract features from the spike streams may not be efficient.
(2) Continuousness of spike streams. Previous work only considers motion between two single moments in spike streams, and it omits the continuousness information in the moving procedure.
(3) Reality of the datasets. Previous work use datasets synthesized by graphics models to train and evaluate the network. However, there is a huge gap between virtual scenes and the real world.
In this paper, we propose the Spike2Flow to estimate flow from continuous spike streams. We propose the differential of spike firing time (DSFT) to transform the binary spikes better represent the procedure of the integration for each spike. A spatial information aggregation (SIA) modulc proposed to aggregate a larger receptive field for each pixel with a self-attention mechanism. The DSFT and SIA form the temporal-spatial representation (TSR) for spike streams. Besides, we propose a joint correlation decoding (JCD) module to use continuous motion clues by simultaneously estimating series of flow fields. To train and evaluate the network in real scenes, based scenes in Slow Flow [25], we generate flow fields and spike streams to construct a dataset, i.e., real scenes with spike and flow (RSSF). The main contributions of this paper can be summarized as follows:
(1) A spike-based optical flow network. Spike2Flow, is proposed. The 2pike2Flow extracts features from binary spikes with comporal-spatio representation and jointly estimates a series of flow fields to utilize the continuousness of the moving procedure.
(2) A dataset for spike-based optical flow, real scenes with spike and flow (RSSF) is proposed. The scenes in RSSF are from the real world, improving the generalization of networks trained by RSSF.
(3) Experiments demonstrate that the Spike2Flow achieves state-of-the-art performance on RSSF, photo-realistic high-speed motion (PHM) dataset, and real data captured by spike cameras.
2 Related Work
Neuromorphic Cameras. Neuromorphic cameras (NeurCams) are kind of vision sensor that gets inspiration from the retina and works asynchronously in each pixel. Event cameras (including DVS [35], DAVIS [43], ATIS [46], CeleX [18] and et al.) and spike cameras [11: 19] are two types of mainstream NeurCams. Both the
cameras record the optical scene asynchronously in each pixel, which brings advantages for NeurCams compared with traditional cameras with a single-exposure imaging pattern, such as ultra- temporal resolution, high dynamic range, low latency, and low energy consumption. Event cameras employ a differential sampling model that only fires events when illuminance change in the logarithmic domain exceeds certain threshold, while spike cameras use an integral sampling model that accumulates photons and fires spikes when the accumulation exceeds a certain threshold. Thus, the spikes output by event cameras are more
2

---

sparse, but fine textures in the scene are lost especially in static regions. More details of the working mechanism of the spike camera can be found [17: 19; 67: 73; 69; 66].
Video-Based Optical Flow. Optical flow estimation aims to find dense pixel correspondences between Two moments in videos, which has lots of applications, such as video enhancement [57: 6], frame interpolation [26; 2; 33] and recognition [3; 55J. FlowNet [12] is the first end-to-end flow estimation network trained by the synthetic FlyingChairs dataset. Subsequent works [47; 23; 52; 20; 21; 53; 68] get inspiration from variational methods [4; 51; 49]. They introduce classical knowledge such as the pyramid, coarse-to-fie to-nine and cost volume to the networks. VCN [62] and DICL [56] design new approaches to build more robust correlation between frames. RAFT [54] builds a multi-scale all-pairs cost volume and performs recurrent refinement in a fixed resolution. RAFT has excellent performance and becomes the baseline of the subsequent works [28; 60; 27; 64]. The methods mentioned above are based on supervised learning. There are also some unsupervised optical flow estimation networks [41; 58; 37; 38; 36; 39; 29; 50]. Besides optical flow between two frames, there are also methods for flow among multi-frames. [24; 38; 36) use temporal context by jointly estimating flow from frame to its previous and future frames. [44; 48; 22] propagate the temporal information from the previous pair of images to the next pair by passing warped flow or latent states of the decoder.
Event-Based Optical Flow. There are also several works on event-based optical flow. EV-FlowNet [71] is the first end-to-end deep network, which is trained based on MVSEC [70] dataset. It is in an mecoer-ddcoder fashion and uses gray images from the active pixel sensor (APS) of event
cameras to construct the photometric loss. Zhu et al. [72] turn to use the average timestamp of warped events to construct the loss function, leaving the help of gray images. Spike-FlowNet [32] uses both spiking neural networks (SNNs) and analog neural networks (ANNs) to encode the events, achieving lower energy consumption. STEFlow [10] uses recurrent networks to encode the event stream and constructs correlation among features from events through time. Hagenaars et al. [15] estimate event- flow using deep networks composed fully of SNNs. Gehrig et al. [14] propose E-RAFT to implement all-pairs correlation and recurrent refinement in event-based flow based on a more complex autonomous driving dataset DSEC [13].
3 Approaches
3.1 Working Mechanism of Spike Cameras
The spike camera is composed of an array of pixels working asynchronously. Each pixel of a spike camera is composed of three main components: photon-receptor, integrator, and comparator. The integrator accumulates the photoelectrons from the photon-receptor and transfers them to the voltage. The comparator compares the accumulation with the threshold continuously. Once the voltage of the integrator exceeds a certain threshold, the camera fires spike and resets the accumulation. The voltage of the accumulator can be formulated as:
A(x, () = f/ a ne-IIxx mod
(I)
where A(x, t) is the voltage of the accumulator at pixel x (z. v). 1(x,T) is the lights intensity in pixel x at time T. 0 is the threshold of the comparator. The above-mentioned working mechanism is called Integral-and-Fin and-Frre (IF). With such an IF procedure, the spike cameras can fire spikes asynchronously and continuously. The reading time of the spikes is quantified with a period T, and the T can reach micro-second level. The spike camera fires spikes at time nT, n E N. Thus, the output of the spike camera is a spatial-temporal binary stream S in H x W x N size. The H and W are the height and width of the sensor, respectively, and N is the temporal steps of the spike stream.
3.2 Problem Statement and Data Generation
Problem Statement. Optical flow estimation for spike cameras is to estimate the pixel-level motion field of the projection from the optical scene to the sensor plane. Suppose that we denote binary spike stream as = (S(x,1) 1) x E Ω, i E N, i S n). if we set lstart = tu as the starting moment, the optical flow be denoted as (w(x, t to) t N, to < t n). The ideally meaning of w can be formulated as:
S (xx wcx, to), ((, EEN, to St5n
(2)
3

---

Spkke Stream Input
Temporal-Spatal epresentation
oint Correlation Decoding
Encoder
Context Feature
Clip
Peturee Encoder
5
Temporal Axis
Feature
Clip
Feature Encoder
Correlation
S1
Feature
Optical Flow
Clio
Encoder
Spike Stream
SN
Figure I: The overall architecture of the Spike2Flow. The input spike stream is firstly clipped to several sub-streams. Each sub-stream extracted be a feature for correlation through temporal- spatial representation, and the first feature constructs correlations with all other features. The recurrent decoder estimates flow fields from all the correlations and the context feature.
where > means pixel-level registration. Given a spike stream S, the target of optical flow estimation estimating w. The data format of the spike stream is continuous binary matrices, which is different from the classic data pattern of video. Correspondingly. the processing method should also be different. Firstly, unlike a pixel in videos, single spike represents the result of the integral procedure rather than the status at the current moment, and a single spike cannot express spatial-temporal information. Secondly, the spike streams more continuous than classic videos. Thus, taking advantage of the continuousness for analyzing the scene is a challenge and an opportunity.
Data Generation. SCFlow proposed two synthetic datasets through graphics simulator for training and evaluation, respectively. However, there are several limitations of the simulating and rendering, such as unrealistic textures, overly simplified lighting conditions, and unreasonable appearance. The huge gap between the synthetic scenes and the real world causes the models often have poor generalization the real domain due to the gap [34; 45; 63; 7]. Thus, it is essential train networks with data from real scenes. To improve the generalization on the real domain, we use high-speed optical flow
dataset Slow Flow [25] with high spatial and temporal resolution to synthesize dataset, i.e., real scenes with spike and flow (RSSF) for spike cameras with real scenes. We use the raw data of Slow Flow generate RSSF. The data have 41 scenes and sum tens of thousands of frames, and there are three kinds of spatial resolution: 2560 x 2048, 2560 x 1440, and 2048 x 1152. We select 11 scenes generate the testing set and the other 30 scenes generate the training set. More details are included in supplementary materials. The generation pipeline is as follows.
Firstly, we imitate the image signal processor (ISP) to process the raw data in Bayer pattern to color images, where the operations include demosaicing, white balance, and intensity mapping. The image frames 2x downsampled spatially and temporally for saving computing and storage sources. Secondly, we use GMA [27] network trained on data mixed by Sintel [5], KITTI [42], HDIK [31] and ElyingThings [40] to generate the reference optical flow of the image frames. Thirdly, We use the reference flow to simulate the motion of the scene and construct virtual spike camera to fire the spike streams. We interpolate 20 time steps of images between every adjacent frames for generating spikes, and we use 10 times temporal oversampling to improve the precision of the scene. The ground truth contains flow for the duration of 20, 40, and 60 interpolated images, denoted as dt = 20, dt = 40, and dt = 60, respectively. It is noted that although the reference flow is not absolutely accurate, still reliable due to the following reasons.
(1) High-quality data. The images for generating the spike have a high spatial and temporal resolution, which approximate the ideal high-speed scenes that world. Estimating the reference flow from such high-quality data is reliable.
hardly get in the real
(2) Correspondence between spike and reference flow. Although the reference flow fields are not absolutely accurate for color frames, they correspond to the spike streams since the spike streams are generated from the reference motion. Thus, the reference flow fields are reliable for the spike streams.

---

Feature Context
Encoder
Adoitio
Dot P Production
CxHxW
Softmax
HW X HW
Conv
CXHXW
Query
DSFT Trans
Feature
Extractin
F
Key
CxHxW
F
Output Feature
P Primary
Feature
Spike Stream
Conv
CxHxW
Temporal Information Aggregation
Extracton
Fealure
Spelial Information Aggregston
Figure Illustration temporal-spatial representation (TSR) for spike streams. spike sub-stream is firstly aggregated temporal information through the differential of spike firing time (DSFT) transform. primary feature is then extracted from the DSFT to the primary feature with downsampling. Then the primary feature is processed by the spatial information aggregation (SIA) module.
(3) Good generalization. Results in section 4.2 show satisfactory performance of network on PHM dataset [17] and real data. For PHM, the ground truth flow nearly absolutely reliable, which indicates the generalization of synthetic data and the reliability of training data. Good performance on real data demonstrates the generalization of our training data on real scenes although there is domain gap between real and simulated spikes.
3.3 Overall Architecture of the network
The overall architecture shown in Fig. 1. The input spike stream S is firstly clipped to spike sub-streams (So. .,...., S. Each sub-stream is extracted be the feature for matching {FM LMM ,..., FM ]. which represents the central moment of the corresponding spike sub-stream, and the first feature So is extracted to be context feature F like RAFT [54]. Noted that it is hard to extract rich features directly from binary spike stream, we propose temporal-spatial representation (TSR) for the above-mentioned feature extraction. To utilize the continuousness of the spike streams, the first matching feature r. constructs all-pairs correlations {Cl,..., CN) with all other matching features. The recurrent decoder jointly estimates flow fields from the central time of Sp to the central time of {SI.... Sy } through all the correlations and the context feature F..
For better embedding the spike streams to the feature domain, we propose to aggregate the information in the temporal and spatial domains. We design the differential of spike firing time (DSFT) in the temporal domain to transform the spike stream from the binary domain, which aggregates information along the temporal axis for each pixel. Based on DSFT, the spike stream is then embedded in high-dimensional feature with a downsampling operation. A spatial information aggregation (SIA) module is proposed to extract the information in the spike further. To utilize the continuousness of spike streams, which reflects the procedure of motion, we propose a joint correlation decoding (JCD) module to jointly estimate series of flow fields starting from the same moment.
3.4 Semporal-Spatial Representation for Spikes
Differential of Spike Firing Time. Each spike in spike streams represents the result of the integral procedure of photons rather than status at the current moment. Different "1" correspond to various light intensities since the "1" in the spike streams represents the number of accumulated photons rather than the arrival rate of the photons. Using features extracted from the binary spikes for matching may not be appropriate to reflect the structures of scenes. We propose to represent the information contained the spike through the firing time, which can better represent the arrival rate of the photons at each pixel, i.e., the light intensity at each pixel. For better aggregating the temporal information, as shown in Fig. 3, we propose to use the differential spike firing time to represent the binary spike. If denote the DSFT of spike stream S S(x, as D D(x, t), the DSFT transform can be
5

---

Temporal Axis
Figure 3: Illustration of the dif- ferential of spike firing time (DSFT) Transform. Each pixel in the binary spike stream (the first row in the figure) is repre- sented as the difference firing time (the second row in the fig- ure) of the corresponding pixel.
DSFT
DSFT
ue Drcouss L.50
DSFT Valuee
F Fring
Siient Pixel
Mobinn Encoder
Weight hored Canvoll ltnn
Decoder
Local Comilation
Concaterete
Respuctivlly
Condictio
Flow Head
Currentt Bunmeted Bow rewiilton
Flow J/k
Caryy Car-diatic
Concad
12
150 12
Now Hood
DM
(WL Noteed
Figure 4: Illustration of the detailed structure of the recurrent decoder. Local correlations are looked up from the correlation and encoded to motion features with the corresponding flow fields. The context feature and all the motion features are used to update the hidden state of the ConvGRUs. A series of flow fields are decoded with different prediction heads.
formulated as:
D(x,t) = D[S(x, t)] Tnext (x, t) T (x. t)
{[min(F] ((p, 1, pED(() (max(T) S(p, t)
Where D the DSFT transform. Thext (x, t) and T (x, t) denote the firing time of the next and previous spike spatial-temporal moment (x, f). Dxx) denote the domain of definition of We use DSFT to reflect the firing rate of the spikes. The firing rate here is statistical concept, and the average of individual spike intervals is connected with the firing rate. The motion in the scenes changes the brightness, and the Poisson process of photon arrivals causes fluctuations in the firing rate. Thus, the firing rate is temporally variational each pixel. DSFT can better recover the spikes' dynamic process than obtaining a more constant firing rate with longer time windows.
Spatial Information Aggregation. A single spike in space-time coordinates hardly describe the scene, and we need a group of spikes to represent each pixel. The DSFT aggregates the information in the temporal domain. However, the DSFT is still fluctuating since the arrival of the photons follows the Poisson process, and the firing rate of spikes exhibits remarkable randomness. The fluctuation of spikes can make the value of features for matching unstable. To enhance the features of spike for better matching, we propose a spatial information aggregation (SIA) module to integrate context information with a larger receptive field. As shown on the right side of Fig. 2, we use the self-attention mechanism to aggregate the spatial information, which can be formulated by:
F F, + softmax (0(F)) ((Fp)²) .g(F))
Where F denotes the feature finally output by the encoder. F. denotes the primary feature before SIA. 0, Φ. and embed the primary feature to query, key, and value, respectively. Through the SIA module, each pixel gets long-range aggregation with pixels in its relative area.
3.5 Joint Decoding of Correlation
There more continuously temporal information spike streams compared with videos. extract the continuous moving procedure of the optical scene in spike, we propose to jointly decode series

---

Scene
RAFT - Spike
GMA - Spike
SCV Spkke
SCFlow
Spike2Fioo low (Ours)
Reference
AEPE: A135 511
ABPE: 0A7
POR:93
AEPE: 9A20
POS 1444 AEPE C283
PO% 7.40
AEPÉ: 0.022 10%
AEPE: 0.255 PON 11.3
AEFE:C.3N
POR: AEPE:R675
POX 37.59 AEPE-C33M
POPe 2112
AEPE:611
POY 352
0106 POX 6.77 AFFF-6.M PORA AFFF: PO% 21.59 AFPF C277
POX: 11.45 AFFF--1101
PO% 2/11
Figure 5: Visual results on RSSF di case, the meaning each column is on the top. PO% means the percentage of outliers. Please enlarge the figures for better comparison.
of motions from the same starting moment, i.e. {w(x,t t sparsely sample the motion and estimate {w; w(x, iT to (o) where 7 is the sampling cycle, is the sampling index, and N the maximum sampling index. For each sampling moment, we construct multi-scale all-pairs correlation [54] from the matching features D and F.M.
fq))- To simplify the computation, we
2
(,,,y,,,,)
1 22
Σ
Σ
(z, FM 2 +p.n 2' +
(5)
Local correlations [L,...., LN| } are then looked up through current estimated flow fields with a radius The local grid for looking up be formulated as:
Mxx) = ((x + w(x, iT to ta))/2' + +y tyly ZZ |yyl rr
The structure the recurrent decoder is shown in Fig. In each iteration, the decoder optimizes the flow fields by estimating their residual through local correlations and context feature F Suppose that the start moment the flow l.s and the sampling cycle of the flow is T. The local correlations [L,) and currently estiamted flow fields (Wa-1,i W (t. + iT ( are extracted to be motion features { [M; by the motion encoder, where & is the iteration index of the GRUs All the motion features {M..... } and F are concatenated and input to the ConvGRUs [8] to update the hidden states. For all the flow fields to be estimated,
single hidden state. Different prediction heads are employed to estimate the residual of different flow fields.
Suppose the recurrent decoder has V iterations. Given ground truth flow fields {w5t function is defined as:
L Ji=1 the loss
w wa
(7)
where y is the decay factor set 0.8.
4 Experimental Results
4.1 Implementation Details
In the experiments, we set N as and T 20, which means we jointly estimate optical flow under 20, 40, and 60 time steps difference. We set the number of input spike frames as 21. For constructing correlation, we set the multi-scale level as 3, and we set the looking-up radius r = 3. The model trained on the training set of the real scenes with the spike and flow (RSSF) dataset. We randomly crop the spike stream to 320 x 448 spatially during the training procedure and set the batch size as 6. We use randomly horizontal and vertical flips data augmentation to balance the motion in the
7

---

Scene
RAFT Spike
GMA Spike
SCV - Spkke
SCFlow
SpkkeFlow Foow (Ours)
Ground Truth
A1F POS--30
AEPE: POS 427
AEPE: POLL 7:11
AEPE-0341 POX-422
AEPE-CIM PON
AEFE/O P0%: 1450 AEPE 0342 90%: 1433 AEPE: 0,462 P0% 33.22 AEPE 0541 POX 29 05 AEPE 0276 90% 8.37
Figure 6: Visual results PHM in dt 10 case, the meaning of each column is the top.
Table 1: Quantitative results on the testing set of RSSF. All the models are retrained on the training set RSSF based on the official code. The best results over each group are bolded. P0%: the percentage of outliers.
Method
di = 20
di = 40
=
60
AEPE
PO%
AEPE
PO%
AEPE
PO%
RAFT [54]
Avging
0.244
0.390
12.41
0.573
14.85
RAFT [54]
Spike
0.181
5.62
0.295
8.69
0.437
10.71
SCV [28]
Avglmg
0.534
33.86
0.762
39.54
1.056
42.29
SCV [28]
Spike
0.570
40.03
0.808
43.90
1.132
44.34
GMA [27]
Avglmg
0.226
7.20
0.388
12.54
0.632
14.81
GMA [27]
Spike
0.230
7.45
0.401
11.60
0.603
14.16
SCFlow [17]
0.389
14.00
0.668
19.00
1.264
23.40
Spike2Flow(nurs) 0.117
2.57
0.197
5.18
0.286
6.98
training dataset. We use Adam optimizer [30] with 81 0.9 and A8 0.999. The learning rate is initially set as 3e-4 and scaled by 0.7 every 10 epoch. The model is trained for 100 epochs.
We use average end-point eиror (AEPE) and percentage of outliers (PO%) evaluating metrics. AEPE is the spatial average of Euclidean distance between optical flow w and the ground truth Wgt
AEPE
HW Σ [/((x) - Wgt(x))
where H and W the height and width of the flow. The percentage of outliers is the percentage of the pixel with end- point error larger than 0.5 and 5% of ground truth at the same time.
Comparative Results
We compare our method with SCFlow [17] and methods straightforwardly designed based on RAFT [54], SCV [28], and GMA [27] for estimating optical flow for spike cameras. There are two ways to input the spikes to the methods for optical flow estimation for spike cameras:
(I) Avglmg. Averaging the spike stream along the temporal axis gray image.
(2) Spike. Directly input the spike streams into the network work as multi-channel image.
We do not use the event based optical flow architecture since it has been proven to be noneffective in [17], and we do not use the training set in [17] since it is synthetic and the generalization ability is limited. We use "RAFT Avgimg" and "RAFT Spike" to represent the direct adaptation based on RAFT with Avgimg and spike, respectively. Adapted methods for other architectures are same. All the methods are retrained under settings in section 4.1 based on the official code on RSSF training set with spike streams as input, respectively. Noted that Spike2Flow jointly estimates flow fields with time duration dt 20, 40, 60. We set the training data for comparable models and

---

Scene
RAFT Spike
GMA Spike
SCV Spike
SCFlow
Spike2F low (Ours)
Figure 7:
Visual results on real data in dt 20 case, the "Scene" is the average of spike stream on temporal axis with gamma transform.
Table 2:
Quantitative results on PHM. All the models are retrained on the training set of RSSF based on the official code. The best results for each group bolded. PO%: the percentage of outliers.
dt
10
dl
20
Method
AEPE
PO%
AEPE
PO%
RAFT [54]
AvgImg
1.635
32.55
2.753
40.85
RAFT [54] Spike
0.622
21.50
1.219
23.79
SCV [28]
Avglmg
1.055
44.17
1.829
54.24
SCV [28]
Spike
0.841
34.18
1.466
41.26
GMA [27]
Avgimg
0.874
28.30
1.784
39.67
GMA [27]
Spike
1.485
27.41
1.888
29.25
SCFlow [17]
1.027
34.54
1.775
38.57
Spikc2Flow(urrs)
0.545
16.07
1.088
19.61
randomly select dt from {20, 40, 60) for each sample in mini-batch for the sake of faimess. We evaluate the models on three kinds of data: (1) The testing set of real scenes with spike and flow (RSSF), (2) photo-realistic high-speed motion (PHM), and (3) real data captured by spike cameras.
Results on real scenes with spike and flow (RSSF). The quantitative results on RSSF is shown in Tab. 1. The Spike2Flow jointly estimates the flow in di 20, 40, 60 cases while other models estimate respectively. Spike2Flow gets best performance in both AEPE and PO% in all cases. The visual results RSSF are shown in Fig. 5. All the optical flow color-coding rule is the same with [1]. More results are included in the supplementary material.
Results on photo-realistic high-speed motion (PHM). For comparative experiments on PHM, we use the same model as experiments on the testing set of RSSF, which is trained on the training set of RSSF. The ground truth of optical flow in PHM is generated from graphics simulator, which highly reliable. PHM contains ground truth flow dt 10 and cases. For 2pike2Flow, we jointly estimate flow fields in di 10, 20, 30 cases. For other models, estimate flow fields in di 10 and = 20 cases, respectively. Noted that we select the scenes PHM except fly since the motion speed of fly is excessive and unrealistic. Quantitative results are shown in Tab. 2, and the results are average among all the scenes we use of the PHM. Spike2Flow still achieves the best performance in all the cases, which demonstrates the good generalization of our method. Compare Tab. with Tab. 1, the performance results on PHM are lower than those on RSSF since the scene in PHM is simulated through graphics model, and the speed motion of the scenes is ultra-high, which more challenging for optical flow estimation. The visual results on PHM are shown in Fig. 6 and more results are included in the supplementary material.
Results on real data. We evaluate the models on real data captured by spike cameras. The visual results are shown in Fig 7. Although we do not have the ground truth of optical flow in real data, we can see that results of 2pik have sharper edges and more clear background, which shows better generalization. More details included in the supplementary material.

---

4.3 Ablation Study
To verify the efficiency of the modules we propose, we implement a series of ablation studies. The quantitative results are shown Tab. 3. We use color gradation from yellow to green for each column. where greener represents better performance. The results demonstrate the effectiveness of DSFT, SIA, and JCD modules. Although the Exp. (D) and (E) have similar performance in dt 20 and dt 40, Exp. (E) performs better than (D) in the percentage of outliers under dt = 60, which shows the effectiveness of the SIA module. Comparison between Exp. (B) and (C) can also show that SIA is effective. Besides proposed modules, we also perform ablation studies the number of input spike frames (NISF). We select {1, 5, , 15, 21 as the NISF candidates. The quantitative results are shown in Tab. 4, where we can see larger NISF makes the performance better.
Table 3: Ablation study of our proposed modules. Greener blocks represent better performance with lower AEPE the percentage of outliers (PO%).
dt
20
dt
40
dt
60
Index Setting of experiment
AEPE
PO%
AEPE
PO%
AEPE
PO%
(A)
Removing DSFT transform
0.236
7.69
0.382
13,49
0.536
16.94
(B)
Removing SIA and JCD module
0.128
3.30
0.218
6.37
0.325
8,40
(C)
Removing JCD module
0.126
3.13
0.210
5.96
0.310
7.88
(D)
Removing module
0.116
2.54
0.199
5.16
0.298
7.24
(E)
Our final model
0.117
2.57
0.197
5.18
0,286
6.98
Average End Point Error
0.6
at 20
Table 4: Ablation study of the number of input spike frames (NISF). Greener blocks represent better performance with lower AEPE or PO%.
0.5
0.3
20
dt
40
dt 60
NISF
0.2
AEPE
PO%
AEPE PO%
AEPE PO%
11
15
21
0.174
4.37
0.315
10.85
0.463
15.01
Number of input spike frames (NISF)
0.153
3.44
0.277
7.70
0.417
11.35
Figure 8: AEPE of ablation NISF, shadow
11
0.125
2.98
0.218
6.30
0.323
8.55
areas represent the average of the standard devi- ation of each scene.
15
0.123
3.01
0.211
5.83
0.312
7.85
21
0.117
2.57
0.197
5.18
0.286
6.98
Conclusions
We propose a method for learning optical flow from continuous spike streams. To improve the generalization spike-based optical flow, we propose a flow dataset RSSF with real scenes. For better extracting information from binary spike streams, we propose DSFT and SIA modules to extract the temporal and spatial information, respectively. To use the continuousness of spike streams motion clues, propose JCD module jointly estimate series of flow fields. Experimental results show that 2lik achieves state-of-the-art performance spike-based optical flow on RSSF, PHM, and real data, and ablation studies verify the effectiveness of proposed modules.
The characteristics of spike firing extremely dark or bright scenes are quite different, so the same representation strategy may not work well all the time. We plan to extend our model to handle these questions future work.
Limitations.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China under Grant 22127807, 62072009, 61931014, and 619721 15, and also supported by National Key R&D Program of China under Grant 2021 YFFO900000.......
10

---

References
[1] Simon Baker, Stefan Roth, Daniel Scharstein, Michael J Black. JP Lewis, and Richard Szeliski. A database and evaluation methodology for optical flow. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1-8, 2007.
[2] Wenho Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang. Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3703-3712, 2019.
[3] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger. Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios? Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2574-2583, 2017.
[4] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical flow estimation based on theory for warping. In Proceedings of the European Conference on Computer Vision (ECCV), pages 25-36, 2004.
[5] Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In Proceedings of the European Conference Computer Vision (ECCV), pages 611 611-625, 2012.
[6] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Understanding deformable aligmment video super-resolution. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 973-981, 2021.
[7] Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization. In Proceedings of the International Conference on Learning Reprexentations (ICLR), 2021.
[8] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: pacoder-decodde approaches. In Proceedings of Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST), pages 103 111,1111200002.. 2014.
[9] Dahjung Chung, Khalid Tahhoub, and Edward J Delp. stream siamese convolutional neural network person re- re-idenification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1983-1991, 2017.
[10] Ziluo Ding, Rui Zhao, Jiyuan Zhang, Tianxiao Gao, Ruiqin Xiong, Zhaofci Yu, and Tiejun Huang. Spatio-temporal recurrent networks for event-based optical flow estimation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
[11] Siwei Dong. Tiejun Huang, and Yonghong Tian. Spike camera and its coding methods. In Proceedings of the Data Compression Conference (DCC), pages 437-437, 2017.
[12] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 2758-2766, 2015.
[13] Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters (RAL), 6(3))4947-44555 2021.
[14] Mathias Gehrig, Mario Millhäusler, Daniel Gehrig, and Davide Scaramuzza. E-raft: Dense optical flow from event cameras. In Proceedings of the IEFE International Conference on 3D Vision (3DV), pages 197-206. 2021.
[15] Jesse Hagenaars, Federico Paredes- Vallés, and Guido De Croon. Self-supervised learning of best- optical flow with spiking neural networks. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
[16] Berthold KP Hom and Brian Schunck. Determining optical flow. Artificial intelligence (Al), 17(1- 30:185-203, 1981.
[17] Liwen Hu, Rui Zhao, Ziluo Ding, Lei Ma. Boxin Shi, Ruiqin Xiong, and Tiejun Huang. Optical flow estimation for spiking camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

---

[18] Jing Huang. Menghan Guo, and Shoushun Chen. A dynamic vision sensor with direct logarithmic output and full-frame piclure-on-demand. In Proceedings the IEEE International Symposium on Circuits and Systems (ISCAS), pages 1-4, 2017.
[19] Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, al. 1000x faster camera and machine vision with ordinary devices. Engineering, 2022.
[20] Tak- Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 8981-8989 2018.
[21] Junhwa Hur and Stefan Roth. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5754-577 1344-5763,220 2019.
[22] Junhwa Hur and Stefan Roth. Self-supervised multi-frana monocular scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2684-2694, 2021.
[23] Eddy Ilg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2462-2470, 2017.
[24] Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger. Unsupervised learning of multi-frame optical flow with occlusions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 690-706, 2018.
[25] Joel Janai, Fatma Guney, Jonas Wulff, Michael Black, and Andreas Geiger. Slow flow: Exploiting high-speed cameras for accurate and diverse optical flow reference data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3597-3607, 2017.
[26] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Sarmed-MMllee and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for vidco interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9000-9008, 2018.
[27] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9772-9781, 2021.
[28] Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning optical flow from a few matches. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 16592-16600, 2021.
[29] Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In Proceedings of the European Conference Computer Vision (ECCV), pages 557- 2020.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[31] Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahinimoghaddam. Sabine Hofmann, Claus Brenner, al. The hei benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRw), pages 19-28, 2016.
[32] Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Danilidiis, and Kaushik Roy. Spoke-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages
2020.
[33] Hycongmin Lec, Tacoh Kim, Tac-young Chung, Dachyun Pak, Yuscok Ban, and Sangyoun Lec. Adacof: Adaptive collaboration of flows for vidco frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5316-5325, 2020.
[34] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV).
pages 5542-5550. 2017.
[35] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128 128 120 db 15ps latency asynchronous temporal contrast vision sensor. IEEE Journal Solid-State Circuits (JSSC), 43(2))566-574 2008.
12

---

[36] Liang Liu, Jiangning Zhang, Ruifei He. Yong Liu, Yabiao Wang. Ying Tai. Donghao Luo, Chengjie Wang. Jilin Li, and Feiyue Huang. Learning by analogy: reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6489-68 6499,, 2020.
[37] Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu. Ddftow: Learning optical flow with unlabeled data distillation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 8770- 8777, 2019.
[38] Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Sclftow: Self-supervised leaming of optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4571 14580, 2019.
[39] Kunming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang. and Jian Sun. Upilow: Upsampling pyramid for unsupervised optical flow lcarning. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1045-1054, 2021.
[40] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040-4048, 2016.
[41] Simon Maister, Junhwa Hur, and Stefan Roth, Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), pages 7251-7259, 2018.
[42] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 3061-3070, 2015.
[43] Diederik Paul Moeys, Federico Corradi, Chenghan Li, Simeon Bamford, Luca Longinotti, Fabian Voigt, Stewart Berry, Gemma Taverni, Fritjof Helmchen, and Tobi Delbruck. A sensitive dynamic and active pixel vision sensor for color neural imaging applications. IEEE Transactions Biomedicul Circuity and Systems (TBCS), 12(1):123-136) 123-136, 2017.
[44] Michal Neoral, Jan Sochman, and Jiff Matas. Continual acclusion and optical flow estimation. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 159-174, 2018.
[45] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and general- ization capacities via ibn-net. In Proceedingy of the European Conference on Compuder Vision (ECCV), pages 464-479, 2018.
[46] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt. A quga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain eds. IEEE Journal of Staie-State Circuits (JSSC), 46(1):259 275, 2010.
[47] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4161-4170, 2017.
[48] Zhile Ren, Orazio Gallo, Deqing Sun, Ming-Hsuan Yang. Erik B Sudderth, and Jan Kautz. A fusion approach for multi-frame optical flow estimation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2077-2086, 2019.
[49] Daniel Scharstein and Richard Szeliski. taxonomy and evaluation of dense two-frame stereo correspon- dence algorithms. International Journal of Computer Vision (IJCV) 472):7744, 2002.
[50] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self- teaching multi-frame unsupervised raft with full full-image warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3887-3896, 2021.
[51] Deqing Sun, Stefan Roth, and Michael J Black. Secrets of optical flow estimation and their principles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2432 2022-243,,200 2010.
[52] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cans for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8934 8943, 2018
13

---

[53] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Models matter, so does training: An empirical study of cans for optical flow estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 4266):1408--142,, 2019.
[54] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402-419, 2020.
[55] Zhigang Tu, Hongyan Li, Dejun Zhang, Justin Dauwels, Baoxin Li, and Junsong Yuan. Action-stage emphasized spatiotemporal vlad for video action recognition. IEEE Transactions on Image Processing (TIP), 28(6):2799 2212, 2019.
[56] Jianyuan Wang. Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacement- invariaat matching cost learning for accurate optical flow estimation. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 01115522-52233 2020.
[57] Longguang Wang, Yulan Guo, Li Liu, Zaiping Lin, Xinpu Deng, and Wei An. Deep video super-resolution using hr optical flow estimation. IEEE Transactions on Image Processing (TIP), 29:4323 4336, 2020.
[58] Yang Wang. Yi Yang, Zhenheng Yang. Liang Zhao, Peng Wang, and Wei Xu. Occlusion aware unsupervised learning of optical flow. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4884-4893, 2018
[59] Xijie Xiang. Lin Zhu, Jianing Li, Yixuan Wang, Tiejun Huang, and Yonghong Tian. Learning super- resolution reconstruction for high temporal resolution spike stream. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021.
[60] Haofei Xu, Jiaolong Yang. Jianfei Cai, Juyong Zhang, and Xin Tong, High-resolution optical flow from Id attention and correlation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
2021.
[61] Jiangtao Xu, I Liang Xu, Zhiyuan Gao, Peng Lin, and Kaiming Nie. denoising method based on pulse interval compensation for high-speed spike-based image sensor. IEFF Transactions on Circuits and Systems for Video Technology (TCSVT), (18)))9966-298)) 2020.
[62] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Proceedings of the Anmual Conference on Neural Information Processing Systems (NeurIPS), 5:12, 2019.
[63] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-rea generalization without access- ing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2100-21 10, 2019.
[64] Feihu Zhang. Oliver Woodford, Victor Adrian Prisacariu, and Philip HS Torr. Separable flow: Learning motion cost volumes for optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10807-10817, 2021.
[65] Jing Zhao, Jiyu Xie, Ruiqin Xiong, Jian Zhang, Zhanfei Yu, and Tiejun Huang. Super resolve dynamic scene from continuous spike streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2533-2542, 2021.
[66] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. SpkZimgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12005, 2021.
[67] Jing Zhao, Ruiqin Xiong, Jiyu Xie, Boxin Shi, Zhaofei Yu, Wen Gao, and Tiejun Huang. Reconstructing clear image for high-speed motion scene with a retina-inspired spike camera. IEEE Transactions on Computational Imaging (TCI), 8:12-27, 2021.
[68] Shengyu Zhao, Yilun Sheng, Yue Dong, Eric Chang, Yan Xu, et al. Maskflownet: Asymmetric feature matching with Icarnable occlusion mask. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6278-6287, 2020.
[69] Yajing Zheng, I. Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 3358-6367, 2021.
[70] Alex Zihao Zhu, Dinesh Thakur, Tolga Özaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multivehicle stereo event camera dataset: An event camera dataset for 3d perception. JEEE Robotics and Automation Letters (RAL). 3(3):0032 20022220992222 2018
14

---

[71] Alex Zihao Zhu and Liangzhe Yuan. Ey-flownet: Self-supervised optical flow estimation for event- cameras. In Proceedings the Robotics: Science and Systems (RSS), 2018.
[72] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danillidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 989-997, 2019.
[73] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image recon- struction spiking neural model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1438 1446, 2020.
[74] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Ultra-high temporal resolution visual reconstruction from fovea-like spike camera via spiking neuron model. IEEE Trunsactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
[75] Lin Zhu, Jianing Li, Xiao Wang. Tiejun Huang, and Yonghong Tian. Neuspike-nett net: High speed video reconstruction via bio-inspiree neuromorphic cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2400-2409, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes) See Section
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See section 4.1.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "_C7XeuntLTp", "reviewer": "Reviewer_p5Ne", "summary": "The authors have proposed a tailored network that extracts information from binary spikes with ST representation based on the differential of spike firing time and spatial information aggregation. In addition, their proposed network utilizes continuous motion clues through joint correlation decoding. ", "questions": "From a technical point of view, there isn't any serious concern regarding the content of this work. I would suggest the authors expand the joint decoding of the correlation section a bit more since it is an important part, but the current status of that section is not very readable to the reviewer. For example, I couldn't get the reasoning behind assigning 0.8 to the decay factor here.", "limitations": "Compared to previous works, the limitations and impact of the work are adequately outlined and addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "All in all, this is a very well-structured paper. An extensive literature review has been performed to support the research objectives of the authors. I especially like the detailed outlining of the architecture of the network. \nThe comparative results on real scenes with spike and flow as well as photo-realistic high-speed motion are adequately represented, and this demonstrates one of the strengths of this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HDlrCXrYgb6", "reviewer": "Reviewer_PUft", "summary": "This paper proposes a method, Spike2Flow, to estimate optical flow from videos recorded by spiking cameras. The input modality is rarely used and not properly introduced, so it hard for me to evaluate the merits of this work. Moreover, the technical contribution seems marginal.", "questions": "1. Please give illustrations of the output (consecutive frames) of spiking cameras, so that readers will have an intuitive understanding what the videos look like, and what challenges come with this new modality.\n2. Why spiking cameras are called with different names in different papers? The two cited references are [11; 19], but in [11] it's called a \"spike camera\" (instead of spiking), and in [19] it's called \"vidar\".\n", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The empirical performance seems good.\n\nWeaknesses:\n1. The presentation is quite unclear. Esp., as spiking cameras are an emerging device and rarely seen, most people are not familiar with its properties, advantages and specific challenges. The authors did not do well when introducing the background of this task.\n2. The architecture presented in Fig. 1 seems very similar to RAFT. The Spatial Information Aggregation in Fig.2 seems very similar to CRAFT [a]. It gives me a feeling that the proposed method has rather limited novelty.\n\n[a] CRAFT: cross-attentional flow transformers for robust optical flow. CVPR 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GoxQM9UD0Tz", "reviewer": "Reviewer_pFP6", "summary": "The authors present an algorithm for optical flow from spiking camera data. The main contributions are:\n\n1) the introduction of a spike based optical flow network, Spike2Flow which extracts features from binary spikes and estimates flow fields\n\n2) a dataset for spike-based optical flow\n\n3) Experimental results demonstrating good comparative results on various datasets and compared to other algorithms", "questions": "See my comments above", "limitations": "N/A: not applicable", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1: the paper is well written overall\n\nS2: the paper deals with an interesting problem, namely using spike cameras for optical flow. Given that spike cameras have high temporal resolution, optical flow is a good application for these cameras\n\nS3: the authors demonstrate good results compared to other similar algorithms\n\nS4: the authors indicate that source code will be released upon publication\n\nWeaknesses:\n\nW1: the algorithm does not really seam terribly innovative. This made me a bit ambivalent in deciding the paper's final ranking, but given the good results, I think it should be of interest to the community that deals with spike and event cameras\n\nW2: Eq 1: clarify it there is at most 1 spike per time intervale T=[T_pre, t]. Do all the spikes emerge from all pixels at the same time of is this asynchronous\n\nW3: How much data is actually produces by this sensor per second? The potential issue I see here is that neuromorphic chips that process event data asynchronously typically have I/O limitations. It is not clear how suitable this algorithm would be for such hardware. Furthermore the use of floating point operations (see softmax in eq 4 for example) may limit the applicability of this algorithm in power efficient hardware. A comment on this would be good.\n\nW4: line 209: typo: \"sparsely sampling\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DE16BQcVrQo", "reviewer": "Reviewer_zkLu", "summary": "The authors propose Spike2Flow, a deep neural network that maps spikes from a spiking camera to optic flow estimates. The method encodes spikes with a differential of spike firing time (DSFT) transform. It achieves results competitive with (better than) the state of the art on a new dataset, real scenes with spikes and flow (RSSF).", "questions": "1. Are you the first to apply the DSFT / what other encoding methods have been proposed? Can you make the relation with firing rate clearer? Can you show how sensitive this encoding is to noise?\n2. Can you apply your trained network to spiking camera inputs? Does it generalize well to non-RSSF, non-\"simulated\" data?\n3. The performance of Spike2Flow is always best. Can the performance of RAFT etc. be mentioned on normal RGB images for the dataset? This puts the performances more in context. For example, does RAFT lose a lot of performance when applying it to AvgImg? \n4. Is j-V in equation 7 correct?\n5. The dataset RSSF is mentioned as a contribution, but if I understand correctly, it is an automatic adaptation of SlowFlow, which was made by others. Is this correct? Can it be clarified earlier in the text?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The network focuses on a new type of spiking camera, potentially allowing for very high-speed optic flow estimation.\n* The encoding scheme may be useful for other work using spiking cameras.\n* The performance of the algorithm seems competitive.\n\nWeaknesses:\n* The encoding scheme seems quite straightforward. A spiking camera accumulates illuminance, which means that a bright pixel will have a higher firing rate. The proposed DSFT basically makes a local estimate of this rate, encoding the time between two subsequent spikes. This seems quite a naive method that may be sensitive to noise. Nothing is mentioned about this. \n* As far as I understand, the network is not applied to actual spiking camera inputs, but to a simulation of  such a camera based on a normal (or high-speed) camera stream. The paper would be much strengthened by using actual spiking camera inputs. \n* RAFT and other sota methods are only applied on the modified  image inputs (AvgImg, Spike). The performance on normal images from the slowflow data set are not mentioned. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept