Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera
Authors: Lujie Xia
Ziluo Ding
Rui Zhao
Jiyuan Zhang
Lei Ma
Zhaofei Yu
Tiejun Huang
Ruiqin Xiong
Venue: NeurIPS 2023 poster (2023)
Keywords: Optical flow
unsupervised learning
spike camera

Abstract: Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method achieves $15\%$ and $19\%$ error reduction on PHM dataset compared to the best spike-based work, SCFlow, in $\Delta t=10$ and $\Delta t=20$ respectively, using the same settings as in previous works. The source code and dataset are available at \href{https://github.com/Bosserhead/USFlow}{https://github.com/Bosserhead/USFlow}.

Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera
Lujie Xia 1.2 Ziluo Ding Rui Zhao 1.2 Jiyuan Zhang 1.2
Lei Ma², Zhaofei Yu Tiejun Huang!? Ruiqin Xiong 1.2.
SScooo of Computer Science, Peking University
National Engineering Research Center of Visual Technology (NERCVT)
BBijing Academy of Artificial Intelligence
'Institute for Artificial Intelligence. Peking University
National Biomedical Imaging Center, College of Future Technology, Peking University (lujie.xia xia, ziluo, lei. .ma, yuzf12, tjhhang, rqxi (erionn)&pk. edu, cn {ruizhao, jyzhang] aaus..uuu..... aku.eddu. cn
Abstract
Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. address this issue, we propose dynamic timing representation for spike streams. Based multi-layers archi- tecture, applies dilated convolutions on temporal dimension to extract features multi-temppral scales with few parameters. And design layer attention dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike- based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method achieves 15% and 19% error reduc- tion PHM dataset compared to the best spike-based work, SCFlow, in A/ 10 and At = 20 respectively, using the same settings as in previous works. The source wwt.tt..oo//eeteeeo
code and dataset are available at attttsss/giiabb.cmm///seeeeaaa
Introduction
Optical flow is defined as the apparent motion of individual pixels on the image plane and is prevalent for being an auxiliary tool for various vision tasks, e.g. frame rate conversion [15]. scene segmentation 38). and object detection [49]. In high-speed scenes, the optical flow estimation may suffer from blurry images from low frame-rate traditional cameras. Obtaining the data with a device that can precisely record the continuous light intensity changes of the scene is the key to addressing this issue. Recently, neuromorphic cameras have been developed greatly, such as event camera and spike camera. They can record light intensity changes in high-speed scenes. Especially, for spike camera, each pixel of it responds independently to the accumulation of photons by generating asynchronous spikes. It records full visual details with ultra-high high temporal resolution (up to 40kHZ), With these features, spike camera has demonstrated superiority in handling some high-speed scenarios, such as video frame interpolation 35 and dynamic scene reconstruction 40
43
48
Since spike camera can record details of high-speed moving objects, has enormous potential for estimating more accurate optical flow in high-speed scenes. Considering that deep learning has
"Corresponinng author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

achieved remarkable success in frame- optical flow estimation [3] 32]. it seems reasonable to directly upply frame-based architecture in spike data. However, the data modality of spike stream output by spike camera is quite different from frame images. For each pixel in spike camera, a spike fired and the accumulation is reset when photons accumulation at a pixel exceeds a set threshold. At each timestamp, the spike camera outputs a binary matrix, denoted as spike frame, representing the presence of spikes at all pixels. Previous work [12] utilizes spike stream as naive input representation for one timestamp, which consists of a series of spike frames within predefined time window. However, a too-long window can incur lots of misleading frames, while a short window is sensitive to noise and cannot provide enough information. Therefore, deliberate modifications are needed for input representation to extract salient information more flexibly and efficiently from spike streams before optical flow estimation architecture takes over.
In addition, the ground truth of optical flow is scarce in the real world, especially for high-speed scenes. To cope with the lack of labeled real-world data, it is necessary to study spike- baske optical flow estimation in an unsupervised manner. As described above, the light intensity information is contained within the spike intervals. This difference in data characteristics makes it unreasonable to directly apply the frame-based unsupervised loss to spike streams. Therefore, the light intensity should first be extracted from spike streams in the spike-based unsupervised loss. This is also the core of constructing the illuminance consistency on spike streams. Moreover, we argue the field of autonomous driving is a good place to validate spike camera since it is suitable for high-speed scenes. In autonomous driving, it is nearly impossible to collect real data for complex, diverse, high-speed extreme scenarios, e.g. vehicle collisions and these scenarios are of great significance to improve the safety of this field and should be highlighted. Therefore, in order to verify that the spike-based algorithm can handle extreme scenarios, we propose a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as the SSES dataset.
ceasstrinn-vehclle accidents. However,
In this paper, we propose an unsupervised method for spike-based optical flow estimation with dynamic timing representation, named USFlow. In our unsupervised loss, we propose two strategies, calli-inteerval-aassee and multi multi-time-wi Hime-window-b bbsee,, to estimate light intensity in regions with different motion speeds. The estimated optical flow is utilized to distinguish regions with different motion speeds and generates corresponding weights for light intensity fusion. Then the final approximate light intensity participates in loss calculation. As for addressing the fixed time window issue, there is a way that apply dynamic time window to the different spike streams. To this end, we propose Temporal Multi-dilated ExpresentationnTMMR for spike streams. In more detail, we apply multi-layer dilated convolutions to operate on the temporal dimension of spike streams. Multi-layer dilated convolutions enable the network to have different receptive fields and each layer can be regarded as summarizing spike stream with one different time window. We also design a Layer Attention(LA) module to extract salient features and filter the redundant ones.
Following the settings in previous works [12). we train our method on SPIFT [2 dataset and evaluate it on PHM [12 and our proposed SSES datasets. We demonstrate its superior generalization ability in different scenarios. Results show that USFlow outperforms all the existing state-of-the-ar methods qualitatively and quantitatively. USFlow shows visually impressive performance real-world data.
2 Related Works
Deep Learning in Optical Flow Estimation. Frame-based optical flow estimation is a classical task in the computer vision area through the years and has been solved well. PWC-Net and Liteflowne: introduce the pyramid and cost volume neural networks for optical flow, warping the features in different levels of the pyramid and learning the flow fields in a coarse-to-fine manner. RAFT[2]] utilizes ConvGRU construct decoders in the network and iteratively decodes the correlation and context information in a fixed resolution. Due to their excellent performance, PWC-
Net and RAFT are the backbones of most algorithms (37 34 41 21 20
[8]
17
36)
16
33 30] in
frame-based optical flow estimation. TransFlow [22] utilizing the long-range temporal association to addition, many frame-based 23 30 were proposed to discard the need for
recover more information by introducing transformer architecture.
unsupervised optical flow networks 25 21
20
18
labeled data. Similar to the traditional optimization- based methods, Yu el al. [14] employ photometric
loss and smoothness loss to train flow estimation network. Unflow (25 applics borward-backwarr

---

check on bidirectional optical flow to estimate the occlusion area, where the backpropagation of photometric loss stopped.
Deep learning has also been applied to event-based optical flow 45 FlowNer(45] can be regarded as the first deep learning work training large datasets with U-Net architecture222 (28). As an updated version of EV-FlowNet, an unsupervised framework has been proposed by Zhu al. [46] based on contrast maximization. Spike-FlowNee(1y) and STE- FlowNet 6 events, respectively.
26 5 EV-
use spiking neural networks and ConvGRU to extract the spatial-temporal features of
The research on spike- based optical flow estimation is just getting started. SCFlow [2]. the first deep-learning method. It proposes large dataset, the SPIFT dataset, train an end-to-end neural network via supervised learning. However, our work aims to fill in the blanks of unsupervised learning.
Event-based and Spike-based Input Representation. Normally, asynchronous event streams are not compatible well with the frame-based deep learning architecture. Therefore, frame-like input representation is needed and is expected capture rich salient information about the input streams. Apart from many handcrafted representations [46] 27 [4] 29 6). some end-to-end learned representations have been proposed, making possible generalize to different vision tasks better. Gehring et al. [T0] simply uses multi-layer perceptrons as trilinear filter to produce a voxel grid grid of Long Short-Term Memory (LSTM) cells that efficiently process events and learn end-to-end task deseddepnndent event surfaces. Similarly, Event-LSTM [T] utilizes LSTM cells process the sequence of events each pixel considering it into single output vector that populates that 2D grid. Moreover, Vemprala er al. [3 presents an event variational autoencoder and shows that it feasible to learn compact representations directly from asynchronous spatio-tempora event data.
of temporal features. Cannici er al. 4 propose
Directly using spike stream as input might incur lots of misleading information or miss something necessary if the time window is not chosen appropriately. Therefore, frame-like input representation should also be carefully designed extract sufficient information from spike stream. SCFlow [12 uses estimated optical flow to align spike frames to eliminate motion blur.
Different from all previous works, aim to train end-to-end input representation with the function of a dynamic time window by layer dilated convolutions.
3 Preliminaries
3.1 Spike Camera
Spike camera works by an "integrate-and-fire" mechanism, which asynchronously accumulates the light on each pixel. The integrator of spike cameras accumulates the electrons transferred from incoming photons. Once the cumulative electrons exceed a set threshold, the camera fires a spike and resets the accumulation. The process of accumulation can be formulated as,
t(x,t)= = S aI(xx r)dr mod 8,
where A(x, is the cumulative electrons pixel x (z, y). 1(x, 7) is the light intensity pixel at time т. a is the photoelectric conversion rate. 0 is the threshold. The reading time of spikes is quantified with period 8 of mieroseconds. The spike camera fires spikes at time T, T nd. Z, and generate an x W spike frame s. As time goes on, the camera produces spatial-temporal binary stream S in H W N size, as shown in Figure The N the temporal length of the spike stream. H and W are the height and width of the sensor, respectively.
3.2 Problem Statement
Given two timestamps to and t1, we have two spike streams centered on to and t1, noted as S and s/. respectively. Then we estimate dense displacement field f (f", f²) from to to t using
3

---

Laye
Layer ttention
Tosp Mutt dilattdd Repp Roptbbatto
Useepevee Lood
Concalasation
Optical
Estimation
Neigle
Bacábone
W Nighte
Spike Streem
Weight Light Interedly
Figure The overall architecture of the USFlow. First, The input spike stream is through the temporal multi-dilated representation, and the layer attention fuses the multi-layer outputs, Second, the fused output is fed to the optical flow estimation backbone. Finally, uses estimated optical flow to learn weights for light intensity estimation.
these two sub-spike streams, mapping each pixel x = (r, y) at to to its corresponding coordinates x' (x', 5²) = (a + f"(x), f* tj.
4
Method
4.1
Overview
To verify the effectiveness of the proposed dynamic timing representation, we present two versions of USFlow. One is the PWC-like version. It adopts variant of PWC-Net [31] which is also the backbone of SCFlow [12]. The other is the RAFT version which adopts an official small version of RAFT 32] as the backbone. More about these two backbones is included in the appendix. Considering one advantage of neuromorphic vision is low latency, our representation part should be lightweight and efficient. The spike streams first pass into the shared dynamic timing representation module separately, whose outputs are sent into existing backbones. In addition, we design an unsupervised loss break the dependency on labeled data. Note that all the components trained end-to-end.
4.2 Dynamic Timing Representation
As stated in 3.J one binary spike frame itself is sensitive to noise and meaningless without contextual connection. Given a bunch of spike data, selecting an appropriate data length is pivotal for subse- quent processing. too-long spike stream not suitable for high-speed regions since time offset accumulation introduces more redundant information. A too-short spike stream won't work either, as can not exhibit light intensity precisely with few binary data. To address this issue, we propose dynamic timing representation for input spike streams.
The Dynamic Timing Representation consists of Temporal Multi-dilated Representation (TMR) module and Layer Attention module (LA). The main ingredient of TMR is dilated convolution. By using dilated convolution, we can extend receptive fields larger range with just few stacked layers, while preserving the input resolution throughout the network as well as computational efficiency. more detail, we apply 1D dilated convolutions to the spike stream for each pixel and the parameters are shared across all the pixels. The TMR can be formulated as follow,
(ptt))))))) i 1.
where D1C(-) represents the ID dilated convolution operation, and x are layer index and spatial coordinate respectively. Note that F(0) (x) is the input spike stream on position x. Figure 2 depicts dilated convolutions for dilations 2, 4, and The higher level of the layer, the larger the receptive is two-fold. First, a different layer can be regarded as summarizing spikes or extracting temporal correlation among spike frames with a different time window. Second, dilated convolution allows the network to operate on a coarser scale more effectively than with a normal convolution, which guarantees the model is lightweight. In Table I parameter size of our USFlow and other methods.
field is. The intuition behind this configuratio
we show the

---

LawkS
Spike Straam
Splecel of
Weight Surtia
Ledt2
Figure 2: The Temporal Multi-dilatee Representation. 1D dilated convolutions are applied to all pixels with shared weights. The dilated convolutions are stacked for larger receptive fields. The higher level the layer is, the larger the receptive field is. Note that the illustration of the spike stream shown in the top-left corner.
Multi-dilated architecture has already greatly expanded the input dimensions but not all layers provide equally useful information. Blindly fusing the output of all layers may impair the learning process. To address this issue, we propose the Layer Attention module to further flexibly emphasize which layer meaningful or which time window is optimal. As illustrated in Figure [
we average the output of n layers, {F() (x) noted as Fx(x to generate one n--iimension layer context descriptor. The descriptor is then forwarded to a multi-layer perceptron(MLP) to produce our layer attention map. The layer attention values are broadcast along the layer dimension and the final concatenation, RF (x), is computed as follows:
RF(x) = o ((MLP(AvpPool((( @ F'(x),
here @ denotes element-wise multiplication. We do this operation on all pixels by sharing weights.
4.3 Unsupervised Loss
Unsupervised learning is necessary for spike-based optical flow estimation due to the lack of labeled training data in high-speed real-world scenes. To this end, we sidestep the need for ground truth of optical flow only spike data are required for training. More specifically, at training time, two spike streams centered on tg and I1, St. and St, are utilized to predict optical flow, f = (f", f*), from to to 41-
Different from traditional RGB frames, binary spike frame can not accurately represent the light intensity at the current timestamp. Therefore, estimating precise current light intensity from binary spike streams is the key to constructing an unsupervised loss function.
For low-speed regions. Since the light intensity accumulation in low-speed regions is an approxi- mately linear process. we count spikes to utilize longer-duration information estimate the light intensity. In this way, we can improve the robustness of light intensity estimation. However, the data length of the spike stream used to estimate light intensity varies with different low-speed motions. For reducing computational complexity, we only set two different time windows. The light intensity estimation can be formulated as follow:
+D. Σ I=
+D Σ I=T-D: Dr
fx(x,
Ws 0
s(x,t)+
Wt
s(x,
(4)
2D, +1
2D: +1
here Ds and Dj are the half length of time windows. They set to 40 and 100 respectively. wy and W/ are the weight factors. The subscripts s and refer to short and long time windows, respectively.
For high-speed regions. Selecting large time span of spike streams to extract information would incur motion blur due to time offset accumulation. Hence, we estimate light intensity during a single spike interval which is typically on the order of microseconds. Since the spike streams have extremely high temporal resolution and the interval between adjacent spikes ultra-short, we can safely assume that the light intensity remains constant during the interval 47). Let us denote s(x, m) and s(x, n)
5

---

as two adjacent spikes at position m and n are the timestamps corresponding to these two spikes. According to the spike camera working mechanism in Equation approximated by:
[
the constant light intensity can be
0
f(x)
(S)
(n m)
In reality, however, the number of incoming photons in an ultra-short interval is a random variable subject to the Poisson distribution even under constant illumination conditions. Therefore, the light intensity calculated by Equation 3 includess errors due to random fluctuations. To address this issue, extend it by multi-intervals and fuse light intensity information with different numbers of intervals,
K
(2k - -
[T(x, Nx) 1) T(x, M,(x) h + 1) where Mx(x)= = arg max (x, < T), N+(x) = arg min (T(x,) 27).
(x(x, r) = Σ (Wk
(6)
k=1
In Equation and 7 T(x. 2) refers to the timestamp corresponding to the zth spike at position x. [x,x, (x) + 1) 1) T(x, (x) k + 1)] is the total time length of these (2 1) intervals. we is the weight factor of the light intensity calculated by using (21: - intervals. Note that is set to and 2 in experiments. This setting ensures that the data length of spike stream used for light intensity estimation in high-speed motion regions much shorter than that used in low-speed regions. Further detailed discussion is presented in the appendix.
Learnable weights of estimated light intensity. Considering scene may contain both high-speed regions and low-speed regions, it is necessary for our unsupervised loss to fuse light intensity estimation methods based on multiple intervals and multiple time windows. We use the estimated optical flow to reflect the motion speed, and choose the most appropriate light intensity estimation strategy for different regions. As shown in Figure L we learn the weights Wa, Wl, Wi from the estimated optical flow. Thence, final approximated light intensity I.
fuse all the terms in Equation 4 and Equation 6 to achieve the
After achieving the i, we then derive the bidirectional photometric loss in a spike-based manner:
Cphoto (f, (f,
[(((T(x,t) to) ī(x + f,t,)) + p(ī(x + fot)) i(x,t,))),
(8)
where p the Charbonnier loss 3 f is the flow from to to t,. f' is the flow from é to to-
Furthermore, we use smoothness loss to regularize the predicted flow. It minimizes the flow difference between neighboring pixels, thus it can enhance the spatial consistency of neighboring flows and mitigate some other issues, e.g. the aperture problem. It can be written as:
Lsmooth (f.8) HW [
where is the difference operator, H is the height and W is the width of the predicted flow. The total loss function consists of two loss terms above-mentioned, which can be written as Liotal Cphoto ACamooth. where À the weight factor.
Experiments
5.1
Implementation Details
In this work, we choose the SPIFT dataset [2] as the training dataset. The PHM dataset [2] and our proposed SSES dataset is used for evaluation. The SPIFT and PHM datasets provide two data
settings, generating optical flow every 10 spike frames (Ai = 10) and 20 spike frames (Ai = 20)
separately from the start to end of sampling. Therefore, we train models for the (At 10) and

---

Table 1: Average end point (AEE) comparison with other methods for estimating optical flow on PHM datasets under At = 10 and At 20 settings. All methods use spike stream as input and trained SPIFT dataset in a supervised manner. The best results are marked in bold.
Method
Param.
Boll
Cook
Doll
Fan
Hand
Jump
Top
Mean
EV-FlowNet
53,43M
0.567
3.030
1.066
1,026
0.939
4.558
0.824
1.306
2.625
1.771
Spike-FlooWeet
13.04M
0.500
3.541
0.666
0.860
0.932
4.886
0.878
0.967
2.624
1.762
SCFlow
0.80M
0.671
1.651
1.190
0.266
0.298
1.692
0.120
1.030
2.166
1.009
RAFT
0.577
1.557
0.296
0.327
1.769
0.141
0.681
0.965
PWC-Net(variant)
0.57M
0.597
2.185
1.288
0.606
0.464
2.551
0.370
1.269
2.602
1.326
RAFT+comY
1044M
0.535
1.874
1.225
0.737
0.447
2.482
0.299
0.957
2.424
1.220
WWC-Nettvariat)+ooo
0.62M
0.511
1.585
0,959
0.187
0.245
1.874
0.077
0,833
2.114
0.932
USFIow((al)
1.0M
0.483
1.228
1.294
0.283
0.345
1.634
0.154
0.764
2.192
0.931
USFlow(pwe)
0.62M
0.430
1,556
0.787
0.210
0.226
1.615
0.105
0.646
2.111
0.854
EV-FlowNer
53,43M
1,051
5,536
1.721
2.057
1.867
8.820
1.803
2.193
5.061
3.345
Spike-FlowNet
13.04M
0.923
7.069
1.131
1,675
1.838
9.829
1.701
1.373
5.257
3.422
SCFlow
0.80M
1.157
3,430
2.205
0.507
0.578
4,018
0.267
1.922
4.327
2.046
20
RAFT
0.99M
1,004
3.378
2.059
0.460
0.561
3.707
0.257
1.416
4.250
1.904
PWC-Net(variant)
0.57M
1.321
4,493
2.601
2.206
1.083
5.654
1.159
2.320
5.143
2.887
RAFT+coRY
104M
2.977
0.533
0.622
0.287
4.313
1.818
WWCC-Netyariant)ecom
0.62M
0
3.198
1.624
0.799
0.476
4,030
0.273
1.424
4.324
1.892
USFlow(raft)
1,04M
0.792
2.734
1,918
0.448
0.569
2.601
0.256
1.231
4.293
1.649
USHoow(ww)
0.62M
0,807
3,075
1.613
0.370
0.377
3.663
0.168
1.216
4.216
1.723
Table 2: Average end point error (AEE) comparison with other methods for estimating optical flow on PHM datasets under At = 10 and At 20 settings. All methods use spike stream as input and trained SPIFT dataset in an unsupervised manner. The best results marked in bold.
Method
Jump
Poler
Top
PWC-Net(variant)
0.642
3,239
1.258
0.579
0.844
4.799
0.587
1.468
2.632
1.783
USFloo((we: Census)
0.640
3.549
0.675
0.815
1018
4.942
0.782
0.756
2.655
1.759
USHow(pwe, SSIM)
0.703
3,546
0.705
0,838
0,980
5.000
0.762
0.819
2.644
1.777
USFlow(pwe)
1.416
0.610
1.159
1.307
2 "WC--Netvariiant
1.480
6.682
1.419
1.407
1.452
9.307
0.905
1.790
5,580
3,336
USFlow(pwe, Consus)
7,049
1.199
L,588
9,854
1.441
1.393
5,284
3,412
USHowipww, SSIM)
1.152
6.882
1,273
1.071
1,260
9.778
3.185
1.534
4,855
3.446
USFlow(pwe)
3.978
2.782
0.873
4.748
2.180
2.413
(Ar = 20) settings separately as SCFlow 12). All training details are included in the appendix. More The color map used in visualization
details on the SSES dataset will be elaborated in Section 5.4 refers to Middlebury (2).
5.2 Comparison Results
Evaluation of Input Representation. To fully validate the effectiveness of the proposed input representation, we first train the model in supervised manner and compare it with other supervised baselines listed in the prior spike- based work, SCFlow [22 In more detail, apart from SCFlow, we compare our network with baselines in west- optical flow, i.e. EV-FlowNet (44 and Spike-FlowNet [9] We also compare our network with frame-based optical flow network, i.e. RAFT [31. Note that these two frame-based networks are lightweight versions as illustrated Section 4.1 and we implement our method on both networks, denoted as USFlow(raft) and (pw)..(pwe). All the methods in Table arre only fed spike streams inputs.
[32 and PWC-Net(varinee
As illustrated in Table I our input representation indeed can improve the performance on top of frame-base backbones. It demonstrates the necessity of directly pre-pros pre-proeessing operations on spike stream information. In addition, USFlow(pwe) achieves the best mean AEE of 0.854 in At 10 setting and USFlow(raft) gets the best mean AEE of 1.649 in At 20 setting, which gets 15% % and 19% error reduction from the best prior deep network, SCFlow, respectively. Note that the mean AEE value is averaged over nine scenes. Meanwhile, our input representation has the least parameters compared to other methods. The parameter size of input representation SCFlow is 0.23M and it in our USFlow is only 0.05M. With the lightweight representation module, the computational time of USFlow for inferring optical flow between two timestamps on 3090 GPU 90.6ms. It gets 61.8% computational time reduction from SCFlow.
7

---

Image
superried
PWC uxsuperiied
USP unspperreed
TumpiSeme
Figure 3: Qualitative results of the evaluation the PHM dataset.
Spika
U.S L.SFlow(purr)-uuuee suppeppeeeee--eeiiiiie
USFlow((ww)
Figure 4: Qualitative results on real scenes. USFlow(pwe) fine-tuned the training set of street scenes.
In order to verify that the performance boost is not brought by the number of parameters increas- ing, we change the dilated convolution to normal convolution with the same input size and feature channel, denoted RAFT+conv PWC-Net(varrann))ooom We found that blindly increasing the number of parameters does not make any sense. Normal convolution only provides a limited perfor- mance improvement. Therefore, claim that dilated convolutions extract salient information more effectively from spike streams. Table 2 shows comparison between PWC-Net( (WC-Net(variant) and USFlow(pwe), indicating that representation also shows superiority with our unsupervised loss.
Evaluation of Unsupervised Loss. Note that only build USFlow OWC-Net(variang) for unsupervised evaluation due to the similar performance of two backbones in Table I Since the metric for measuring appearance similarity is critical for any unsupervised optical flow technique [18] in the frame-based domain, we compare with some metrics, i.e. the structural similarity index (SSIM) and the Census loss. As illustrated in Table [ our proposed unsupervised loss help the model achieve significant performance improvements. Components of our loss are analyzed in Section 5.3
Qualitative Results. Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the PHM dataset visualized in Figure 3 Note that PWC(variant) and USFlow(pwe) trained unsupervised. USFlow(pwe) can predict more detailed flow than PWC-Net(varicg in many regions. However, there still exists a performance gap between the unsupervised method and the supervised method. Moreover, as for supervised methods, the directions of predicted flows (viewed in color) of USFlow(pwe) are closer to the ground truth than SCFlow, especially in object edges.
Fine-tuned Results. We collect some real data in street scenes and split them into training and evaluation set. Due to the advantage of unsupervised learning over supervised learning, we can fine-tune the unsupervised model on real data, which has no ground truth, to bridge the domain gap. The fine-tuned model has achieved better qualitative results than the supervised model trained on SPIFT in the evaluation set of street scenes. Parts of qualitative results can be found in Figure More clarifications placed in the appendix.

---

5.3 Ablation Study
Dynamic Timing Representation. We do ablation studies on dynamic timing representation in both supervised and unsupervised manners. Table g again verifies that dilated convolu- tions can effectively extract salient information to build promising input representation. More analysis regarding dilated convolutions are in the appendix. Though it seems that layer attention can only improve the performance marginally, we can find out this part makes the training process more stable and faster as illustrated in Figure O
KGB
Ground Trush
PM Cinusiaat)
USa USHloggred
Figure 5: Qualitative results on the SSES dataset. The positions of vehicles and pedestrians highlighted by black boxes.
Unsupervised Loss. Since estimating the light intensity from spike streams is the core of our unsupervised loss, do ablation studies it. As shown in Table 4 the mean AEE of the experiment (A) is higher than that of (B). The reason is there are fewer high-speed motion regions than low-speed motion regions the PHM dataset. Compared with experiments (A) and (B), our unsupervised loss can handle regions with different motion speeds and get the best performance.
5.4 SSES Dataset
Based on the CARLA 91. we build a synthetic dataset for extreme scenarios autonomous driving. CARLA is open-source simulator for autonomous driving research, which provides open digital assets (urban layouts, buildings, vehicles) to build specific scenarios. Moreover, the simulation platform supports flexible specifications of sensor suites, environmental conditions, and much more. In addition, CARLA can provide the ground truth of optical flow, instance segmentation, and depth.
In the proposed dataset SSES, design ten extreme scenarios, mainly focusing traffic accidents caused by violating traffic rules or vision-blind areas. We also include various street scenes, back- ground vehicles, and weather conditions to make scenes more diverse. The demonstration of sample cases and more descriptions of the extreme scenarios are in the appendix.
In all scenarios, the speed setting range is 10 ~ 16 m/s for cars, 5 ~ 8 m/s for pedestrians and bicycles, and the frame rates for rendered RGB frames and spike frames are 500 fps and 40K fps respectively. Regarding the generation of spike frames, we first increase the frame rate of RGB frames 40K fps through flow-based interpolation method and then generate spikes by treating pixel value as light intensity and simulating the integrate-and-Ho mechanism [40]. Note that the ground truth of optical flow obtained from time aligned with RGB frames. The sequence duration about 0.5 1.5s.
Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the SSES dataset are visualized in Figure 5 USFlow(pwe) can successfully predict the optical flow in regions where vehicles and pedestrians exist (highlighted by black boxes), which can help decision decision-making in autonomous driving. Table 5 shows the quantitative evaluation of the SSES dataset.
6 Conclusions
We propose an unsupervised method for learning optical flow from continuous spike streams. Specifi- cally, we design a dynamic timing representation for spike streams. We also propose unsupervised loss function a spike-based manner. Moreover, we simulate extreme scenarios in autonomous driving and propose a validation dataset SSES for testing the robustness of optical flow estima- tion in high-speed scenes. Experiment results show that our USFlow achieves the ofate-of-the-ar performance on PHM. SSES, and real data.

---

Ablation studies of our design choices for input representation. The present value is the mean AFE, averaged over nine scenes, PHM. The best results marked in bold.
Table 3:
4.0
USR lewipwd
3.5
mean AEE
Supervised
Unsupervised
TMR LA
At 10 At 20 At
10 &
20
X
0.943
1.797
1.783
3.336
5
10
15
20 epoch
25
30
35
40
USFlow
/
0.854 x
0.888
1,699
1.355
2.461
(pwe)
/
1.723
Figure 6: The mean AEE on PHM dataset during supervised training. The shadowed area is enclosed by the min and max values of three training runs, and the solid line in the middle is the mean value.
USFlow (raft)
x
0.965
1.902
0.952
1.659
/
0.931
1.649
Table 4: Ablation studies of unsupervised Table 5: Mean AEE averaged over ten scenes The present value is the mean AEE over the PHM on SSES dataset. All models are trained on the dataset. Best in bold.
SPIFT dataset.
Setting of Experiment
At = 10
At
20
USFlow(pwo) USFlow(pwe)) supervised
Remove Ty from
2,071
3.260
unsupervised
USHlow (pwe) (R) Remove 7 from our loss
1.484
2.563
mean AEE
2.967
3.122
(C) our unsupervised loss
1.307
2.413
At 20
2.234
3.130
The characteristics of spike streams generated extremely dark scenes are quite different from those in bright scenes, so the length of the time window in the unsupervised loss may need be reset during fine-tuning. We plan to extend our method address this issue in future work.
Limitations.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 22127807, 62072009, and in part by the National Key R&D Program of China under Grant 202IYFF0900501.
References
Lakshmi Aunamalai. Vignesh Ramanathan, and Chetan Singh Thakur: Event-LSTM: An unsupervised and asynchronous learning-basor representation for data. IEEE Robotics and Automation Letters (RAL), 7(2):6788 4685, 2022.
Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision (IJCV), 92(1):1-31, 2011.
[3] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/Kanade meets Hom/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision (IJCV), 61:211-231 2005.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. A differentiable recurrent surface for asynchronous event-based data. In European Conference on Computer Vision (ECCV), pages 136-152, 2020.
[5] Javier Cuadrado, Ulysse Rançon, Benoit Cottercau, Francisco Barranco, and Timothée Masquelier. Optical flow estimation from event-based cameras and spiking neural networks, Frontiers in Neuroscience, 17:1160034, 2023.
[6] Ziluo Ding, Rui Zhao, Jiyuan Zhang. Tianxiao Gao, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang. Spatio- temporal recurrent networks for event event-based optical flow estimation. AAAI Conference on Artificial Intelligence (AAAI), 2021.
10

---

[7] Yanchen Dong, Jing Zhao. Ruiqin Xiong, and Tiejun Huang. 3D residual interpolation for spike camera demosaicing. In IEEE International Conference on Image Processing (ICIP). pages 1461-1465, 2022.
[8] Yanchen Dong, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. High-speed scene reconstruction from low- light spike streams. In IEEE International Conference on Visual Communications and Image Processing (VCIP), pages 1-5, 2022.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladien Koltun. CARLA: An open urban driving simulator. In Annual Conference on Robot Learning (CoRL), pages 1-16, 2017.
[10] Daniel Gehrig, Antonio Loquercio, Konstantinos Derpanis, and Davide Scaramuzza. End-to to-ed learning of representations for asynchronous event-based data, In IEEE/CVF International Conference on Computer Vision (ICCV), pages 5633-5643, 2019.
[11] Jesse Hagenaars, Federico Paredes-Vallés and Guido De Croon, Self-supervised learning of event-based optical flow with spiking neural networks. In Annual Conference Neural Information Processing Systems (NeurlPS), 2021.
[12] Liwen Hu, Rui Zhao, Ziluo Ding, Ruiqin Xiong, Lei Ma, and Tiejun Huang. SCFlow: Optical flow estimation for spiking camera. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[13] Tak- Wai Hui, Xiaoou Tang, and Chen Change Loy. LiteFlowNet: A lightweight convolutional neural network for optical flow estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8981-8989, 2018.
[14] Yu Jason, Adam W Harley, and Konstantinos G Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In European Conference on Computer Vision (ECCV), pages 10, 2016.
[15] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang. Erik Learned-Miller and Jan Kautz. Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9000-9008. 2018
[16] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In JEEF/CVE International Conference on Computer Vision (ICCV), pages 9772-9781, 2021.
[17] Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning optical flow from a few matches. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (6592-16600) 2021.
[18] Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In European Conference Computer Vision (ECCV), pages 557-572, 2020.
[19] Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Danillidis, and Kaushik Roy. Spike-FlowNet: event-based optical flow estimation with energy-efficient hybrid neural networks. In European Conference on Computer Vision (ECCV), pages 366-382, 2020.
[20] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang. Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: reliable supervision from transformations for unsupervised optical flow estimation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (C (CVPR), pages 6489-666 64996, 2020.
[21] Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised leaminng of optical flow. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4571-45800 2019.
[22] Yawen Lu, Qifan Wang, Siqi Ma, Tong Geng, Yingjie Victor Chen, Huaijin Chen, and Dongfang Liu. TransFlow: Transformer flow learner. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18063-18073, 2023.
[23] Kumming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang, and Jian Sun. UPflow: Upsampling pyramid for unsupervised optical flow learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1045-1054, 2021.
[24] Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event- based vision meets deep learning on steering prediction for self delf-driving cars. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages $419-5427, 2018
11

---

[25] Simon Meister, Junhwa Hur. and Stefan Roth. Unflow: Unsupervised learning of optical flow with bidirectional census loss. In AAAI Conference Artificial Intelligence (AAAI), volume 32, 2018.
[26] Federico Paredes-Vallee Kirk YW Scheper, Christophe De Wagter, and Guido CHE de Croon. Taming contrast maximization for learning sequential. low-latency, event-based optical flow. arXiv preprint arXiv:2303.05214, 2023.
[27] Henri Rehecq, Timo Horstschaefer, and Davide Scaramuzza, Real-time visual-inertial odometry for event cameras using keyframe- based nonlinear optimization. 2017.
[28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Ner: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assinn Intervention (MICCAI), pages 234 241, 2015.
[29] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7731-1740, 2018.
[30] Austin Stone. Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. SMURF: Self- teaching multi-frame unsupervised raft with full-image warping. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3887-3896, 2021.
[31] Deqing Sun, Xiaodong Yang. Ming- Liu. and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8934-8944 2018.
[32] Zachary Teed and Jia Deng. RAFT: Recurrent all pairs field transforms for optical flow, In European Conference on Computer Vision (ECCV), pages 402 419, 2020.
[33] Sai Vemprala, Sami Mian, and Ashish Kapoor. Representation learning for event-based visuomotor policies. Annual Conference on Neural Information Processing Systems (NeurIPS), 34, 2021.
[34] Jianyuan Wang, Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacement- invariant matching cost learning for accurate optical flow estimation. Annual Conference on Neural Information Processing Systems (NeurIPS), 33:15220-15231 2020.
[35] Lujie Xia, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. SVFI: Spiking-based video frame interpolation for high-speed motion. In AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 2910 19-0-29,8, 2023.
[36] Haofei Xu, Jiaolong Yang. Jianfei Cai, Juyong Zhang, and Xin Tong. High-resolution optical flow from Id attention and correlation. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 10498-10507, 2021.
[37] Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Annual Conference on Neural Information Processing Systems (NeurIPS), 5:12, 2019.
[38] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic- static bootstrapping. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2826 2066-2836,622 2021.
[39] Feihu Zhang, Oliver J Woodford, Victor Adrian Prisacariu, and Philip HS Torr. Separable Flow: Learning motion cost volumes for optical flow estimation. In JEEL/CVF International Conference Computer Vision (ICCV), pages 10807-1080 10817, 2021.
[40] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11996- 12005, 2021.
[41] Shengyu Zhao, Yilun Sheng, Yue Dong, Eric Chang, Yan Xu, et al. MaskFlownet: Asymmetric feature matching with learnable occlusion mask. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6278 6287, 2020.
[42] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6358 6358-6367, 2021.
[43] Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao. Motion-attentiv, transition for zero-shot video object segmentation. In AAAI Conference on Artificial Intelligence (AAAI). pages 13066- 13073, 2020.
12

---

[44] Alex Zihao Zhu, Nikolay Atanasov, and Kostas Danillidis. Event-based feature tracking with probabilistic data association. In IEEE International Conference on Robotics and Automation (ICRA), pages 4465-4470, 2017.
[45] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. EV-FlowNet: supe-sspe optical flow estimation for event-based cameras. In Robotics: Science and Systems (RSS), 2018.
[46] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 989 997, 2019.
[47] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian, retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432-1437, 2019.
[48] Lin Zhu, Siwei Dong, Jianing Li. Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruc- tion via spiking neural model. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1438-1446, 2020.
[49] Xizhou Zhu, Yujie Wang. Jifeng Dai, Lu Yuan, and Yichen Wei. Flow guided feature aggregation for vidco object detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 408 417, 2017.
13


Reviews:
[{"review_id": "JNxtbIhAWM", "reviewer": "Reviewer_fVdD", "summary": "The paper introduces a method for optical flow estimaiton for spike cameras. Because of the high temporal resolution of the camera, a preprocessing step based on temporal dilated convolution and attention layers is used to automatically select the best temporal scale for a given sequence.\nThe authors also introduce a unsupervised loss based on the reconstruction of intensity values form spike data combined with the photometric loss.\n", "questions": "please refer to \"weaknesess\" section", "limitations": "The authors could add a general discussion of advantages and limitations of the spike camera and of optical flow from spike cameras.\nFor example any the challenge related to data rate, computational time and power consumption of these devices.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are the temporal dilated convolution preprocessing that allow for automatic temporal scale selection, as well as the unsupervised loss.\nThe description of the method is clear and the experimental section complete, including ablation studies and comparision with previous works.\n\n", "weaknesses": "The paper does not address the question of latency and computational time, which are key aspect of the spike camera.\nAlso, since frames are available in the simulate datasets it would have been interesting to compare the accuracy of the proposed method on standard rgb frames, to quantify the benefits of using a high temporal resolution sensor for optical flow estimation.\nFinally, for people not familiar with the spike camera, it would be beneficial a more complete introduction to this type of sensor. There is no reference in the paper to any publications explaining what a spike camera is, it's HW implementation and carachteristics.   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and reviews.\n\n**Q1.The latency and computational time of the proposed method.**\\\nIn USFlow, to estimate the optical flow from timestamp t0 to timestamp t1, a spike stream should be collected containing (100 + dt) spike frames from t0, where dt represents the number of spike frames between t0 and t1. For instance, using a first-generation spike camera with a frequency of 20 kHz, when dt is set to 10, the necessary time to collect the spike stream for optical flow estimation would be approximately 5.5 ms. During inference, when utilizing a 3090 Ti GPU to estimate the optical flow between two timestamps, the computational time of our USFlow is 90.6 ms, whereas the computational time of the comparative method SCFlow is 236.9 ms.\n\n**Q2.Comparing the accuracy of the proposed method on standard rgb frames to show the benefits of using a high temporal resolution sensor for optical flow estimation.**\\\nTypical RGB cameras usually operate at frequencies between 30Hz and 120Hz. The spike stream output by the spike camera takes advantage of its ultra-high temporal resolution, which can compensate for the deficiencies of low-frame-rate RGB frame sequences in optical flow estimation task. Taking a 120Hz RGB camera as an example, within a time span of (1/120)s, the RGB camera can only capture two RGB frames. Estimating nonlinear intricate motion between these two adjacent RGB frames by frame-based methods is often challenging. However, the spike stream, with its inherent advantage of extremely high temporal resolution, is capable of continuously recording dynamic processes. Thence, spike stream can be used to estimate nonlinear and non-uniform motion. Specifically, the (1/120)s can be divided into multiple short time spans. The motion within each short time span can be estimated on the corresponding spike stream. Eventually, by combining the motion information within these short time spans, the complete intricate motion within the (1/120)s can be estimated.\n\n**Q3.For people not familiar with the spike camera, it would be beneficial a more complete introduction to this type of sensor.**\\\nIn Section 3.1, we introduce the fundamental working mechanism of the spike camera. Some previously works, such as [a] and [b], provide a more comprehensive introduction to the spike camera. Thank you for your suggestions, we will cite these references containing detailed introductions of the spike camera.\\\n[a] J. Zhao, et al. Reconstructing Clear Image for High-Speed Motion Scene With a Retina-Inspired Spike Camera. TCI 2022.\\\n[b] T. Huang, et al. 1000x Faster Camera and Machine Vision with Ordinary Devices. Engineering 2022."}, {"review_id": "EQ5D6kNldi", "reviewer": "Reviewer_eenq", "summary": "This paper works on the optical flow estimation problem, especially in the unsupervised setting, for high-frequency spike camera inputs. Specifically, a dynamic timing representation module and a spike-based unsupervised loss are proposed to improve performance. A new synthetic dataset, namely SSES, is created to test extreme scenes in autonomous driving. Experiment results show the proposed method has achieved state-of-the-art performance compared with previous methods.\n", "questions": "1. If I understand it correctly, the method uses the predicted optical flow to determine whether each pixel belongs to a low-speed or high-speed region and then uses different ways to approximate light intensities $\\tilde{I}$. At the starting iterations of training, the predictions could be totally off, so it is likely that the method will pick a wrong way to approximate intensities. Did you try to tackle this issue?\n2. The data used in Fig 4 do not have ground-truth labels, right? Only qualitative evaluation is possible, so this evaluation may be weak. \n3. There is also a large part of supervised results, so stating \"unsupervised\" in the title could be misleading.\n4. I am curious how a normal RGB-based optical flow network would perform as a comparison. For example, a naïve approach is to first convert spike inputs to RGB and then apply a state-of-the-art RGB-based optical flow network of similar size (parameters, memory cost). What would the results be like? This could be a baseline to strengthen the motivation of developing optical flow networks specific to spike camera inputs.\n\n\n\nAdditional comments:\n1. Grammatical errors: run-on sentences (Line 103-105, line 140-141, Line 163-165, Line 310-311).\n2. Eq 1: The \"mod\" operator is traditionally only defined for integer division, but the value here is clearly continuous numbers, so it is better to define \"mod\" in the text in the continuous context.\n3. Chapter 3.1: There are some confusions here. (1) I assume $A(\\mathbf x, t)$ is continuous, but Line 119 says the output stream is binary, so there is missing gap in between that need explanations. Maybe adding a definition, something like $S_t^N(\\mathbf x) = \\mathbb I(\\sup_{\\tau\\in[t-\\Delta t, t+\\Delta t]} A(\\mathbf x, \\tau) =\\theta)$ will make it clear. (2) Line 113-114 say the camera \"fires a spike\" when the cumulative electrons hits the threshold, so I assume it is asynchronous for each pixel, but then Line 117-118 say the camera \"fires a spike\" at time T, which is synchronous and periodic. Maybe you could change the expression here to avoid confusion.\n4. Line 130: \"officially\" -> \"official\".\n5. Line 135: \"dependence\" -> \"dependency\".\n6. Fig 2: Text is too small. It is better to make text size close to that of the main paper so that readers can read the figure clearly without needing to zoom in.\n7. Line 147: It is better to add that your convolution is 1D on the time dimension to avoid confusion.\n8. Line 182-183: Does your method guarantee $\\omega_s + \\omega_l = 1$? Eq 6: Does $\\sum_k\\omega_k=1$? If so, state so explicitly.\n9. Eq 7 could be merged into the text. It does not have to an equation.\n10. Line 175-184: What do you mean by \"speed\"? The frequency of camera readings, or fast motion? Please clarify.\n11. Line 207: The notation $\\omega_s, \\omega_l, \\omega_{k=1}, \\omega_{k=2}$ is messy. Try not to use both textual abbreviations (\"s\", \"l\") and values ($k$) as subscripts for the same letter $\\omega$. Use other letters like $\\alpha_s, \\alpha_l$, $\\beta_k$ instead; or, you could do superscripts with parentheses $\\omega^{(s)}, \\omega^{(l)}, \\omega_k$.\n12. Line 209: \"approximate\" -> \"approximated\".\n13. Line 212: define or cite \"Charbonnier loss\"; Line 213: define or cite \"smoothness loss\" specifically. There are many ways to define \"smoothness\".\n14. Table 1 and 2: It is better to add citations into the table, so we know which data are generated from your experiments and which data are cited from other papers. The current table assumes that all data are credited to yourself. \n15. Table 1 and 2: I recommend adding a vertical line between the \"Top\" and \"Mean\" columns, so we know \"Top\" is the name of the scene instead of the top error across all scenes.\n16. Line 277: \"through\" -> \"though\".\n17. Line 284: \"Figure 4\" -> \"Table 4\".", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated by introducing the importance of optical flow estimation for high-frequency inputs collected from latest developed sensors like spike cameras. This topic has not been extensively studied so far.\n2. The paper is overall clear and well-structured.\n3. The proposed method is reasonable. Explanations on each module design are provided.\n4. The results are compelling.\n", "weaknesses": "1. There are still some confusions in the text or figures that need to be clarified. See additional comments below.\n2. It will be better to polish the language of the paper before publication. Correct grammatical errors and use more formal language. Avoid informal expressions like \"a bunch of\" (Line 138), \"won't\" (Line 140), etc. See additional comments below.\n3. Authors are encouraged to share their code and generated datasets for better reproducibility. Such plan has not been declared in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and reviews.\n\n**Q1. Do authors have plan to share their code and generated datasets ?**\\\nWe will release the code, models and datasets.\n\n**Q2. The problem about the network may choose wrong way to approximate light intensity at the starting iterations of training.**\\\nIn high-speed regions, the light intensity estimated by the method which is suitable for low-speed regions will be affected by motion blur. In low-speed regions, the light intensity estimated by the method which is suitable for high-speed regions will contain noise. While the quality of these light intensities estimated in the wrong ways is not good, they are still sufficient for initiating the training of the network.\n\n**Q3. The real spike streams output by spike camera lack ground truth.**\\\nThere are two approaches to obtaining the ground truth of optical flow from real-world data: 1. The first method is used for indoor scenes. The optical flow can be obtained with the assistance of ultraviolet (UV) light. 2. The second method is used for outdoor scenes. It uses LiDAR to collect optical flow. However, this method will introduce calibration and measurement errors, and the collected optical flow field is sparse. Obtaining the ground truth of optical flow in outdoor scenes is challenging. Therefore, we show the performance of USFlow by comparing the visualization results of USFlow and other methods on real spike data.\n\n**Q4. The paper shows some supervised results, so stating \"unsupervised\" in the title could be misleading.**\\\nIn Table 1, the comparative methods are all supervised methods. Therefore, in Table 1, we also employ supervised training in order to compare the performance of the representation module. The method presented in Table 1 is not the final method proposed in this paper. After validating the effectiveness of our representation module in Table 1, we proceed to Table 2 where our USFlow is trained by unsupervised learning. Our proposed method includes a novel unsupervised loss function, and the USFlow trained using unsupervised learning is our proposed method in this paper.\n\n**Q5. First convert spike inputs to RGB and then apply SOTA RGB-based optical flow estimation networks. What would the results be like?**\\\nReconstructing spike data into RGB images is currently a developing research topic, involving intricate considerations. Simple principle-based methods for reconstructing RGB images from spike data often have issues with noise and blurriness. Even the state-of-the-art frame-based optical flow estimation methods use such poor-quality reconstructed RGB images as input, the accuracy of the estimated optical flow remains challenging. A well-designed reconstruction method is highly complex, such as Zhao's Spk2imgnet, which needs to consider many complex factors, and the reconstruction process involves a motion analysis module. First reconstructing the spike data into RGB images and then estimating the optical flow is a kind of solution with a large space for exploration. Our proposed method which directly estimates optical flow on spike data belongs to another kind of solution.\n\n**Q6. In L175 and L184, what do you mean by \"speed\"?**\\\nThe \"speed\" at here means motion speed.\n\n**Q7. \"Charbonnier loss\" and \"smoothness loss\".**\\\nThe Charbonnier loss we use is the basic version: $\\\\rho(x)=(x^2+\\epsilon^2)^r\\$. The smoothness loss we use is also a simple version which is defined as: $\\mathcal{L}{\\rm smooth}\\left( f, f' \\right) = \\frac{1}{HW} \\sum{\\mathbf{x}} \\vert \\nabla f (\\mathbf{x}) \\vert + \\vert \\nabla f' (\\mathbf{x}) \\vert$, where $f$ and $f'$ are bidirectional estimated optical flow, $( \\nabla )$ is the difference operator, H is the height and W is the width of the optical flow. We will add definitions about them.\n\n**Q8. Additional comments.**\\\nThank you for your concern. We will correct grammatical errors, add citations in tables and clarify confusion."}, {"review_id": "VJSWv3NC0H", "reviewer": "Reviewer_2gct", "summary": "This paper propose an unsupervised learning framework for spike-based optical flow estimation, which is mainly developed for spike input representation and spike loss function. In general, I think the method has merit, but the experiment is unconvincing in its lack of clarity. ", "questions": "1. L158, Blindly fusing the output of all layers may impair the learning process. It is desirable to have experiments to support this belief. \n4. L170 seems duplicate to L122.\n5. I suggest to complete the experiments of raft backbone model.", "limitations": "1. L132, advantage of neuromorphic vision is low latency. However, the authors do not present the runtime analysis of the proposed method.\n2. See weaknesses above.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Propose a lightweight end-to-end spike data representation with the function of a dynamic time window.\n2. Propose a two-stage unsupervised loss to model light intensity in regions with different motion speeds.", "weaknesses": "1. In Fig. 5, there are significant quantization/discontinuities in the optical flow gt of the SSES dataset, concentrated in the ground area. To my knowledge, this is not present in other optical flow datasets. The reliability of the optical flow ground-truth is a concern, please provide an explanation. \n2. In Fig. 4 and supplementary video, the results of these methods using only spike as input look bad. The major problem is that the results are concentrated only the textures and are not consistent for the same object., which cannot be called ``dense`` optical flow. To my knowledge, this phenomenon is not present in unsupervised two-frame based optical flow  estimation methods. I suggest comparing the outputs of two-frame based methods and analyzing the spike-based failure cases. \n3. The performance of using spike cameras for optical flow estimation is of concern. In L27, spike camera can record details of high-speed moving objects, it has enormous potential for estimating more accurate optical flow in high-speed scenes. But comparing the state-of-the-art optical flow estimation methods using image cameras, I believe this claim is not convincing. The visualization results of existing spike-based methods, including the authors', are significantly worse than existing image-based methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and reviews.\n\n**Q1. About the reliability of the optical flow ground-truth.**\\\nThe SSES dataset is a verification dataset containing various corner cases in the autonomous driving field and it is generated by CARLA. CARLA is an open-source simulator for autonomous driving research and it can provide the ground truth of optical flow. The proposed SSES dataset is used to verify the effectiveness of the methods in extreme cases. However, the temporal resolution of spike data is ultra-high, which results in relatively small motions between spike frames. Therefore, although the generated ground truth is a smooth optical flow field, it is limited by the quantization accuracy of the CARLA, and quantization appears in the visualization of the ground truth. For high-speed motion scenes, collecting optical flow by using LiDAR can circumvent the need for using simulators. However, the frequency of LiDAR is relatively low, and the process of calibration and measurement can introduce larger errors to the ground truth.\n\n**Q2. The performance of spike-based optical flow estimation methods is not better than frame-based methods.**\\\nThe frequency of spike cameras can reach up to 40 kHz, so spike cameras can continuously record high-speed motion scenes. The spike camera serves to compensate for the drawbacks of traditional cameras in high-speed motion scenes, where information loss arises from insufficient frame rates. If there is complex nonlinear and non-uniform motion between two adjacent RGB frames, the common frame-based methods usually approximate the complex motion as linear motion. However, with the help of spike data containing continuous motion information, the complex motion can be estimated. Specifically, the spike data containing the information of complex motion is divided into multiple spike data with shorter data lengths. Then corresponding small motions can be estimated from these spike data with shorter data lengths. Finally, all these small motions are merged into a complete complex nonlinear and non-uniform motion.\n\n**Q3. Questions about the quality of the results shown in Figure4.**\\\nDirectly estimating optical flow on spike data using unsupervised learning is a new and difficult topic. It is very challenging to infer light intensity from spike data, because it is necessary to consider many factors, such as the brightness of scenes and the motion speed of different objects. Thence, directly estimating optical flow on spike data is more challenging than estimating optical flow on RGB images. The data corresponding to the Figure 4 is captured by a spike camera in the real-world and it only contains spike data. Limited by the frame rate of traditional camera, we don’t have RGB frames corresponding to these real spike data. The scenes on the left column of Figure 4 are reconstructed by a professional reconstruction method, and these reconstructed RGB images contain more noise than real RGB images. When these reconstructed RGB images are used in frame-based methods, the accuracy of estimation will be affected. Thence, the real spike data corresponding to Figure 4 cannot be used to compare the performance of spike-based methods and frame-based methods. We just reconstruct the scenes to tell the readers what contents are in the scenes. In the practical applications of spike cameras, there is only spike data can be used.  As shown in Figure 4, the proposed method shows significant advantages over other spike-based methods, and it also promotes the exploration of unsupervised learning in spike-based optical flow estimation task. For spike-based unsupervised methods, there is still space for exploration. We may further study this issue in our future works.     \n\n**Q4. It is desirable to have experiments to support L58.**\\\nThe \"blindly fusing\" here means not using layer attention, but simply averaging the output of all layers. In the ablation study on the layer attention module, simply averaging all layer outputs is used to replace the layer attention module. As shown by the orange curve in **Figure 6**, it can be seen that simply averaging all layer outputs does impair the learning process.\n\n**Q5. L170 seems duplicate to L122.**\\\nIn order to start the discussion of unsupervised loss function design, L170 briefly recalls the problem setting and clarifies again that the proposed method doesn’t need ground truth in training. Thank you for your advice, we will simplify the expression.\n\n**Q6. Suggest to complete the experiments of raft backbone model.**\\\nWe test the USFlow(raft) which is trained by  unsupervised learning on PHM dataset, and AEEs are shown below. As indicated by Table 2, it can be observed that both USFlow(raft) and USFlow(pwc) show the similar performance.\n|  $\\Delta$ t |  method  |  Ball  |  Cook  |  Dice  |  Doll  |  Fan  |  Hand  |  Jump  |  Poker  |  Top  |  Mean  |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| $\\Delta$ t=10 | USFlow(raft) | 0.760 | 1.902 | 1.923 | 0.631 | 0.724 | 2.439 | 0.490 | 1.217 | 2.556 | 1.405 |\n| $\\Delta$ t=20 | USFlow(raft) | 1.249 | 3.507 | 2.980 | 0.964 | 1.112 | 3.440 | 0.624 | 2.216 | 4.628 | 2.303 |\n\n**Q7. About runtime.**\\\nDuring the process of designing the network, one of our goals is to shorten the runtime by designing a lightweight model. Estimating the optical flow between two timestamps on a 3090 Ti GPU, the runtime of our proposed USFlow is 90.6ms, and the runtime of the comparison method SCFlow is 236.9ms."}, {"review_id": "vPtYeQw1ND", "reviewer": "Reviewer_Hpzy", "summary": "The paper proposes a method for estimating optical flow from event streams captured with a spike camera. The approach is based on unsupervised learning and achieves good results in the comparison to other works in the SotA, including conventional and event-based approaches. Authors collect their own dataset and evaluate on it, specially, in an attempt to show the advantage of finely estimating for low and high-speed regions (although, this is not clearly demonstrated with the current experimental section). Finally, authors also include a section with an ablation study to assess the impact of the different parts of the model (loss terms) and the supervised vs. unsupervised learning approaches.", "questions": "- Regarding the difference between event and spike cameras. Authors should make clear what kind of data they are handling. Event cameras produce asynchronous events and in fact, one of the advantages of these cameras is the lack of need for reconstructing the dynamics of the scene due to their high temporal resolution. This is great e.g. for optical flow estimation. However, it seems that spike cameras produce frames at very high resolution (something closer to a high-speed camera). This could also be interesting but it carries the burden of analyzing a lot of full frames, which cannot be ignored as a drawback. I really think that, since they are comparing to both conventional and event cameras, this should be clarified as soon as possible in the paper. \nIn fact, this is mixed up in the paper, for example, at some point authors mention something about spike frames but then, they point out that pixels respond asynchronously (?)\n- When comparing to event camera processing, performance should be part of the comparison as well (or latency).\n- In the abstract, lns 12-14 do not make reference to the dataset (I think this should be added here as well)\n- In page 2, I believe the correct term is luminance (not illuminance)\n- In line 68, authors mention \"shows visually impressive performance\", what do you mean? Is that qualitative results? I guess authors mean that they are appealing? Are they only qualitative?\n- The sentence in Lns 107-108 seems incomplete\n- In Ln 182, are 40 and 100 ms? or frames? \n- In ln. 196 \"different numbers of intervals\" \n- Lsmooth is not defined\n- There are some recent works that use SNNs such as \"Optical flow estimation from event-based cameras and spiking neural networks\", from J. Cuadrado et al. that could help to improve the state of the art section.\n- Citations are weirdly formatted. Authors should make sure that they are using the conference format\n- In section 5.3 authors (unsupervised loss) authors mention Figure 4 when they mean Table 4.\n- What is the contribution of the LA module? According to Fig. 6 it seems it is not helping at all ... in such a case, why do authors keep it? is there another case in which the LA module does add a significant contribution?\n- At the end of section 5.4, authors discuss results on SSES dataset but they mention before that they are also doing some real-world sequences, where are the results on these sequences?", "limitations": "The work includes a limitations section that seems adequate", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ Authors prepare and collect a dataset with is always a hard but valuable task\n+ The unsupervised learning approach provides a valuable alternative for optical flow estimation and shows good results on the evaluated datasets\n+ It is unsupervised, which is another good keypoint to bear in mind\n+ The ablation study on the loss terms impact and the comparison between supervised vs. unsupervised are very valuable", "weaknesses": "- After reading the work, I don't think authors make clear the differences between spike and event cameras. There is a confusion all over the introduction and methodology and this makes me (and the reader) hesitate about the relevance of this work. \n- The work needs extensive English proofreading, some sentences are hard to understand.\n- The conclusions should not be a summary of the work\n- In the experimental section, authors compared on PHM and propose their own dataset but they are not comparing to the widely used MVSEC and DSEC datasets (that are the most common for event-based processing). This will improve not only completeness but also the comparison to other works in the state of the art. \n- From the paper and since authors propose an artificially generated dataset, it is not clear if 1) the spike camera is a real sensor and 2) in case it is, if they have access to real sequences recorded with it.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and reviews.\n\n**Q1. Authors do not make clear the differences between spike and event camera.**\\\nBoth spike camera and event camera have ultra-high frequency, and they can continuously record high-speed motions. However, the working mechanism of them are different. In the spike camera, when the accumulated photons reach a preset threshold, a spike will be fired. The event camera records the light intensity change information on the corresponding pixels whose light intensity changes exceed the threshold. Spike camera and event camera have their own application advantages. In the analysis of high-speed motion scenes, there are indeed some related event-based methods. The method proposed in this paper uses spike data output by spike camera to estimate optical flow and it belongs to the spike-based method.\n\n**Q2. Authors should make clear what kind of data they are handling. Spike camera carries the burden of analyzing a lot of full frames.**\\\nThe proposed method uses the spike data output by the spike camera to estimate optical flow. Spike camera and event camera have their own advantages. The event camera records motion information according to the relative changes in light intensity and outputs relatively sparse event streams. Thus, event camera has great application value in tasks such as motion estimation, and objects tracking. The spike camera outputs binary spike data at each pixel position according to photon accumulation. This full-pixel recording feature makes spike camera suitable for pixel-dense tasks, for example, high-speed motion scene reconstruction. When using a spike camera to reconstruct a high-speed motion scene, optical flow estimation is usually involved, so there is a need to estimate optical flow based on spike data. Thank you for your suggestions. We will clarify in the paper that spike camera records scene information at full resolution.\n\n**Q3. In the experimental section, authors compared on PHM and propose their own dataset but they are not comparing to the widely used MVSEC and DSEC datasets (that are the most common for event-based processing).**\\\nThis paper focuses on using spike data which is output by the spike camera to estimate optical flow. The input of the whole model is spike data. The MVSEC and DSEC datasets do not contain spike data, so these two datasets cannot be used as test datasets for the spike-based methods. \n\n**Q4. if the spike camera is a real sensor? In case it is, if they have access to real sequences recorded with it.**\\\nThe spike camera is a real camera. We use the first-generation spike camera (20 kHz, 250 pixels × 400 pixels) to collect a series of real spike data in real-world. In our paper, Figure 4 shows some test results on these real spike data. The SSES dataset is generated by CARLA simulator. We plan to release all new datasets used in our paper in the future.\n\n**Q5. When comparing to event camera processing, performance should be part of the comparison as well (or latency).**\\\nIt is not the goal of this paper to compare spike camera with event camera in optical flow estimation. Since spike camera applications usually involve motion estimation, this paper aims to study how to estimate more accurate optical flow from spike data which is output by spike camera.\n\n**Q6. In the abstract, lns 12-14 do not make reference to the dataset.**\\\nWe will make reference to PHM dataset here.\n\n**Q7. What does \"shows visually impressive performance\" mean?**\\\nIt means the visualization results of USFlow on real spike data are significantly better than other methods.\n\n**Q8. The sentence in Lns 107-108 seems incomplete.**\\\nAt the end of the subsection that introduces various types of representation works, Lns 107-108 is a preview of the proposed representation method.\n\n**Q9. In Ln 182, are 40 and 100 ms? or frames?**\\\n40 and 100 represent 40 spike frames and 100 spike frames, respectively.\n\n**Q10. What does \"different numbers of intervals\" mean?**\\\nThe interval is defined in Lns 186-188. Multiple intervals are spliced to form a time span, and we use multiple such time spans to infer light intensity. These time spans contain (2k-1) intervals, the k is a hyper-parameters in experiments.\n\n**Q11. Smooth loss function is not defined.**\\\nThe $L{smooth}$ used in our method is a basic version which is defined as: $\\mathcal{L}{\\rm smooth}\\left( f, f' \\right) = \\frac{1}{HW} \\sum_{\\mathbf{x}} \\vert \\nabla f (\\mathbf{x}) \\vert + \\vert \\nabla f' (\\mathbf{x}) \\vert$, where $f$ and $f'$ are bidirectional estimated optical flow, $( \\nabla )$ is the difference operator, H is the height and W is the width of the optical flow.\n\n**Q12. There are some recent works that use SNNs. These works could help to improve the state-of-the-art section.**\\\nThe proposed USFlow estimates optical flow based on spike data. Although the output of spike camera and event camera have different data structures, some event-based methods and SNN works may indeed be instructive. We will keep our attention on such methods.\n\n**Q13. What is the contribution of LA module?**\\\nAlthough the LA module cannot bring a huge improvement in performance, it can make the network converge faster and the training process more stable.\n\n**Q14. Where are the results on real spike data captured by the spike camera?**\\\nIn Figure 4, we show some results on real spike data. We capture these real spike data by a first-generation spike camera (20 kHz, 250 pixels × 400 pixels) from different angles.\n\n**Q15. Writing advice.**\\\nThanks for your concern. We will correct these issues."}, {"review_id": "fMZldAl8W6", "reviewer": "Reviewer_PCU5", "summary": "The paper titled \"Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera\" presents a novel approach for unsupervised optical flow estimation using spike cameras. The authors propose a method that leverages dynamic timing representation to estimate optical flow without the need for explicit supervision. They demonstrate the effectiveness of their approach through extensive experiments and comparisons with existing methods.", "questions": "1.How does the proposed method handle occlusions or complex scenes where multiple objects are moving simultaneously? Are there any specific limitations or challenges in such scenarios?\n2. How does the dynamic timing representation used in the spike camera contribute to the accuracy of optical flow estimation compared to traditional frame-based approaches?\n", "limitations": "The proposed method's performance and generalizability might be limited to specific datasets or conditions. It would be valuable to explore its effectiveness on a wider range of datasets, including more challenging and diverse scenarios.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a unique and innovative approach to unsupervised optical flow estimation. By utilizing spike cameras and dynamic timing representation, the authors provide a fresh perspective on tackling this problem. This can potentially open new avenues for research and development in the field.\n2. The paper presents the methodology in a clear and concise manner. The authors provide sufficient details and explanations, making it easier for readers to understand the technical aspects of their approach. They also include relevant diagrams and figures to aid in comprehension.", "weaknesses": "1. The paper lacks a thorough discussion of the limitations of the proposed method. While the authors demonstrate its effectiveness through experiments, it would be beneficial to address potential challenges or scenarios where the method might not perform optimally. This would provide a more comprehensive understanding of the approach.\n2. The authors primarily focus on benchmark datasets and quantitative evaluations, but they do not provide real-world examples or scenarios where their method could be applied. Including such examples would enhance the practical relevance of the research and highlight its potential impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and reviews.\n\n**Q1. Lack a thorough discussion of the limitations.**\\\nWe fix the data length of spike streams which participate in the calculation of our unsupervised loss function. In scenes with normal brightness, this fixed-length spike data can provide sufficient information for inferring light intensity. However, the intervals between spikes are relatively large in the ultra-dark scenes, and light intensity cannot be accurately inferred using the spike streams of this fixed data length. We note relevant examples in real-world data. For example, when estimating the optical flow for a dark car in the shadow of an overpass, we note that different positions of the car have different optical flows. In future work, we plan to perform interval analysis on the input spike data and dynamically select the appropriate-length spike data to participate in the unsupervised loss function according to the statistical distribution of the interval length.\n\n**Q2. The author does not provide real-world examples or scenarios where their method could be applied.**\\\nOur method is suitable for high-speed motion scenes with normal brightness in the real world. The Figure 4 shows the test results of USFlow and comparison methods on the real spike data captured by a spike camera in real-world. The shooting device we use is the first-generation spike camera with a spatial resolution of 250 × 400 and a frequency of 20 kHz. For these real spike data, there are more detailed descriptions in the supplementary material. However, in outdoor scenes, obtaining the ground truth of optical flow can only rely on LiDAR, and the ground truth collected by this way has measurement and calibration errors. We cannot collect the accurate ground truth in these real street scenes. We can only demonstrate the performance of our method by visually comparing it with other spike-based methods on these real spike data.\n\n**Q3. How to handle the occlusions or complex scenes where multiple objects are moving simultaneously? Are there any specific limitations or challenges?**\\\nThe spike camera has an ultra-high frequency, and it can continuously record high-speed motion scenes. In the application of the spike camera, complex motion is usually divided into multiple small motions along the time dimension. Note that the time span corresponding to each small motion is very small, so the occluded area changes very little in each small time span. Then the spike data within these short time spans are used to estimate the corresponding small motions respectively. Finally, all these estimated small motions make up the complete complex motion. Therefore, the influence of occlusion problem in spike-based methods is less than in frame-based methods, and we did not design a particular structure for this problem. The proposed method currently does not deal well with the problem of occlusion caused by the simultaneous movement of multiple objects. It addresses the main issues of the spike-based methods: the effective representation of spike data and the lack of sufficient datasets with ground truth.  In future works, we will consider designing a structure to handle the occlusion issue. \n\n**Q4. Compared to frame-based methods, how does the dynamic timing representation used in the spike camera contribute to the accuracy of optical flow estimation?**\\\nCompared with traditional cameras, spike camera has an ultra-high frequency and the ability to continuously record high-speed motion with a high temporal resolution. Given two adjacent RGB frames, frame-based methods usually model the motion between two RGB frames as linear motion. However, with the help of the continuous spike data between these two frames, the motion to be estimated can be divided into multiple small motions along the time dimension to estimate separately. These estimated small motions are then used to compose the complete motion. Therefore, nonlinear and non-uniform complex motions can be estimated by using spike data. For spike-based methods, how to represent the spike data is the key point because the data structure of spike data is different from the data structure of RGB frames. In spike representation, selecting an appropriate data length is pivotal. A too-long spike stream is not suitable for high-speed regions since time offset accumulation introduces more redundant information and a too-short cannot exhibit light intensity precisely with few binary data. The proposed Dynamic timing representation breaks the limitation of the fixed time window and it dynamically selects the spike data of the appropriate length according to the motion speed at each pixel position for representation. It improves the quality of representation and thus the accuracy of optical flow.\n\n**Q5. It would be valuable to explore its effectiveness on a wider range of datasets.**\\\nThe benchmark dataset used in the experiments for testing is the PHM dataset which is generated by a graphics-based simulator. It is the most common dataset in the spike-based optical flow estimation task. In other tasks, there are some spike datasets, in which the spike data is generated by simulating the working mechanism of the spike camera based on RGB frame sequences. The frame rates of these RGB frame sequences are much lower than that of spike camera, so the motion information between two adjacent RGB frames can only be generated by frame-based methods. Thus, there is usually a difference between the real spike data captured by the spike camera and the simulated spike data generated by algorithms simulating the spike camera working mechanism. In order to verify the effectiveness of the proposed method on a wider range of data, we use a spike camera to capture a series of real spike data from different angles in the real world. Then we test and compare the performances of methods on these real spike data. The visualization results are shown in Figure 4 of our paper."}]

Author rebuttal:

Meta-review: The paper introduces a novel approach to unsupervised optical flow estimation using spike cameras. The methodology is well-explained, and the unique dataset provided could potentially be valuable for the research community. The paper was evaluated by five reviewers. Out of the five, four gave weak accept ratings, while one provided a borderline rating without expressing strong objections to accepting the paper. The authors are encouraged to consider the reviewers' feedback and incorporate the suggested revisions in the final version of the paper.
Decision: Accept (poster)