Title: Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optical flow

unsupervised learning

spike camera
Abstract: Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method achieves $15\%$ and $19\%$ error reduction on PHM dataset compared to the best spike-based work, SCFlow, in $\Delta t=10$ and $\Delta t=20$ respectively, using the same settings as in previous works. The source code and dataset are available at \href{https://github.com/Bosserhead/USFlow}{https://github.com/Bosserhead/USFlow}.

Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera

Abstract

Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method achieves 15% and 19% error reduction on PHM dataset compared to the best spike-based work, SCFlow, in ∆ t = 10 and ∆ t = 20 respectively, using the same settings as in previous works. The source code and dataset are available at https://github.com/Bosserhead/USFlow.

1 Introduction

Optical flow is defined as the apparent motion of individual pixels on the image plane and is prevalent for being an auxiliary tool for various vision tasks, e.g. frame rate conversion [15], scene segmentation [43, 38], and object detection [49]. In high-speed scenes, the optical flow estimation may suffer from blurry images from low frame-rate traditional cameras. Obtaining the data with a device that can precisely record the continuous light intensity changes of the scene is the key to addressing this issue. Recently, neuromorphic cameras have been developed greatly, such as event camera and spike camera. They can record light intensity changes in high-speed scenes. Especially, for spike camera, each pixel of it responds independently to the accumulation of photons by generating asynchronous spikes. It records full visual details with an ultra-high temporal resolution (up to 40kHZ). With these features, spike camera has demonstrated superiority in handling some high-speed scenarios, such as video frame interpolation [35] and dynamic scene reconstruction [40, 42, 7, 8, 48].

Since spike camera can record details of high-speed moving objects, it has enormous potential for estimating more accurate optical flow in high-speed scenes. Considering that deep learning has

achieved remarkable success in frame-based optical flow estimation [31, 32], it seems reasonable to directly apply frame-based architecture in spike data. However, the data modality of spike stream output by spike camera is quite different from frame images. For each pixel in spike camera, a spike is fired and the accumulation is reset when photons accumulation at a pixel exceeds a set threshold. At each timestamp, the spike camera outputs a binary matrix, denoted as spike frame, representing the presence of spikes at all pixels. Previous work [12] utilizes spike stream as naïve input representation for one timestamp, which consists of a series of spike frames within a predefined time window. However, a too-long window can incur lots of misleading frames, while a short window is sensitive to noise and cannot provide enough information. Therefore, deliberate modifications are needed for input representation to extract salient information more flexibly and efficiently from spike streams before optical flow estimation architecture takes over.

In addition, the ground truth of optical flow is scarce in the real world, especially for high-speed scenes. To cope with the lack of labeled real-world data, it is necessary to study spike-based optical flow estimation in an unsupervised manner. As described above, the light intensity information is contained within the spike intervals. This difference in data characteristics makes it unreasonable to directly apply the frame-based unsupervised loss to spike streams. Therefore, the light intensity should first be extracted from spike streams in the spike-based unsupervised loss. This is also the core of constructing the illuminance consistency on spike streams. Moreover, we argue the field of autonomous driving is a good place to validate spike camera since it is suitable for high-speed scenes. In autonomous driving, it is nearly impossible to collect real data for complex, diverse, high-speed extreme scenarios, e.g. vehicle collisions and pedestrian-vehicle accidents. However, these scenarios are of great significance to improve the safety of this field and should be highlighted. Therefore, in order to verify that the spike-based algorithm can handle extreme scenarios, we propose a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as the SSES dataset.

In this paper, we propose an unsupervised method for spike-based optical flow estimation with dynamic timing representation, named USFlow. In our unsupervised loss, we propose two strategies, multi-interval-based and multi-time-window-based, to estimate light intensity in regions with different motion speeds. The estimated optical flow is utilized to distinguish regions with different motion speeds and generates corresponding weights for light intensity fusion. Then the final approximate light intensity participates in loss calculation. As for addressing the fixed time window issue, there is a way that apply a dynamic time window to the different spike streams. To this end, we propose Temporal Multi-dilated Representation(TMR) for spike streams. In more detail, we apply multi-layer dilated convolutions to operate on the temporal dimension of spike streams. Multi-layer dilated convolutions enable the network to have different receptive fields and each layer can be regarded as summarizing spike stream with one different time window. We also design a Layer Attention(LA) module to extract salient features and filter the redundant ones.

Following the settings in previous works [12], we train our method on SPIFT [12] dataset and evaluate it on PHM [12] and our proposed SSES datasets. We demonstrate its superior generalization ability in different scenarios. Results show that USFlow outperforms all the existing state-of-the-art methods qualitatively and quantitatively. USFlow shows visually impressive performance on real-world data.

2 Related Works

Deep Learning in Optical Flow Estimation. Frame-based optical flow estimation is a classical task in the computer vision area through the years and has been solved well. PWC-Net[31] and Liteflownet[13] introduce the pyramid and cost volume to neural networks for optical flow, warping the features in different levels of the pyramid and learning the flow fields in a coarse-to-fine manner. RAFT[32] utilizes ConvGRU to construct decoders in the network and iteratively decodes the correlation and context information in a fixed resolution. Due to their excellent performance, PWCNet and RAFT are the backbones of most algorithms[37, 34, 41, 21, 20, 18, 17, 36, 16, 39, 30] in frame-based optical flow estimation. TransFlow [22] utilizing the long-range temporal association to recover more information by introducing transformer architecture. In addition, many frame-based unsupervised optical flow networks[25, 21, 20, 18, 23, 30] were proposed to discard the need for labeled data. Similar to the traditional optimization-based methods, Yu et al. [14] employ photometric loss and smoothness loss to train a flow estimation network. Unflow[25] applies a forward-backward

check on bidirectional optical flow to estimate the occlusion area, where the backpropagation of photometric loss is stopped.

Deep learning has also been applied to event-based optical flow[45, 46, 19, 6, 11, 26, 5]. EVFlowNet[45] can be regarded as the first deep learning work training on large datasets with a U-Net architecture[28]. As an updated version of EV-FlowNet, an unsupervised framework has been proposed by Zhu et al. [46] based on contrast maximization. Spike-FlowNet[19] and STEFlowNet[6] use spiking neural networks and ConvGRU to extract the spatial-temporal features of events, respectively.

The research on spike-based optical flow estimation is just getting started. SCFlow[12], is the first deep-learning method. It proposes a large dataset, the SPIFT dataset, to train an end-to-end neural network via supervised learning. However, our work aims to fill in the blanks of unsupervised learning.

Event-based and Spike-based Input Representation. Normally, asynchronous event streams are not compatible well with the frame-based deep learning architecture. Therefore, frame-like input representation is needed and is expected to capture rich salient information about the input streams. Apart from many handcrafted representations[46, 27, 24, 29, 6], some end-to-end learned representations have been proposed, making it possible to generalize to different vision tasks better. Gehring et al. [10] simply uses multi-layer perceptrons as a trilinear filter to produce a voxel grid of temporal features. Cannici et al. [4] propose Matrix-LSTM, a grid of Long Short-Term Memory (LSTM) cells that efficiently process events and learn end-to-end task-dependent event surfaces. Similarly, Event-LSTM [1] utilizes LSTM cells to process the sequence of events at each pixel considering it into a single output vector that populates that 2D grid. Moreover, Vemprala et al. [33] presents an event variational autoencoder and shows that it is feasible to learn compact representations directly from asynchronous spatio-temporal event data.

Directly using spike stream as input might incur lots of misleading information or miss something necessary if the time window is not chosen appropriately. Therefore, frame-like input representation should also be carefully designed to extract sufficient information from spike stream. SCFlow [12] uses estimated optical flow to align spike frames to eliminate motion blur.

Different from all previous works, we aim to train an end-to-end input representation with the function of a dynamic time window by multi-layer dilated convolutions.

3 Preliminaries

3.1 Spike Camera

Spike camera works by an "integrate-and-fire" mechanism, which asynchronously accumulates the light on each pixel. The integrator of spike cameras accumulates the electrons transferred from incoming photons. Once the cumulative electrons exceed a set threshold, the camera fires a spike and resets the accumulation. The process of accumulation can be formulated as,

where A ( x , t ) is the cumulative electrons at pixel x = ( x, y ) . I ( x , τ ) is the light intensity at pixel x at time τ . α is the photoelectric conversion rate. θ is the threshold. The reading time of spikes is quantified with a period δ of microseconds. The spike camera fires spikes at time T , T = nδ, n ∈ Z , and generate an H × W spike frame s . As time goes on, the camera produces a spatial-temporal binary stream S N t in H × W × N size, as shown in Figure 2. The N is the temporal length of the spike stream. H and W are the height and width of the sensor, respectively.

3.2 Problem Statement

Given two timestamps t 0 and t 1 , we have two spike streams centered on t 0 and t 1 , noted as S L t 0 and S L t 1 , respectively. Then we estimate a dense displacement field f = ( f u , f v ) from t 0 to t 1 using

Pooling

MLP

Weights

Multi-Layer Output

Concatenation

x

y

t



Layer Attention

Optical Flow

Estimation

Backbone

Optical Flow

Estimation

Backbone

Optical Flow

Estimation

Backbone

Conv

ReLU

Conv

Softmax

Weights

Weights in Light Intensity Estimation

Temporal Multi-dilated

Representation

Temporal Multi-dilated

Representation

Unsupervised Loss

s



l



1

k

 =

2

k

 =

s



l



1

k

 =

2

k

 =

these two sub-spike streams, mapping each pixel x = ( x, y ) at t 0 to its corresponding coordinates x ′ = ( x ′ , y ′ ) = ( x + f u ( x ) , y + f v ( x )) at t 1 .

4 Method

4.1 Overview

To verify the effectiveness of the proposed dynamic timing representation, we present two versions of USFlow. One is the PWC-like version. It adopts a variant of PWC-Net [31] which is also the backbone of SCFlow [12]. The other is the RAFT version which adopts an official small version of RAFT [32] as the backbone. More about these two backbones is included in the appendix. Considering one advantage of neuromorphic vision is low latency, our representation part should be lightweight and efficient. The two spike streams first pass into the shared dynamic timing representation module separately, whose outputs are sent into existing backbones. In addition, we design an unsupervised loss to break the dependency on labeled data. Note that all the components are trained end-to-end.

4.2 Dynamic Timing Representation

As stated in 3.1, one binary spike frame itself is sensitive to noise and meaningless without contextual connection. Given a bunch of spike data, selecting an appropriate data length is pivotal for subsequent processing. A too-long spike stream is not suitable for high-speed regions since time offset accumulation introduces more redundant information. A too-short spike stream won't work either, as it can not exhibit light intensity precisely with few binary data. To address this issue, we propose a dynamic timing representation for input spike streams.

The Dynamic Timing Representation consists of Temporal Multi-dilated Representation (TMR) module and a Layer Attention module (LA). The main ingredient of TMR is dilated convolution. By using dilated convolution, we can extend receptive fields to a larger range with just a few stacked layers, while preserving the input resolution throughout the network as well as computational efficiency. In more detail, we apply 1D dilated convolutions to the spike stream for each pixel and the parameters are shared across all the pixels. The TMR can be formulated as follow,

where D1C( · ) represents the 1D dilated convolution operation, i and x are layer index and spatial coordinate respectively. Note that F (0) ( x ) is the input spike stream on position x . Figure 2 depicts dilated convolutions for dilations 1, 2, 4, and 8. The higher level of the layer, the larger the receptive field is. The intuition behind this configuration is two-fold. First, a different layer can be regarded as summarizing spikes or extracting temporal correlation among spike frames with a different time window. Second, dilated convolution allows the network to operate on a coarser scale more effectively than with a normal convolution, which guarantees the model is lightweight. In Table 1, we show the parameter size of our USFlow and other methods.

Input

Level 1

Dilation = 1

Level 2

Dilation = 2

Level 3

Dilation = 4

Level 4

Dilation = 8

Level 5

Dilation = 16

Level 6

Dilation = 32

Spikes of each pixel

Dilated 1D Conv

Dilated 1D Conv

Dilated 1D Conv

Weights-Sharing

Pixel A

Pixel B

Pixel C

s(

x

,t) = 0

s(

x

,t) = 1

Pixels

(Show Connections)

Pixels

(Hide Connections)

Connections

in 1D Conv

x

y

t

Spikes of each pixel



口

Multi-dilated architecture has already greatly expanded the input dimensions but not all layers provide equally useful information. Blindly fusing the output of all layers may impair the learning process. To address this issue, we propose the Layer Attention module to further flexibly emphasize which layer is meaningful or which time window is optimal. As illustrated in Figure 1, we average the output of n layers, { F ( i ) ( x ) } n i =1 , noted as F ′ ( x ) to generate one n -dimension layer context descriptor. The descriptor is then forwarded to a multi-layer perceptron(MLP) to produce our layer attention map. The layer attention values are broadcast along the layer dimension and the final concatenation, RF ( x ) , is computed as follows:

here ⊗ denotes element-wise multiplication. We do this operation on all pixels by sharing weights.

4.3 Unsupervised Loss

Unsupervised learning is necessary for spike-based optical flow estimation due to the lack of labeled training data in high-speed real-world scenes. To this end, we sidestep the need for ground truth of optical flow as only spike data are required for training. More specifically, at training time, two spike streams centered on t 0 and t 1 , S L t 0 and S L t 1 , are utilized to predict optical flow, f = ( f u , f v ) , from t 0 to t 1 .

Different from traditional RGB frames, a binary spike frame can not accurately represent the light intensity at the current timestamp. Therefore, estimating precise current light intensity from binary spike streams is the key to constructing an unsupervised loss function.

For low-speed regions. Since the light intensity accumulation in low-speed regions is an approximately linear process, we count spikes to utilize longer-duration information to estimate the light intensity. In this way, we can improve the robustness of light intensity estimation. However, the data length of the spike stream used to estimate light intensity varies with different low-speed motions. For reducing computational complexity, we only set two different time windows. The light intensity estimation can be formulated as follow:

here D s and D l are the half length of time windows. They are set to 40 and 100 respectively. ω s and ω l are the weight factors. The subscripts s and l refer to short and long time windows, respectively.

For high-speed regions. Selecting a large time span of spike streams to extract information would incur motion blur due to time offset accumulation. Hence, we estimate light intensity during a single spike interval which is typically on the order of microseconds. Since the spike streams have extremely high temporal resolution and the interval between adjacent spikes is ultra-short, we can safely assume that the light intensity remains constant during the interval [47]. Let us denote s ( x , m ) and s ( x , n )

as two adjacent spikes at position x . m and n are the timestamps corresponding to these two spikes. According to the spike camera working mechanism in Equation 1, the constant light intensity can be approximated by:

In reality, however, the number of incoming photons in an ultra-short interval is a random variable subject to the Poisson distribution even under constant illumination conditions. Therefore, the light intensity calculated by Equation 5 includes errors due to random fluctuations. To address this issue, we extend it by multi-intervals and fuse light intensity information with different numbers of intervals,

In Equation 6 and 7, T ( x , z ) refers to the timestamp corresponding to the z th spike at position x . [ T ( x , N τ ( x ) + k -1) -T ( x , M τ ( x ) -k +1)] is the total time length of these (2 k -1) intervals. ω k is the weight factor of the light intensity calculated by using (2 k -1) intervals. Note that k is set to 1 and 2 in our experiments. This setting ensures that the data length of spike stream used for light intensity estimation in high-speed motion regions is much shorter than that used in low-speed regions. Further detailed discussion is presented in the appendix.

Learnable weights of estimated light intensity. Considering a scene may contain both high-speed regions and low-speed regions, it is necessary for our unsupervised loss to fuse light intensity estimation methods based on multiple intervals and multiple time windows. We use the estimated optical flow to reflect the motion speed, and choose the most appropriate light intensity estimation strategy for different regions. As shown in Figure 1, we learn the weights ω s , ω l , ω k from the estimated optical flow. Thence, we can fuse all the terms in Equation 4 and Equation 6 to achieve the final approximated light intensity ˜ I .

After achieving the ˜ I , we then derive the bidirectional photometric loss in a spike-based manner:

where ρ is the Charbonnier loss [3], f is the flow from t 0 to t 1 , f ′ is the flow from t 1 to t 0 .

Furthermore, we use smoothness loss to regularize the predicted flow. It minimizes the flow difference between neighboring pixels, thus it can enhance the spatial consistency of neighboring flows and mitigate some other issues, e.g. the aperture problem. It can be written as:

where ∇ is the difference operator, H is the height and W is the width of the predicted flow. The total loss function consists of two loss terms above-mentioned, which can be written as L total = L photo + λ L smooth , where λ is the weight factor.

5 Experiments

5.1 Implementation Details

In this work, we choose the SPIFT dataset [12] as the training dataset. The PHM dataset [12] and our proposed SSES dataset is used for evaluation. The SPIFT and PHM datasets provide two data settings, generating optical flow every 10 spike frames (∆ t = 10) and 20 spike frames (∆ t = 20) separately from the start to end of sampling. Therefore, we train models for the (∆ t = 10) and

(∆ t = 20) settings separately as SCFlow [12]. All training details are included in the appendix. More details on the SSES dataset will be elaborated in Section 5.4. The color map used in visualization refers to Middlebury [2].

5.2 Comparison Results

Evaluation of Input Representation. To fully validate the effectiveness of the proposed input representation, we first train the model in a supervised manner and compare it with other supervised baselines listed in the prior spike-based work, SCFlow [12]. In more detail, apart from SCFlow, we compare our network with baselines in event-based optical flow, i.e. EV-FlowNet [44] and Spike-FlowNet [19]. We also compare our network with frame-based optical flow network, i.e. RAFT [32] and PWC-Net(variant) [31]. Note that these two frame-based networks are lightweight versions as illustrated in Section 4.1 and we implement our method on both networks, denoted as USFlow(raft) and USFlow(pwc). All the methods in Table 1 are only fed spike streams as inputs.

As illustrated in Table 1, our input representation indeed can improve the performance on top of frame-based backbones. It demonstrates the necessity of directly pre-processing operations on spike stream information. In addition, USFlow(pwc) achieves the best mean AEE of 0.854 in ∆ t = 10 setting and USFlow(raft) gets the best mean AEE of 1.649 in ∆ t = 20 setting, which gets 15% and 19% error reduction from the best prior deep network, SCFlow, respectively. Note that the mean AEE value is averaged over nine scenes. Meanwhile, our input representation has the least parameters compared to other methods. The parameter size of input representation in SCFlow is 0.23M and it in our USFlow is only 0.05M . With the lightweight representation module, the computational time of USFlow for inferring optical flow between two timestamps on a 3090 GPU is 90.6ms. It gets 61.8% computational time reduction from SCFlow.

RGB Image

Ground Truth

SCFlow-supervised

USFlow(pwc)-supervised

PWC(variant)-unsupervised

USFlow(pwc)-unsupervised

Fan Scene

3

USFlow(pwc)

SCFlow

USFlow(pwc)-supervised

Spike frame

Scene

In order to verify that the performance boost is not brought by the number of parameters increasing, we change the dilated convolution to normal convolution with the same input size and feature channel, denoted as RAFT+conv or PWC-Net(variant)+conv. We found that blindly increasing the number of parameters does not make any sense. Normal convolution only provides a limited performance improvement. Therefore, we claim that dilated convolutions can extract salient information more effectively from spike streams. Table 2 shows a comparison between PWC-Net(variant) and USFlow(pwc), indicating that our representation also shows superiority with our unsupervised loss.

Evaluation of Unsupervised Loss. Note that we only build USFlow on PWC-Net(variant) for unsupervised evaluation due to the similar performance of two backbones in Table 1. Since the metric for measuring appearance similarity is critical for any unsupervised optical flow technique [18] in the frame-based domain, we compare with some metrics, i.e. the structural similarity index (SSIM) and the Census loss. As illustrated in Table 2, our proposed unsupervised loss can help the model achieve significant performance improvements. Components of our loss are analyzed in Section 5.3.

Qualitative Results. Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the PHM dataset are visualized in Figure 3. Note that PWC(variant) and USFlow(pwc) are trained unsupervised. USFlow(pwc) can predict more detailed flow than PWC-Net(variant) in many regions. However, there still exists a performance gap between the unsupervised method and the supervised method. Moreover, as for supervised methods, the directions of predicted flows (viewed in color) of USFlow(pwc) are closer to the ground truth than SCFlow, especially in object edges.

Fine-tuned Results. We collect some real data in street scenes and split them into training and evaluation set. Due to the advantage of unsupervised learning over supervised learning, we can fine-tune the unsupervised model on the real data, which has no ground truth, to bridge the domain gap. The fine-tuned model has achieved better qualitative results than the supervised model trained on SPIFT in the evaluation set of street scenes. Parts of qualitative results can be found in Figure 4. More clarifications are placed in the appendix.

5.3 Ablation Study

Dynamic Timing Representation. We do ablation studies on dynamic timing representation in both supervised and unsupervised manners. Table 3 again verifies that dilated convolutions can effectively extract salient information to build promising input representation. More analysis regarding dilated convolutions are in the appendix. Though it seems that layer attention can only improve the performance marginally, we can find out this part makes the training process more stable and faster as illustrated in Figure 6.

Unsupervised Loss. Since estimating the light intensity from spike streams is the core of our unsupervised loss, we do ablation studies on it. As shown in Table 4, the mean AEE of the experiment (A) is higher than that of (B). The reason is there are fewer high-speed motion regions than low-speed motion regions in the PHM dataset. Compared with experiments (A) and (B), our unsupervised loss can handle regions with different motion speeds and get the best performance.

5.4 SSES Dataset

Based on the CARLA [9], we build a synthetic dataset for extreme scenarios of autonomous driving. CARLA is an open-source simulator for autonomous driving research, which provides open digital assets (urban layouts, buildings, vehicles) to build specific scenarios. Moreover, the simulation platform supports flexible specifications of sensor suites, environmental conditions, and much more. In addition, CARLA can provide the ground truth of optical flow, instance segmentation, and depth.

In the proposed dataset SSES, we design ten extreme scenarios, mainly focusing on traffic accidents caused by violating traffic rules or vision-blind areas. We also include various street scenes, background vehicles, and weather conditions to make scenes more diverse. The demonstration of sample cases and more descriptions of the extreme scenarios are in the appendix.

In all scenarios, the speed setting range is 10 ∼ 16 m/s for cars, 5 ∼ 8 m/s for pedestrians and bicycles, and the frame rates for rendered RGB frames and spike frames are 500 fps and 40K fps respectively. Regarding the generation of spike frames, we first increase the frame rate of RGB frames to 40K fps through a flow-based interpolation method and then generate spikes by treating pixel value as light intensity and simulating the integrate-and-fire mechanism [40]. Note that the ground truth of optical flow is obtained from time aligned with RGB frames. The sequence duration is about 0 . 5 ∼ 1 . 5 s .

Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the SSES dataset are visualized in Figure 5. USFlow(pwc) can successfully predict the optical flow in regions where vehicles and pedestrians exist (highlighted by black boxes), which can help decision-making in autonomous driving. Table 5 shows the quantitative evaluation of the SSES dataset.

6 Conclusions

We propose an unsupervised method for learning optical flow from continuous spike streams. Specifically, we design a dynamic timing representation for spike streams. We also propose an unsupervised loss function in a spike-based manner. Moreover, we simulate extreme scenarios in autonomous driving and propose a validation dataset SSES for testing the robustness of optical flow estimation in high-speed scenes. Experiment results show that our USFlow achieves the state-of-the-art performance on PHM, SSES, and real data.

RGB Image

Ground Truth

USFlow(pwc)-supervised

PWC(variant)

USFlow(pwc)

0

5

10

15

20

25

30

35

40

2.0

2.5

3.0

3.5

4.0

mean AEE

USFlow(pwc)

USFlow(pwc) without layer attention

Limitations. The characteristics of spike streams generated in extremely dark scenes are quite different from those in bright scenes, so the length of the time window in the unsupervised loss may need to be reset during fine-tuning. We plan to extend our method to address this issue in future work.

7 Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 22127807, 62072009, and in part by the National Key R&D Program of China under Grant 2021YFF0900501.

References

Lakshmi Annamalai, Vignesh Ramanathan, and Chetan Singh Thakur. Event-LSTM: An unsupervised and asynchronous learning-based representation for event-based data. IEEE Robotics and Automation Letters (RAL) , 7(2):4678-4685, 2022.

Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision (IJCV) , 92(1):1-31, 2011.

Andrés Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision (IJCV) , 61:211-231, 2005.

Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. A differentiable recurrent surface for asynchronous event-based data. In European Conference on Computer Vision (ECCV) , pages 136-152, 2020.

Javier Cuadrado, Ulysse Rançon, Benoit R Cottereau, Francisco Barranco, and Timothée Masquelier. Optical flow estimation from event-based cameras and spiking neural networks. Frontiers in Neuroscience , 17:1160034, 2023.

Ziluo Ding, Rui Zhao, Jiyuan Zhang, Tianxiao Gao, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang. Spatiotemporal recurrent networks for event-based optical flow estimation. AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Yanchen Dong, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. 3D residual interpolation for spike camera demosaicing. In IEEE International Conference on Image Processing (ICIP) , pages 1461-1465, 2022.

Yanchen Dong, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. High-speed scene reconstruction from lowlight spike streams. In IEEE International Conference on Visual Communications and Image Processing (VCIP) , pages 1-5, 2022.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Annual Conference on Robot Learning (CoRL) , pages 1-16, 2017.

Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End-to-end learning of representations for asynchronous event-based data. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5633-5643, 2019.

Jesse Hagenaars, Federico Paredes-Vallés, and Guido De Croon. Self-supervised learning of event-based optical flow with spiking neural networks. In Annual Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Liwen Hu, Rui Zhao, Ziluo Ding, Ruiqin Xiong, Lei Ma, and Tiejun Huang. SCFlow: Optical flow estimation for spiking camera. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. LiteFlowNet: A lightweight convolutional neural network for optical flow estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8981-8989, 2018.

J Yu Jason, Adam W Harley, and Konstantinos G Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In European Conference on Computer Vision (ECCV) , pages 3-10, 2016.

Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9000-9008, 2018.

Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9772-9781, 2021.

Shihao Jiang, Yao Lu, Hongdong Li, and Richard Hartley. Learning optical flow from a few matches. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16592-16600, 2021.

Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In European Conference on Computer Vision (ECCV) , pages 557-572, 2020.

Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Daniilidis, and Kaushik Roy. Spike-FlowNet: event-based optical flow estimation with energy-efficient hybrid neural networks. In European Conference on Computer Vision (ECCV) , pages 366-382, 2020.

Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. Learning by analogy: reliable supervision from transformations for unsupervised optical flow estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6489-6498, 2020.

Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised learning of optical flow. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4571-4580, 2019.

Yawen Lu, Qifan Wang, Siqi Ma, Tong Geng, Yingjie Victor Chen, Huaijin Chen, and Dongfang Liu. TransFlow: Transformer as flow learner. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18063-18073, 2023.

Kunming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang, and Jian Sun. UPflow: Upsampling pyramid for unsupervised optical flow learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1045-1054, 2021.

Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Eventbased vision meets deep learning on steering prediction for self-driving cars. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5419-5427, 2018.

Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In AAAI Conference on Artificial Intelligence (AAAI) , volume 32, 2018.

Federico Paredes-Vallés, Kirk YW Scheper, Christophe De Wagter, and Guido CHE de Croon. Taming contrast maximization for learning sequential, low-latency, event-based optical flow. arXiv preprint arXiv:2303.05214 , 2023.

Henri Rebecq, Timo Horstschaefer, and Davide Scaramuzza. Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization. 2017.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) , pages 234-241, 2015.

AmosSironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1731-1740, 2018.

Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. SMURF: Selfteaching multi-frame unsupervised raft with full-image warping. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3887-3896, 2021.

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8934-8943, 2018.

Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV) , pages 402-419, 2020.

Sai Vemprala, Sami Mian, and Ashish Kapoor. Representation learning for event-based visuomotor policies. Annual Conference on Neural Information Processing Systems (NeurIPS) , 34, 2021.

Jianyuan Wang, Yiran Zhong, Yuchao Dai, Kaihao Zhang, Pan Ji, and Hongdong Li. Displacementinvariant matching cost learning for accurate optical flow estimation. Annual Conference on Neural Information Processing Systems (NeurIPS) , 33:15220-15231, 2020.

Lujie Xia, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. SVFI: Spiking-based video frame interpolation for high-speed motion. In AAAI Conference on Artificial Intelligence (AAAI) , volume 37, pages 2910-2918, 2023.

Haofei Xu, Jiaolong Yang, Jianfei Cai, Juyong Zhang, and Xin Tong. High-resolution optical flow from 1d attention and correlation. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10498-10507, 2021.

Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Annual Conference on Neural Information Processing Systems (NeurIPS) , 5:12, 2019.

Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamicstatic bootstrapping. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2826-2836, 2021.

Feihu Zhang, Oliver J Woodford, Victor Adrian Prisacariu, and Philip HS Torr. Separable Flow: Learning motion cost volumes for optical flow estimation. In IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10807-10817, 2021.

Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11996-12005, 2021.

Shengyu Zhao, Yilun Sheng, Yue Dong, Eric I Chang, Yan Xu, et al. MaskFlownet: Asymmetric feature matching with learnable occlusion mask. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6278-6287, 2020.

Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6358-6367, 2021.

Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao. Motion-attentive transition for zero-shot video object segmentation. In AAAI Conference on Artificial Intelligence (AAAI) , pages 13066-13073, 2020.

Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis. Event-based feature tracking with probabilistic data association. In IEEE International Conference on Robotics and Automation (ICRA) , pages 4465-4470, 2017.

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In Robotics: Science and Systems (RSS) , 2018.

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 989-997, 2019.

Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME) , pages 1432-1437, 2019.

Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1438-1446, 2020.

Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 408-417, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera\" presents a novel approach for unsupervised optical flow estimation using spike cameras. The authors propose a method that leverages dynamic timing representation to estimate optical flow without the need for explicit supervision. They demonstrate the effectiveness of their approach through extensive experiments and comparisons with existing methods.\n\nSTRENGTHS:\n1. The paper introduces a unique and innovative approach to unsupervised optical flow estimation. By utilizing spike cameras and dynamic timing representation, the authors provide a fresh perspective on tackling this problem. This can potentially open new avenues for research and development in the field.\n2. The paper presents the methodology in a clear and concise manner. The authors provide sufficient details and explanations, making it easier for readers to understand the technical aspects of their approach. They also include relevant diagrams and figures to aid in comprehension.\n\nWEAKNESSES:\n1. The paper lacks a thorough discussion of the limitations of the proposed method. While the authors demonstrate its effectiveness through experiments, it would be beneficial to address potential challenges or scenarios where the method might not perform optimally. This would provide a more comprehensive understanding of the approach.\n2. The authors primarily focus on benchmark datasets and quantitative evaluations, but they do not provide real-world examples or scenarios where their method could be applied. Including such examples would enhance the practical relevance of the research and highlight its potential impact.\n\nQUESTIONS:\n1.How does the proposed method handle occlusions or complex scenes where multiple objects are moving simultaneously? Are there any specific limitations or challenges in such scenarios?\n2. How does the dynamic timing representation used in the spike camera contribute to the accuracy of optical flow estimation compared to traditional frame-based approaches?\n\nLIMITATIONS:\nThe proposed method's performance and generalizability might be limited to specific datasets or conditions. It would be valuable to explore its effectiveness on a wider range of datasets, including more challenging and diverse scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for estimating optical flow from event streams captured with a spike camera. The approach is based on unsupervised learning and achieves good results in the comparison to other works in the SotA, including conventional and event-based approaches. Authors collect their own dataset and evaluate on it, specially, in an attempt to show the advantage of finely estimating for low and high-speed regions (although, this is not clearly demonstrated with the current experimental section). Finally, authors also include a section with an ablation study to assess the impact of the different parts of the model (loss terms) and the supervised vs. unsupervised learning approaches.\n\nSTRENGTHS:\n+ Authors prepare and collect a dataset with is always a hard but valuable task\n+ The unsupervised learning approach provides a valuable alternative for optical flow estimation and shows good results on the evaluated datasets\n+ It is unsupervised, which is another good keypoint to bear in mind\n+ The ablation study on the loss terms impact and the comparison between supervised vs. unsupervised are very valuable\n\nWEAKNESSES:\n- After reading the work, I don't think authors make clear the differences between spike and event cameras. There is a confusion all over the introduction and methodology and this makes me (and the reader) hesitate about the relevance of this work. \n- The work needs extensive English proofreading, some sentences are hard to understand.\n- The conclusions should not be a summary of the work\n- In the experimental section, authors compared on PHM and propose their own dataset but they are not comparing to the widely used MVSEC and DSEC datasets (that are the most common for event-based processing). This will improve not only completeness but also the comparison to other works in the state of the art. \n- From the paper and since authors propose an artificially generated dataset, it is not clear if 1) the spike camera is a real sensor and 2) in case it is, if they have access to real sequences recorded with it.\n\nQUESTIONS:\n- Regarding the difference between event and spike cameras. Authors should make clear what kind of data they are handling. Event cameras produce asynchronous events and in fact, one of the advantages of these cameras is the lack of need for reconstructing the dynamics of the scene due to their high temporal resolution. This is great e.g. for optical flow estimation. However, it seems that spike cameras produce frames at very high resolution (something closer to a high-speed camera). This could also be interesting but it carries the burden of analyzing a lot of full frames, which cannot be ignored as a drawback. I really think that, since they are comparing to both conventional and event cameras, this should be clarified as soon as possible in the paper. \nIn fact, this is mixed up in the paper, for example, at some point authors mention something about spike frames but then, they point out that pixels respond asynchronously (?)\n- When comparing to event camera processing, performance should be part of the comparison as well (or latency).\n- In the abstract, lns 12-14 do not make reference to the dataset (I think this should be added here as well)\n- In page 2, I believe the correct term is luminance (not illuminance)\n- In line 68, authors mention \"shows visually impressive performance\", what do you mean? Is that qualitative results? I guess authors mean that they are appealing? Are they only qualitative?\n- The sentence in Lns 107-108 seems incomplete\n- In Ln 182, are 40 and 100 ms? or frames? \n- In ln. 196 \"different numbers of intervals\" \n- Lsmooth is not defined\n- There are some recent works that use SNNs such as \"Optical flow estimation from event-based cameras and spiking neural networks\", from J. Cuadrado et al. that could help to improve the state of the art section.\n- Citations are weirdly formatted. Authors should make sure that they are using the conference format\n- In section 5.3 authors (unsupervised loss) authors mention Figure 4 when they mean Table 4.\n- What is the contribution of the LA module? According to Fig. 6 it seems it is not helping at all ... in such a case, why do authors keep it? is there another case in which the LA module does add a significant contribution?\n- At the end of section 5.4, authors discuss results on SSES dataset but they mention before that they are also doing some real-world sequences, where are the results on these sequences?\n\nLIMITATIONS:\nThe work includes a limitations section that seems adequate", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose an unsupervised learning framework for spike-based optical flow estimation, which is mainly developed for spike input representation and spike loss function. In general, I think the method has merit, but the experiment is unconvincing in its lack of clarity.\n\nSTRENGTHS:\n1. Propose a lightweight end-to-end spike data representation with the function of a dynamic time window.\n2. Propose a two-stage unsupervised loss to model light intensity in regions with different motion speeds.\n\nWEAKNESSES:\n1. In Fig. 5, there are significant quantization/discontinuities in the optical flow gt of the SSES dataset, concentrated in the ground area. To my knowledge, this is not present in other optical flow datasets. The reliability of the optical flow ground-truth is a concern, please provide an explanation. \n2. In Fig. 4 and supplementary video, the results of these methods using only spike as input look bad. The major problem is that the results are concentrated only the textures and are not consistent for the same object., which cannot be called ``dense`` optical flow. To my knowledge, this phenomenon is not present in unsupervised two-frame based optical flow  estimation methods. I suggest comparing the outputs of two-frame based methods and analyzing the spike-based failure cases. \n3. The performance of using spike cameras for optical flow estimation is of concern. In L27, spike camera can record details of high-speed moving objects, it has enormous potential for estimating more accurate optical flow in high-speed scenes. But comparing the state-of-the-art optical flow estimation methods using image cameras, I believe this claim is not convincing. The visualization results of existing spike-based methods, including the authors', are significantly worse than existing image-based methods.\n\nQUESTIONS:\n1. L158, Blindly fusing the output of all layers may impair the learning process. It is desirable to have experiments to support this belief. \n4. L170 seems duplicate to L122.\n5. I suggest to complete the experiments of raft backbone model.\n\nLIMITATIONS:\n1. L132, advantage of neuromorphic vision is low latency. However, the authors do not present the runtime analysis of the proposed method.\n2. See weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on the optical flow estimation problem, especially in the unsupervised setting, for high-frequency spike camera inputs. Specifically, a dynamic timing representation module and a spike-based unsupervised loss are proposed to improve performance. A new synthetic dataset, namely SSES, is created to test extreme scenes in autonomous driving. Experiment results show the proposed method has achieved state-of-the-art performance compared with previous methods.\n\nSTRENGTHS:\n1. The paper is well-motivated by introducing the importance of optical flow estimation for high-frequency inputs collected from latest developed sensors like spike cameras. This topic has not been extensively studied so far.\n2. The paper is overall clear and well-structured.\n3. The proposed method is reasonable. Explanations on each module design are provided.\n4. The results are compelling.\n\nWEAKNESSES:\n1. There are still some confusions in the text or figures that need to be clarified. See additional comments below.\n2. It will be better to polish the language of the paper before publication. Correct grammatical errors and use more formal language. Avoid informal expressions like \"a bunch of\" (Line 138), \"won't\" (Line 140), etc. See additional comments below.\n3. Authors are encouraged to share their code and generated datasets for better reproducibility. Such plan has not been declared in the paper.\n\nQUESTIONS:\n1. If I understand it correctly, the method uses the predicted optical flow to determine whether each pixel belongs to a low-speed or high-speed region and then uses different ways to approximate light intensities $\\tilde{I}$. At the starting iterations of training, the predictions could be totally off, so it is likely that the method will pick a wrong way to approximate intensities. Did you try to tackle this issue?\n2. The data used in Fig 4 do not have ground-truth labels, right? Only qualitative evaluation is possible, so this evaluation may be weak. \n3. There is also a large part of supervised results, so stating \"unsupervised\" in the title could be misleading.\n4. I am curious how a normal RGB-based optical flow network would perform as a comparison. For example, a naïve approach is to first convert spike inputs to RGB and then apply a state-of-the-art RGB-based optical flow network of similar size (parameters, memory cost). What would the results be like? This could be a baseline to strengthen the motivation of developing optical flow networks specific to spike camera inputs.\n\n\n\nAdditional comments:\n1. Grammatical errors: run-on sentences (Line 103-105, line 140-141, Line 163-165, Line 310-311).\n2. Eq 1: The \"mod\" operator is traditionally only defined for integer division, but the value here is clearly continuous numbers, so it is better to define \"mod\" in the text in the continuous context.\n3. Chapter 3.1: There are some confusions here. (1) I assume $A(\\mathbf x, t)$ is continuous, but Line 119 says the output stream is binary, so there is missing gap in between that need explanations. Maybe adding a definition, something like $S_t^N(\\mathbf x) = \\mathbb I(\\sup_{\\tau\\in[t-\\Delta t, t+\\Delta t]} A(\\mathbf x, \\tau) =\\theta)$ will make it clear. (2) Line 113-114 say the camera \"fires a spike\" when the cumulative electrons hits the threshold, so I assume it is asynchronous for each pixel, but then Line 117-118 say the camera \"fires a spike\" at time T, which is synchronous and periodic. Maybe you could change the expression here to avoid confusion.\n4. Line 130: \"officially\" -> \"official\".\n5. Line 135: \"dependence\" -> \"dependency\".\n6. Fig 2: Text is too small. It is better to make text size close to that of the main paper so that readers can read the figure clearly without needing to zoom in.\n7. Line 147: It is better to add that your convolution is 1D on the time dimension to avoid confusion.\n8. Line 182-183: Does your method guarantee $\\omega_s + \\omega_l = 1$? Eq 6: Does $\\sum_k\\omega_k=1$? If so, state so explicitly.\n9. Eq 7 could be merged into the text. It does not have to an equation.\n10. Line 175-184: What do you mean by \"speed\"? The frequency of camera readings, or fast motion? Please clarify.\n11. Line 207: The notation $\\omega_s, \\omega_l, \\omega_{k=1}, \\omega_{k=2}$ is messy. Try not to use both textual abbreviations (\"s\", \"l\") and values ($k$) as subscripts for the same letter $\\omega$. Use other letters like $\\alpha_s, \\alpha_l$, $\\beta_k$ instead; or, you could do superscripts with parentheses $\\omega^{(s)}, \\omega^{(l)}, \\omega_k$.\n12. Line 209: \"approximate\" -> \"approximated\".\n13. Line 212: define or cite \"Charbonnier loss\"; Line 213: define or cite \"smoothness loss\" specifically. There are many ways to define \"smoothness\".\n14. Table 1 and 2: It is better to add citations into the table, so we know which data are generated from your experiments and which data are cited from other papers. The current table assumes that all data are credited to yourself. \n15. Table 1 and 2: I recommend adding a vertical line between the \"Top\" and \"Mean\" columns, so we know \"Top\" is the name of the scene instead of the top error across all scenes.\n16. Line 277: \"through\" -> \"though\".\n17. Line 284: \"Figure 4\" -> \"Table 4\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper introduces a method for optical flow estimaiton for spike cameras. Because of the high temporal resolution of the camera, a preprocessing step based on temporal dilated convolution and attention layers is used to automatically select the best temporal scale for a given sequence.
The authors also introduce a unsupervised loss based on the reconstruction of intensity values form spike data combined with the photometric loss.

STRENGTHS:
The main strengths of the paper are the temporal dilated convolution preprocessing that allow for automatic temporal scale selection, as well as the unsupervised loss.
The description of the method is clear and the experimental section complete, including ablation studies and comparision with previous works.

WEAKNESSES:
The paper does not address the question of latency and computational time, which are key aspect of the spike camera.
Also, since frames are available in the simulate datasets it would have been interesting to compare the accuracy of the proposed method on standard rgb frames, to quantify the benefits of using a high temporal resolution sensor for optical flow estimation.
Finally, for people not familiar with the spike camera, it would be beneficial a more complete introduction to this type of sensor. There is no reference in the paper to any publications explaining what a spike camera is, it's HW implementation and carachteristics.

QUESTIONS:
please refer to "weaknesess" section

LIMITATIONS:
The authors could add a general discussion of advantages and limitations of the spike camera and of optical flow from spike cameras.
For example any the challenge related to data rate, computational time and power consumption of these devices.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05