SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Average end point error (AEE) comparison with other methods for estimating optical flow on PHM datasets under  $\Delta t = {10}$  and  $\Delta t = {20}$  settings. All methods use spike stream as input and are trained on SPIFT dataset in a supervised manner. The best results are marked in bold.
Method	Param.	Ball	Cook	Dice	Doll	Fan	Hand	Jump	Poker	Top	Mean
10	EV-FlowNet	53.43M	0.567	3.030	1.066	1.026	0.939	4.558	0.824	1.306	2.625	1.771
Spike-FlowNet	13.04M	0.500	3.541	0.666	0.860	0.932	4.886	0.878	0.967	2.624	1.762
SCFlow	0.80M	0.671	1.651	1.190	0.266	0.298	1.692	0.120	1.030	2.166	1.009
RAFT	0.99M	0.577	1.557	1.135	0.296	0.327	1.769	0.141	0.681	2.198	0.965
PWC-Net(variant)	0.57M	0.597	2.185	1.288	0.606	0.464	2.551	0.370	1.269	2.602	1.326
RAFT+conv	1.04M	0.535	1.874	1.225	0.737	0.447	2.482	0.299	0.957	2.424	1.220
PWC-Net(variant)+conv	0.62M	0.511	1.585	0.959	0.187	0.245	1.874	0.077	0.833	2.114	0.932
USFlow(raft)	1.04M	0.483	1.228	1.294	0.283	0.345	1.634	0.154	0.764	2.192	0.931
USFlow(pwc)	0.62M	0.430	1.556	0.787	0.210	0.226	1.615	0.105	0.646	2.111	0.854
20	EV-FlowNet	53.43M	1.051	5.536	1.721	2.057	1.867	8.820	1.803	2.193	5.061	3.345
Spike-FlowNet	13.04M	0.923	7.069	1.131	1.675	1.838	9.829	1.701	1.373	5.257	3.422
SCFlow	0.80M	1.157	3.430	2.205	0.507	0.578	4.018	0.267	1.922	4.327	2.046
RAFT	0.99M	1.004	3.378	2.059	0.460	0.561	3.707	0.257	1.416	4.250	1.904
PWC-Net(variant)	0.57M	1.321	4.493	2.601	2.206	1.083	5.654	1.159	2.320	5.143	2.887
RAFT+conv	1.04M	0.983	2.977	1.864	0.533	0.622	3.421	0.287	1.361	4.313	1.818
PWC-Net(variant)+conv	0.62M	0.881	3.198	1.624	0.799	0.476	4.030	0.273	1.424	4.324	1.892
USFlow(raft)	1.04M	0.792	2.734	1.918	0.448	0.569	2.601	0.256	1.231	4.293	1.649
USFlow(pwc)	0.62M	0.807	3.075	1.613	0.370	0.377	3.663	0.168	1.216	4.216	1.723
Table 2: Average end point error (AEE) comparison with other methods for estimating optical flow on PHM datasets under  $\Delta t = {10}$  and  $\Delta t = {20}$  settings. All methods use spike stream as input and are trained on SPIFT dataset in an unsupervised manner. The best results are marked in bold.
Method	Ball	Cook	Dice	Doll	Fan	Hand	Jump	Poker	Top	Mean
10	PWC-Net(variant)	0.642	3.239	1.258	0.579	0.844	4.799	0.587	1.468	2.632	1.783
USFlow(pwc, Census)	0.640	3.549	0.675	0.815	1.018	4.942	0.782	0.756	2.655	1.759
2	USFlow(pwc, SSIM)	0.703	3.546	0.705	0.838	0.980	5.000	0.762	0.819	2.644	1.777
USFlow(pwc)	0.705	2.170	1.416	0.555	0.610	2.219	0.438	1.159	2.488	1.307
20	PWC-Net(variant)	1.480	6.682	1.419	1.407	1.452	9.307	0.905	1.790	5.580	3.336
USFlow(pwc, Census)	1.117	7.049	1.199	1.513	1.858	9.854	1.441	1.393	5.284	3.412
21	USFlow(pwc, SSIM)	1.152	6.882	1.273	1.071	1.260	9.778	3.185	1.554	4.855	3.446
USFlow(pwc)	1.259	3.978	2.782	0.784	0.873	4.748	0.524	2.180	4.589	2.413
$(\Delta t = 20)$  settings separately as SCFlow [12]. All training details are included in the appendix. More details on the SSEs dataset will be elaborated in Section 5.4. The color map used in visualization refers to Middlebury [2].
Evaluation of Input Representation. To fully validate the effectiveness of the proposed input representation, we first train the model in a supervised manner and compare it with other supervised baselines listed in the prior spike-based work, SCFlow [12]. In more detail, apart from SCFlow, we compare our network with baselines in event-based optical flow, i.e. EV-FlowNet [44] and Spike-FlowNet [19]. We also compare our network with frame-based optical flow network, i.e. RAFT [32] and PWC-Net(variant) [31]. Note that these two frame-based networks are lightweight versions as illustrated in Section 4.1 and we implement our method on both networks, denoted as USFlowRAFT and USFlow(pwc). All the methods in Table 1 are only fed spike streams as inputs.
As illustrated in Table 1, our input representation indeed can improve the performance on top of frame-based backbones. It demonstrates the necessity of directly pre-processing operations on spike stream information. In addition, USFlow(pwc) achieves the best mean AEE of 0.854 in  $\Delta t = 10$  setting and USFlow(raft) gets the best mean AEE of 1.649 in  $\Delta t = 20$  setting, which gets  $15\%$  and  $19\%$  error reduction from the best prior deep network, SCFlow, respectively. Note that the mean AEE value is averaged over nine scenes. Meanwhile, our input representation has the least parameters compared to other methods. The parameter size of input representation in SCFlow is 0.23M and it in our USFlow is only 0.05M. With the lightweight representation module, the computational time of USFlow for inferring optical flow between two timestamps on a 3090 GPU is 90.6ms. It gets  $61.8\%$  computational time reduction from SCFlow.
Figure 3: Qualitative results of the evaluation of the PHM dataset.
Figure 4: Qualitative results on real scenes. USFlow(pwc) is fine-tuned in the training set of street scenes.
In order to verify that the performance boost is not brought by the number of parameters increasing, we change the dilated convolution to normal convolution with the same input size and feature channel, denoted as  $\mathrm{RAFT + conv}$  or PWC-Net(variant)+conv. We found that blindly increasing the number of parameters does not make any sense. Normal convolution only provides a limited performance improvement. Therefore, we claim that dilated convolutions can extract salient information more effectively from spike streams. Table 2 shows a comparison between PWC-Net(variant) and USFlow(pwc), indicating that our representation also shows superiority with our unsupervised loss.
Evaluation of Unsupervised Loss. Note that we only build USFlow on PwC-Net(variant) for unsupervised evaluation due to the similar performance of two backbones in Table 1. Since the metric for measuring appearance similarity is critical for any unsupervised optical flow technique [18] in the frame-based domain, we compare with some metrics, i.e. the structural similarity index (SSIM) and the Census loss. As illustrated in Table 2, our proposed unsupervised loss can help the model achieve significant performance improvements. Components of our loss are analyzed in Section 5.3.
Qualitative Results. Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the PHM dataset are visualized in Figure 3. Note that PWC(variant) and USFlow(pwc) are trained unsupervised. USFlow(pwc) can predict more detailed flow than PWC-Net(variant) in many regions. However, there still exists a performance gap between the unsupervised method and the supervised method. Moreover, as for supervised methods, the directions of predicted flows (viewed in color) of USFlow(pwc) are closer to the ground truth than SCFlow, especially in object edges.
Fine-tuned Results. We collect some real data in street scenes and split them into training and evaluation set. Due to the advantage of unsupervised learning over supervised learning, we can fine-tune the unsupervised model on the real data, which has no ground truth, to bridge the domain gap. The fine-tuned model has achieved better qualitative results than the supervised model trained on SIFT in the evaluation set of street scenes. Parts of qualitative results can be found in Figure 4. More clarifications are placed in the appendix.
Dynamic Timing Representation. We do ablation studies on dynamic timing representation in both supervised and unsupervised manners. Table 3 again verifies that dilated convolutions can effectively extract salient information to build promising input representation. More analysis regarding dilated convolutions are in the appendix. Though it seems that layer attention can only improve the performance marginally, we can find out this part makes the training process more stable and faster as illustrated in Figure 6.
Figure 5: Qualitative results on the SSES dataset. The positions of vehicles and pedestrians are highlighted by black boxes.
Unsupervised Loss. Since estimating the light intensity from spike streams is
the core of our unsupervised loss, we do ablation studies on it. As shown in Table 4, the mean AEE of the experiment (A) is higher than that of (B). The reason is there are fewer high-speed motion regions than low-speed motion regions in the PHM dataset. Compared with experiments (A) and (B), our unsupervised loss can handle regions with different motion speeds and get the best performance.
5.4 SES Dataset
Based on the CARLA [9], we build a synthetic dataset for extreme scenarios of autonomous driving. CARLA is an open-source simulator for autonomous driving research, which provides open digital assets (urban layouts, buildings, vehicles) to build specific scenarios. Moreover, the simulation platform supports flexible specifications of sensor suites, environmental conditions, and much more. In addition, CARLA can provide the ground truth of optical flow, instance segmentation, and depth.
In the proposed dataset SSES, we design ten extreme scenarios, mainly focusing on traffic accidents caused by violating traffic rules or vision-blind areas. We also include various street scenes, background vehicles, and weather conditions to make scenes more diverse. The demonstration of sample cases and more descriptions of the extreme scenarios are in the appendix.
In all scenarios, the speed setting range is  $10 \sim 16 \mathrm{~m/s}$  for cars,  $5 \sim 8 \mathrm{~m/s}$  for pedestrians and bicycles, and the frame rates for rendered RGB frames and spike frames are 500 fps and 40K fps respectively. Regarding the generation of spike frames, we first increase the frame rate of RGB frames to 40K fps through a flow-based interpolation method and then generate spikes by treating pixel value as light intensity and simulating the integrate-and-fire mechanism [40]. Note that the ground truth of optical flow is obtained from time aligned with RGB frames. The sequence duration is about  $0.5 \sim 1.5s$ .
Parts of RGB images, ground truth flow, and the corresponding predicted flow images of the SSEs dataset are visualized in Figure 5. USFlow(pwc) can successfully predict the optical flow in regions where vehicles and pedestrians exist (highlighted by black boxes), which can help decision-making in autonomous driving. Table 5 shows the quantitative evaluation of the SSEs dataset.
We propose an unsupervised method for learning optical flow from continuous spike streams. Specifically, we design a dynamic timing representation for spike streams. We also propose an unsupervised loss function in a spike-based manner. Moreover, we simulate extreme scenarios in autonomous driving and propose a validation dataset SSES for testing the robustness of optical flow estimation in high-speed scenes. Experiment results show that our USFlow achieves the state-of-the-art performance on PHM, SSES, and real data.
Table 3: Ablation studies of our design choices for input representation. The present value is the mean AEE, averaged over nine scenes, on PHM. The best results are marked in bold.
TMR LA	Supervised	Unsupervised
Δt = 10	Δt = 20	Δt = 10	Δt = 20
USFlow (pwc)	X	X	0.943	1.797	1.783	3.336
✓	X	0.888	1.699	1.355	2.461
✓	✓	0.854	1.723	1.307	2.413
USFlow (raft)	X	X	0.965	1.902	-	-
✓	X	0.952	1.659	-	-
✓	✓	0.931	1.649	-	-
Figure 6: The mean AEE on PHM dataset during supervised training. The shadowed area is enclosed by the min and max values of three training runs, and the solid line in the middle is the mean value.
Table 4: Ablation studies of unsupervised loss. Table 5: Mean AEE averaged over ten scenes The present value is the mean AEE over the PHM on SSES dataset. All models are trained on the dataset. Best in bold. SPIFT dataset.
Setting of Experiment	Δt=10	Δt=20
USFlow (pwc)	(A) Remove I_T from our loss	2.071	3.260
(B) Remove I_I from our loss	1.484	2.563
(C) our unsupervised loss	1.307	2.413
USFlow(pwc) supervised	USFlow(pwc) unsupervised
mean AEE	Δt=10	2.967	3.122
Δt=20	2.234	3.130
Limitations. The characteristics of spike streams generated in extremely dark scenes are quite different from those in bright scenes, so the length of the time window in the unsupervised loss may need to be reset during fine-tuning. We plan to extend our method to address this issue in future work.
This work was supported in part by the National Natural Science Foundation of China under Grants 22127807, 62072009, and in part by the National Key R&D Program of China under Grant 2021YFF0900501.
[1] Lakshmi Annamalai, Vignesh Ramanathan, and Chetan Singh Thakur. Event-LSTM: An unsupervised and asynchronous learning-based representation for event-based data. IEEE Robotics and Automation Letters (RAL), 7(2):4678-4685, 2022.
[2] Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision (IJCV), 92(1):1-31, 2011.
[3] Andres Bruhn, Joachim Weickert, and Christoph Schnörr. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision (IJCV), 61:211-231, 2005.
[4] Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci. A differentiable recurrent surface for asynchronous event-based data. In European Conference on Computer Vision (ECCV), pages 136-152, 2020.
[5] Javier Cuadrado, Ulysse Rançon, Benoit R Cotterreau, Francisco Barranco, and Timothee Masquelier. Optical flow estimation from event-based cameras and spiking neural networks. Frontiers in Neuroscience, 17:1160034, 2023.
[6] Ziluo Ding, Rui Zhao, Jiyuan Zhang, Tianxiao Gao, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang. Spatiotemporal recurrent networks for event-based optical flow estimation. AAAI Conference on Artificial Intelligence (AAAI), 2021.
[44] Alex Zihao Zhu, Nikolay Atanasov, and Kostas Daniilidis. Event-based feature tracking with probabilistic data association. In IEEE International Conference on Robotics and Automation (ICRA), pages 4465-4470, 2017.
[45] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In Robotics: Science and Systems (RSS), 2018.
[46] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. Unsupervised event-based learning of optical flow, depth, and egomotion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 989-997, 2019.
[47] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432-1437, 2019.
[48] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1438-1446, 2020.
[49] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 408-417, 2017.