SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We jointly train the proposed SpkDeblurNet. The spike reconstruction branch obtains high-resolution image priors from the initial deblurred result, enabling the recovery of better image-like features. The features from the spike branch in turn guide the deblurring branch, providing time-specific texture information. For both the initial deblurred result  $\overline{I}_t$  and the refined deblurred result  $\hat{I}_t$ , we utilize the L1 loss for supervision during training. The loss function of the entire network can be formulated as follows:
$$
\mathcal {L} = \left\| I _ {t} - \hat {I} _ {t} \right\| _ {1} + \lambda_ {1} \left\| I _ {t} ^ {g} - \hat {I} _ {t} ^ {g} \right\| _ {1} + \lambda_ {2} \left\| I _ {t} - \bar {I} _ {t} \right\| _ {1}, \tag {13}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are hyperparameters that control the loss terms.
3.6 Datasets
This paper is the first study on spike-based motion deblurring, and no large-scale training datasets are currently available. Therefore, we built two large-scale spike-based datasets based on existing real image datasets [39, 46] for training and validation. The first one, named 'Spk-X4K1000FPS', is built on the dataset X4K1000FPS which contains clear images captured by 1000fps cameras. To generate spikes, we utilize the interpolation method [46] to further increase the frame rate four times. Then we build a simple spike simulator according to the spike generation mechanism. It inputs continuous sharp images as light signals and converts them to spikes. For clips in the dataset, we use the average of neighboring  $e$  sharp images around the ground truth images to generate blurry images, where  $e = 33, 65$ . The second one, named 'Spk-GoPro' is built on the GoPro [39] dataset in a similar way as above. More implementation details are attached in the supplementary materials.
4.1 Comparative Results
In this section, we present comparative results on both Spk-X4K1000FPS and Spk-GoPro datasets. For Spk-X4K1000FPS, we generate blurry images by setting an exposure window of  $e = 33$  as well as an extreme exposure window of  $e = 65$ . For Spk-GoPro, we use the provided blurry images. Implementation details can be found in supplementary materials.
Results on Spk-X4K1000FPS. Tab. 1 and Fig. 5 present quantitative and qualitative comparisons of our method with other approaches under two different exposure window settings. For EFNet and REFID, we use their networks for spike-assisted deblurring. For EFNet, we employed input representations similar to its SCER approach, excluding the EMGC module due to its event-related nature. Regarding REFID, we utilized similar spike voxel representations. All comparative methods
Figure 5: Visualized results on Spk-X4K1000FPS dataset, compared with NAFNet [3] and HINet [4].
were trained from scratch on the proposed dataset. We observe that both image-domain methods perform well in recovering image details with a smaller exposure window setting  $(e = 33)$ . However, when the exposure window becomes larger  $(e = 65)$ , pure image-based methods struggle to find cues from extremely blurry scenes to restore clear images, resulting in significant loss of texture details or generating artifacts. In the second row of the first sample sequence in Fig. 5, NAFNet and HINet mistakenly recover non-existent stair edges. In contrast, our multimodal approach leverages spike streams to find cues and restore correct and clear textures. Additionally, the networks in event-assisted methods fall short of our network. This is due to our network's superior bidirectional information transmission nature.
Results on Spk-GoPro. As shown in Tab. 2, our proposed SpkDeblurNet outperforms the state-of-the-art model REFID [49] and other competitive models in terms of PSNR On the GoPro dataset. We have observed that both our SpkDeblurNet and the SOTA event-based methods have significantly outperformed image-based methods, which demonstrates the substantial potential of neuromorphic data in complementing information for the image domain. Fig. 5 visualizes the deblurred results of different methods, from which we demonstrate that the quality of sharp images predicted by the SpkDeblurNet is more promising and satisfying than others. In the first sample of Fig. 5, our approach successfully restored the clear license plate number while also effectively recovering the fallen leaves on the ground, while other methods only achieved a relatively blurred restoration of the ground. Furthermore, our method excelled in restoring the clearest facial features of the person in the second sample. Zoom in for more details. Our framework, by leveraging the complementary information from both image and spike domains, has surpassed event-based methods that solely rely on unidirectional information transfer in terms of both quantitative and qualitative results.
Results on Real-World Scenes. To validate the generalization of our method in real-world scenarios, we construct a hybrid camera system to capture real-world spike-image data. We use a beam splitter to connect a spike camera and a conventional frame-based camera, enabling both cameras to have the same view. We rapidly wave objects in front of the cameras to simulate high-speed motion. We synchronize the two cameras and manually aligned their spatial and temporal outputs. Fig. 7 presents visual results of applying our model trained on the Spk-X4K1000FPS dataset with a window size of 33 to two real-world sequences. We can observe desirable results with clear textures and accurate colors. This demonstrates the good generalization of our proposed algorithm and dataset, highlighting the potential of our algorithm for real-world applications. More results and the hybrid camera system diagram can be found in the supplementary materials.
Table 2: Comparison of various motion deblurring methods on Spk-GoPro. HINet+: event-enhanced versions of HINet [4].
Method	Extra Data	PSNR ↑	SSIM ↑	#Param
Spk-GoPro
DeblurGAN-v2 [26]	29.55	0.934	-
D2Nets [43]	Event	31.60	0.940	-
LEMD [18]	Event	31.79	0.949	-
MPRNet [61]	32.66	0.959	20.0M
HINet [4]	32.71	0.959	88.7M
Restormer [59]	32.92	0.961	26.1M
ERDNet [14]	Event	32.99	0.935	-
HINet+ [4]	Event	33.69	0.961	88.7M
NAFNet [3]	33.69	0.967	67.9M
EFNet [48]	Event	35.46	0.972	8.5M
REFID [49]	Event	35.91	0.973	15.9M
SpkDeblurNet (Ours)	Spike	36.12	0.971	13.5M
Figure 6: Visualized results on Spk-GoPro dataset.
In this section, we conduct a series of ablation studies on Spk-X4K1000FPS dataset.
Effectiveness of the proposed modules. As shown in Tab. 3, removing the CAMMA branch resulted in a performance degradation of  $0.2 - 0.3\mathrm{dB}$ , confirming the necessity of introducing features from the deblurring branch and the effectiveness of the CAMMA module's learnable magnitude masking mechanism. Additionally, replacing the TCAF module with concatenation led to minor performance degradation, which we attribute to the explicit supervision of the spike reconstruction branch enabling the branch to learn image-like features, thereby the modality gap is reduced.
Information supplementation from image domain to spike domain. We further conduct ablation studies on spike reconstruction with different combinations of three input representations and the usage of the CAMMA branch. To be specific, we employ either coarse reconstruction CSFI or a symmetric cumulative spike representation similar to EFNet [48]. For the latter, we accumulate spikes from the center towards both ends using three window lengths and concatenate the results along the channel dimension. As shown in Tab. 4, CAMMA consistently significantly improves spike reconstruction across all input representations. For settings using CSFI and cumulated spikes as inputs, CAMMA enhances performance by  $1.5 - 3.5\mathrm{dB}$ . This is due to the simpler nature of CSFI and cumulated spikes, which contain less texture and motion information. High-resolution image priors from the deblurring branch compensate for this, further enhancing spike reconstruction. Even with spike streams as input, CAMMA still yields improvements of  $0.4 - 0.7\mathrm{dB}$ . This set of ablation experiments demonstrates the effectiveness of the proposed scheme in incorporating high-resolution features from the deblurring branch into the spike branch, thereby enhancing spike reconstruction performance and providing better guidance and cues for the deblurring task.
Real-World Blurry Image
HINet
NAFNet
Ours
Figure 7: Qualitative Results on Real-World Scenes.
Table 3: Ablation study of our proposed modules and input representations.
Experiment settings	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑
Spk-X4K1000FPS	e = 33	e = 65
Replace input representation with CSFI	36.08	0.957	34.22	0.950
Replace input representation with cumulate spikes	36.34	0.955	34.92	0.955
Replace TCAF with concatenation	37.36	0.968	35.81	0.965
Remove CAMMA branch	37.20	0.967	35.64	0.965
Our final SpkDeblurNet	37.42	0.968	35.94	0.966
Table 4: Ablation study of CAMMA on spike reconstruction with different inputs.
Experiment settings	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑
Input Representation	CAMMA	e = 33	e = 65
CSFI	X	33.67	0.923	33.69	0.922
CSFI	✓	37.25	0.939	36.13	0.935
Cumulated Spikes	X	35.58	0.926	35.11	0.914
Cumulated Spikes	✓	37.54	0.935	36.65	0.936
Spike Streams	X	38.72	0.953	38.91	0.952
Spike Streams	✓	39.41	0.955	39.32	0.954
Information supplementation from spike domain to image domain. We investigate the complementary role of the spike domain to the image domain by replacing the input representation. Quantitative results in Tab. 3 indicated that using coarse reconstruction as input still yielded competitive performance, suggesting that the rough texture information contained in CSFI serves as a good complement to the deblurring branch. On the other hand, using cumulated spikes as input further improved the performance. As a comparison, we achieved the highest performance using pure spike stream as input, demonstrating the network's ability to fully excavate the rich temporal information and clues embedded in spike streams to recover clear images. We further conduct ablation studies on different combinations of three input representations and the usage of the CAMMA module for the deblurring task, the results can be found in the supplementary materials.
In this paper, we propose the first spike-based motion deblurring model with bidirectional information complementarity, in which we propose a content-aware motion magnitude attention module and a transposed cross-attention fusion module. We build two large-scale synthesized datasets used for both training and validation, which contain high-temporal-resolution spikes, blurry images, and sharp images. Experimental results demonstrate that the proposed method outperforms other deblurring methods and achieves desirable results.
Limitations. Our future work will prioritize enhancing the resolution of the spike stream to improve its auxiliary capability for image-domain tasks, considering the limitation of the current low resolution of spike cameras and the significant impact of spatial super-resolution on applications.
This work was supported by the National Natural Science Foundation of China under Grant No. 62176003 and No. 62088102.