Enhancing Motion Deblurring in High-Speed Scenes with Spike Streams
Authors: Shiyan Chen
Jiyuan Zhang
Yajing Zheng
Tiejun Huang
Zhaofei Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: spike camera
neuromorphic vision sensors
motion deblurring
high speed imaging

Abstract: Traditional cameras produce desirable vision results but struggle with motion blur in high-speed scenes due to long exposure windows. Existing frame-based deblurring algorithms face challenges in extracting useful motion cues from severely blurred images. Recently, an emerging bio-inspired vision sensor known as the spike camera has achieved an extremely high frame rate while preserving rich spatial details, owing to its novel sampling mechanism. However, typical binary spike streams are relatively low-resolution, degraded image signals devoid of color information, making them unfriendly to human vision. In this paper, we propose a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes. 
We propose the first spike-based motion deblurring model with bidirectional information complementarity. We introduce a content-aware motion magnitude attention module that utilizes learnable mask to extract relevant information from blurry images effectively, and we incorporate a transposed cross-attention fusion module to efficiently combine features from both spike data and blurry RGB images.
Furthermore, we build two extensive synthesized datasets for training and validation purposes, encompassing high-temporal-resolution spikes, blurry images, and corresponding sharp images. The experimental results demonstrate that our method effectively recovers clear RGB images from highly blurry scenes and outperforms state-of-the-art deblurring algorithms in multiple settings.

Enhancing Motion Deblurring in High-Speed Scenes with Spike Streams
Shiyan Chen
Jiyuan Zhang 1.2- Yajing Zhengl.2 1.21 Tiejun Huang 223 1223
Zhaofei Yu 12,31
School of Computer Science, Peking University Nationaa Key Laboratory for Multimedia Information Processing, Peking University Institute for Artificial Intelligence, Peking University vyzhang)&stu. pku. edu. {yj zheng, yuzf t (aauang)@pkir edu. en
Abstract
Traditional cameras produce desirable vision results but struggle with motion blur in high-speed scenes due long exposure windows. Existing frame-based deblurring algorithms face challenges in extracting useful motion cues from severely blurred images. Recently, an emerging bio-inspired vision sensor known as the spike camera has achieved an extremely high frame rate while preserving rich spatial details, owing its novel sampling mechanism. However, typical binary spike streams are relatively low-resolution, degraded image signals devoid of color information, making them unfriendly to human vision. In this paper, we propose a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes. We propose the first spike-based motion deblurring model with bidirectional information complementarity. We introduce a content-aware motion magnitude attention module that utilizes learnable mask to extract relevant information from blurry images effectively, and we incorporate a transposed cross-attention fusion module to efficiently combine features from both spike data and blurry RGB images. Furthermore, we build two extensive synthesized datasets for training spikes, blurry
and validation purposes, encompassing highhteemporal-resolutee images, and corresponding sharp images. The experimental results demonstrate that method effectively recovers clear RGB images from highly blurry scenes and outperforms state-of of-the-art deblurring algorithms in multiple settings.
Introduction
Traditional computer vision is well developed and models on visual tasks are satisfying 10 [1]. As mature industrial product, frame-based cameras have also been fully developed whose imaging quality high. Due the existence of the exposure window, high-speed scenes or moving objects will cause serious blur effects as illustrated in Fig. Ta). which degrades or invalidates the model performance. Thus, deblurring task remains an important demand for applications. With the development of convolutional neural networks and Transformers, models like Restormer [59) and SwinIR (32) used for deblurring tasks perform well. Nevertheless, our main concern is that the performance of these models is not reliable the blurry image lacks sufficient contextual information to recover the corresponding clear image. When the blurry artifact is severe, models only input with blurry images may fail.
Equua contributors. Corresponning authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Exposure Window
ABC
ABC
ABC
ABC
(a) Discrete Images from Frame-based Camera
ABC
Spikes Images Deblurring
(b) Continuous Spikes from Spike Camera
Figure/ 1: Illustration the motivation of this paper. (a) The exposure window leads to motion blur in basee cameras. (b) Continuous spikes can recover low-resolution gray images yet with high temporal resolution. (c) Our method aims to fuse two modalities for motion deblurring.
Neuromorphic vision sensors, including event cameras [| 33 and spike cameras [16], have gained recent attention due to advantages such as high temporal resolution, dynamic range, and low power consumption. Event cameras capture changes in light intensity, while spike cameras record absolute intensity in unique way. They accumulate voltage converted from the light signal asynchronously and emit spikes when a preset threshold reached, achieving time resolution of up to 40,000Hz. Spike cameras are pootion-sensitee
and capture comprehensive texture information, offering the potential to mitigate motion blur. However, the spike stream is not directly interpretable by humans like RGB images. Initial research aimed at reconstructing grayscale images from spikes (see Fig. [b)). Subsequent studies leveraged spikes for high-level vision tasks, including object tracking, optical flow estimation, and depth estimation. Presently, spike cameras face challenges in these tasks due to their lower spatial resolution and absence of color. We consider utilizing spike cameras assist in image deblurring, Our intuition is that the information in the RGB images and the neuromorphic data are complementary. The image domain contains rich spatial texture information with high-fidelity color, while the neuromorphic domain provides abundant temporal information 72 9 promises them to record fast motions. Some studies utilize event cameras to assist image deblurring [57 63, 49).
However, most event-based methods unidirectionally utilize information from the event domain to assist the image domain, without achieving the complementarity information from both domains. Unlike event cameras, spike cameras maintain high temporal resolution while also recording texture information in the form of spike intervals (72]. The complete texture information, as opposed to solely motion edge information, Fig. c). we aim to simultaneously consider the bidirectional transmission of information from both modalities, thereby achieving more efficient motion deblurring.
offer more valuable cues for deblurring tasks. As shown in
Therefore, we propose the first spike-based RGB image deblurring model named SpkDeblurNet. The model takes high- high-resolution blurry RGB images and low-resolution spike streams input, each of which passes through one Transformer based branch. We design a content-aware motion-mape notion-magnitude attention module to fuse the high-resolution features in images into the spike branch. We further propose transposed cross-attention to perform a bidirectional fusion of cross-modal features between two branches. Moreover, establish two high-quality datasets M4k-X4K1000FFP and Spk-GoPro for both training and validation. In experiments, we compare with various deblurring methods and prove the effective assistance of spikes. The complementarity of the two modalities is also verified. Our contributions to this work can be summarized as follows:
We propose the first baske- motion deblurring model with bidirectional information complementarity. Specifically, we propose a content-aware motion magnitude attention module based on learnable mask to utilize effective information in blurry images. A transposed cross-attention fusion module is built to efficiently fuse features from spikes and blurry RGB images.
We generate large-scale synthesized datasets used for training and validation, which contain high-temporal-resoluuig spikes, blurry images, and sharp images.
Experimental results demonstrate that the proposed method achieves ideal results, outper- forming other deblurring methods.

---

2 Related Works
Research on Spike Camera. Spike cameras possess the advantages of ultra-high temporal resolution and high dynamic range, which bring the potential to solve various visual tasks, especially in high-speed scenes. Early studies focus recovering clear gray-scale images from spikes. Zhu et al. (72) propose the first work by directly analyzing the spike firing rate and the inter-spike interval. Some works [73 [69] take biological plausibility into considcration. Spk2imgnet [65] achieves satisfying performance by training deep convolutional neural networks (CNNs) in a supervised manner. Chen al. [5] introduce the self-supervised framework. Besides, some studies [56] 64 explore recovering super-resolution images from spikes. Recently, more researchers have used spike cameras to solve various visual tasks. Zhao al. [66] build new spike-based dataset for object recognition. Zheng et al. (68) propose a bio-inspired framework for object tracking. Hu al. 15] contribute spike-based optical flow estimation by proposing the SCFlow net and new datasets. Zhao el al. [67] proposed to jointly estimate a series of flow fields. Chen el al. [6] propose self-supervised framework for joint learning of optical flow and reconstruction. Besides, Spike Transformer [62] and SSDEFormer (53) are proposed for monocular/stereo depth estimation. Additionally, we observed similarities between single-photon cameras (SPCs) and spike cameras in their binary output. Nonetheless, SPCs predominantly depends on single-photon avalanche diode
while the spike camera is based on CMOS sensors like traditional cameras and employs standard semiconductor manufacturing processes, rendering them fundamentally distinct, with the latter being more cost-efficient.
(SPAD) detector (35 17 27 or quanta image sensors (QIS) 38] 7 2 L3
Frame-Based Motion Deblurring. Traditional frame-based motion deblurring methods typically 30, (58) and blur kernel estimation 28
employ optimization techniques [12 24 29]
55
42
2T In
recent years, with the advancement of deep learning, end-to-end approaches based on CNNs and transformers have demonstrated impressive performance in image deblurring. Multi-scale [39] 50).
multi-stage [6] 4. and progressive strategies [8] have been proposed to improve the effectiveness
of end-to-end training. Some works [25] 26 proposed to utilize Generative Adversarial Networks
(GANs) to provide better human perception. On the other hand, universal networks [ 31 32
54 61
(59 60 4 have been introduced address series of low-level problems including image deblurring.
Further research applied the powerful transformer framework [52] 31 32
[9] 23 to motion debluring,
achieving state-of-the-arl (SOTA) performance. Unlike the conventional one-to-one mapping from blurry image to a single sharp image, blurry frame interpolation or blur decomposition aims to
recover multiple sharp images from a single blurry image [20] 19 45
40
Additionally,
directional anbiguity can be avoided in methods based on blurry video inputs
19
45
70
due to the motion cues of adjacent frames.
Event-Based Motion Deblurring. There has been increasing interest in utilizing event cameras to utilized optical flow to provide motion information. NEST [5]] proposed an efficient event representation based on bidirectional LSTM for deblurring.
assist image deblurring. Several works 18 43
Kim al. [22] extended the task to images with unknown exposure time. EFNet 48] introduced multi-head attention module to better fuse event and image information. Some works [34 41
52
57 have utilized events to assist blurry frame interpolation or blur decomposition. In addition, a series of high-qualit, event-image datasets have been proposed [51] 49
49
63
18
48).
providing convenience for subsequent research. Most event-based methods merely utilize event- information to assist the image domain task. In this paper, aim to the bidirectional transmission of information from both modalities to achieve more efficient motion deblurring.
3 Methods
3.1 Spike Generation Mechanism
Spike cameras, different from the differential sampling of event cameras, adopt the integral sampling mechanism. A spike camera with the spatial resolution of H x W is composed of an array of units on each pixel. The working mechanism of the spike camera is illustrated in Fig. 2 For a unit at pixel (xr, u) that a E [0, ), y E [0, H) continuously receives coming photons at any timestamps I and converts the light intensity (1) into the electric current, and increases its voltage Vl.,((). A voltage threshold θ is pre-defined. A spike will be triggered once the threshold is reached, and the V Ve,y (t) will be reset to O. The process can be formulated as follows:
3

---

Light Intensity
L
t
Voltage V
Threshold
7
25
t
Readout
1
1
1
1
0
1
1
Figure 2: Illustration of the mechanism of generating spikes as the light intensity changes.
V a L., if V < O, 0,
V (1)
otherwise,
where Ve.g(( and (t) denotes the voltage before and after receiving the electric current, and the is the photoelectric conversion factor. The back-end circuit reads out spikes continuously with very small interval 1 (25us). A pixel at (x, y) will output Siz, y, n2) when the threshold is reached at t E (n7n 1), mm). otherwise output S(x, y, n) = 0. Thus. for T-times readout, a spike train with the size of H W x T would be generated.
3.2 Problem Statement
spike camera outputs binary spike streams with very-high temporal resolution. conventional frame-based camera outputs an image sequence and each image is generated after exposure time Te- Though scenes in the world are continuous, images are discrete. We regard spikes approximately continuous signals. When coming scenarios with high-speed motions, the exposure window in frame-based cameras leads to blurred image B., while spikes S, during the exposure time finely records the motion dynamics. In this work, aim utilize the spike stream S. during the T, to deblur the RGB image B.. Moreover, the current spatial resolution of spike cameras is relatively low, so we the spatial resolution of Se half of B. to simulate the real situation.
3.3 Architecture
To achieve effective integration of the high resolution color texture information in conventional images and the high temporal resolution offered by spike streams, we present novel two-branch fusion spike-based image deblurring framework, SpkDeblurNer (Fig 3). Our framework combines the benefits of both image deblurring and spike-based image processing. In the first branch of SpkDeblurNet, we input the blurred RGB image B., while the second branch takes a continuous spike stream S. centered around the latent clear image. B. has twice the spatial resolution of S..
The input blurry image H x3 first passed through two 3 x convolutional layers with a stride of 2 for downsampling. Subsequently, the downscaled features are fed into series of continuous Residual Swin Transformer blocks (RSTB) 332 37]. These RSTBs play a crucial role extracting features that encapsulate enhanced spatiotemporal semantics. Mathematically, we can express this process as follows:
f RSTB (F) down (B Hx33))
(2)
f/ = RSTB (FL.).
(3)
where the Fl denotes the downsampling operations.
Similarly, the spike stream S#xWxT undergoes processing through singular convolutional layer, facilitating the embedding of spike streams within the feature domain. Furthermore, sequence of

---

Reudual Transformer Block
TCAF
(RSTB)
Burny Image B,
Transformer Block (RSTB)
Conter Spike Firing interval (CSFI)
RGB image
CAMIMA
Spike Stream
Vaast
Clear
Resiual Swin Transformer Block
Residual Swin Transformer Block (RSTB)
(RSTB)
TCAF
x1
Figure 3: The architecture of the SpkDeblurNet. Arrows in red denote the image branch, the blue denotes the spike branch, and the green denotes the bidirectional cross-model feature fusion process.
three RSTBs is employed to execute profound spatiotemporal semantic extraction:
fi = RSTB (F down (S (S#xW*T))
(4)
fi RS-B
(5)
To effectively merge the comprehensive high resolution color texture information available in the image domain with the spatiotemporal semantic extraction performed on the spike stream, we design novel Content-Awar Motion Magnitude Attention module (CAMMA). This module facilitates the extraction of features specifically from the regions of clarity within the initial deblurred image, thereby producing a robust motion mask represented as M. By combining this motion mask with the initial deblurred image F (f%) extracted from the blurry image and integrating with feature f3 obtained from the spike domain using the Transposed Cross-Attentic Fusion (TCAF) module, clear grayscale image denoted as i is generated through an RSTB and post-upsampling process. The mathematical expression for this procedure is as follows:
Steaf TCAF (M F'b (12).
p = F (RSTB
with denotes the element wiss multiplication, and i denotes the predicted grayscale image the center timestamp f of window T..
We then combine the spike features / with the feature information ʃ3 obtained by the blurry image through the TCAF module, and output refined sharp color image 1e with series of RSTBs and upsampling, which can be summarized as follows:
feaa T (fi. {2),
(8)
i Fb. (RSTBs" (Fi.n)))
In the blurry decomposition task, a common issue directional ambiguity 70 reconstruct the center position of the exposure window in the spike branch, the output of RSTBB contains time-specific spike encoding information, which can avoid this problem. During online inference, can decompose the blurry image into different timestamps by moving the position of the spike stream, and the frame rate of the decomposed images depends on the sampling rate of the spike camera.
Since we
3.4 Cross-Modal Information Complementing
Spike Reconstruction Based on High Resolution Prior Image Domain. Each pixel in the spike camera asynchronously outputs high-time-resolltionn spikes according to the brightness proportional to the input light intensity. This mechanism enables the spike stream effectively capture and represent both temporal and spatial information of visual inputs. However, the current spatial resolution of the
5

---

Lavcnn m
sin Cone
An X.
T,
Nop Fanking
fml
Lvi Cone
Domm
X
SwT Cofe
3-8 DCoi
gul
Content-Awar Motion Magnitude Attention
(b) Transposed Crass coss-Attention Fusion (TCAF)
Module (CAMMA)
Figure 4: Illustration of the Content-Awaid Motion Magnitude Attention Module and the Transposed Cross-Attention Fusion Module.
spike camera is relatively low. In our module, explicitly reconstruct high-resolution grayscale images the spike stream to enable the network to learn image-like features. Given the ground truth grayscale image I at the moment of the exposure window center, the loss function is formulated as:
Crccon |11 1
(10)
Reconstructing nigh-resolution frames solely from the spike stream itself remains challenging. To fully leverage the advantages of both input modalities, propose to utilize information from the image domain as a rishuuusolutton prior to spike reconstruction. However, the input images are mostly highly blurred, which hinders the provision of high-resolution clues for reconstruction. Therefore, we further introduce the CAMMA module to adaptively search for regions with smaller motion magnitudes in the initial deblurred image, and fuse them into the spike branch.
Content Aware Motion Magnitude Attention Module. order to incorporate high-resolution prior knowledge from the image domain into the spike domain, intuitive idea is to directly fuse the input blurred image with the spike branch. The blurry foreground caused by moving objects cannot provide effective texture information, which be handled by using mask retain sharp backgrounds [47). Moreover, the global blur caused by camera motion unable to provide effective high-resolution prior. Therefore, we propose CAMMA module to encode the motion magnitude from the initial deblurred result. As shown in Fig. we first pass the S through shallow convolution and upsampling to obtain the initial deblurred image T,. Through training, the blur in T, is initially reduced. Subsequently, we need a coarse sharp image as a reference to obtain the blurry region by comparing with 1,, We adopt the basic statistical method text-from-interval (TFI) 72 to center-spike firing interval (CSFI) to obtain the central coarse sharp image. CSFI estimates the pixel value the current pixel based on the observation that the spike camera's firing frequency each pixel is proportional to the real light intensity by statistically analyzing the spike interval. The process is formulated as:
C
(11)
At'
where At represents the inter-spike interval corresponding moment and the C refers to the maxi- mum dynamic range of the reconstruction. Subsequently, we perform differencing and normalization on the upsampled Ieefs and I and then apply thresholding to obtain the hard thresholding motion magnitude mask. However, since r is merely a basic reconstruction and suffers from noise (72). In order to enable the network to learn a more robust motion magnitude mask, we concatenate re and T, and use simplified channel attention
3 with the hard thresholding mask to obtain a content-aware motion magnitude mask M. After multiplying the motion magnitude mask with Ie, utilize the TCAF module fuse it into the spike branch. We refer to the CAMMA module, the CSFT process, and the F' F.)(()) module as the CAMMA branch, with which the high-resolution prior knowledge from the image domain can be transferred effectively into the spike domain.
Upike--Gided Motion Deblurring. As mentioned in Sec. 3.3 our SpkDeblurNet involves informa- tion transfer from image to spike and spike to the image. The modalities differ significantly for the former, while the differences
minor for the latter. In order to better integrate the two modalities, we propose the TCAF Module. The query (Qm is derived from one modality m1 with e₁ channels, que while the key (K (2) and value (Vm2) are obtained from another modality m2 with e2 channels. The attention is computed using the following equation:
fml pml + mm2
G(K Qml (a).
(12)

---

Table 1: Comparison of various motion deblurring methods on Spk-X4K 100-X4KKOOOFFPP
Method
Extra Data
PSNR
SSIM 1
PSNR
SSIM
#Param
Spk- 000000SSSS
33
65
HIINet A
33.98
0.933
29.19
0.867
88.7M
NAFNet 8
34.13
0.937
29.06
0.878
67.9M
EFNet (8
Spike
36.36
0.960
33.53
0.937
8.5M
REFID
Spike
36.30
0.962
33.47
0.945
15,9M
SpkDeblurNet (Ours)
Spike
0.968
35.94
0.966
13.5M
Where f and fml are the features before and after fusion for modality m1, G denotes the softmax function and is a learnable scaling parameter. However, the calculation of attention faces two problems: 1) The spike branch is lightweight with fewer channels compared to the deblurring branch, which makes traditional attention computation in the spatial dimension not directly applicable. 2) The computational cost of traditional attention grows quadratically with the input size, which is not acceptable for image restoration tasks. In the main network, mitigate these issues by employing the RSTB [37 36]. which calculates attention within windows to reduce computation. In TCAF, we adopt the Transposed Attention proposed in [59], which performs attention computation along the channel dimension, thereby addressing the aforementioned problems. We apply the TCAF module to the fusion of features in both directions.
3.5 Joint Learning of Deblurring and Reconstruction
We jointly train the proposed SpkDeblurNet. The spike reconstruction branch obtains high-resolution image priors from the initial deblurred result, enabling the recovery of better image-like features. The features from the spike branch in turn guide the deblurring branch, providing time-specific texture information. For both the initial deblurred result /, and the refined deblurred result Í,, we utilize the LI loss for supervision during training. The loss function of the entire network can formulated as follows:
L = || f.ll1 + A 1211 >2 Tella,
(13)
where d and 22 are hyperparameters that control the loss terms.
3.6
Datasets
This paper is the first study on spike-based motion deblurring, and no large-scale training datasets are currently available. Therefore, we built two large-scale spike-based datasets based on existing real image datasets [39] (46] for training and validation. The first one, named 'Spk-X4K 100-XXKS is built on the dataset 1000FSS which contains clear images captured by 1000fps cameras. To generate spikes, we utilize the interpolation method [б] to further increase the frame rate four times. Then we build a simple spike simulator according to the spike generation mechanism. It inputs continuous sharp images as light signals and converts them to spikes. For clips in the dataset, we use the average of neighboring e sharp images around the ground truth images to generate blurry images, where = 33, 65. The second one, named "Spk-GoPro is built on the GoPro 33 dataset in a similar way as above. More implementation details are attached in the supplementary materials.
Experiments
4.1 Comparative Results
In this section, present comparative results both Spk-X4KI000FPS and Spk-GoPro datasets. For D0X-X4K1000FPP we generate blurry images by setting an exposure window = 33 as well extreme exposure window of 65. For Spk-GoPro, we use the provided blurry images. Implementation details can be found supplementary materials.
Results on Spk-X4K 10000FSS. Tab. [ and Fig. [5 present quantitative and qualitative comparisons of our method with other approaches under two different exposure window settings. For BFNet and REFID, we use their networks for spike-assisted deblurring. For EFNet, we employed input representations similar to SCER approach, excluding the EMGC module due to its event-related nature. Regarding REFID, we utilized similar spike voxel representations. All comparative methods
7

---

Truth
Blurry Image (e=33)
36.41dB
Blurry Image (e-65)
NAFNet 28.85d8
HiNet 29.52dB
Ours 36. 12d8
Ground Truth
Blurry Image (e-33)
NAFNet 29.44dB
HINet 26.77dB
Ours 30.15dB
Blurry (e=65)
NAFNet 24.21dB
HINet 22.99dB
Ours 28.47dB
Figure 5: Visualized results on Spk-X4K dataset, compared with NAFNet
3 and HINet 4
trained from scratch on the proposed dataset. We observe that both image-domain methods perform well in recovering image details with a smaller exposure window setting (e = 33). However, when the exposure window becomes larger (e 65), pure image-based methods struggle to find from extremely blurry scenes to restore clear images, resulting in significant loss of texture details or generating artifacts. In the second row of the first sample sequence in Fig. 5 NAFNet and HINet mistakenly recover non-existent stair edges. In contrast, our multimodal approach leverages spike streams to find cues and restore correct and clear textures. Additionally, the networks in event-assisted methods fall short of our network. This is due to our network's 's superior bidirectional information transmission nature.
Results on Spk-GoPro. As shown in Tab. 2 our proposed SpkDeblurNet outperforms the state-of- the-art model REFID 49) and other competitive models in terms of PSNR On the GoPro dataset. We have observed that both our SpkDeblunNet and the SOTA event-based methods have significantly outperformed image-based methods. which demonstrates the substantial potential of neuromorphic data in complementing information for the image domain. Fig. [5] visualizes the deblurred results of different methods, from which we demonstrate that the quality of sharp images predicted by the SpkDeblurNet is more promising and satisfying than others. In the first sample of Fig. [3] our approach successfully restored the clear license plate number while also effectively recovering the fallen leaves on the ground, while other methods only achieved relatively blurred restoration of the ground. Furthermore, our method excelled in restoring the clearest facial features of the person in the second sample. Zoom in for more details. Our framework, by leveraging the complementary information from both image and spike domains, has surpassed event-based methods that solely rely on unidirectional information transfer in terms of both quantitative and qualitative results.
Results on Real-World Scenes. To validate the generalization of our method in real-world scenarios. we construct a hybrid camera system to capture real-world spike-image data. We use beam splitter connect spike camera and conventional frame-based camera, enabling both cameras have the same view. We rapidly wave objects in front of the cameras to simulate high-speed motion. We synchronize the two cameras and manually aligned their spatial and temporal outputs. Fig. presents visual results of applying our model trained on the Spk-X4K 1000FPS dataset with window size of 33 to two real-world sequences. colors. This demonstrates the good generalization of our proposed algorithm and dataset, highlighting the potential of our algorithm for real-world applications. More results and the hybrid camera system diagram can be found in the supplementary materials.
can observe desirable results with clear textures and accurate

---

Table 2: Comparison of various motion deblurring methods on Spk-GoPro. HINett: event-enhanced
versions of HINet
Method
Extra Data
PSNR t
SSIM
#Param
Spk-GoPro
DeblurGAN-V. 26
29.55
0,934
Nete 43
Event Event
31.60
0,940
LEMD 18
31.79
0,949
MPRNet
61
32.66
0,959
20.0M
HINet
32.71
0,959
88.7M
Restormer
59
32.92
0,961
26.1M
ERDNet 14
Event
32,99
0,935
HINet+ 4
Event
33.69
0,961
88.7M
NAFNet
33.69
0,967
67.9M
EFNet
Event
35.46
0,972
8.5M
REFID 49
Event
35.91
0,973
15.9M
SpkDeblurNet (Ours)
Spike
36.12
0.971
13.5M
Blurry Image
HINet
HINet+
TSPN
Ground Truth
EFNet
Ours
Blurry Image
HINet
HINet+
Ground Truth
EFNet
Ours
Figure 6: Visualized results on Spk-GoPro dataset.
Ablation Studies
In this section, we conduct series of ablation studies on Spk-X4p X0OFFS dataset. Effectiveness of the proposed modules. As shown Tab. 3 removing the CAMMA branch resulted in performance degradation of 0.2-0.3 dB, confirming the necessity of introducing features from the deblurring branch and the effectiveness of the CAMMA module's Icarnable magnitude masking mechanism. Additionally, replacing the TCAF module with concatenation led to minor performance degradation, which we attribute to the explicit supervision the spike reconstruction branch enabling the branch to learn image-liks features, thereby the modality gap is reduced.
Information supplementation from image domain spike domain. We further conduct ablation studies on spike reconstruction with different combinations of three input representations and the usage of the CAMMA branch. To be specific, employ either coarse reconstruction CSFI symmetric cumulative spike representation similar to EFNet [48). For the latter, we accumulate spikes from the center towards both ends using three window lengths and concatenate the results along the channel dimension. As shown in Tab. 4 CAMMA consistently significantly improves spike reconstruction across all input representations. For settings using CSFI and cumulated spikes as inputs, CAMMA enhances performance by 1.5-3.5 dB. This is due to the simpler nature of CSFI and cumulated spikes, which contain less texture and motion information. High-resolution image priors from the deblurring branch compensate for this, further enhancing spike reconstruction. Even with spike streams as input, CAMMA still yields improvements of 0.4-0.7 dB. This set of ablation experiments demonstrates the effectiveness of the proposed scheme in incorporating high-resolution features from the deblurring branch into the spike branch, thereby enhancing spike reconstruction performance and providing better guidance and cues for the deblurring task.

---

Dann
Dann
Dana
Danu
TRAOO
TEOO
TRADE
TRADIT
BUTTER COO
BUTTER COC
BUTTER COC
BUTTER COC
Real-World Blurry Image
HINet
NAFNet
Ours
Figure 7: Qualitative Results on Real- World Scenes.
Table 3: Ablation study of our proposed modules and input representations.
Experiment settings
PSNR
SSIM f
PSNR f
SSIM 1
Spk-X4KI000 TO00FPS
33
65
Replace imput representation with CSFI
36.08
0.957
34.22
0.950
Replace input representation with cumulate spikes
36.34
0.955
34.92
0.955
Replace TCAF with concatenation
37.36
0,968
35.81
0,965
Remove CAMMA branch
37.20
0.967
35.64
0.965
Our final SpkDeblurNet
37.42
0.968
35.94
0.966
Table 4:
Ablation study of CAMMA spike reconstruction with different inputs.
Experiment settings
PSNR
SSIM 1
PSNR f
SSIM
Input Representation
CAMMA
33
65
CSFI
33.67
0,923
33.69
0,922
CSFI
37.25
0,939
36.13
0.935
Cumulated Spikes
X
35.58
0,926
35.11
0.914
Cumulated Spikes
37.54
0.935
36.65
0.936
Spike Streams
X
38.72
0.953
38.91
0.952
Spike Streams
39.41
0.955
39.32
0.954
Information supplementation from spike domain to image domain. We investigate the plementary role of the spike domain to the image domain by replacing the input representation. Quantitative results in Tab. inddicatd that using coarse reconstruction input still yielded competi- tive performance, suggesting that the rough texture information contained in CSFI serves as good complement the deblurring branch. On the other hand, using cumulated spikes input further improved the performance. As a comparison, we achieved the highest performance using pure spike stream as input, demonstrating the network's ability fully excavate the rich temporal information and clues embedded in spike streams to recover clear images. We further conduct ablation studies on different combinations of three imput representations and the usage of the CAMMA module for the deblurring task, the results can be found in the supplementary materials.
5 Conclusion
In this paper, we propose the first spike-based motion deblurring model with bidirectional information complementarity, in which we propose a content-aware motion magnitude attention module and transposed cross-attention fusion module. We build two large-scale synthesized datasets used for both training and validation, which contain high-temppora-resolutiie spikes, blurry images, and sharp images. Experimental results demonstrate that the proposed method outperforms other deblurring methods and achieves desirable results.
Limitations. Our future work will prioritize enhancing the resolution of the spike stream to improve its auxiliary capability for image-domall tasks, considering the limitation of the current low resolution of spike cameras and the significant impact of spatial super-resolution on applications.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant No. 62 17603 and No. 62088102.
10

---

References
[1] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240x 180 130 μus latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits. 49(10))2333-2344 2014.
[2] Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, and Andreas Kolb. bit too much? high speed imaging from sparse photon counts. In 2019 IEFE International Conference on Computational Photography (ICCP), pages 1-9. IEFE, 2019.
[3] Liangyu Chen. Xiaojie Chu. Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII. pages 17-33. Springer, 2022.
[4] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEFF/CVP Conference on Computer Vision and Pattern Recognition, pages 182-192, 2021.
[5] Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI, pages 2859-2866, 2022.
[6] Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In Proceedings the AAAI Conference on Artificial Intelligence, volume 37, pages 350-358, 2023.
Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, and Stanley H Chan. Dynamic low-light imaging with quanta image sensors. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 122-138. Springer, 2020.
[8] Sung-Jin Jin Cho, Seo-Won Ji. Jun-Pyo Hong, Seung-Wor Jung. and Sung-Jea Ko. Rethinking coarse-to- fine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4641-4653 2021.
[9] Siwei Dong, Lin Zhu, Daoyuan Xu, Yonghong Tian, and Tiejun Huang. An efficient coding method for spike camera using inter-spike intervals. In 2019 Data Compression Conference (DCC), pages 568-568. IEFE, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.II1 11929, 2020.
[11] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Fva: Exploring the limits of masked visual representation learning scale. arXiv preprint arXiv:221 1.076666, 2022.
[12] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam Roweis, and William T Freeman. Removing camera shake from a single photograph. In Acm Siggraph 2006 Papers, pages 787-794, 2006.
[13] Eric Fossum, Jiaju Ma. Saleh Masoodian, Leo Anzagira, and Rachel Zizza. The quanta image sensor: Every photon counts. Sensors, 1268),1200, 2016.
[14] Chen Haoyu, Teng Minggui, Shi Boxin, Wang Ylzhou, and Huang Tiejun. Leaminng to deblur and generate high frame rate video with an event camera. arXiv preprint arXiv:2003.0084. 2020.
[15] Liwen Hu, Rui Zhao, Ziluo Ding, Lei Ma. Boxin Shi. Ruiqin Xiong, and Tiejun Huang. Optical flow estimation for spiking camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17844-1 17853, 2022.
[16] Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma. Junwei Zhao, Siwei Dong, Lin Zhu, al. 1000x faster camera and machine vision with ordinary devices. Engineering, 2022.
[17] Atul Ingle, Andreas Velten, and Mohit Gupta. High flux passive imaging with singlc-photon sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6760-6769. 2019.
[18] Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu. Learning event-based mo- tion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 3320 3329, 2020.
11

---

[19] Meiguang Jin, Zhe Hu, and Paolo Favaro. Learning to extract flawless slow motion from blurry videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognittin, pages 8112-8121, 2019.
[20] Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning extract video sequence from single motion- blurred image. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6334-6342, 2018.
[21] Tae Hyun Kim and Kyoung Mu Lec. Generalized video deblurring for dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1426-5434, 2015.
[22] Taewoo Kim, Jeongmin Lee, Lin Wang, and Kuk-Jin Yoon. Event-guided dehlurring of unknown exposure time videos. In Computer Vision-FCCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII. pages 519-538. Springer, 2022.
[23] Lingshun Kong. Jiangxin Dong, Mingqiang Li, Jianjun Ge, and Jinshan Pan. Efficient frequeney domain- based transformers for high-qualit image deblurring. arXiv preprint arXiv:2211.12250, 2022.
[24] Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using normalized sparsity measure. In CVPR 2011. pages 233-240. IEEE, 2011.
[25] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych. Dmytro Mishkin, and Jiff Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183-8192, 2018.
[26] Orest Kupyn. Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (orders- of-magnitud) faster and better. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8878-8887, 2019.
[27] Martin I. Laurenzis. Single photon range, intensity and photon flux imaging with kilohertz frame rate and high dynamic range. Optics express, 27126)33331-33840, 2019.
[28] Anat Levin. Blind motion deblurring using image statistics. In Advances in Neural Information Processing Systems, pages 841-848, 2007.
[29] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1964-1971. IEEE, 2009.
[30] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Efficient marginal likelihood optimization in blind deconvolution. In CVPR 2011, pages 2657-2664. IEEE, 2011.
[31] Jingyun Liang, Jiczhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawci Li, Radu Timofte, and Luc Van Gool, Vrt: A vidco restoration transformer. arXiv preprint arXiv:2201. /22888 2022.
[32] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833-1844, 2021.
[33] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. 128x 128 120 db 15 us latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566-576, 2008
[34] Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang. Jing Chen, and Jimmy Ren. Learning event-driven video deblurring and interpolation. In Computer Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 695-710. Springer, 2020.
[35] Yuhao Liu, Felipe Birierrez-Barragan, Atul Ingle, Mohit Gupta, and Andreas Velten. Single-photon camera guided extreme dynamic range imaging. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1575 1585, 2022.
[36] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Sealing up capacity and resolution. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2021.
12

---

[38] Sizhuo Ma, Shantanu Gupta, Arin C Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta. Quanta burst photography, ACM Transactions on Graphicx (TOG), 39(4):79-1, 2020.
[39] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883-3891, 2017.
[40] Jihyong Oh and Munchurl Kim. Demfi: Deep joint deblurring and multi-frame interpolation with flow- guided attentive correlation and recursive boosting. arXiv preprint arXiv:2/11. 1.09985, 2021.
[41] Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hardey. Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6820-6829, 2019.
[42] Wenqi Ren, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang. Video deblurring via semantic segmenta- tion and pixel-wise non-linear kernel. In Proceedings of the IEEE International Conference on Computer Vision, pages 1077-1085, 2017.
[43] Wei Shang, Dongwei Ren, Dongqing Zou, Jimmy S Ren, Ping Luo, and Wangmeng Zuo. Bringing events into vidco deblurring with non-consecutively blurry frames. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531 4531-4540, 2021.
[44] Wang Shen, Wenbo Bao, Guangtao Zhai. Chen, Xiongkuo Min, and Zhiyong Gao. Blurry video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 51 14-5123, 2020.
[45] Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao. Video frame interpolation and enhancement via pyramid recurrent framework. IEEE Transactions on Image Processing. 30:277 292, 2020.
[46] Hyconjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In Proceedings of the IEEE/CVF international conference on computer vision, pages #4489-14498, 2021.
[47] Maitreya Suin and AN Rajagopalan. Gated gpatio-tempont attention attention-guided video deblurring. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7802-78 7811, 2021.
[48] Lei Sun, Christos Sakaridis, Jingyun Liang. Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang. and Lue Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XVIII, pages 412 428. Springer, 2022.
[49] Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang. and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. arXiv preprint arXiv:2301.00 001123,,. 2023.
[50] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8174-8182, 2018.
[51] Minggui Teng, Chu Zhou, Hanyue Lou, and Boxin Shi. Nest: Neural event stack for event- image enhancement. In Computer Vision Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part VI, pages 660 676. Springer, 2022.
[52] Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang. Event enhanced high-quality image recovery. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020. Proceedings, Part XIII 16, pages 171. Springer, 2020.
[53] Yixuan Wang. Jianing Li, Lin Zhu, Xijie Xiang, Tiejun Huang, and Yonghong Tian. Learning stereo depth estimation with bio-inspired spike cameras. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1 6. JEEE, 2022.
[54] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17683-17693, 2022.
[55] Jonas Wulff and Michael Julian Black. Modeling blurred video with layers. In European Conference on Computer Vision, pages Springer, 2014.
13

---

[56] Xijie Xiang, Lin Zhu, Jianing Li, Yixuan Wang, Tiejun Huang. and Yonghong Tian. Learning super- resolution reconstruction for high temporal resolution spike stream. IEEE Transactions on Circuits and Systems for Video Technology. 2021.
[57] Fang Xu, Lei Yu, Bishan Wang, Wen Yang. Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion dehlurring with real events. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2583-2592, 2021.
[58] Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image dehlurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1107-1114, 2013.
[59] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming King-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.
[60] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hssun Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 492-511. Springer, 2020.
[61] Syed Waqus 7amir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang. and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821-14831, 2021.
[62] Jiyuan Zhang. Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part VII. pages 34-52. 52. Springer. 2022.
[63] Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022.
[64] Jing Zhao, Jiyu Xie, Ruiqin Xiong, Jian Zhang, Zhaofci Yu, and Tiejun Huang. Super resolve dynamic scene from continuous spike streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2542, 2021.
[65] Jing Zhao, Ruigin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings the Vision and pottern Recognition, pages 11996-12005, 2021.
the IEFF/CVF Conference Computer
[66] Junwei Zhao, Shiliang Zhang, Zhaofei Yu, and Tiejun Huang. Spireco: Fast and efficient recognition of high-speed moving objects with spike cameras. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2023.
[67] Rui Zhao, Ruiqin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. Advances in Neural Information Processing Systems, 35:7905-79222 2022.
[68] Yajing Zheng, Zhaofei Yu, Song Wang, and Tiejun Huang. Spike-hased motion estimation for object tracking through bio-inspired unsupervised learning. IEEE Transactions on Image Processing, 32:335-349, 2023.
[69] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi. Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6358-6367, 2021.
[70] Zhihang Zhong, Mingdeng Cao, Xiang Ji, Yinqiang Zheng, and Imari Sato. Blur interpolation transformer for real-world motion from blur. arXiv preprint arXiv:2211.1 1. 2022.
[71] Zhihang Zhong, Xiao Sun, Zhirong Wu, Yinqiang Zheng, Stephen Lin, and Imari Sato. Animation from blur: Multi-modal blur decomposition with motion guidance. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 27, 2022, Proceedings, Part XIX, pages 599-615. Springer, 2022.
[72] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian, A retina-inspired sampling method for visual texture reconstruction. In 2019 IEFE International Conference on Multimedia and Expo (ICME), pages 1432-1437. IEEE, 2019.
[73] Lin Zhu, Siwei Dong, Jianing Li. Tiejun Huang, and Yonghong Tian. Retina-like visual image recon- struction via spiking neural model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1438- 1446, 2020.
14


Reviews:
[{"review_id": "13OCxFa2Ve", "reviewer": "Reviewer_Rugk", "summary": "This paper introduces a novel approach that combines traditional cameras and spike cameras to address motion blur in high-speed scenes. By leveraging spike streams as auxiliary visual cues, the proposed spike-based motion deblurring model effectively extracts relevant information from blurry images using content-aware motion magnitude attention and transposed cross-modal attention fusion modules.", "questions": "The related works and the comparison section should ideally have a detailed comparison with the QIS based methods.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes to novel technique to deblur a blurry scene using a traditional camera frame along with spike camera frames. The proposed method is able to leverage the high SNR information from the traditional camera frame and the motion information from the spike camera frames to reconstruct the final image with high quality.", "weaknesses": "The authors have missed a major branch of related works using Quanta Image Sensors (QIS). QIS are a very similar family of image sensors that operate at high speed and produce single-bit frames, and has a lot of previous works on deblurring and denoising. For eg.,  [1] Ma, S., Gupta, S., Ulku, A.C., Bruschini, C., Charbon, E. and Gupta, M., 2020. Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), pp.79-1. [2] Chi, Y., Gnanasambandam, A., Koltun, V. and Chan, S.H., 2020. Dynamic low-light imaging with quanta image sensors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16 (pp. 122-138). Springer International Publishing. [3] Chandramouli, P., Burri, S., Bruschini, C., Charbon, E. and Kolb, A., 2019, May. A bit too much? High speed imaging from sparse photon counts. In 2019 IEEE International Conference on Computational Photography (ICCP) (pp. 1-9). IEEE.\n\nIn fact, \"[4] Liu, Y., Gutierrez-Barragan, F., Ingle, A., Gupta, M. and Velten, A., 2022. Single-photon camera guided extreme dynamic range imaging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1575-1585). \" uses a very similar idea of combining two imaging modalities, but for a different task - HDR imaging.\n\nEDIT:\nWith the promised change, my major concern will be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***1. The authors have missed a major branch of related works using Quanta Image Sensors (QIS). The related works and the comparison section should ideally have a detailed comparison with the QIS-based methods.}***\n\nThanks for recommending these works. We appreciate the recognition of Quanta Image Sensors' (QIS) ability to capture high-speed motion and perform well across various tasks. In the revised paper, we have thoroughly reviewed and incorporated the following references[1-4] into our introduction, related work, and comparative experiments. We will incorporate the discussion on these works in our final version, where the one additional page allows us to extend the current Sections with more content and illustrations.\n\nFurthermore, we would like to gently emphasize that while both the spike camera and QIS are capable of capturing high-speed scenes, there exist significant differences between the two technologies. QIS primarily relies on the single-photon avalanche diode (SPAD) detector technology, whereas the spike camera is constructed from CMOS sensors similar to traditional cameras and utilizes regular semiconductor manufacturing processes, resulting in a more cost-effective solution. Exploiting the fact that the time sensitivity of CMOS photosensitive devices widely used today has reached tens of nanoseconds, the spike camera achieves exceptional temporal resolution by mimicking the sampling mechanism of the primate fovea.\n\nAdditionally, we would like to clarify that the works of [1-3] focus on reconstruction tasks specific to QIS and involve single-modal data only. Moreover, while both our method and the QIS-assisted HDR imaging method[4] incorporate two modalities, this does not imply a similarity between our work and [4]. Firstly, multi-sensor modality fusion is a common task in the field of autonomous driving. Secondly, [4] simply concatenates the features of two modalities during skip connections for fusion. In contrast, our method takes a bidirectional complementary approach to leverage information from both modalities. We introduce the initial deblurred output from the image branch as a high-resolution image domain prior to guiding the spike branch to aid spike reconstruction, and then we introduce the refined spike features into the image branch to guide deblurring in the image domain. Additionally, we propose the TCAF module for the effective fusion of the two branches. \n\nSince our method and [4] are two different tasks and cannot be directly compared, to validate the effectiveness of our proposed framework, we adapted the network of [4] for our spike-assisted deblurring task under their experiment protocol. Experimental results in Tab.R0 indicate that our framework achieves superior performance over [4] and can better utilize the complementarity between different modalities to improve performance.\n\n**Table.R2: Comparisons between different networks on Spk-X4K1000FPS dataset.**\n|  Method   | PSNR ($e=33/e=65$) | SSIM ($e=33/e=65$) |\n|  ----  | ---- | ---- |\n|  CMOS-SPC [4]  |33.65/31.59 |0.926/0.912 |\n| SpkDeblurNet (Ours) |**37.42/35.94** |**0.968/0.966** |\n\n[1] Ma, S., Gupta, S., Ulku, A.C., Bruschini, C., Charbon, E. and Gupta, M., 2020. Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), pp.79-1. \n\n[2] Chi, Y., Gnanasambandam, A., Koltun, V. and Chan, S.H., 2020. Dynamic low-light imaging with quanta image sensors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16 (pp. 122-138). Springer International Publishing. \n\n[3] Chandramouli, P., Burri, S., Bruschini, C., Charbon, E. and Kolb, A., 2019, May. A bit too much? High speed imaging from sparse photon counts. In 2019 IEEE International Conference on Computational Photography (ICCP) (pp. 1-9). IEEE.\n\n[4] Liu, Y., Gutierrez-Barragan, F., Ingle, A., Gupta, M. and Velten, A., 2022. Single-photon camera guided extreme dynamic range imaging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1575-1585)."}, {"review_id": "mnDQUeeM6n", "reviewer": "Reviewer_DRAW", "summary": "The paper proposes a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes, introducing a content-aware motion magnitude attention module and transposed \n cross-modal attention fusion module.", "questions": "See weaknesses.", "limitations": "See weaknesses.\n\nI encourage the authors to actively and rigorously prepare the rebuttal and I will raise the rating if my concerns are well-addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Authors propose the first spike-based motion deblurring model equipped with content-aware motion magnitude attention and a cross-modal transposed attention fusion module.\n\n2. Extensive experiments have demonstrated the effectiveness of the proposed model \n\n3. Overall, the paper is well-written and technically sound.\n", "weaknesses": "1. Although the authors conducted the validation on two synthetic datasets that the proposed method is effective, the performance on real scenarios is still unknown. Especially, it is hard to simultaneously obtain aligned pairs of spike streams and RGB images. So, have authors ever considered constructing a camera system to truly capture the spike data and blur images to better demonstrate the proposed setup is feasible and has practical value ?\n\n2. Considering the extra cost of spike cameras, is it really worth using such an expensive device for deblurring rather than using a high-speed global shutter camera directly capture relatively sharp images? Authors should provide more motivations for using spike cameras from the perspective of reality.\n\n\n3. What about the generalization ability of the proposed method? Authors should present related experiments. Moreover, the computational complexity and inference time are also supposed to be compared.\n\n4. Some minor issues: typos (e.g. line 15)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive feedback. We are encouraged that you find our method effective. We would like to address your concerns and answer your questions here.\n\n***1. Have authors ever considered constructing a camera system to truly capture the spike data and blur images to better demonstrate the proposed setup is feasible and has practical value ?***\n\nThanks for your suggestion. Please refer to Response 1 in **To all reviewers**.\n\n***2. Authors should provide more motivations for using spike cameras from the perspective of reality.***\n\nWe would like to clarify that the spike camera is built using the same CMOS sensors as traditional cameras, and it leverages consumer-grade integrated circuits through regular semiconductor manufacturing processes, making it a cost-effective solution [1]. In contrast, conventional high-speed cameras, such as Phantom cameras, necessitate specialized sensors and shutters that are considerably expensive. Specifically, the spike camera employs an innovative temporal domain sampling method with continuous photon capture by photosensitive units, as opposed to synchronous exposure with a fixed exposure time. When accumulated intensity surpasses a predefined threshold, a spike is generated. These spikes, generated by each photosensitive unit, are spatially organized to form a spike stream array. This sampling technique enables achieving exceptionally high sampling rates on conventional CMOS sensors. Considering the low cost of the spike camera and its advantages of high speed and high dynamic imaging and recording rich spatial texture information, we believe that the spike camera holds promising prospects for various applications.\n\n***3. What about the generalization ability of the proposed method? Moreover, the computational complexity and inference time are also supposed to be compared.}***\n\nWe showcase visual results in Supplementary Fig. S2 by applying our model trained on the Spk-X4K1000FPS dataset with a window size of 33 to real-world sequences. These results demonstrate the generalization ability of our method to real data. \nAdditionally, we conduct experiments to test models trained on Spk-X4K1000FPS ($e=33$) on data with $e=65$, as well as cross-dataset testing by applying models trained on Spk-X4K1000FPS to the GoPro dataset. The visualized results are provided in the PDF file of the global response. All these results demonstrate the robust generalization of our method across diverse blurring conditions and scenes. \n\nWe have added comparisons regarding computational complexity and inference time in Tab.R3 (note that the settings for e=33 and e=65 are only applicable to the Spk-X4K1000FPS dataset). Our approach achieves a favorable balance between complexity and performance, as evident from our results.\n\n**Table.R3: Comparisons between different methods regarding computational complexity and inference time.**\n|  Method    | MACs | Params | Inference Time | PSNR (on GoPro) |\n|  ----  | ---- | ---- |---- |---- |\n| HINet  |170.49G |88.67M |20.2ms |33.69|\n| NAFNet  |63.06G|67.79M |27.8ms |33.69|\n| EFNet  |107.93G |7.73M |14.9ms |35.46|\n| REFID  |4.36T |88.81M |781.2ms |35.91|\n| Ours ($e=65$)  |53.25G |12.93M |140.6ms |N/A|\n| Ours ($e=33$)  |53.18G |12.92M |107.9ms |N/A|\n| Ours ($e=56$)  |53.23G |12.93M |130.1ms |36.12|\n\n\n\n***4. Some minor issues: typos (e.g. line 15)}***\n\nWe appreciate your keen attention to detail. The mentioned typos have been rectified in the revised version of the paper.\n\n[1] Huang T, Zheng Y, Yu Z, et al. 1000× faster camera and machine vision with ordinary devices[J]. Engineering, 2022."}, {"review_id": "nRYmjoCJdw", "reviewer": "Reviewer_wVmq", "summary": "This paper proposes a motion deblurring method that integrates RGB images and binary spike streams. In detail, it has a content-aware motion magnitude attention module and a transposed cross-modal attention fusion module. Experiments demonstrate state-of-the-art performance on deblurring datasets.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, the first spike-based motion deblurring model.\n\n2, the two large-scale synthesized datasets for spike-based motion deblurring\n\n3, state-of-the-art results.", "weaknesses": "1, The Transposed Cross-Attention Fusion (TCAF) is close to the multi-Dconv head transposed attention (MDTA) in Restormer.\n\n2, The TCAF module is similar to EICA in EFNet.\n\n3, The spike camera is similar to event camera, and the method based on these two methods are also similar except the format of event/spike stream. I suspect that event-based methods also works with spike camera. Because the paper mainly forcuses on the algorithm design, and the proposed SpkDeblurNet makes no huge differences with the existing EFNet, from this aspect the novelty of the paper is limited.\n\n4, In the abstract, \"preserving rich spatial details\" and \"are low-resolution image signals\" are conflicting and confusing.\n\n5, Typo in abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your summary, and we appreciate you for pointing out the strengths of our paper. My clarification and answers for the weaknesses you summarized are as follows.\n\n***1. The Transposed Cross-Attention Fusion (TCAF) is close to the multi-Dconv head transposed attention (MDTA) in Restormer, and the TCAF module is similar to EICA in EFNet.***\n\nThanks for pointing it out. As described in Section 3.4.3, unlike the original usage of MDTA solely for self-attention computation, in TCAF, we have adapted and modified the MDTA module proposed in Restormer to address two specific challenges in our multi-modal fusion: 1) The lightweight nature of the spike branch with fewer channels compared to the deblurring branch makes conventional spatial attention computation not directly applicable. 2) The computational cost of traditional attention mechanisms grows quadratically with input size, which is impractical for image restoration tasks.\n\nFurthermore, as an effective module, the MDTA module has been drawn upon by multiple studies[1,2]. In contrast to EFNet's EICA module, our TCAF differs in that the two modalities within our module possess differing channel counts, with the spike branch having fewer channels. We also employ depth-wise convolutions to enhance local context before computing the attention map, aiming to improve generalization.\n\n***2. The spike camera is similar to the event camera, and the method based on these two methods are also similar except for the format of the event/spike stream. I suspect that event-based methods also work with spike camera.***\n\nPlease refer to Response 2 in **To all reviewers**.\n\n***3. In the abstract, \"preserving rich spatial details\" and \"are low-resolution image signals\" are conflicting and confusing.***\n\nWe would like to clarify that the phrase \"preserving rich spatial details\" refers to the spike cameras' ability, **in contrast to event cameras** that primarily capture temporal information, to capture per-pixel spatial information alongside temporal data. On the other hand, \"are low-resolution image signals\" reflects the current state of spike cameras, where the sensor array is limited to $250\\times 400$ photosensitive units, resulting in low-resolution signals **compared to high-resolution RGB images**. We will revise the wording in the revised version to mitigate any potential misunderstandings.\n\n***4. Typo in the abstract.***\n\nThank you for your careful observation. We have revised these typos in the paper.\n\n[1] Sun L, Sakaridis C, Liang J, et al. Event-based fusion for motion deblurring with cross-modal attention[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 412-428.\n\n[2] Song J, Mou C, Wang S, et al. Optimization-Inspired Cross-Attention Transformer for Compressive Sensing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 6174-6184.\n\n[3] Sun L, Sakaridis C, Liang J, et al. Event-Based Frame Interpolation with Ad-hoc Deblurring[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 18043-18052."}, {"review_id": "AbYhFXyK4u", "reviewer": "Reviewer_9w9w", "summary": "The paper attempts to remove motion blur from high-speed scenes making use of spike streams. Most deep learning-based deblurring algorithms predict sharp frames relying only on the input blurry frames and are not robust when the blurry artifact is severe. This work proposes to use spike streams that could be obtained from spike cameras along with the blurry input for motion deblurring. The authors propose a framework that integrates both modalities in a manner where information is shared bi-directionally. In addition, the work builds two synthetic datasets on top of the GoPro and X4K1000FPS datasets for training and evaluation. Experimental comparisons are presented with several baselines.  ", "questions": "Please refer to the \"Weaknesses\" section and address the raised concerns carefully.", "limitations": "The authors adequately address the limitations of their work.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The proposed problem formulation of motion deblurring using spike streams is quite new and a promising direction for further research\n* The joint training of RGB and gray image reconstruction is an intuitive approach to enforce the proposed network to use both modalities\n* The Spk-X4K1000FPS and Spk-GoPro datasets would be a useful contribution to the community if they become publicly available\n* The qualitative results look good", "weaknesses": "* The motivation of the work is not convincingly justified\n\n   There are several previous works [1,2] that use event information for motion deblurring. Compared to these works, it is not clear why the paper chooses spike information over events. The explanation in L52-53 that \"...spike cameras record low-resolution texture information and ... this serves as a stronger guidance for deblurring task\" is very vague and not convincing. What does low-resolution texture information have to do with motion deblurring? and why should it be a stronger guidance compared to high temporal resolution? \n\n   The claim in L49-51 that, \"Most event-based methods unidirectionally utilize information from the event domain to assist the image domain, without achieving the complementarity of information from both domains\", is not true. There are several event-based works [1] that jointly use both the RGB and event information for motion deblurring. Hence, this claim should be toned down. \n\n* The experimental results in the paper are weak and unfairly done\n\n   The experimental comparison on X4K1000FPS in Table 1 is not very meaningful as it is clearly biased to the proposed method. The other baselines HiNet and NAFNet do not use extra data and hence, it is not surprising that they underperform. A more thorough and fair comparison should be done with previous event-based methods following their experiment protocol.\n\n   The results for the GoPro dataset in Table 2 are simply copied (quoted) from previous papers. However, the authors follow a different experiment setting in their paper. Hence, how can the authors conclude that the performance gain is coming from the proposed approach and not necessarily from the different experimental protocols?\n\n    The qualitative analysis both in the main paper and supplementary ignores to compare with the state-of-the-art approach, REFID [2]. Why is that?\n\n* More ablations are needed to justify the different design choices in the proposed approach\n\n   There has to be an ablation experiment that only uses the image stream to prove the benefit of incorporating the spike stream\n   \n   I want to see the ablation for using the claims in L188-193 (directly fusing the input blurred image with the spike branch). Removing the whole CSFI, CAMMA, and initial deblurring branch only results in a PSNR decrease of 0.2 dB. Hence, how worse would it be if we naively fuse the blurred input with the spike branch?\n\n   What is the point of doing several experiments with different input representations? How does that lead to a better understanding of the proposed approach? I get that the spike branch leads to a good performance. However, all other modules seem to be very redundant with very minor contributions to network performance. \n\n    It will be good to provide the visualization of the motion magnitude mask in L208, to verify if the CAMMA module is indeed doing what it is claimed to be doing. \n\n\n* The writing of the paper could be improved\n\n   The submission will greatly benefit from re-writing the methodology part by removing too many unnecessarily coined acronyms and by restructuring the explanation of the different modules in the proposed approach. Moreover, important details such as train-test set splits and results on real-world blurred images should be in the main paper instead of the supplementary. \n\n\nReferences\n\n[1]. Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention, ECCV 2022\n\n[2]. Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring, CVPR 2023\n   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***1. What does low-resolution texture information have to do with motion deblurring? Why should it be a stronger guidance compared to high temporal resolution?***\n\nWe would like to clarify that the low resolution of spike camera is due to current hardware limitation. Our emphasis is on the texture information within spike data (rather than resolution), which contributes positively to motion deblurring. To validate this assertion, We conducted a simple experiment. We use both the blurry RGB images in Spk-X4K1000FPS ($e=65$) and the grayscale images of the corresponding sharp RGB images as input to NAFNet for training. As shown in Tab.R4, we find that a grayscale image with rich texture can effectively guide the restoration of blurred RGB images. \n\n**Table.R4: Toy experiment using NAFNet.**\n|  Input    | PSNR ($e=65$) | SSIM ($e=65$) |\n|  ----  | ---- | ---- |\n| Blurry RGB  |29.06 |0.878 |\n| Blurry RGB + Gray Ground Truth |**39.84** |**0.999** |\n\nThus, we assert that sharp texture information can effectively guide deblurring. In other words, the texture information within a grayscale image can remap color information in the blurred RGB image, even in cases of extreme blur ($e=65$). Our method explicitly reconstructs in the spike branch and strives to introduce sharper texture information into the deblurring branch to guide deblurring. Our CAMMA module aims at integrating image priors from the deblurring branch into the spike branch to enhance the spike reconstruction and subsequently provide improved guidance for the deblurring branch. Supplementary Tab.S1 demonstrates that the CAMMA enhances the quality of spike branch reconstruction. Tab.S2 illustrates that the CAMMA also improve the deblurring performance.\n\nMoreover, both temporal and texture information provides clues for reconstruction and deblurring. We apologize for the overly absolute description used in the paper. We will rectify this in the revised version.\n\n***2. The claim in L49-51 that, \"Most event-based methods unidirectionally utilize information from the event domain to assist the image domain\" is not accurate.***\n\nPlease refer to Response 2 in **To all reviewers**.\n\n***3. A more thorough and fair comparison on X4K1000FPS should be done with previous event-based methods.***\n\nPlease refer to Response 2 in **To all reviewers**.\n\n***4. The authors follow a different experiment setting on GoPro dataset in their paper.***\n\nThe data presented in Tab.2 is primarily cited from REFID[2]. We observed that REFID used different experiment settings in their experiments compared to EFNet[1]. This discrepancy could be attributed to REFID's utilization of recurrent units and multi-frame outputs, which may lead to higher GPU memory consumption and make it hard to use larger batchsize. In our case, we have conducted additional experiments on GoPro dataset using the same training settings as EFNet, as shown in Tab.R5. \n\n**Table.R5: Additional experiments of our method on GoPro dataset using the same training settings as EFNet.**\n|  Method    |Extra Data| PSNR  | SSIM  |\n|  ----  | ---- | ---- | ---- |\n| EFNet  |Event| 35.46 |0.972 |\n| REFID |Event|35.91 |0.973 |\n| Ours (old training settings) |Spike|36.12 |0.971 |\n| Ours (the same training settings as EFNet) |Spike|**36.83** |**0.976** |\n\nThese results confirm the performance improvements achieved by our method even under EFNet's training parameter settings. We believe that this performance gain is attributed to higher learning rates and an increased number of total iterations (In our previous experiments, our network can converge in around 100k iterations, whereas EFNet requires approximately 300k iterations).\n\n\n***5. The qualitative analysis ignores to compare with the state-of-the-art approach, REFID***\n\nWe apologize for not including a comparison with REFID. Due to the lack of code before NeurIPS 2023 deadline and  suitable comparative images in the original REFID paper, we couldn't compare in the inital submission. But now with open-source code, we're addressing this gap by including a qualitative REFID comparison in the revised paper.\n\n***6. The ablation experiment that only uses the image stream to prove the benefit of incorporating the spike stream.***\n\nWe have conducted the requested ablation experiment. The results are provided in Tab.R6. This experiment validates that the introduction of spike data effectively improves performance.\n\n**Table.R6: Additional ablation experiments.**\n|  Method   | PSNR ($e=33/e=65$) | SSIM ($e=33/e=65$) |\n|  ----  | ---- | ---- |\n| Only uses the image stream  |32.45/28.25 |0.895/0.841 |\n| Our final SpkDeblurNet |**37.42/35.94** |**0.968/0.966** |\n\n***7. The ablation for using the claims in L188-193 (directly fusing the input blurred image with the spike branch).***\n\nWe have carried out the suggested experiments, involving the direct fusion of the input blurred image with the spike branch. The results are shown in Tab.R7. We observed that comparing to removing the CAMMA branch (eliminating information flow from image to spike branch), fusing the blurred image directly provides a performance gain, highlighting the importance of bidirectional information complementarity. With CAMMA, performance further improves, demonstrating its effectiveness in transferring image information to the spike branch efficiently.\n\n**Table.R7: Additional ablation experiments.**\n|  Method   | PSNR ($e=33/e=65$) | SSIM ($e=33/e=65$) |\n|  ----  | ---- | ---- |\n| Remove CAMMA branch  |37.20/35.64 |0.967/0.965 |\n| Directly fuse the input blurred image  |37.31/35.80 |0.967/0.965 |\n| Our final SpkDeblurNet |**37.42/35.94** |**0.968/0.966** |\n\n***8. What is the point of doing several experiments with different input representations?***\n\nPlease refer to Response 3 in To all reviewers.\n \n***9. The visualization of the motion magnitude mask in L208.***\n\nPlease refer to Response 4 in To all reviewers.\n\n***10. About the writing.***\n\nWe appreciate it and will carefully revise the writing."}, {"review_id": "Y9UHfXB2JT", "reviewer": "Reviewer_kDLr", "summary": "This paper proposed the first deblurring method that uses two types of inputs: RGB frames and spike streams. Two synthetic datasets are introduced. They are also used for training and evaluating the proposed method. This method outperforms other state-of-the-art methods on these synthetic datasets.", "questions": "Addressing any of the above weaknesses would be great, especially about the real data.", "limitations": "Some limitations are addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper proposes the first spike-based motion deblurring model, which is based on transformers.\n* The authors claim that they can reconstruct sub-frame sharp images at any timestamp, which is a great advantage.\n* The datasets can be of research value to the community. \n* The proposed method outperforms SOTA on those synthetic datasets.", "weaknesses": "* The datasets use synthetic blurry images and a synthetic spike camera simulator. The method is evaluated on a real spike stream only on a few images in the supplementary materials (which is not even mentioned in the main paper). Moreover, this brings up another issue: synchronizing a real spike camera with an RGB camera is not trivial. Some hybrid camera setup is shown in the supplementary materials, but it requires manually aligned spatial and temporal outputs. More thorough evaluation is needed.\n* The main loss (13) is not clear. The first and the last terms seem the same. Why are they separated?\n* The paper contains some typos: L205 (noisy), L76/L98 (Researches), L15 (vFurthermore)\n* Intro: not true that deep learning models always find a clear image (L32) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments, summary of our paper, and affirmation of the performance. We would like to address your concerns and answer your questions here.\n\n***1. More thorough evaluation in real-world scenarios is needed.***\n\nThanks for your suggestion! Please refer to Response 1 in **To all reviewers**.\n\n***2. The main loss (13) is not clear. The first and the last terms seem the same. Why are they separated?***\n\nThanks for pointing it out. As presented in Sections 3.3 and 3.4.2, in the first term of the loss (Equation 13), $\\hat{I}_t$ represents the final deblurred output from the deblurring branch, while in the third term, $\\overline{I}_t$ represents the initial deblurred result generated by the deblurring branch. The separation of these terms serves a distinct purpose. We introduce $\\overline{I}_t$ through the CAMMA module as a high-resolution image domain prior to guiding the spike branch to aid spike reconstruction. By performing the loss function on $\\overline{I}_t$ and ${I}_t$, we aim to minimize the blurriness present in $\\overline{I}_t$, thereby providing higher-quality image-domain priors to guide the spike branch's reconstruction. The loss function between $\\overline{I}_t$ and ${I}_t$ constrains the final deblurring result. We will clarify it in the revised version.\n\n***3. The paper contains some typos: L205 (noisy), L76/L98 (Researches), L15 (vFurthermore).***\n\nWe greatly appreciate your meticulous review. We have revised these typos in the paper.\n\n***4. Intro: not true that deep learning models always find a clear image (L32).***\n\nWe acknowledge that our statement might not be precise. We have revised the statement to emphasize the potential of deep learning models when trained on large and diverse datasets, while also considering their limitations when trained with only a single modality as input."}]

Author rebuttal:
We appreciate all reviewers for their helpful feedbacks. We are encouraged that they found our paper "well-written and technically sound" [DRAW], our method "novel" [Rugk] and "intuitive" [9w9w], and the proposed problem "new" [9w9w] and "is a promising direction for further research". We are delighted that they acknowledge that our method outperforms state-of-the-art methods [wVmq,kDLr] and the qualitative results are good [9w9w]. We are glad that they recognize our dataset [wVmq,9w9w,kDLr] and evaluate it as "a useful contribution"[9w9w] with "research value"[kDLr]. We will incorporate all feedback.
 
In this general response, we would like to address some cruical concenrns.

***1. More evaluation in real-world scenarios.***
> **[Reviewer DRAW]** Have authors ever considered constructing a camera system to truly capture the spike data and blur images to better demonstrate the proposed setup is feasible and has practical value?

Yes, actually we have provided relevant information about the prototype of our hybrid camera system in Supplementary Sec.D.1 and Fig.S1. We used this system to capture pairs of blurred images and spike sequences from real-world scenes and applied the proposed method to obtain deblurred results, as shown in Supplementary Fig.S2.

> **[Reviewer kDLr]** More thorough evaluation in real-world scenario is needed.

We achieve temporal synchronization by simultaneously triggering the capture programs of both cameras through a script. We use a feature point matching algorithm to achieve spatial registration. We are currently in the process of improving the hybrid camera system prototype and extending our evaluation to include a broader range of real-world scenarios. We have added additional results in real-world scenes in Fig.R4 in the PDF in the global response.

***2. More comparative experiments that applying event-based methods to spike camera.***
> **[Reviewer wVmq]** The spike camera is similar to event camera, and the method based on these two methods are also similar except the format of event/spike stream. I suspect that event-based methods also works with spike camera.

> **[Reviewer 9w9w]** A more thorough and fair comparison on X4K1000FPS should be done with previous event-based methods.

While both camera types are neuromorphic cameras, event cameras utilize differential sampling, while spike cameras employ integral sampling. This fundamental difference leads to intrinsic distinctions. Both capture rich temporal information, yet spike cameras uniquely excel in capturing detailed spatial-temporal textures. This trait motivates our incorporation of image-domain priors into spike domains for explicit reconstruction. These refined spike features enhance the image branch for guiding deblurring. The mutual complementation of information between modalities enhances performance. In contrast, event camera-based methods often involve one-way information flow from the event branch to the image branch, highlighting a significant difference from our approach.

Despite these disparities, both approaches fall under the multi-modal paradigm. Hence, we conducted supplemental experiments using EFNet and REFID networks for spike-assisted deblurring. For EFNet, we employed input representations similar to its SCER approach, excluding the EMGC module due to its event-related nature. Regarding REFID, we utilized similar spike voxel representations. Tab.R1 results demonstrate our approach's superiority with spike data, yielding enhanced outcomes.


**Table.R1: Comparisons between different networks on Spk-X4K1000FPS dataset.**
|  Method   | PSNR ($e=33/e=65$) | SSIM ($e=33/e=65$) |
|  ----  | ---- | ---- |
| EFNet  |36.36/33.53 |0.960/0.937 |
| REFID  |36.30/33.47 |0.962/0.945 |
| SpkDeblurNet (Ours)  |**37.42/35.94** |**0.968/0.966** |

>**[Reviewer 9w9w]** The claim in L49-51 that, "Most event-based methods unidirectionally utilize information ... without achieving the complementarity of information" is not accurate.

Our phrase "achieving the complementarity of information from both domains" signifies bidirectional information flow between modal branches, enhancing each other's tasks. EFNet, in contrast, unidirectionally incorporates event data into the image branch for RGB deblurring, without reciprocating by including image data into the event branch. Our approach facilitates mutual bidirectional information exchange between branches, fostering mutual assistance. We will include a clearer elaboration in the revised version. 

***3.[Reviewer 9w9w] What is the point of doing several experiments with different input representations?***

Apart from validating enhanced deblurring performance with diverse input representations via the spike branch, our motivation for exploring different input representations also involves evaluating the effectiveness of our CAMMA module. Supplementary Tab.S1 shows CAMMA's enhancement of spike branch reconstruction quality across various inputs. Tab.S2 shows CAMMA's contribution to improved overall deblurring performance. These experiments present CAMMA's potential to enhance performance across diverse inputs.


***4. [Reviewer 9w9w] The visualization of the motion magnitude mask in L208.***

We've added visualizations of the motion magnitude mask in our global response PDF. In Fig.R3, we display the hard thresholding mask and the content-aware mask for both spike stream and CSFI input representations. The hard thresholding mask roughly masks high motion magnitude regions, while the content-aware mask offers refined masking. Notably, the CSFI representation yields a more pronounced content-aware mask due to its reduced information content compared to the spike stream, requiring greater reliance on image-domain prior knowledge. Supplementary Tab.S1 and Tab.S2 also demonstrate the significant improvement brought about by the introduction of CAMMA module when utilizing the CSFI representation.

Meta-review: The paper has positive final ratings except for a single reviewer who articulated certain remaining concerns and gave a score of borderline reject. The AC read the paper, all the reviews, the authors' responses, and discussion.

After taking everything into consideration, the AC finds the paper to be novel in proposing the use of spike cameras (along with RGB ones) to help with motion de-blurring. The inference approach is also sufficiently interesting/novel to clear the bar.

The AC acknowledges the BR reviewer's main remaining complaints, but does not find them to be a bar for acceptance. 

(1) The AC thinks the question of whether there's enough "motivation/intuition" for why spike cameras would be better than event cameras is moot --- intuition is never going to be precise, and the experimental results show their method is better than results reported with event cameras (reviewer complains about not comparing with a CVPR 2023 paper in the main paper and only giving quantitative comparisons in the rebuttal --- the AC thinks that's reasonable on part of the authors, the quantitative results are better, it's contemporary work anyway, and in any event, the novelty and impact of using a new kind of camera for deblurring cues is novel).

Having said that, authors should tone down the assertion that spike data is intrinsically more informative than event data for deblurring.

(2) The fact that one component of the method showed limited impact in ablations is also not a bar --- at least the ablation is there, and so the reader has all the information they need.

Therefore, the AC is happy to recommend acceptance.
Decision: Accept (poster)