Title: Enhancing Motion Deblurring in High-Speed Scenes with Spike Streams
Venue: NeurIPS 2023
Accepted: yes
Keywords: spike camera

neuromorphic vision sensors

motion deblurring

high speed imaging
Abstract: Traditional cameras produce desirable vision results but struggle with motion blur in high-speed scenes due to long exposure windows. Existing frame-based deblurring algorithms face challenges in extracting useful motion cues from severely blurred images. Recently, an emerging bio-inspired vision sensor known as the spike camera has achieved an extremely high frame rate while preserving rich spatial details, owing to its novel sampling mechanism. However, typical binary spike streams are relatively low-resolution, degraded image signals devoid of color information, making them unfriendly to human vision. In this paper, we propose a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes. 
We propose the first spike-based motion deblurring model with bidirectional information complementarity. We introduce a content-aware motion magnitude attention module that utilizes learnable mask to extract relevant information from blurry images effectively, and we incorporate a transposed cross-attention fusion module to efficiently combine features from both spike data and blurry RGB images.
Furthermore, we build two extensive synthesized datasets for training and validation purposes, encompassing high-temporal-resolution spikes, blurry images, and corresponding sharp images. The experimental results demonstrate that our method effectively recovers clear RGB images from highly blurry scenes and outperforms state-of-the-art deblurring algorithms in multiple settings.

Enhancing Motion Deblurring in High-Speed Scenes with Spike Streams

Abstract

Traditional cameras produce desirable vision results but struggle with motion blur in high-speed scenes due to long exposure windows. Existing frame-based deblurring algorithms face challenges in extracting useful motion cues from severely blurred images. Recently, an emerging bio-inspired vision sensor known as the spike camera has achieved an extremely high frame rate while preserving rich spatial details, owing to its novel sampling mechanism. However, typical binary spike streams are relatively low-resolution, degraded image signals devoid of color information, making them unfriendly to human vision. In this paper, we propose a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes. We propose the first spike-based motion deblurring model with bidirectional information complementarity. We introduce a content-aware motion magnitude attention module that utilizes learnable mask to extract relevant information from blurry images effectively, and we incorporate a transposed cross-attention fusion module to efficiently combine features from both spike data and blurry RGB images. Furthermore, we build two extensive synthesized datasets for training and validation purposes, encompassing high-temporal-resolution spikes, blurry images, and corresponding sharp images. The experimental results demonstrate that our method effectively recovers clear RGB images from highly blurry scenes and outperforms state-of-the-art deblurring algorithms in multiple settings.

1 Introduction

Traditional computer vision is well developed and models on visual tasks are satisfying [10, 37, 36, 11]. As a mature industrial product, frame-based cameras have also been fully developed whose imaging quality is high. Due to the existence of the exposure window, high-speed scenes or moving objects will cause serious blur effects as illustrated in Fig. 1(a), which degrades or invalidates the model performance. Thus, deblurring task remains an important demand for applications. With the development of convolutional neural networks and Transformers, models like Restormer [59] and SwinIR [32] used for deblurring tasks perform well. Nevertheless, our main concern is that the performance of these models is not reliable as the blurry image lacks sufficient contextual information to recover the corresponding clear image. When the blurry artifact is severe, models only input with blurry images may fail.

†

ABC

(a) Discrete Images from  Frame-based Camera

(c) Fusion of Spikes and Images for Deblurring

ABC

ABC

Neuromorphic vision sensors, including event cameras [1, 33] and spike cameras [16], have gained recent attention due to advantages such as high temporal resolution, dynamic range, and low power consumption. Event cameras capture changes in light intensity, while spike cameras record absolute intensity in a unique way. They accumulate voltage converted from the light signal asynchronously and emit spikes when a preset threshold is reached, achieving a time resolution of up to 40,000Hz. Spike cameras are motion-sensitive and capture comprehensive texture information, offering the potential to mitigate motion blur. However, the spike stream is not directly interpretable by humans like RGB images. Initial research aimed at reconstructing grayscale images from spikes (see Fig. 1(b)). Subsequent studies leveraged spikes for high-level vision tasks, including object tracking, optical flow estimation, and depth estimation. Presently, spike cameras face challenges in these tasks due to their lower spatial resolution and absence of color. We consider utilizing spike cameras to assist in image deblurring. Our intuition is that the information in the RGB images and the neuromorphic data are complementary. The image domain contains rich spatial texture information with high-fidelity color, while the neuromorphic domain provides abundant temporal information [72, 9] promises them to record fast motions. Some studies utilize event cameras to assist image deblurring [57, 63, 49]. However, most event-based methods unidirectionally utilize information from the event domain to assist the image domain, without achieving the complementarity of information from both domains. Unlike event cameras, spike cameras maintain high temporal resolution while also recording texture information in the form of spike intervals [72]. The complete texture information, as opposed to solely motion edge information, can offer more valuable cues for deblurring tasks. As shown in Fig. 1(c), we aim to simultaneously consider the bidirectional transmission of information from both modalities, thereby achieving more efficient motion deblurring.

Therefore, we propose the first spike-based RGB image deblurring model named SpkDeblurNet . The model takes high-resolution blurry RGB images and low-resolution spike streams as input, each of which passes through one Transformer-based branch. We design a content-aware motion-magnitude attention module to fuse the high-resolution features in images into the spike branch. We further propose transposed cross-attention to perform a bidirectional fusion of cross-modal features between two branches. Moreover, we establish two high-quality datasets Spk-X4K1000FPS and Spk-GoPro for both training and validation. In experiments, we compare with various deblurring methods and prove the effective assistance of spikes. The complementarity of the two modalities is also verified. Our contributions to this work can be summarized as follows:

We propose the first spike-based motion deblurring model with bidirectional information complementarity. Specifically, we propose a content-aware motion magnitude attention module based on learnable mask to utilize effective information in blurry images. A transposed cross-attention fusion module is built to efficiently fuse features from spikes and blurry RGB images.

We generate two large-scale synthesized datasets used for training and validation, which contain high-temporal-resolution spikes, blurry images, and sharp images.

Experimental results demonstrate that the proposed method achieves ideal results, outperforming other deblurring methods.

2 Related Works

Research on Spike Camera. Spike cameras possess the advantages of ultra-high temporal resolution and high dynamic range, which bring the potential to solve various visual tasks, especially in high-speed scenes. Early studies focus on recovering clear gray-scale images from spikes. Zhu et al. [72] propose the first work by directly analyzing the spike firing rate and the inter-spike interval. Some works [73, 69] take biological plausibility into consideration. Spk2imgnet [65] achieves satisfying performance by training deep convolutional neural networks (CNNs) in a supervised manner. Chen et al. [5] introduce the self-supervised framework. Besides, some studies [56, 64] explore recovering super-resolution images from spikes. Recently, more researchers have used spike cameras to solve various visual tasks. Zhao et al. [66] build a new spike-based dataset for object recognition. Zheng et al. [68] propose a bio-inspired framework for object tracking. Hu et al. [15] contribute to spike-based optical flow estimation by proposing the SCFlow net and new datasets. Zhao et al. [67] proposed to jointly estimate a series of flow fields. Chen et al. [6] propose a self-supervised framework for joint learning of optical flow and reconstruction. Besides, Spike Transformer [62] and SSDEFormer [53] are proposed for monocular/stereo depth estimation. Additionally, we observed similarities between single-photon cameras (SPCs) and spike cameras in their binary output. Nonetheless, SPCs predominantly depends on single-photon avalanche diode (SPAD) detector [35, 17, 27] or quanta image sensors (QIS) [38, 7, 2, 13], while the spike camera is based on CMOS sensors like traditional cameras and employs standard semiconductor manufacturing processes, rendering them fundamentally distinct, with the latter being more cost-efficient.

Frame-Based Motion Deblurring. Traditional frame-based motion deblurring methods typically employ optimization techniques [12, 24, 29, 30, 58] and blur kernel estimation [28, 55, 42, 21]. In recent years, with the advancement of deep learning, end-to-end approaches based on CNNs and transformers have demonstrated impressive performance in image deblurring. Multi-scale [39, 50], multi-stage [61, 4], and progressive strategies [8] have been proposed to improve the effectiveness of end-to-end training. Some works [25, 26] proposed to utilize Generative Adversarial Networks (GANs) to provide better human perception. On the other hand, universal networks [3, 31, 32, 54, 61, 59, 60, 4] have been introduced to address a series of low-level problems including image deblurring. Further research applied the powerful transformer framework [54, 31, 32, 59, 23] to motion debluring, achieving state-of-the-art (SOTA) performance. Unlike the conventional one-to-one mapping from a blurry image to a single sharp image, blurry frame interpolation or blur decomposition aims to recover multiple sharp images from a single blurry image [20, 19, 44, 45, 40, 70]. Additionally, directional ambiguity can be avoided in methods based on blurry video inputs [19, 44, 45, 40, 70] due to the motion cues of adjacent frames.

Event-Based Motion Deblurring. There has been increasing interest in utilizing event cameras to assist image deblurring. Several works [18, 43] utilized optical flow to provide motion information. NEST [51] proposed an efficient event representation based on bidirectional LSTM for deblurring. Kim et al. [22] extended the task to images with unknown exposure time. EFNet [48] introduced a multi-head attention module to better fuse event and image information. Some works [34, 41, 52, 49, 63, 57] have utilized events to assist blurry frame interpolation or blur decomposition. In addition, a series of high-quality event-image datasets have been proposed [51, 49, 18, 48], providing convenience for subsequent research. Most event-based methods merely utilize event-based information to assist the image domain task. In this paper, we aim to use the bidirectional transmission of information from both modalities to achieve more efficient motion deblurring.

3 Methods

3.1 Spike Generation Mechanism

Spike cameras, different from the differential sampling of event cameras, adopt the integral sampling mechanism. A spike camera with the spatial resolution of H × W is composed of an array of units on each pixel. The working mechanism of the spike camera is illustrated in Fig. 2. For a unit at pixel ( x, y ) that x ∈ [0 , W ) , y ∈ [0 , H ) continuously receives coming photons at any timestamps t and converts the light intensity L x,y ( t ) into the electric current, and increases its voltage V x,y ( t ) . A voltage threshold Θ is pre-defined. A spike will be triggered once the threshold is reached, and the V x,y ( t ) will be reset to 0 . The process can be formulated as follows:

0

1

1

0

1

1

0

1

1

0

1

0

1

0

0

1

0

1

0

0

Voltage

Readout

Threshold

Light Intensity

where V -x,y ( t ) and V + x,y ( t ) denotes the voltage before and after receiving the electric current, and the α is the photoelectric conversion factor. The back-end circuit reads out spikes continuously with a very small interval η (25 µ s). A pixel at ( x, y ) will output S ( x, y, n ) = 1 when the threshold is reached at t ∈ ( η ( n -1) , ηn ] , otherwise output S ( x, y, n ) = 0 . Thus, for a T -times readout, a spike train with the size of H × W × T would be generated.

3.2 Problem Statement

A spike camera outputs binary spike streams with very-high temporal resolution. A conventional frame-based camera outputs an image sequence and each image is generated after an exposure time T e . Though scenes in the world are continuous, images are discrete. We regard spikes approximately as continuous signals. When coming to scenarios with high-speed motions, the exposure window in frame-based cameras leads to a blurred image B e , while spikes S e during the exposure time finely records the motion dynamics. In this work, we aim to utilize the spike stream S e during the T e to deblur the RGB image B e . Moreover, the current spatial resolution of spike cameras is relatively low, so we set the spatial resolution of S e to half of B e to simulate the real situation.

3.3 Architecture

To achieve effective integration of the high resolution color texture information in conventional images and the high temporal resolution offered by spike streams, we present a novel two-branch fusion spike-based image deblurring framework, SpkDeblurNet (Fig. 3). Our framework combines the benefits of both image deblurring and spike-based image processing. In the first branch of SpkDeblurNet, we input the blurred RGB image B e , while the second branch takes a continuous spike stream S e centered around the latent clear image. B e has twice the spatial resolution of S e .

The input blurry image B H × W × 3 e is first passed through two 3 × 3 convolutional layers with a stride of 2 for downsampling. Subsequently, the downscaled features are fed into a series of continuous Residual Swin Transformer blocks (RSTB) [32, 37]. These RSTBs play a crucial role in extracting features that encapsulate enhanced spatiotemporal semantics. Mathematically, we can express this process as follows:

where the F b down ( · ) denotes the downsampling operations.

Similarly, the spike stream S H × W × T e undergoes processing through a singular convolutional layer, facilitating the embedding of spike streams within the feature domain. Furthermore, a sequence of

x

y

t

LayerNorm

(S)W-MSA

LayerNorm

MLP

×6

×3

Residual Swin Transformer Block (RSTB)

Residual Swin Transformer Block

(RSTB)

×3

Residual Swin Transformer Block

(RSTB)

×1

Residual Swin Transformer Block

(RSTB)

×1

Residual Swin Transformer Block

(RSTB)

×3

Residual Swin Transformer Block

(RSTB)

×3

Swin Transformer Block

Center-Spike Firing Interval

(CSFI)

CAMMA

TCAF

Conv

TCAF

Clear

RGB Image

Clear

Gray Image

Conv

Conv

Conv

Conv

Conv

Mask

*

T

three RSTBs is employed to execute profound spatiotemporal semantic extraction:

To effectively merge the comprehensive high resolution color texture information available in the image domain with the spatiotemporal semantic extraction performed on the spike stream, we design a novel Content-Aware Motion Magnitude Attention module (CAMMA). This module facilitates the extraction of features specifically from the regions of clarity within the initial deblurred image, thereby producing a robust motion mask represented as M . By combining this motion mask with the initial deblurred image F ′ b up ( f b 3 ) extracted from the blurry image and integrating it with the feature f s 3 obtained from the spike domain using the Transposed Cross-Attention Fusion (TCAF) module, a clear grayscale image denoted as ˆ I g t is generated through an RSTB and a post-upsampling process. The mathematical expression for this procedure is as follows:

with ∗ denotes the element-wise multiplication, and ˆ I g t denotes the predicted grayscale image at the center timestamp t of window T e .

We then combine the spike features f s 4 with the feature information f b 3 obtained by the blurry image through the TCAF module, and output a refined sharp color image ˆ I t with a series of RSTBs and upsampling, which can be summarized as follows:

In the blurry decomposition task, a common issue is directional ambiguity [70, 71]. Since we reconstruct the center position of the exposure window in the spike branch, the output of RSTB s contains time-specific spike encoding information, which can avoid this problem. During online inference, we can decompose the blurry image into different timestamps by moving the position of the spike stream, and the frame rate of the decomposed images depends on the sampling rate of the spike camera.

3.4 Cross-Modal Information Complementing

Spike Reconstruction Based on High Resolution Prior in Image Domain. Each pixel in the spike camera asynchronously outputs high-time-resolution spikes according to the brightness proportional to the input light intensity. This mechanism enables the spike stream to effectively capture and represent both temporal and spatial information of visual inputs. However, the current spatial resolution of the

(a) Content-Aware Motion Magnitude Attention

Module (CAMMA)

(b) Transposed Cross-Attention Fusion (TCAF)

UpSample

Cat

Normalize

Thresholding

Conv

Cat

1×1 Conv

1×1 Conv

1×1 Conv

3×3 DConv

3×3 DConv

Reshape

1×1 Conv

3×3 DConv

Reshape

Reshape

Reshape

1×1 Conv

LayerNorm

LayerNorm

*

spike camera is relatively low. In our module, we explicitly reconstruct high-resolution grayscale images of the spike stream to enable the network to learn image-like features. Given the ground truth grayscale image I g t at the moment of the exposure window center, the loss function is formulated as:

Reconstructing high-resolution frames solely from the spike stream itself remains challenging. To fully leverage the advantages of both input modalities, we propose to utilize information from the image domain as a high-resolution prior to spike reconstruction. However, the input images are mostly highly blurred, which hinders the provision of high-resolution clues for reconstruction. Therefore, we further introduce the CAMMA module to adaptively search for regions with smaller motion magnitudes in the initial deblurred image, and fuse them into the spike branch.

Content Aware Motion Magnitude Attention Module. In order to incorporate high-resolution prior knowledge from the image domain into the spike domain, an intuitive idea is to directly fuse the input blurred image with the spike branch. The blurry foreground caused by moving objects cannot provide effective texture information, which can be handled by using a mask to retain sharp backgrounds [47]. Moreover, the global blur caused by camera motion is unable to provide effective high-resolution prior. Therefore, we propose a CAMMA module to encode the motion magnitude from the initial deblurred result. As shown in Fig. 4, we first pass the f b 3 through shallow convolution and upsampling to obtain the initial deblurred image I t . Through training, the blur in I t is initially reduced. Subsequently, we need a coarse sharp image as a reference to obtain the blurry region by comparing it with I t . We adopt the basic statistical method text-from-interval (TFI) [72] to center-spike firing interval (CSFI) to obtain the central coarse sharp image. CSFI estimates the pixel value of the current pixel based on the observation that the spike camera's firing frequency at each pixel is proportional to the real light intensity by statistically analyzing the spike interval. The process is formulated as:

where ∆ t represents the inter-spike interval corresponding to moment t , and the C refers to the maximum dynamic range of the reconstruction. Subsequently, we perform differencing and normalization on the upsampled I csfi t and I t and then apply thresholding to obtain the hard thresholding motion magnitude mask. However, since I csfi t is merely a basic reconstruction and suffers from noise [72]. In order to enable the network to learn a more robust motion magnitude mask, we concatenate I csfi t and I t and use simplified channel attention [3] with the hard thresholding mask to obtain a content-aware motion magnitude mask M . After multiplying the motion magnitude mask with I t , we utilize the TCAF module to fuse it into the spike branch. We refer to the CAMMA module, the CSFI process, and the F ′ b up ( · ) module as the CAMMA branch, with which the high-resolution prior knowledge from the image domain can be transferred effectively into the spike domain.

Spike-Guided Motion Deblurring. As mentioned in Sec. 3.3, our SpkDeblurNet involves information transfer from image to spike and spike to the image. The modalities differ significantly for the former, while the differences are minor for the latter. In order to better integrate the two modalities, we propose the TCAF Module. The query ( Q m 1 ) is derived from one modality m 1 with c 1 channels, while the key ( K m 2 ) and value ( V m 2 ) are obtained from another modality m 2 with c 2 channels. The attention is computed using the following equation:

Where f m 1 and ˆ f m 1 are the features before and after fusion for modality m 1 , G denotes the softmax function and α is a learnable scaling parameter. However, the calculation of attention faces two problems: 1) The spike branch is lightweight with fewer channels compared to the deblurring branch, which makes traditional attention computation in the spatial dimension not directly applicable. 2) The computational cost of traditional attention grows quadratically with the input size, which is not acceptable for image restoration tasks. In the main network, we mitigate these issues by employing the RSTB [37, 36], which calculates attention within windows to reduce computation. In TCAF, we adopt the Transposed Attention proposed in [59], which performs attention computation along the channel dimension, thereby addressing the aforementioned problems. We apply the TCAF module to the fusion of features in both directions.

3.5 Joint Learning of Deblurring and Reconstruction

We jointly train the proposed SpkDeblurNet. The spike reconstruction branch obtains high-resolution image priors from the initial deblurred result, enabling the recovery of better image-like features. The features from the spike branch in turn guide the deblurring branch, providing time-specific texture information. For both the initial deblurred result I t and the refined deblurred result ˆ I t , we utilize the L1 loss for supervision during training. The loss function of the entire network can be formulated as follows:

where λ 1 and λ 2 are hyperparameters that control the loss terms.

3.6 Datasets

This paper is the first study on spike-based motion deblurring, and no large-scale training datasets are currently available. Therefore, we built two large-scale spike-based datasets based on existing real image datasets [39, 46] for training and validation. The first one, named 'Spk-X4K1000FPS', is built on the dataset X4K1000FPS which contains clear images captured by 1000fps cameras. To generate spikes, we utilize the interpolation method [46] to further increase the frame rate four times. Then we build a simple spike simulator according to the spike generation mechanism. It inputs continuous sharp images as light signals and converts them to spikes. For clips in the dataset, we use the average of neighboring e sharp images around the ground truth images to generate blurry images, where e = 33 , 65 . The second one, named 'Spk-GoPro' is built on the GoPro [39] dataset in a similar way as above. More implementation details are attached in the supplementary materials.

4 Experiments

4.1 Comparative Results

In this section, we present comparative results on both Spk-X4K1000FPS and Spk-GoPro datasets. For Spk-X4K1000FPS, we generate blurry images by setting an exposure window of e = 33 as well as an extreme exposure window of e = 65 . For Spk-GoPro, we use the provided blurry images. Implementation details can be found in supplementary materials.

Results on Spk-X4K1000FPS. Tab. 1 and Fig. 5 present quantitative and qualitative comparisons of our method with other approaches under two different exposure window settings. For EFNet and REFID, we use their networks for spike-assisted deblurring. For EFNet, we employed input representations similar to its SCER approach, excluding the EMGC module due to its event-related nature. Regarding REFID, we utilized similar spike voxel representations. All comparative methods

Ground Truth

NAFNet 28.85dB

NAFNet 35.22

HINet 29.52dB

HINet 34.54dB

Ours 36.12dB

Ours 36.41dB

Ground Truth

NAFNet 24.21dB

NAFNet 29.44dB

HINet 22.99dB

HINet 26.77dB

Ours 28.47dB

Ours 30.15dB

were trained from scratch on the proposed dataset. We observe that both image-domain methods perform well in recovering image details with a smaller exposure window setting ( e = 33 ). However, when the exposure window becomes larger ( e = 65 ), pure image-based methods struggle to find cues from extremely blurry scenes to restore clear images, resulting in significant loss of texture details or generating artifacts. In the second row of the first sample sequence in Fig. 5, NAFNet and HINet mistakenly recover non-existent stair edges. In contrast, our multimodal approach leverages spike streams to find cues and restore correct and clear textures. Additionally, the networks in event-assisted methods fall short of our network. This is due to our network's superior bidirectional information transmission nature.

Results on Spk-GoPro. As shown in Tab. 2, our proposed SpkDeblurNet outperforms the state-ofthe-art model REFID [49] and other competitive models in terms of PSNR On the GoPro dataset. We have observed that both our SpkDeblurNet and the SOTA event-based methods have significantly outperformed image-based methods, which demonstrates the substantial potential of neuromorphic data in complementing information for the image domain. Fig. 5 visualizes the deblurred results of different methods, from which we demonstrate that the quality of sharp images predicted by the SpkDeblurNet is more promising and satisfying than others. In the first sample of Fig. 5, our approach successfully restored the clear license plate number while also effectively recovering the fallen leaves on the ground, while other methods only achieved a relatively blurred restoration of the ground. Furthermore, our method excelled in restoring the clearest facial features of the person in the second sample. Zoom in for more details. Our framework, by leveraging the complementary information from both image and spike domains, has surpassed event-based methods that solely rely on unidirectional information transfer in terms of both quantitative and qualitative results.

Results on Real-World Scenes. To validate the generalization of our method in real-world scenarios, we construct a hybrid camera system to capture real-world spike-image data. We use a beam splitter to connect a spike camera and a conventional frame-based camera, enabling both cameras to have the same view. We rapidly wave objects in front of the cameras to simulate high-speed motion. We synchronize the two cameras and manually aligned their spatial and temporal outputs. Fig. 7 presents visual results of applying our model trained on the Spk-X4K1000FPS dataset with a window size of 33 to two real-world sequences. We can observe desirable results with clear textures and accurate colors. This demonstrates the good generalization of our proposed algorithm and dataset, highlighting the potential of our algorithm for real-world applications. More results and the hybrid camera system diagram can be found in the supplementary materials.

HINet+

HINet

EFNet

Ours

Ground Truth

HINet+

HINet

EFNet

Ours

Ground Truth

Przaht

Phizaht

248022

Phzahut

Pizatht

PzaMht

Pzaht

4.2 Ablation Studies

In this section, we conduct a series of ablation studies on Spk-X4K1000FPS dataset.

Effectiveness of the proposed modules. As shown in Tab. 3, removing the CAMMA branch resulted in a performance degradation of 0.2-0.3 dB, confirming the necessity of introducing features from the deblurring branch and the effectiveness of the CAMMA module's learnable magnitude masking mechanism. Additionally, replacing the TCAF module with concatenation led to minor performance degradation, which we attribute to the explicit supervision of the spike reconstruction branch enabling the branch to learn image-like features, thereby the modality gap is reduced.

Information supplementation from image domain to spike domain. We further conduct ablation studies on spike reconstruction with different combinations of three input representations and the usage of the CAMMA branch. To be specific, we employ either coarse reconstruction CSFI or a symmetric cumulative spike representation similar to EFNet [48]. For the latter, we accumulate spikes from the center towards both ends using three window lengths and concatenate the results along the channel dimension. As shown in Tab. 4, CAMMA consistently significantly improves spike reconstruction across all input representations. For settings using CSFI and cumulated spikes as inputs, CAMMA enhances performance by 1.5-3.5 dB. This is due to the simpler nature of CSFI and cumulated spikes, which contain less texture and motion information. High-resolution image priors from the deblurring branch compensate for this, further enhancing spike reconstruction. Even with spike streams as input, CAMMA still yields improvements of 0.4-0.7 dB. This set of ablation experiments demonstrates the effectiveness of the proposed scheme in incorporating high-resolution features from the deblurring branch into the spike branch, thereby enhancing spike reconstruction performance and providing better guidance and cues for the deblurring task.

HINet

NAFNet

Ours

qw

D

anu

9

MADU

TRADIT

BUTTER

COC

BUTTER

COO

Information supplementation from spike domain to image domain. We investigate the complementary role of the spike domain to the image domain by replacing the input representation. Quantitative results in Tab. 3 indicated that using coarse reconstruction as input still yielded competitive performance, suggesting that the rough texture information contained in CSFI serves as a good complement to the deblurring branch. On the other hand, using cumulated spikes as input further improved the performance. As a comparison, we achieved the highest performance using pure spike stream as input, demonstrating the network's ability to fully excavate the rich temporal information and clues embedded in spike streams to recover clear images. We further conduct ablation studies on different combinations of three input representations and the usage of the CAMMA module for the deblurring task, the results can be found in the supplementary materials.

5 Conclusion

In this paper, we propose the first spike-based motion deblurring model with bidirectional information complementarity, in which we propose a content-aware motion magnitude attention module and a transposed cross-attention fusion module. We build two large-scale synthesized datasets used for both training and validation, which contain high-temporal-resolution spikes, blurry images, and sharp images. Experimental results demonstrate that the proposed method outperforms other deblurring methods and achieves desirable results.

Limitations. Our future work will prioritize enhancing the resolution of the spike stream to improve its auxiliary capability for image-domain tasks, considering the limitation of the current low resolution of spike cameras and the significant impact of spatial super-resolution on applications.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant No. 62176003 and No. 62088102.

References

Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240 × 180 130 db 3 µ s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits , 49(10):2333-2341, 2014.

Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, and Andreas Kolb. A bit too much? high speed imaging from sparse photon counts. In 2019 IEEE International Conference on Computational Photography (ICCP) , pages 1-9. IEEE, 2019.

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII , pages 17-33. Springer, 2022.

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 182-192, 2021.

Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI , pages 2859-2866, 2022.

Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 350-358, 2023.

Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, and Stanley H Chan. Dynamic low-light imaging with quanta image sensors. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16 , pages 122-138. Springer, 2020.

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-tofine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4641-4650, 2021.

Siwei Dong, Lin Zhu, Daoyuan Xu, Yonghong Tian, and Tiejun Huang. An efficient coding method for spike camera using inter-spike intervals. In 2019 Data Compression Conference (DCC) , pages 568-568. IEEE, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022.

Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. Removing camera shake from a single photograph. In Acm Siggraph 2006 Papers , pages 787-794. 2006.

Eric R Fossum, Jiaju Ma, Saleh Masoodian, Leo Anzagira, and Rachel Zizza. The quanta image sensor: Every photon counts. Sensors , 16(8):1260, 2016.

Chen Haoyu, Teng Minggui, Shi Boxin, Wang YIzhou, and Huang Tiejun. Learning to deblur and generate high frame rate video with an event camera. arXiv preprint arXiv:2003.00847 , 2020.

Liwen Hu, Rui Zhao, Ziluo Ding, Lei Ma, Boxin Shi, Ruiqin Xiong, and Tiejun Huang. Optical flow estimation for spiking camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17844-17853, 2022.

Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, et al. 1000 × faster camera and machine vision with ordinary devices. Engineering , 2022.

Atul Ingle, Andreas Velten, and Mohit Gupta. High flux passive imaging with single-photon sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6760-6769, 2019.

Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu. Learning event-based motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3320-3329, 2020.

Meiguang Jin, Zhe Hu, and Paolo Favaro. Learning to extract flawless slow motion from blurry videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8112-8121, 2019.

Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning to extract a video sequence from a single motionblurred image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6334-6342, 2018.

Tae Hyun Kim and Kyoung Mu Lee. Generalized video deblurring for dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5426-5434, 2015.

Taewoo Kim, Jeongmin Lee, Lin Wang, and Kuk-Jin Yoon. Event-guided deblurring of unknown exposure time videos. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII , pages 519-538. Springer, 2022.

Lingshun Kong, Jiangxin Dong, Mingqiang Li, Jianjun Ge, and Jinshan Pan. Efficient frequency domainbased transformers for high-quality image deblurring. arXiv preprint arXiv:2211.12250 , 2022.

Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using a normalized sparsity measure. In CVPR 2011 , pages 233-240. IEEE, 2011.

Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiˇ rí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8183-8192, 2018.

Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (ordersof-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8878-8887, 2019.

Martin Laurenzis. Single photon range, intensity and photon flux imaging with kilohertz frame rate and high dynamic range. Optics express , 27(26):38391-38403, 2019.

Anat Levin. Blind motion deblurring using image statistics. In Advances in Neural Information Processing Systems , pages 841-848, 2007.

Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 1964-1971. IEEE, 2009.

Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Efficient marginal likelihood optimization in blind deconvolution. In CVPR 2011 , pages 2657-2664. IEEE, 2011.

Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288 , 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1833-1844, 2021.

Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128 × 128 120 db 15 µ s latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits , 43(2):566-576, 2008.

Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren. Learning event-driven video deblurring and interpolation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16 , pages 695-710. Springer, 2020.

Yuhao Liu, Felipe Gutierrez-Barragan, Atul Ingle, Mohit Gupta, and Andreas Velten. Single-photon camera guided extreme dynamic range imaging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1575-1585, 2022.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009-12019, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Sizhuo Ma, Shantanu Gupta, Arin C Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta. Quanta burst photography. ACM Transactions on Graphics (TOG) , 39(4):79-1, 2020.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3883-3891, 2017.

Jihyong Oh and Munchurl Kim. Demfi: Deep joint deblurring and multi-frame interpolation with flowguided attentive correlation and recursive boosting. arXiv preprint arXiv:2111.09985 , 2021.

Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6820-6829, 2019.

Wenqi Ren, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang. Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In Proceedings of the IEEE International Conference on Computer Vision , pages 1077-1085, 2017.

Wei Shang, Dongwei Ren, Dongqing Zou, Jimmy S Ren, Ping Luo, and Wangmeng Zuo. Bringing events into video deblurring with non-consecutively blurry frames. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4531-4540, 2021.

Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao. Blurry video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5114-5123, 2020.

Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao. Video frame interpolation and enhancement via pyramid recurrent framework. IEEE Transactions on Image Processing , 30:277-292, 2020.

Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 14489-14498, 2021.

Maitreya Suin and AN Rajagopalan. Gated spatio-temporal attention-guided video deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7802-7811, 2021.

Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII , pages 412-428. Springer, 2022.

Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. arXiv preprint arXiv:2301.05191 , 2023.

Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8174-8182, 2018.

Minggui Teng, Chu Zhou, Hanyue Lou, and Boxin Shi. Nest: Neural event stack for event-based image enhancement. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI , pages 660-676. Springer, 2022.

Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang. Event enhanced high-quality image recovery. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16 , pages 155-171. Springer, 2020.

Yixuan Wang, Jianing Li, Lin Zhu, Xijie Xiang, Tiejun Huang, and Yonghong Tian. Learning stereo depth estimation with bio-inspired spike cameras. In 2022 IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6. IEEE, 2022.

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17683-17693, 2022.

Jonas Wulff and Michael Julian Black. Modeling blurred video with layers. In European Conference on Computer Vision , pages 236-252. Springer, 2014.

Xijie Xiang, Lin Zhu, Jianing Li, Yixuan Wang, Tiejun Huang, and Yonghong Tian. Learning superresolution reconstruction for high temporal resolution spike stream. IEEE Transactions on Circuits and Systems for Video Technology , 2021.

Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion deblurring with real events. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2583-2592, 2021.

Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1107-1114, 2013.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2022.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16 , pages 492-511. Springer, 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14821-14831, 2021.

Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII , pages 34-52. Springer, 2022.

Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17765-17774, 2022.

Jing Zhao, Jiyu Xie, Ruiqin Xiong, Jian Zhang, Zhaofei Yu, and Tiejun Huang. Super resolve dynamic scene from continuous spike streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2533-2542, 2021.

Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11996-12005, 2021.

Junwei Zhao, Shiliang Zhang, Zhaofei Yu, and Tiejun Huang. Spireco: Fast and efficient recognition of high-speed moving objects with spike cameras. IEEE Transactions on Circuits and Systems for Video Technology , pages 1-1, 2023.

Rui Zhao, Ruiqin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. Advances in Neural Information Processing Systems , 35:7905-7920, 2022.

Yajing Zheng, Zhaofei Yu, Song Wang, and Tiejun Huang. Spike-based motion estimation for object tracking through bio-inspired unsupervised learning. IEEE Transactions on Image Processing , 32:335-349, 2023.

Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6358-6367, 2021.

Zhihang Zhong, Mingdeng Cao, Xiang Ji, Yinqiang Zheng, and Imari Sato. Blur interpolation transformer for real-world motion from blur. arXiv preprint arXiv:2211.11423 , 2022.

Zhihang Zhong, Xiao Sun, Zhirong Wu, Yinqiang Zheng, Stephen Lin, and Imari Sato. Animation from blur: Multi-modal blur decomposition with motion guidance. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX , pages 599-615. Springer, 2022.

Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME) , pages 1432-1437. IEEE, 2019.

Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1438-1446, 2020.

Peer reviews:
comments: SUMMARY:
This paper proposed the first deblurring method that uses two types of inputs: RGB frames and spike streams. Two synthetic datasets are introduced. They are also used for training and evaluating the proposed method. This method outperforms other state-of-the-art methods on these synthetic datasets.

STRENGTHS:
* This paper proposes the first spike-based motion deblurring model, which is based on transformers.
* The authors claim that they can reconstruct sub-frame sharp images at any timestamp, which is a great advantage.
* The datasets can be of research value to the community. 
* The proposed method outperforms SOTA on those synthetic datasets.

WEAKNESSES:
* The datasets use synthetic blurry images and a synthetic spike camera simulator. The method is evaluated on a real spike stream only on a few images in the supplementary materials (which is not even mentioned in the main paper). Moreover, this brings up another issue: synchronizing a real spike camera with an RGB camera is not trivial. Some hybrid camera setup is shown in the supplementary materials, but it requires manually aligned spatial and temporal outputs. More thorough evaluation is needed.
* The main loss (13) is not clear. The first and the last terms seem the same. Why are they separated?
* The paper contains some typos: L205 (noisy), L76/L98 (Researches), L15 (vFurthermore)
* Intro: not true that deep learning models always find a clear image (L32)

QUESTIONS:
Addressing any of the above weaknesses would be great, especially about the real data.

LIMITATIONS:
Some limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to remove motion blur from high-speed scenes making use of spike streams. Most deep learning-based deblurring algorithms predict sharp frames relying only on the input blurry frames and are not robust when the blurry artifact is severe. This work proposes to use spike streams that could be obtained from spike cameras along with the blurry input for motion deblurring. The authors propose a framework that integrates both modalities in a manner where information is shared bi-directionally. In addition, the work builds two synthetic datasets on top of the GoPro and X4K1000FPS datasets for training and evaluation. Experimental comparisons are presented with several baselines.\n\nSTRENGTHS:\n* The proposed problem formulation of motion deblurring using spike streams is quite new and a promising direction for further research\n* The joint training of RGB and gray image reconstruction is an intuitive approach to enforce the proposed network to use both modalities\n* The Spk-X4K1000FPS and Spk-GoPro datasets would be a useful contribution to the community if they become publicly available\n* The qualitative results look good\n\nWEAKNESSES:\n* The motivation of the work is not convincingly justified\n\n   There are several previous works [1,2] that use event information for motion deblurring. Compared to these works, it is not clear why the paper chooses spike information over events. The explanation in L52-53 that \"...spike cameras record low-resolution texture information and ... this serves as a stronger guidance for deblurring task\" is very vague and not convincing. What does low-resolution texture information have to do with motion deblurring? and why should it be a stronger guidance compared to high temporal resolution? \n\n   The claim in L49-51 that, \"Most event-based methods unidirectionally utilize information from the event domain to assist the image domain, without achieving the complementarity of information from both domains\", is not true. There are several event-based works [1] that jointly use both the RGB and event information for motion deblurring. Hence, this claim should be toned down. \n\n* The experimental results in the paper are weak and unfairly done\n\n   The experimental comparison on X4K1000FPS in Table 1 is not very meaningful as it is clearly biased to the proposed method. The other baselines HiNet and NAFNet do not use extra data and hence, it is not surprising that they underperform. A more thorough and fair comparison should be done with previous event-based methods following their experiment protocol.\n\n   The results for the GoPro dataset in Table 2 are simply copied (quoted) from previous papers. However, the authors follow a different experiment setting in their paper. Hence, how can the authors conclude that the performance gain is coming from the proposed approach and not necessarily from the different experimental protocols?\n\n    The qualitative analysis both in the main paper and supplementary ignores to compare with the state-of-the-art approach, REFID [2]. Why is that?\n\n* More ablations are needed to justify the different design choices in the proposed approach\n\n   There has to be an ablation experiment that only uses the image stream to prove the benefit of incorporating the spike stream\n   \n   I want to see the ablation for using the claims in L188-193 (directly fusing the input blurred image with the spike branch). Removing the whole CSFI, CAMMA, and initial deblurring branch only results in a PSNR decrease of 0.2 dB. Hence, how worse would it be if we naively fuse the blurred input with the spike branch?\n\n   What is the point of doing several experiments with different input representations? How does that lead to a better understanding of the proposed approach? I get that the spike branch leads to a good performance. However, all other modules seem to be very redundant with very minor contributions to network performance. \n\n    It will be good to provide the visualization of the motion magnitude mask in L208, to verify if the CAMMA module is indeed doing what it is claimed to be doing. \n\n\n* The writing of the paper could be improved\n\n   The submission will greatly benefit from re-writing the methodology part by removing too many unnecessarily coined acronyms and by restructuring the explanation of the different modules in the proposed approach. Moreover, important details such as train-test set splits and results on real-world blurred images should be in the main paper instead of the supplementary. \n\n\nReferences\n\n[1]. Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention, ECCV 2022\n\n[2]. Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring, CVPR 2023\n\nQUESTIONS:\nPlease refer to the \"Weaknesses\" section and address the raised concerns carefully.\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a motion deblurring method that integrates RGB images and binary spike streams. In detail, it has a content-aware motion magnitude attention module and a transposed cross-modal attention fusion module. Experiments demonstrate state-of-the-art performance on deblurring datasets.

STRENGTHS:
1, the first spike-based motion deblurring model.

2, the two large-scale synthesized datasets for spike-based motion deblurring

3, state-of-the-art results.

WEAKNESSES:
1, The Transposed Cross-Attention Fusion (TCAF) is close to the multi-Dconv head transposed attention (MDTA) in Restormer.

2, The TCAF module is similar to EICA in EFNet.

3, The spike camera is similar to event camera, and the method based on these two methods are also similar except the format of event/spike stream. I suspect that event-based methods also works with spike camera. Because the paper mainly forcuses on the algorithm design, and the proposed SpkDeblurNet makes no huge differences with the existing EFNet, from this aspect the novelty of the paper is limited.

4, In the abstract, "preserving rich spatial details" and "are low-resolution image signals" are conflicting and confusing.

5, Typo in abstract.

QUESTIONS:
See Weaknesses

LIMITATIONS:
See Weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach that integrates the two modalities from two branches, leveraging spike streams as auxiliary visual cues for guiding deblurring in high-speed motion scenes, introducing a content-aware motion magnitude attention module and transposed \n cross-modal attention fusion module.\n\nSTRENGTHS:\n1. Authors propose the first spike-based motion deblurring model equipped with content-aware motion magnitude attention and a cross-modal transposed attention fusion module.\n\n2. Extensive experiments have demonstrated the effectiveness of the proposed model \n\n3. Overall, the paper is well-written and technically sound.\n\nWEAKNESSES:\n1. Although the authors conducted the validation on two synthetic datasets that the proposed method is effective, the performance on real scenarios is still unknown. Especially, it is hard to simultaneously obtain aligned pairs of spike streams and RGB images. So, have authors ever considered constructing a camera system to truly capture the spike data and blur images to better demonstrate the proposed setup is feasible and has practical value ?\n\n2. Considering the extra cost of spike cameras, is it really worth using such an expensive device for deblurring rather than using a high-speed global shutter camera directly capture relatively sharp images? Authors should provide more motivations for using spike cameras from the perspective of reality.\n\n\n3. What about the generalization ability of the proposed method? Authors should present related experiments. Moreover, the computational complexity and inference time are also supposed to be compared.\n\n4. Some minor issues: typos (e.g. line 15)\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.\n\nI encourage the authors to actively and rigorously prepare the rebuttal and I will raise the rating if my concerns are well-addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach that combines traditional cameras and spike cameras to address motion blur in high-speed scenes. By leveraging spike streams as auxiliary visual cues, the proposed spike-based motion deblurring model effectively extracts relevant information from blurry images using content-aware motion magnitude attention and transposed cross-modal attention fusion modules.\n\nSTRENGTHS:\nThe paper proposes to novel technique to deblur a blurry scene using a traditional camera frame along with spike camera frames. The proposed method is able to leverage the high SNR information from the traditional camera frame and the motion information from the spike camera frames to reconstruct the final image with high quality.\n\nWEAKNESSES:\nThe authors have missed a major branch of related works using Quanta Image Sensors (QIS). QIS are a very similar family of image sensors that operate at high speed and produce single-bit frames, and has a lot of previous works on deblurring and denoising. For eg.,  [1] Ma, S., Gupta, S., Ulku, A.C., Bruschini, C., Charbon, E. and Gupta, M., 2020. Quanta burst photography. ACM Transactions on Graphics (TOG), 39(4), pp.79-1. [2] Chi, Y., Gnanasambandam, A., Koltun, V. and Chan, S.H., 2020. Dynamic low-light imaging with quanta image sensors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16 (pp. 122-138). Springer International Publishing. [3] Chandramouli, P., Burri, S., Bruschini, C., Charbon, E. and Kolb, A., 2019, May. A bit too much? High speed imaging from sparse photon counts. In 2019 IEEE International Conference on Computational Photography (ICCP) (pp. 1-9). IEEE.\n\nIn fact, \"[4] Liu, Y., Gutierrez-Barragan, F., Ingle, A., Gupta, M. and Velten, A., 2022. Single-photon camera guided extreme dynamic range imaging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1575-1585). \" uses a very similar idea of combining two imaging modalities, but for a different task - HDR imaging.\n\nEDIT:\nWith the promised change, my major concern will be addressed.\n\nQUESTIONS:\nThe related works and the comparison section should ideally have a detailed comparison with the QIS based methods.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}