Title: SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams
Venue: NeurIPS 2024
Accepted: yes
Keywords: Motion Deblur

Spike Camera

Self-supervised
Abstract: Reconstructing a sequence of sharp images from the blurry input is crucial for enhancing our insights into the captured scene and poses a significant challenge due to the limited temporal features embedded in the image. Spike cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Nonetheless, existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios that diverge from the synthetic training data domain. To address these challenges, we propose the first self-supervised framework for the task of spike-guided motion deblurring. Our approach begins with the formulation of a spike-guided deblurring model that explores the theoretical relationships among spike streams, blurry images, and their corresponding sharp sequences. We subsequently develop a self-supervised cascaded framework to alleviate the issues of spike noise and spatial-resolution mismatching encountered in the deblurring model. With knowledge distillation and re-blurring loss, we further design a lightweight deblur network to generate high-quality sequences with brightness and texture consistency with the original input. Quantitative and qualitative experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization of the proposed framework. Our code, data and trained models are available at \url{https://github.com/chenkang455/S-SDM}.

SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams

Abstract

Reconstructing a sequence of sharp images from the blurry input is crucial for enhancing our insights into the captured scene and poses a significant challenge due to the limited temporal features embedded in the blurry image. Spike cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Nonetheless, existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios that diverge from the synthetic training data domain. To address this challenge, we propose the first self-supervised framework for the task of spike-guided motion deblurring. Our approach begins with the formulation of a spike-guided deblurring model that explores the theoretical relationships among spike streams, blurry images, and their corresponding sharp sequences. We subsequently develop a self-supervised cascaded framework to alleviate the issues of spike noise and spatial-resolution mismatching encountered in the deblurring model. With knowledge distillation and reblur loss, we further design a lightweight deblur network to restore high-quality sequences with brightness and texture consistency with the original input. Quantitative and qualitative experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization of the proposed framework. Our code, data and trained models are available at https://github.com/chenkang455/S-SDM .

1 Introduction

Traditional cameras, constrained by their exposure-based imaging mechanism, often produce blurry images when capturing fast-moving objects or during camera movement throughout the exposure process [19, 35]. While these blurry images lose significant details, the ability to recover dynamic motion trajectories from the static blurry input becomes critically important. However, the inherent challenge lies in the limited motion features available within blurry frames, leading to potential ambiguities such as multiple motion trajectories corresponding to the same blurry input. This is exemplified by scenarios where two objects move along the same trajectory but in opposite directions [23, 24, 38], rendering the task of motion deblurring ill-posed. Recent advancements in learningbased approaches [40, 12, 46] seek to address this challenge by establishing direct mappings from

Synthetic

Blur/Spike

𝟎𝟎

𝑻𝑻

Real-world

Blur/Spike

𝑻𝑻

𝟎𝟎

Supervised

Self-Supervised (Ours)

…

…

w/o GT

w/ GT

…

…

…

…

w/o GT

…

…

w/o GT

Train

Test

Train

Finetune



color



light



texture

Deterioration



color



light



texture

Generalization

Methods

Datatype

Spike-guided

Deblur-Net

blurry inputs to sharp sequences in a supervised learning manner. Despite these efforts, traditional cameras struggle to capture fine details in high-speed motion due to their exposure constraints, thus limiting the effectiveness of these methods in scenarios not covered by the training datasets.

In recent years, neuromorphic cameras [9, 10], leveraging their ultra-high temporal resolution and high dynamic range, have found widespread use in many fields, including computer vision and robotics. These cameras, including event and spike cameras, are distinguished by their ability to produce high temporal resolution outputs directly tied to changes in light intensity. Specifically, event cameras generate events in areas where light intensity changes [4], while spike cameras capture the absolute brightness of the scene at each pixel, offering a stream of spikes as output [10]. This distinctive feature endows spike cameras with a significant advantage [47, 45, 48, 5, 41, 6] in capturing and recovering sharp texture from scenes with rapid motion.

Recent studies [2, 8] have explored the potential of RGB-Spike fusion, i . e ., harnessing the strengths of both traditional and spike cameras to reconstruct sharp sequences from blurry inputs. However, their frameworks are constrained within the supervised learning paradigm, which necessitates extensive datasets comprising pairs of blurry and sharp images, as well as spike sequences. While synthetically acquiring such paired data, as demonstrated in previous studies [8, 4, 24], is feasible, collecting them in real-world scenarios presents the following challenges: (1) high-speed cameras are prohibitively expensive and not readily deployable in many settings; (2) spatial-temporal calibration between spike cameras and high-speed RGB cameras complicates the data collection process. These problems render the fine-tuning of supervised methods on real-world datasets challenging, further leading to their performance deterioration in such environments as shown in Fig. 1. The resulting degradation in image quality, manifesting as color distortion, brightness inconsistency, and inaccurate texture restoration, is mainly caused by the disparity between synthetic and real-world datasets, especially in terms of the density of spike stream, spike generation mechanism, and blurry image generation. Moreover, the effectiveness of supervised methods is inherently limited by the ground truth sequences created through motion analysis interpolation algorithms [22, 11], which inherently differs from the real-world scene and thus affects the model's generalization ability.

To overcome these issues, we propose the first-of-its-kind S elf-supervised S pike-guided D eblurring M odel (S-SDM) , capable of recovering the continuous sharp sequence from a single blurry input with the assistance of low-resolution spike streams. We begin with a theoretical analysis of the

relationship between spike streams, blurry images, and sharp sequences, leading to the development of our Spike-guided Deblurring Model (SDM). We further construct a self-supervised processing pipeline by cascading the denoising network and the super-resolution network to reduce the sensitivity of the SDM to spike noise and its reliance on spatial-resolution matching between the two modalities. To reduce the computational cost and enhance the utilization of spatial-temporal spike information within this pipeline, we further design a Lightweight Deblurring Network (LDN) and train it based on pseudo-labels from the teacher model, i . e ., the established self-supervised processing pipeline. Further introducing reblur loss during LDN training, we achieve better restoration performance and faster processing speed than the processing-lengthy and structure-complicated teacher model. To validate the performance of our S-SDM across various scenarios, we build an RGB-Spike binocular system and propose the first spatially-temporally calibrated Real-world Spike Blur (RSB) dataset in this community. Quantitative and qualitative experiments conducted on the real-world and synthetic datasets validate the superiority of our method. In summary, our key contributions are:

We develop a self-supervised spike-guided image deblurring framework, addressing the performance degradation due to the synthetic-real domain gap in supervised methods.

We perform an in-depth theoretical analysis of the fusion between the spike stream and blurry image, leading to the development of the SDM.

We propose a real-world dataset RSB and experiments on GOPRO and RSB datasets validate the superior generalization of our S-SDM.

2 Related Work

Spike Camera. The spike camera, inspired by the primate retina, stands apart from conventional cameras with its ability to generate synchronous spike streams for each pixel at extremely low latency. This distinct feature provides significant advantages in various applications such as highspeed imaging [47, 45, 48, 5, 41, 6, 33], optical flow estimation [43], object detection [44], 3D reconstruction [30], depth estimation [34], motion deblurring [8, 32], and occlusion removal [31].

Spike-guided Motion Deblurring. While the spike camera boasts an ultra-high temporal resolution, its development is currently impeded by the low spatial resolution. Additionally, the single-channel output from the spike camera restricts previous methods from recovering the image color information. To address these issues, a promising approach is establishing an RGB-Spike hybrid imaging system [2]. The binocular system achieves the multi-modality fusion of High-spatial/Low-temporal RGB blurry input and High-temporal/Low-spatial spike stream, thereby also serving as a spike-guided motion deblurring method [8]. However, to the best of our knowledge, existing spike-guided deblurring methods [2, 8] predominantly rely on supervised training on synthetic datasets. This reliance results in significant performance degradation when these methods are evaluated in real-world scenarios due to the domain discrepancies between synthetic and real datasets as illustrated in Fig. 1.

Event-based Motion Deblurring. Event camera [21] can asynchronously generate events that record log-intensity changes at the pixel level with minimal latency, which contains a rich set of motion features beneficial for motion deblurring tasks. Numerous supervised methods [16, 4, 3, 16, 23, 24] have been proposed to learn the mapping from the blurry input, events to the sharp outcome. Despite these advancements, a major hurdle remains in obtaining real blurry-sharp image pairs for training. To overcome the domain gap between synthetic and real-world datasets, recent methods [29, 38, 39] explored the mutual constraint between the blurry image and event stream, enabling the training of networks on real-world blur datasets.

3 Method

3.1 Preliminaries

Spike Camera Mechanism. Consider L ( t ) to represent the latent sharp frame at time t . Each pixel p in the spike camera [10] has an integrator that accumulates the incoming photons at a high frequency. Once the cumulative intensity exceeds a predefined threshold C at time t e , pixel p emits a spike, and the accumulation of photons is reset to zero. This process can be mathematically described as

follows:

where t s denotes the firing time of the previous spike. While the spike camera is capable of generating asynchronous spike streams akin to that of event cameras, its effectiveness is constrained by the inherent limitations of its physical circuitry, which necessitates reading spikes at a predetermined sampling rate. We denote the generated spike stream as S ∈ { 0 , 1 } K × 1 × H × W , where H and W signify the height and width of the image, and K represents the length of the spike sequence.

Problem Formulation. In traditional photography, motion blur occurs when there is relative movement between the camera and the scene during the exposure period. According to the motion blur physical model [9], the blurry image B can be represented as the average of the latent frame L ( t ) over the exposure T , i . e .:

where T represents the exposure period. Despite the spike camera's superior temporal resolution, its spatial resolution remains comparatively low. This limitation is primarily attributed to the constraints in data transmission bandwidth and the challenges inherent in the manufacturing process. Here, we postulate that the spatial resolution of the spike camera is approximately one-quarter that of a conventional RGB camera.

In this paper, we aim to enhance the High-spatial/Low-temporal resolution blurry input B ∈ R 1 × 3 × H × W into a sequence of High-Quality images { L ( t i ) } K i =1 ∈ R K × 3 × H × W with the aiding of High-temporal/Low-spatial resolution spike stream S T ∈ { 0 , 1 } K × 1 × H 4 × W 4 , which can be mathematically formulated as:

In Eq. (3), Deblur ( · ) represents the Spike-guided Deblur-Net as shown in Fig. 1, i refers to the i -th frame in the spike stream S T , and t i is the timestamp associated with this frame.

3.2 Theoretical Analysis

The spike camera, with its photodetector tailored to capture the single-channel light intensity, faces difficulties in obtaining the color information that the multi-channel RGB camera can effortlessly capture. Therefore, we modify the color intensity L ( t ) in Eq. (1) to the grayscale value L g ( t ) for further analysis:

In this formulation, L g ( t ) = w r · L r ( t ) + w gre · L gre ( t ) + w b · L b ( t ) , where L c ( t ) , w c denote the intensity and weight of channel c ∈ { r, gre, b } respectively.

Given the blurry input B and its corresponding spike stream S T , we incorporate Eq. (4) into the motion blur model presented in Eq. (2). This integration formulates a link between the two modalities as outlined below:

where B g denotes the grayscale version of the blurry input, and N T denotes the total number of spikes accumulated over the exposure period. The accumulation N T is calculated as N T = ∑ K i =1 S [ i ] , with S [ i ] indicating the i -th frame of the spike stream.

Within the exposure T , we consider a shorter spike sequence centered around the t moment S T ′ ∈ { 0 , 1 } K ′ × 1 × H × W , satisfying K ′ glyph[lessmuch] K , t ∈ T , and T ′ ⊂ T . Similar to Eq. (5), we can derive the relationship between the short-exposure gray image E g ( t, T ′ ) and the short spike stream S T ′ as follows:

where T ′ glyph[lessmuch] T represents the short exposure period.

预训练网络

Spike-Nerf

实验

RGB

Spike

Student

Time

Teacher

Reconstruction

Image

Encoder

C

LDN

Blur

Spike

Spike

Encoder

Decoder

BSN

EDSR

SDM

Exposure

Given the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the short-exposure image E ( t, T ′ ) are identical. This assumption implies that the intensity proportion among RGB channels in the blurry input α B c and the short-exposure image α E c ( t, T ′ ) is approximately equivalent, satisfying α B c = B g / B c and α E c ( t, T ′ ) = E g ( t, T ′ ) / E c ( t, T ′ ) . More details can be found in the supplementary materials.

Upon establishing it, we move forward to build a mathematical relation between the blurry image and the spike stream. By substituting the gray channel g with color channel c and dividing Eq. (5) by Eq. (6), we efficiently eliminate the unknown threshold C and weights α B c /α E c ( t, T ′ ) , leading to the following equation:

By applying Eq. (7) to RGB three channels, we explicitly establish the relationship between the color blurry input B , the color short-exposure image E ( t, T ′ ) and the spike stream S T as shown in Fig. 18. Since the exposure period T ′ is relatively short, it is reasonable to assume that the scene remains static. In this context, we interpret the short-exposure image E ( t, T ′ ) as the latent sharp frame L ( t ) , allowing us to modify Eq. (7) as follows:

To this end, we have conducted a comprehensive theoretical analysis of the spike-guided motion deblurring task, which is neglected in prior learning-based motion deblur methodologies [2, 8]. For further discussion readability, we refer to Eq. (8) as the Spike-guided Deblurring Model (SDM) , which is analogous to the baseline motion deblur model EDI [18] in event camera.

3.3 Self-supervised Spike-guided Deblurring Model

3.3.1 Processing Pipeline

While SDM theoretically allows for the fusion of the blurry image and the spike stream, its practical deployment faces the following obstacles:

The deblurred image E ( t, T ′ ) suffers from noise-related degradation due to the lack of adequate spike information during the short exposure T ′ .

The spatial resolution of the spike camera is approximately one-quarter of the RGB camera, rendering the SDM implementation impractical.

To overcome these limitations, we further cascade the self-supervised denoising network to eliminate the spike noise in N T ′ and super-resolution network to match spatial resolutions of the blurry image and spike stream, with the processing pipeline illustrated in the bottom of Fig. 2.

Denoising Network. We leverage the Blind Spot Network (BSN) [13, 1, 27, 5, 14, 7] to predict the clean spike accumulation N T ′ from the input short-exposure spike stream S T ′ . The core idea of BSN is to design the blind-spot strategy that compels the convolutional layer to estimate the clean value of each pixel solely based on its surrounding pixels.

Under the premise that the spike stochastic thermal noise is independent identically distributed [42], the BSN is trained to deduce sharp spike frames from the input, with the loss function formulated as:

where the denoised spike frame BSN ( S T ′ ; Θ 1 ) is denoted by L bsn ( t ) for further analysis.

Super-Resolution Network. In this task, we observe that the blurry input B g and the long-exposure spike frame N T exhibit the same texture features as shown in Eq. (5). This observation motivates us to train the Super-Resolution (SR) network based on pairs of the blurry images and the long-exposure spike frames.

We leverage the well-explored Enhanced Deep Super-Resolution network (EDSR) [15] as the backbone of our SR network, with the loss function formulated as follows:

With the training of the SR network completed, we freeze its parameters and apply it to the denoised spike frame L bsn ( t ) , yielding the resolution-enhanced spike frame L ↑ bsn ( t ) .

3.3.2 Knowledge Distillation Framework

While the aforementioned processing pipeline achieves the multi-modality fusion of the blurry input and the spike stream, several aspects still need refinement:

The framework is lengthy and computationally demanding, which hinders its suitability for real-time system deployment.

The blind-spot strategy of the BSN limits the full utilization of the spatial information inherent in the spike stream.

The representation of the short-exposure image does not fully reflect the advantages of the high temporal resolution inherent in the spike stream.

To improve them, we further build a knowledge distillation framework building upon the existing processing pipeline. This pipeline serves as the teacher model, providing the reconstructed sequence as pseudo-labels for the training of the student model LDN, as illustrated in Fig. 2.

Lightweight Deblur Network. LDN adheres to a similar input and output pattern as previous research [8], i . e ., taking the blurry input B and the short spike stream S T ′ centered around moment t as inputs, with the output being the reconstructed sharp image L ldn ( t ) , mathematically formulated as follows:

Full details regarding the LDN structure are available in the supplementary materials.

To avoid the scenario where the LDN exactly replicates the mapping of the teacher model, we design the teacher loss L tea based on the LPIPS [36] loss and further introduce the blur reconstruction loss. The reblur loss L reblur measures the difference between the blurry input B and the re-synthesized blurry image ˜ B , satisfying:

where L ldn ( t m ) represents the m -th recovered image within the exposure period T and M is the total number of reconstructed images. Finally, we sum up two loss functions with the weighting parameter

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

321

312

322

313

323

314

324

325

315

326

327

316

328

317

329

318

330

319

331

332

320

333

334

335

336

337

10

ECCV 2024 Submission #3728

Input

Blur

BiT

LEVS

Spk

Deblur

Net

Motion-ETR

Ours

Blur

BiT

LEVS

Spk

Deblur

Net

Motion-ETR

Ours

Blur

BiT

LEVS

Spk

Deblur

Net

Motion-ETR

Ours

Blur

BiT

LEVS

Spk

Deblur

Net

Motion-ETR

Ours

Fig. 4:

4

4.1

ECCV 2024 Submission #3728

11

.

our self-supervised framework in real-world scenarios, we construct an RGB-

to Motion-ETR [32] and 'Spk-Net' is the abbreviation of SpkDeblurNet [8].

Experiments

Dataset

Synthetic Data.

For quantitative and qualitative analysis of our spike-guided

motion deblurring task, we construct the synthetic dataset based on the widely

employed motion deblur dataset GOPRO [17] with the resolution of

1280

×

720

Initially, we utilize the interpolation algorithm XVFI [21] to augment the video

frame by interpolating additional

7

frames between each pair of consecutive sharp

images. To generate the spike stream that mimics reality closely, we downsample

the interpolated video to the resolution of

320

×

180

and simulate the spike stream

based on the spike simulator [34] designed according to

the spike

generation

physical model. To replicate real-world motion blur, we synthesize each blurry

input by averaging

97

frames from interpolated video sequences.

Real-world Data.

To comprehensively assess the deblurring performance of

(a) Blur

(b) SpkDeblurNet

0

LEVS

M

otion

-ETR

BiT

Spk

Deblur

Net

Ours

(c) Ours

T/2

T

321

322

323

quence restoration evaluation as listed in Tab. 1. Given that the principal contri- bution of our method is self-supervised learning, it is foreseeable that our method

324

325

might be slightly inferior to the supervised method SpkDeblurNet on the syn-

326

4.2 Experimental Results 4 Experiments

thetic GOPRO dataset. Revisiting the motion ambiguity problem mentioned earlier, we observe that frame-based methods LEVS and Motion-ETR suffer

We conduct both quantitative and qualitative comparisons of our S-SDM against 4.1 Dataset

327

328

329

330

331

332

state-of-the-art (SOTA) methods, including frame-based Motion-ETR [32], LEVS [12], video-based BiT [38] and spike-based SpkDeblurNet [8] on the GOPRO and RSB datasets. Since SpkDeblurNet does not fully consider the real spatialresolution gap between two modalities in real-world scenarios, we modify its spike significant performance degradation in transitioning from single frame restoration to sequence reconstruction, with decreases of 4.425 dB ↓ and 5.866 dB ↓ respectively. In contrast, our method alleviates this issue benefiting from the incorporation of a high temporal-resolution spike stream. Synthetic Data. For quantitative analysis of our spike-guided motion deblurring task, we construct the synthetic dataset based on the widely employed GOPRO [17] dataset. We initially utilize the interpolation algorithm XVFI [22] to augment the video frame by interpolating additional 7 frames between each pair of consecutive sharp images. To generate the spike stream that mimics reality closely, we downsample the interpolated video to the resolution of 320 × 180 and simulate the spike

RSB Dataset.

We further present visualizations of single-frame reconstruc- tion and sequence reconstruction results on the real-world RSB dataset. For the 7

single-frame restoration

qualitative experiments as illustrated

in

Fig. 4, both frame-based and video-based methods fail to replicate fine textures and intri-

cate details present in the blurry input. While SpkDeblurNet is capable of re-

333

334

335

336

337

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 1

Ours

𝑉𝑉

= 1

Motion-ETR

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 2

Ours

𝑉𝑉

= 2

BiT

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 4

Ours

𝑉𝑉

= 4

𝑡𝑡𝑡

𝑡𝑡𝑡

𝑡𝑡𝑡

stream based on the spike simulator [42]. To replicate real-world motion blur, we synthesize each blurry input by averaging 97 frames from interpolated video sequences.

Real-world Data. We construct an RGB-Spike binocular system and propose the first R eal-world S pike-guided B lur dataset ( RSB ) in this community. This system consists of a fixed-exposure RGB camera (Basler acA1920-150uc) and a spike camera [10], enabling us to capture the blurry image and the corresponding spike stream simultaneously. Further details about our RSB and the spatial-temporal calibration for our binocular system are provided in the supplementary materials.

4.2 Experimental Results

We conduct both quantitative and qualitative comparisons of our S-SDM against state-of-the-art (SOTA) motion deblurring methods, including frame-based Motion-ETR [40], LEVS [12], videobased BiT [46], event-based TRMD [4], REFID [25], RED [29] and spike-based SpkDeblurNet [8] on the GOPRO and RSB datasets. For event-based methods, we replace the event stream with the spike stream and adopt the same input representation in these methods [29, 25, 4]. We further cascade the image super-resolution technique DASR [26] as in [39] for the deblurred sequence to overcome the modality resolution inconsistency which is not considered in these methods. We reconstruct 7 images from one blurry input for sequence restoration evaluation [39] as listed in Tab. 1.

glyph[negationslash]

Results on GOPRO. To simulate the spike density domain gap as depicted in Fig. 1, we train all supervised spike-based deblurring methods on the GOPRO dataset under spike threshold [42] V th = 1 and evaluate them on datasets with spike thresholds V th = 1 , 2 , 4 . Quantitative comparison results are listed in Tab. 1 and the visual comparison is demonstrated in Fig. 5. Given that the principal contribution of our S-SDM is self-supervised learning, it is foreseeable that our method might be slightly inferior to the supervised methods SpkdDeblurNet on the dataset with V th = 1 . While these supervised methods deteriorate on datasets with V th = 1 , our method achieves great generalization benefiting from the self-supervision design. Specifically, SpkDeblurNet tends to produce darker and blurrier reconstructions on images with high thresholds as shown in Fig. 5. Besides, our method achieves better restoration performance than the self-supervised method RED due to our consideration of the spatial-resolution mismatch between two modalities and the designed teacher loss, which imposes a stronger constraint than the optical loss in RED.

Results on RSB. We further present visualizations of single frame and sequence reconstruction comparisons on the real-world RSB dataset as depicted in Fig. 3 and 4. Both frame-based and videobased approaches fail to replicate fine textures and detailed elements present in the blurry input. While SpkDeblurNet is capable of recovering structural details and the motion trajectory, it is deteriorated by significant noise and color distortion under conditions with lower spike density than those simulated in the GOPRO dataset. Similarly, in scenarios of higher spike stream densities, the restoration of SpkDeblurNet tends to exhibit over-exposure, resulting in brightness inconsistency compared to the blurry input. This over-exposure affects the dynamic range of the reconstructed image, ultimately compromising the overall perceptual quality and uniformity of the restored sequence. Our method addresses these challenges by finetuning on the RSB dataset, ensuring that the restored sequence aligns with the real-captured blurry input and spike stream. More comparative experiments and analyses are accessible in the supplementary material.

4.3 Ablation Study

We perform ablation experiments on the GOPRO and RSB datasets to evaluate the performance of each module within S-SDM, the validity of our designed network architecture, as well as the overall effectiveness of our distillation learning framework. In this subsection, we evaluate the performance based on the single middle frame for simplicity.

Modules Cascading. Building upon the SDM, we sequentially cascade the BSN, the SR, and the LDN to evaluate their respective effectiveness, with quantitative results on GOPRO presented in Tab. 3. We employ bilinear interpolation as a substitute for the SR network in experiments I-1 and I-2 to align the spatial resolution of two modalities.

Qualitative ablation experiments are illustrated in Fig. 6. These comparisons reveal that while the SDM effectively removes motion blur, it struggles with significant noise and detail loss due to the spike noise and the low resolution of the spike stream. While the BSN mitigates noise and the SR network improves spatial resolution explicitly, the LDN trained via distillation learning further refines these enhancements, enabling the recognition of intricate textural features in the images, such as the license plate and the door number shown in Fig. 6.

Network Architecture. While our designed LDN mirrors the Scale-aware Network (SAN) proposed in the GEM [39], we replace the SAN with the LDN to compare the performance difference between the two architectures as depicted in Tab. 2. Despite the simple design of our LRN, which consists only of convolutional layers, ResBlocks, and basic modules such as CBAM [28], it outperforms SAN in both PSNR and SSIM while requiring fewer parameters and less computation, demonstrating that LDN is both sufficient and efficient for this task. This conclusion is consistent with our previous discussion, i . e ., the restoration performance of the self-supervised learning framework primarily depends on the quality of pseudo-labels rather than the network architecture.

I-4

I-3

GT

Blur

I-1

I-2

I-4

I-3

GT

Blur

I-1

I-2

II-6

II-5

II-7

II-2

II-3

II-4

II-6

II-5

II-7

II-2

II-3

II-4

224002172

2400212

ABA

Distillation Learning. We focus on analyzing the contribution of the teacher loss L tea and the reblur loss L reblur within the distillation framework, with quantitative results listed in Tab. 4. Without the teacher loss L tea, the LDN tends toward learning identity mapping from the blurry input. While under the guidance of the teacher model, the reblur loss L reblur not only enforces motion consistency in the reconstructed sequence but also enriches the LDN with high-resolution details from the non-blurry regions of the input, thus improving the performance on GOPRO as listed in Tab. 4.

We further apply the LDN trained on GOPRO to the real-world dataset RSB, with qualitative visualization illustrated in Fig. 7. As observed in the figure, the absence of the reblur loss L reblur leads to significant noise in the recovered image, which predominantly arises from the disparity in spike density and generation mechanism between the simulated and real-captured spike stream. This discrepancy causes the LDN to overestimate the spike number, resulting in black holes in regions with lower spike density than simulated, which reflects the drawback inherent in the supervised learning strategies as discussed in Sec. 1. Increasing the weight of the reblur loss L reblur allows the LDN to incorporate more information from the blurry input, thereby mitigating this issue. However, this adjustment also leads to the presence of blurry edges. We follow the parameters set in Experiment II-6 and retrain the LDN on the RSB dataset (referred to as Experiment II-7). The retrained LDN effectively recovers the sharp edge of the calibration board and suppresses the spike noise in the background, validating the feasibility of our self-supervised framework in real-world scenarios.

5 Conclusion

In this paper, we introduce a novel self-supervised spike-guided motion deblurring framework S-SDM, which reconstructs sequences of sharp images from real-world blurry inputs with the spike stream. Additionally, we construct an RGB-Spike binocular system and propose the first spatially-temporally calibrated real-world dataset RSB in this community. Quantitative and qualitative experiments validate the superior generalization capabilities of our proposed S-SDM.

Limitation. The limitation of our S-SDM lies in its dependence on strict spatial-temporal calibration. Misalignment will lead to color shifts and quality degradation in the deblurred sequence.

Acknowledgments

Wesincerely appreciate Yuyan Chen (HUST) for her valuable suggestions and for polishing the figures. This work was supported by the National Natural Science Foundation of China (62176003, 62088102, 62306015), the China Postdoctoral Science Foundation (2023T160015), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and the Beijing Nova Program (20230484362).

References

Jaeseok Byun, Sungmin Cha, and Taesup Moon. Fbi-denoiser: Fast blind image denoiser for poissongaussian noise. In CVPR , pages 5768-5777, 2021.

Yakun Chang, Chu Zhou, Yuchen Hong, Liwen Hu, Chao Xu, Tiejun Huang, and Boxin Shi. 1000 fps hdr video with a spike-rgb hybrid camera. In CVPR , pages 22180-22190, 2023.

Haoyu Chen, Minggui Teng, Boxin Shi, Yizhou Wang, and Tiejun Huang. A residual learning approach to deblur and generate high frame rate video with an event camera. IEEE TMM , 2022.

Kang Chen and Lei Yu. Motion deblur by learning residual from events. IEEE TMM , 2024.

Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. IJCAI, 2022.

Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In AAAI , volume 37, pages 350-358, 2023.

Shiyan Chen, Jiyuan Zhang, Zhaofei Yu, and Tiejun Huang. Exploring asymmetric tunable blind-spots for self-supervised denoising in real-world scenarios. arXiv preprint arXiv:2303.16783 , 2023.

Shiyan Chen, Jiyuan Zhang, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Enhancing motion deblurring in high-speed scenes with spike streams. In NeurIPS , 2023.

Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE TPAMI , 44(1):154-180, 2020.

Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, et al. 1000 × faster camera and machine vision with ordinary devices. Engineering , 25:110-119, 2023.

Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV , pages 624-642. Springer, 2022.

Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning to extract a video sequence from a single motion-blurred image. In CVPR , pages 6334-6342, 2018.

Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. NeurIPS , 32, 2019.

Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In CVPR , pages 17725-17734, 2022.

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPR , pages 136-144, 2017.

Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren. Learning event-driven video deblurring and interpolation. In ECCV , pages 695-710. Springer, 2020.

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR , July 2017.

Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive at high frame-rate with an event camera. In CVPR , pages 6820-6829, 2019.

Bryan Peterson. Understanding exposure: how to shoot great photographs with any camera . AmPhoto books, 2016.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Teresa Serrano-Gotarredona and Bernabé Linares-Barranco. A 128 × 128 1.5% contrast sensitivity 0.9% fpn 3 µ s latency 4 mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers. IEEE Journal of Solid-State Circuits , 48(3):827-838, 2013.

Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In ICCV , pages 14489-14498, 2021.

Chen Song, Qixing Huang, and Chandrajit Bajaj. E-cir: Event-enhanced continuous intensity recovery. In CVPR , pages 7803-7812, 2022.

Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In ECCV , pages 412-428. Springer, 2022.

Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. In CVPR , pages 18043-18052, 2023.

Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In CVPR , pages 10581-10590, 2021.

Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image denoising with visible blind spots. In CVPR , pages 2027-2036, 2022.

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV , pages 3-19, 2018.

Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion deblurring with real events. In ICCV , pages 2583-2592, 2021.

Jiyuan Zhang, Kang Chen, Shiyan Chen, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Spikegs: 3d gaussian splatting from spike streams with high-speed camera motion. arXiv preprint arXiv:2407.10062 , 2024.

Jiyuan Zhang, Shiyan Chen, Yajing Zheng, Zhaofei Yu, and Tiejun Huang. Unveiling the potential of spike streams for foreground occlusion removal from densely continuous views. arXiv preprint arXiv:2307.00821 , 2023.

Jiyuan Zhang, Shiyan Chen, Yajing Zheng, Zhaofei Yu, and Tiejun Huang. Spike-guided motion deblurring with unknown modal spatiotemporal alignment. In CVPR , pages 25047-25057, 2024.

Jiyuan Zhang, Shanshan Jia, Zhaofei Yu, and Tiejun Huang. Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI , volume 37, pages 137-147, 2023.

Jiyuan Zhang, Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. In ECCV , pages 34-52. Springer, 2022.

Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Björn Stenger, Ming-Hsuan Yang, and Hongdong Li. Deep image deblurring: A survey. IJCV , 130(9):2103-2130, 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , pages 586-595, 2018.

Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In CVPR , pages 14071-14081, 2023.

Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. In CVPR , pages 17765-17774, 2022.

Xiang Zhang, Lei Yu, Wen Yang, Jianzhuang Liu, and Gui-Song Xia. Generalizing event-based motion deblurring in real-world scenarios. In ICCV , pages 10734-10744, 2023.

Youjian Zhang, Chaoyue Wang, Stephen J Maybank, and Dacheng Tao. Exposure trajectory recovery from motion blur. PAMI , 44(11):7490-7504, 2021.

Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR , pages 11996-12005, 2021.

Junwei Zhao, Shiliang Zhang, Lei Ma, Zhaofei Yu, and Tiejun Huang. Spikingsim: A bio-inspired spiking simulator. In ISCAS , pages 3003-3007. IEEE, 2022.

Rui Zhao, Ruiqin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. NeurIPS , 35:7905-7920, 2022.

Yajing Zheng, Zhaofei Yu, Song Wang, and Tiejun Huang. Spike-based motion estimation for object tracking through bio-inspired unsupervised learning. IEEE TIP , 32:335-349, 2022.

Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In CVPR , pages 6358-6367, 2021.

Zhihang Zhong, Mingdeng Cao, Xiang Ji, Yinqiang Zheng, and Imari Sato. Blur interpolation transformer for real-world motion from blur. In CVPR , pages 5713-5723, 2023.

Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In ICME , pages 1432-1437. IEEE, 2019.

Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In CVPR , pages 1438-1446, 2020.

A Appendix

This supplementary material provides a comprehensive elaboration on the methodologies and experiments in this paper. It is organized into four distinct sections: Theory Analysis in appendix A.1, Network Settings in appendix A.2, RSB Dataset in appendix A.3, Experimental Details in appendix A.4 and Additional Figure in appendix A.5.

A.1 Theory Analysis

We define k B 1 and k B 2 as the ratios of the red channel to the green and blue channels in the blurry input B respectively, i . e .,

Since the gray image is the weighted sum of RGB channels, the ratio of the gray to the red image α B r is formulated as:

where w c denote the weight of channel c ∈ { r, gre, b } respectively.

Similarly, we define k E 1 ( t, T ′ ) and k E 2 ( t, T ′ ) as the fractions of the red channel relative to the green and blue channels in the short-exposure image E ( t, T ′ ) , resulting in:

Given the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the shortexposure image E ( t, T ′ ) are identical. This assumption implies that the intensity proportion among RGB channels in two images is approximately equivalent, i . e ., k B 1 ≈ k E 1 ( t, T ′ ) and k B 2 ≈ k E 2 ( t, T ′ ) . Drawing from Eq. (19) and Eq. (20), We further deduce the following relation:

which can be readily generalized across channels, leading to α B c ≈ α E c ( t, T ′ ) .

Blur

预训练网络

Spike-Nerf

实验

(b)

C

LDN

BSN

R

(a)

R

↓ ↓

5×5 Conv

stride = 2

R

Rotate

Transpose

Conv

CBAM

Predict

ResBlock

1×1 Conv

3×3 Conv

stride = 1

↓

C

Concat

LEVS

BiT

Motion-ETR

B

S'

SpkDeblurNet

A.2 Network Settings

A.2.1 Blind Spot Network

We construct our BSN based on the blind-spot strategy outlined in [13]. We first rotate the input image four times, then concatenate them into the U-Net [20] structure like denoising network. To prevent direct mapping from the input pixel to the output pixel, a single-pixel offset strategy is employed in the convolutional kernel to separate its receptive field from the central pixel. Finally, the denoising output is obtained by merging the results of four branches via a 1 × 1 convolution, as shown in Fig. 9(a). S-SDM (Ours)

A.2.2 Lightweight Deblur Network

The network structure is depicted in Fig. 9(b), where the encoder for blurry images consists of two layers of down-sampling convolutions to align the spatial resolution of two modalities. In contrast to the intricate cross-attention mechanism outlined in [4, 8], we implement the fusion of two modalities by the simple operation of Concat ( · ) , with the cascaded CBAM [28] and residual blocks for further feature fusion.

A.3 RSB Dataset

We detail the construction of our RGB-Spike hybrid system as illustrated in Fig. 10. The system comprises a Spike Camera-001T-Gen2 with a resolution of 400 × 250 pixels, paired with a Basler acA1920-150uc RGB Camera, offering a higher resolution of 1920 × 1200 pixels.

RGB Camera

1920 ×1200

Spike Camera

400 ×250

Calibration  Board

(a) RGB Camera

(b) RGB (Calibrated)

(c) Spike Camera

Ours

SpkDeblurNet

BiT

Motion-ETR

LEVS

Blur

Earphone

Face

Board-H

Board-M

Board-L

To achieve the spatial calibration of two cameras, we perform simultaneous captures of the calibration board using the RGB-Spike system as shown in Fig. 10. The RGB images are cropped to a resolution of 1600 × 1000, aligning them to be fourfold the resolution of the spike camera. We convert the cropped RGB image to grayscale and take the TFP [47] image reconstruction result as the reference from the spike camera. We utilize the MATLAB calibration toolbox to implement the calibration process, with the results detailed in Fig. 11.

RGB trans (a) RGB Camera (b) RGB (Calibrated) (c) Spike Camera Our RSB dataset contains 10 video sequences under different conditions, captured under varied conditions including scene brightness levels ( e . g ., Low, Middle, and High light) and motion patterns ( e . g ., camera shake and object motion), which introduce different types of motion blur. Besides, the RSB dataset comprises a large amount of blur-spike pairs with each blurry input corresponding to 400 spike frames.

Spike tfp A.4 Experimental Details

A.4.1 Comparison

To assess the performance of our method on the GOPRO dataset, we utilize the Peak Signal-toNoise Ratio (PSNR) and the Structural Similarity Index (SSIM) as the quantitative metrics, which are commonly used in motion deblurring tasks. In real-world datasets, where obtaining ground truth sharp sequence is challenging, we opt for the non-reference image quality assessment method Language-Image Quality Evaluator (LIQE) [37] as a reference. By assessing visual quality through

Board-Mid

Board-High

Face

Earphone

Blur

SpkDeblurNet [8]

Ours

(a)

(b)

(c)

(d)

Ours

Blur

SpkDeblurNet [8]

(a)

(b)

(c)

(d)

Ours

Ours

the computation of joint probabilities from visual-textual embeddings, LIQE adeptly identifies the clarity and blurriness of images independently of the ground truth, making it ideally suited for our task.

Comparison on the RSB dataset. Qualitative and quantitative experiments of the single frame restoration task on the RSB dataset are shown in Fig. 12 and Tab. 5 respectively. The visual results coupled with the LIQE metrics demonstrate that our method outperforms other methods in handling the real-world RSB dataset. While the supervised SpkDeblurNet encounters substantial noise and overexposure challenges in both low-light and high-light environments, our approach demonstrates superior restoration performance, which is attributed to the designed self-supervised framework.

Blur

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 1

Ours

𝑉𝑉

= 1

Motion-ETR

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 2

Ours

𝑉𝑉

= 2

BiT

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 4

Ours

𝑉𝑉

= 4

Azzaf

PhzaHht

Phzaht

Pezaht

Pizaht

Phzaht

𝑡𝑡𝑡

𝑡𝑡𝑡

𝑡𝑡𝑡

Blur

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 1

Ours

𝑉𝑉

= 1

Motion-ETR

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 2

Ours

𝑉𝑉

= 2

BiT

SpkDeblurNet

𝑉𝑉

𝑡𝑡𝑡

= 4

Ours

𝑉𝑉

= 4

𝑡𝑡𝑡

𝑡𝑡𝑡

𝑡𝑡𝑡

Moreover, as depicted in Fig. 13 and Fig. 8, S-SDM exhibits outstanding performance in precisely retrieving luminance information and texture details while ensuring the motion consistency of reconstructed sequences. In contrast, the video reconstruction of the BiT suffers from poor image quality and shows inadequate sequence continuity in the restored sequence. Besides, SpkDeblurNet encounters issues with color distortion, brightness inconsistency, and inaccurate texture restoration owing to the domain gap between synthetic and real-world datasets. These observations further highlight the superior performance of our S-SDM in real-world scenarios.

Comparison on the GOPRO dataset. We provide additional visual comparison of our method against other SOTA methods on the GOPRO dataset as shown in Fig. 14 and 15.

A.4.2 Implementation

To augment the dataset and accelerate the training process, we randomly crop 512 × 512 image from each blurry frame, along with the 128 × 128 spike stream. We use PyTorch to build and train our S-SDM using an NVIDIA GeForce GTX 4090 GPU and AMD EPYC 7742 64-Core Processor. The training of our LDN consumes about 4 hours on the GOPRO. During the testing phase, we feed the entire image and the spike stream into the network to assess performance.

We complete the training of BSN on the GOPRO dataset, employing an initial learning rate of 3 e -4 and spanning 1000 epochs. The training uses the Adam optimizer with a cosine scheduler and sets the batch size to 8 for each epoch. Adopting the same settings as BSN, EDSR is trained on the blur-spike

I-4

I-3

GT

Blur

I-1

I-2

I-4

I-3

GT

Blur

I-1

I-2

II-4

II-1

II-5

II-2

II-6

II-3

II-7 (retrained II-6)

framework on the RSB dataset. The details corresponding to the experiment ID can be viewed through the Tab. 3. paired data for 70 epochs. Subsequently, LDN undergoes the training of 100 epochs with the learning rate adjusted to 1 e -3 .

401

brown board of Fig. 7, which reflects the drawback inherent in the supervised

401

learning strategies as discussed in Sec. 1. Increasing the weight of the reblur loss 402 A.4.3 Ablation study

402

L reblur allows the LDN to incorporate more information from the blurry input, 403 403 thereby mitigating this issue. However, this adjustment also leads to the pres404 404 ence of blurry edges. We further retrain the LDN on the RSB dataset (referred 405 405 We provide additional ablation visualizations to demonstrate the effectiveness of our designed modules and the distillation framework as shown in Fig. 16 and 17.

406

to as Experiment II-7), following the parameters set in Experiment II-6. The

A.5 Additional Figure

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

预训练网络

Spike-Nerf

实验

retrained LDN effectively recovers the sharp edge of the calibration board and suppresses the spike noise in the background, validating the feasibility of our

self-supervised framework in real-world scenarios.

5

Conclusion

In this paper, we have looked into a novel self-supervised framework S-SDM

for spike-guided motion deblurring, which reconstructs sequences of sharp im- ages from real-world blurry inputs alongside the corresponding spike stream. We

initially propose the physical model SDM for this task and subsequently con- struct a self-supervised processing pipeline to overcome the challenges of spike

noise and spatial resolution mismatch inherent in the SDM. We further design the distillation-based LDN to enhance the restoration performance. In addition,

we construct an RGB-Spike binocular system and propose the first spatially- temporally calibrated real-world dataset RSB in this community. Quantitative

and qualitative experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization of our proposed S-SDM.

RGB

Spike

S

S

D

SDM

Divide

Times

S

Sum

D

time

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See lines 72-77.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See lines 298-299.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: This paper provided the theoretical analysis on the spike-guided motion deblurring task.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiments are reproduceable.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have released the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, they are included in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We don't report it.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, they are included in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, they are correctly used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe SpikeReveal paper uses a blurry RGB image and leverages pulse data captured by a spike camera of the corresponding scene to guide the image deblurring task. In terms of network design, the authors cascade a blind-spot network for denoising, a super-resolution network, and a deblurring network, employing a self-supervised approach to accomplish the deblurring task.\n\nSTRENGTHS:\nThe SpikeReveal paper uses a blurry RGB image and leverages pulse data captured by a spike camera of the corresponding scene to guide the image deblurring task. In terms of network design, the authors cascade a blind-spot network for denoising, a super-resolution network, and a deblurring network, employing a self-supervised approach to accomplish the deblurring task.\n\nWEAKNESSES:\nThe paper has several areas that could be improved to better achieve its stated goals.  Addressing these issues would improve the robustness and validity of the comparative analysis.\n\nQUESTIONS:\n[Definition of t_s]\nThere is an issue with the definition of t_s in line 109. When the first spike occurs, there is no previous spike to reference. Therefore, this definition needs to be revised.\n[Missing Ablation Study]\nTable 2: The module cascading ablation on GOPRO should include with BSN without SR with LDN and without BSN with SR with LDN. This would allow us to determine which component, BSN, SR, or LDN, contributes the most to the final performance.\n[Comparison with SpkDeblurNet]\nIn Table 1 and Figure 4, has SpkDeblurNet been retrained using the same simulation data as the authors' method? Why do the restoration results of SpkDeblurNet in Figure 4 appear noticeably darker? Additionally, why does SpkDeblurNet perform better than the authors' method at Vth=1V_{th}=1?\n\nLIMITATIONS:\nThe authors could consider collecting paired datasets to better address the limitations of simulating pulse camera data. While the authors mention using real data, it serves only as input to the network without providing a genuine ground truth for comparison. Addressing this by capturing paired real-world data could significantly enhance the validity and applicability of their findings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The work focuses on improving image sharpness from blurry inputs using spike cameras with high-motion capture rates. Addressing limitations of supervised learning in real-world scenarios, the authors introduce a self-supervised framework for spike-guided motion deblurring. Validation through extensive experiments on both real-world and synthetic datasets confirms its superior performance and generalization ability.

STRENGTHS:
This paper introduces a pioneering approach to spike-guided motion deblurring, addressing the challenge of recovering sharp images from blurry inputs captured by spike cameras. It demonstrates high-quality research with rigorous theoretical foundations and thorough experimental validations on synthetic and real-world datasets.

WEAKNESSES:
1，I'm curious about how the order of applying BSN (Blur to Sharp Network) and EDSR (Enhanced Deep Super-Resolution) influences model performance. Specifically, I wonder if this sequence could introduce additional artifacts during image restoration.

QUESTIONS:
see the weakness

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a spike-guided self-supervised image deblurring algorithm that combines the high spatial resolution of RGB cameras with the high temporal resolution of spike cameras to obtain sharp RGB images in real-world scenarios. The self-supervised network addresses performance degradation issues found in existing supervised spike deblurring algorithms. The spike-guided module considers noise and spatial resolution alignment between the two cameras. Both subjective and quantitative experimental results show that the proposed algorithm achieves excellent generalization performance in both real and simulated scenes, surpassing previous spike-guided deblurring algorithms.\n\nSTRENGTHS:\n1. The proposed self-supervised spike-guided RGB deblurring algorithm effectively addresses the synthetic-real domain gap performance degradation of previous algorithms.\n2. The authors analyze the relationships between spike data, blurry RGB, and sharp RGB images, providing a theoretical foundation for the Spike-Guided Deblurring Model (SDM).\n3. The paper's clear writing and figures make it easy to follow, with comprehensive experiments demonstrating the RGB-Spike binocular system's spatiotemporal alignment and generalization in real scenarios.\n\nWEAKNESSES:\n1. While many existing image-based deblurring algorithms perform adequately, the introduction of spike cameras presents challenges such as aligning the two modalities. The authors have not sufficiently compared or explained the advantages of the spike-guided approach.\n2. The robustness of the model to noise lacks systematic analysis and discussion. Is this primarily influenced by the BSN Loss described in Equation 9?\n\nQUESTIONS:\nThis method primarily targets optical flow motion estimation. For more complex motion In addition to addressing the performance degradation issues of some supervised models in real-world scenarios, what other advantages does the proposed unsupervised model offer?\n\nLIMITATIONS:\nThe authors have stated the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper combines the RGB camera and the spike camera for image deblur. The key contributions consist a self-supervised learning framework for deblur and a real-world dataset RSB.

STRENGTHS:
This paper presents a novel self-supervised framework for image deblur with spike camera. The network design is interesting. The paper is well-written.

WEAKNESSES:
1 The originality is marginal. The whole framework is similar to [36]. Please clarify more on the difference between this paper and [36].

2. The proposed LDN is also similar to the DCN in [36]. It would be better to see in the ablation whether LDN is better than DCN.

3. It can be observed that smaller V_th leads to better performance. What if V_th == 0.5?

4. The size and diversity of the RSB dataset is limited. It seems the RSB dataset only contains indoor scenes. Outdoor scenes with various objects are desired.

QUESTIONS:
See the weaknesses


------------------------------------
The response addressed my concerns. I raised my score to 6.

LIMITATIONS:
See the weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13