SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams
Authors: Kang Chen
Shiyan Chen
Jiyuan Zhang
Baoyue Zhang
Yajing Zheng
Tiejun Huang
Zhaofei Yu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Motion Deblur
Spike Camera
Self-supervised
Primary area: machine_vision

Abstract: Reconstructing a sequence of sharp images from the blurry input is crucial for enhancing our insights into the captured scene and poses a significant challenge due to the limited temporal features embedded in the image. Spike cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Nonetheless, existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios that diverge from the synthetic training data domain. To address these challenges, we propose the first self-supervised framework for the task of spike-guided motion deblurring. Our approach begins with the formulation of a spike-guided deblurring model that explores the theoretical relationships among spike streams, blurry images, and their corresponding sharp sequences. We subsequently develop a self-supervised cascaded framework to alleviate the issues of spike noise and spatial-resolution mismatching encountered in the deblurring model. With knowledge distillation and re-blurring loss, we further design a lightweight deblur network to generate high-quality sequences with brightness and texture consistency with the original input. Quantitative and qualitative experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization of the proposed framework. Our code, data and trained models are available at \url{https://github.com/chenkang455/S-SDM}.

SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams
Kang Chen
Shiyan Chen 1,23+ 12,3+
Jiyuan Zhang1.
Baoyue Zhang
Yajing Zheng 2
1.2m
Tiejun Huang 12.3 Zhaofei Yu 1:,36
School of Computer Science, Peking University
2 State Key Laboratory for Multimedia Information Processing, Peking University Institute for Artificial Intelligence, Peking University
fur Inrchenkang stteericiiiie chia002, Jyzhang, byzhang) @ett..tttuuuuu pku. edu cn {yj . zheng, t jijhang, yuzf 12)@pku. edu. en
Abstract
Reconstructing a sequence of sharp images from the blurry input is crucial for enhancing our insights into the captured scene and poses significant challenge due the limited temporal features embedded in the blurry image. Spike cameras, sam- pling rates up to 40,000 Hz, have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Nonetheless, existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios that diverge from the synthetic training data domain. To address this challenge, propose the first self-supervised framework for the task of spike-guided motion deblurring. Our approach begins with the formulation of spike-guided deblurring model that explores the theo- retical relationships among spike streams, blurry images, and their corresponding sharp sequences. We subsequently develop self-supervised cascaded framework to alleviate the issues of spike noise and spatial-resolution mismatching encoun- tered in the deblurring model. With knowledge distillation and reblur loss, we further design a lightweight deblur network to restore high-quality sequences with brightness and texture consistency with the original input. Quantitative and qualita- tive experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization the proposed framework. Our code, data and trained models are available at https: //g thub. com/ /henkang455/s SDM.
1 Introduction
Traditional cameras, constrained by their exposure-based imaging mechanism, often produce blurry images when capturing fast-moving objects or during camera movement throughout the exposure process [19, 35]. While these blurry images lose significant details, the ability to recover dynamic motion trajectories from the static blurry input becomes critically important. However, the inherent challenge lies in the limited motion features available within blurry frames, leading potential ambiguities such as multiple motion trajectories corresponding to the same blurry input. This is exemplified by scenarios where two objects move along the same trajectory but in opposite directions [23, 24, 38], rendering the task motion deblurring ill-posed. Recent advancements in learning- based approaches [40, 12, 46] seek to address this challenge by establishing direct mappings from
Equal contributors.
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Methods
Supervised
Self-Supervised (Ours)
Datatype
Synthetic Blur/Spike
w/ GT
w/o GT
Deterioration * colar light texture
Generalization color
Spike-guided Deblur-Net
Train
Train
Dataset Gap
light
Test
Finetune
texture
Real-world Blur/Spike
w/o GT
w/o GT
Figure 1: Illustration of the superiority of our self-supervised framework (S-SDM) over supervised methods. Supervised methods, while effective on synthetic datasets, suffer from significant perfor- mance decline when applied to real-world datasets, primarily due to data distribution discrepancies. In contrast, our self-supervised framework, necessitating no Ground Truth (GT) for training, seamlessly bridges this dataset gap through fine-tuning on real-world datasets.
blurry inputs to sharp sequences supervised learning manner. Despite these efforts, traditional cameras struggle to capture fine details in high-speed motion due to their exposure constraints, thus limiting the effecttivnness of these methods scenarios not covered by the training datasets.
In recent years, neuromorphic cameras [9, 10], leveraging their ultra-high temporal resolution and high dynamic range, have found widespread use in many fields, including computer vision and robotics. These cameras, including event and spike cameras, are distinguished by their ability to produce high temporal resolution outputs directly tied changes in light intensity. Specifically, event cameras generate events in areas where light intensity changes [4], while spike cameras capture absolute brightness of the scene at each pixel, offering stream of spikes as output [10]. This distinctive feature endows spike cameras with significant advantage [47, 45, 48, 5, 41, 6) in capturing and recovering sharp texture from scenes with rapid motion.
Recent studies [2. 8] have explored the potential of RGB-Spike fusion, i.e., harnessing the strengths of both traditional and spike cameras to reconstruct sharp sequences from blurry inputs. However, their frameworks are constrained within the supervised learning paradigm, which necessitates extensive datasets comprising pairs of blurry and sharp images, as well as spike sequences. While synthetically acquiring such paired data, as demonstrated in previous studies [8, 4, 24], is feasible, collecting them real-world scenarios presents the following challenges: (1) high-speee cameras are prohibitively expensive and not readily deployable in many settings; (2) spatial-temporal calibration between spike cameras and high-speed RGB cameras complicates the data collection process. These problems render the fine-tuning of supervised methods on real-world datasets challenging, further leading to their performance deterioration in such environments as shown in Fig. 1. The resulting degradation in image quality, manifesting color distortion, brightness inconsistency, and inaccurate texture restoration, is mainly caused by the disparity between synthetic and woal-world datasets, especially in terms of the density of spike stream, spike generation mechanism, and blurry image generation. Moreover, the effectiveness of supervised methods is inherently limited by the ground truth sequences created through motion analysis interpolation algorithms [22, 11], which inherently differs from the real-world scene and thus affects the model's generalization ability.
To overcome these issues, propose the trst-of-its-kind Self-supervised Spike-guided Deblurring Model (S-SDM), capable recovering the continuous sharp sequence from single blurry input with the assistance of low-resolutio spike streams. We begin with theoretical analysis of the

---

relationship between spike streams, blurry images, and sharp sequences, leading to the development of our Spike-guided Deblurring Model (SDM). We further construct a self-supervised processing pipeline by cascading the denoising network and the super-resolution network to reduce the sensitivity of the SDM spike noise and its reliance on spatial-resolution matching between the two modalities. To reduce the computational cost and enhance the utilization of spatial-temporal spike information within this pipeline, we further design a Lightweight Deblurring Network (LDN) and train it based on pseudo-labels from the teacher model, i.e., the established self-supervised processing pipeline. Further introducing reblur loss during LDN training, we achieve better restoration performance and faster processing speed than the processing-lengteee and structure-complicatee teacher model. To validate the performance of our S-DDM across various scenarios, we build an KGB-Spike binocular system and propose the first temporolly calibrated Real-world Spike Blur (RSB) dataset in this community. Quantitative and qualitative experiments conducted on the worldworld and synthetic datasets validate the superiority of our method. In summary, our key contributions are:
We develop a self-supervised balkee-uiied image deblurring framework, addressing the performance degradation due to the synthetic-real domain gap in supervised methods. We perform an in-depth theoretical analysis of the fusion between the spike stream and blurry image, leading to the development of the SDM.
We propose a real-world dataset RSB and experiments on GOPRO and RSB datasets validate the superior generalization of our S-SDM.
2
Related Work
Spike Camera. The spike camera, inspired by the primate retina, stands apart from conventional cameras with its ability to generate synchronous spike streams for each pixel at extremely low latency. This distinct feature provides significant advantages in various applications such high- speed imaging [47, 45, 48, 5, 41, 6, 33]. optical flow estimation [43], object detection [44], 3D reconstruction [30]. depth estimation [34], motion deblurring [8, 32], and occlusion removal [31].
Spike-guided Motion Deblurring. While the spike camera boasts an ultra-high temporal resolution, its development is currently impeded by the low spatial resolution. Additionally, the single-channel output from the spike camera restricts previous methods from recovering the image color information. To address these issues, a promising approach is establishing an RGB- (Gike hybrid imaging system [2]. The binocular system achieves the multi-modalin fusion of High-spatial/I Low-temporal RGB blurry input and Higghttmmpral//Loww-ss soa-spaial spike stream, thereby also serving as a spike-guides motion deblurring method [8]. However, to the best of our knowledge, existing spike-guided deblurring methods [2. 8] predominantly rely on supervised training on synthetic datasets. This reliance results in significant performance degradation when these methods are evaluated in real-world scenarios due to the domain discrepancies between synthetic and real datasets illustrated in Fig. 1.
Event-based Motion Deblurring, Event camera [21] can asynchronously generate events that record log- reginntenstty changes at the pixel level with minimal latency, which contains rich set of motion features beneficial for motion deblurring tasks. Numerous supervised methods [16, 4, 3, 16, 23, 24] have been proposed learn the mapping from the blurry input, events the sharp outcome. Despite these advancements, a major hurdle remains in obtaining real blurry-sharp image pairs for training. To overcome the domain gap between synthetic and real-world datasets, recent methods [29, 38, 39] explored the mutual constraint between the blurry image and event stream, enabling the training of networks on wellworld blur datasets.
3 Method
3.1 Preliminaries
Spike Camera Mechanism. Consider L(l) to represent the latent sharp frame at time I. Each pixel p in the spike camera [10] has integrator that accumulates the incoming photons at a high frequency. Once the cumulative intensity exceeds a predefined threshold C at time les pixel p emits spike, and the accumulation of photons is reset to zero. This process can be mathematically described as
3

---

follows:
L" L(tddt 2 C,
where is denotes the firing time of the previous spike. While the spike camera is capable of generating asynchronous spike streams akin to that of event cameras, its effectiveness is constrained by the inherent limitations of its physical circuitry, which necessitates reading spikes at a predetermined sampling rate. We denote the generated spike stream as S 10, xxxx*x where H and W signify the height and width of the image, and K' represents the length of the spike sequence.
Problem Formulation. In traditional photography, motion blur occurs when there is relative movement between the camera and the scene during the exposure period. According to the motion blur physical model [9]. the blurry image B can be represented as the average of the latent frame L(t) over the exposure T, i.e.:
/ L(t)dt. T JEET
1
B
(2)
where 7 represents the exposure period. Despite the spike camera's superior temporal resolution, its spatial resolution remains comparatively low. This limitation is primarily attributed to the constraints in data transmission bandwidth and the challenges inherent in the manufacturing process. Here, we postulate that the spatial resolution of the spike camera is approximately one-quarter that of conventional RGB camera.
In this paper, we aim to enhance the fiigh-spatil/Loowteemooe resolution blurry input B E I. x3x NWW into a sequence of Sigh-Quaiity images [[(t,)]== R with the aid- resolution spike stream Sy E {0, 10,1)xxxx4*4 which can be
ing of High-temporal/L.ow mathematically formulated as:
(L(4) Deblur((;; B,
In Eq. (3), Deblur(-) represents the Spike-guided Deblur- shown in Fig. 1, i refers to the i-th frame in the spike stream Sy, and t; is the timestamp associated with this frame.
3.2 Theoretical Analysis
The spike camera, with its photodetector tailored to capture the single-channel light intensity, faces difficulties in obtaining the color information that the multi-channel RGB camera can effortlessly capture. Therefore, we modify the color intensity L()) in Eq. (1) to the grayscale value (l) for further analysis:
1.a L (t) C. C.
(4)
In this formulation, (t) = wi (t) + Wore (Ir + wn, [t)(), where Lt), we denote the intensity and weight of channel E (r. gre, b) respectively.
Given the blurry input B and its corresponding spike stream S7, we incorporate Eq. (4) into the motion blur model presented in Eq. (2). This integration formulates a link between the two modalities as outlined below:
BN NT T
where By denotes the grayscale version of the blurry input, and Ny denotes the total number of spikes accumulated over the exposure period. The accumulation Ny is calculated as Ny = Σ S(i), with Sli] indicating the i-th frame of the spike stream.
Within the exposure T, we consider a shorter spike sequence centered around the moment Sy E (0. 1) xxxx))xxxxxxxx x satisfying K' & K, E T. and T' T. Similar to Eq. (5), we can derive the relationship between the short-exposure gray image (1,(TT)) and the short spike stream S7, as follows:
(s)ds C.N T"
Et,
(6)
T" SET'
where T" <T represeenss the short exposure period.

---

Student
Exposure
B R RXXxXXX
Lebibur
B
M
Luto (tw)
Encoder
RGB
Decoder
Spike
Encoder
Spike
LDN
Unan (tw)
Spike
C tea
B IR 1x3xXxx
SDM
[ (tw))
Time [4m]
Blurry Input
Reconstruction
BSN
EDSR
Spike Sequence
Spiike Frames LL (t,,))
Teacher
{
Figure 2:
The schematic diagram of our proposed distillation self-supervised framework. The "Ie"" indicates that certain computations executed in non non-network manner.
Given the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the short-exposure image E(,, T') are identical. This assumption implies that the intensity proportion among RGB channels in the blurry input a and the short-exposure image ((, T') is approximately cquivalent, satisfying a BB and a T') = (t, (t,, T'). More details can be found in the supplementary materials.
Upon establishing it, we move forward build a mathematical relation between the blurry image and the spike stream. By substituting the gray channel with color channel and dividing Eq. (5) by Eq. (6), we efficiently eliminate the unknown threshold C' and weights a /a F T"), leading the following equation:
Et, Ny. T NT T
(7)
By applying Eq. (7) to RGB three channels, explicitly establish the relationship between the color blurry input B, the color short-exposure image E(t, T') and the spike stream ST as shown in Fig. 18. Since the exposure period T' is relatively short, it is reasonable assume that the scene remains static. In this context, we interpret the short-exposure image E(t, T') as the latent sharp frame L(f), allowing to modify Eq. (7) follows:
L(I) = N T NT T"
8
To this end, we have conducted a comprehensive theoretical analysis of the spike- motion deblurring task, which neglected prior learning-based motion deblur methodologies [2, 8). For further discussion readability, refer to Eq. (8) as the Spike-guided Deblurring Model (SDM), which is analogous to the baseline motion deblur model EDI [18] in event camera.
3.3
Self-supervised Spike-guided Deblurring Model
3.3.1
Processing Pipeline
While SDM theoretically allows for the fusion of the blurry image and the spike stream, its practical deployment faces the following obstacles:
The deblurred image E(t, T') suffers from noise-related degradation due to the lack of adequate spike information during the short exposure T'.
5

---

The spatial resolution of the spike camera is approximately one-quarter of the RGB camera, rendering the SDM implementation impractical.
To overcome these limitations, we further cascade the self self- supervise denoising network to eliminate the spike noise in Ny. and super-resolution network to match spatial resolutions of the blurry image and spike stream, with the processing pipeline illustrated in the bottom of Fig. 2.
Denoising Network. We leverage the Blind Spot Network (BSN) [13, 1, 27, 14, 7] predict the clean spike accumulation N from the input short-exposure spike stream ST The core idea of BSN design the blind-spot strategy that compels the convolutional layer estimate the clean value of each pixel solely based on its surrounding pixels.
Under the premise that the spike stochastic thermal noise is independent identically distributed [42], the BSN is trained to deduce sharp spike frames from the input, with the loss function formulated as:
CASs [BSN(S+;;() O1) NT )12:
where the denoised spike frame SSN ()) is denoted by Loan for further analysis.
Super-Resolution Network. In this task, we observe that the blurry input B, and the long-ex ong-exxosure spike frame Nf exhibit the same texture features as shown Eq. (5). This observation motivates us to train the Super- Respluteeoottion (SR) network based on pairs of the blurry images and the long-exposure spike frames.
We leverage the well-explored Enhanced Deep Super-Resolution network (EDSR) [15] as the back- bone of our SR network, with the loss function formulated as follows:
LEDSR = |EDSR(NT;; θ2) - -Byl
(10)
With the training of the SR network completed, we freeze its parameters and apply the denoised spike frame Losa yielding the resolution-enhaneed spike frame b.a
3.3.2 Knowledge Distillation Framework
While the aforementioned processing pipeline achieves the multi-modality fusion of the blurry input and the spike stream, several aspects still need refinement:
The framework is lengthy and computationally demanding, which hinders its suitability for real-time system deployment.
The blind-spot strategy of the BSN limits the full utilization of the spatial information inherent the spike stream.
The representation of the short-exposure image does not fully reflect the advantages of the high temporal resolution inherent in the spike stream.
To improve them, we further build knowledge distillation framework building upon the existing processing pipeline. This pipeline serves as the teacher model, providing the reconstructed sequence pseudo-labels for the training of the student model LDN, illustrated in Fig. 2.
Lightweight Deblur Network. LDN adheres a similar input and output pattern as previous research [8], iee., taking the blurry input B and the short spike stream S7 centered around moment t as inputs, with the output being the reconstructed sharp image [tan mathematically formulated as follows:
Lian (t) LDN(B, ST;(,a)
(11)
Full details regarding the LDN structure are available in the supplementary materials.
To avoid the scenario where the LDN exactly replicates the mapping the teacher model, we design the teacher loss Liea based on the LPIPS [36] loss and further introduce the blur reconstruction loss. The reblur loss Lreblur measures the difference between the blurry input and the re-synthesized blurry image B, satisfying:
M Lian (tm),
Σ
(12)
M
m=1
where Lián (4m) represents the m-th recovered image within the exposure period and M the total number of reconstructed images. Finally, we sum up two loss functions with the weighting parameter

---

Blar
Moticn ETR
Blur
LEVS
Modion ETR
Blurry Input
BIT
SpiDeelerNet
Ours
Blurry Input
BIT
SpkDeblarNet
Ours
LEVS
LEVS
Motion-EIX
V+
Blurry Input
HET
SpkDeblurNe
Oure
Blurry Input
BET
SpkDeblurd
Ours
Figure 3: Qualitative comparison for the single frame restoration on the RSB dataset.
LEVS
Me0 EIT
(a) Blur
BTT
SpXDoHurNca
(b) SpkDeblurNet
Ours
(c) Ours
0
T/2
Figure 4: Qualitative comparison for the sequence reconstruction on the RSB dataset.
X, and the final loss function is formulated as follows:
C Liaa + Lysblur
(13)
M
Lires (tm), Lidn (tmn)t))))= + A EMSE (B, B).
(14)
4 Experiments
4.1 Dataset
Synthetic Data. For quantitative analysis of our spike-guided motion deblurring task, we construct the synthetic dataset based on the widely employed GOPRO [17] dataset. We initially utilize interpolation algorithm XVFI [22] to augment the video frame by interpolating additional 7 frames between each pair of consecutive sharp images. generate the spike stream that mimics reality closely, downsample the interpolated video to the resolution of 320 180 and simulate the spike

---

Table 1: Quantitative comparison of the sequence reconstruction task on the GOPRO dataset.
V4+==
Vix=2
Vna =4
Methods
Spike
Params
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
LEVS[12]
X
21.155
0.601
21.155
0.601
21.155
0.601
4.97M
Motion-ETR[40]
X
21.955
0.610
21.955
0.610
21.955
0.610
6,55M
BiT[46]
23.644
0.698
23.644
0.698
23.644
0.698
11.3M
TRMD[4)+DASR1201
V
27.323
0.784
21.198
0.601
18.567
0.523
19.3M
REDI29]+DASSR26]
24.456
0.741
23.178
0.674
21.942
0.608
9.76M
REFIDD]25/4DDSSR2261
28.124
0.819
15.288
0.339
13.623
0.274
15.9M
SpkDeblurNer(81
28.307
0.834
14,406
0.299
11.621
0.202
13.4M
S-SDM (Ours)
26.893
0.757
26.367
0.740
25,433
0.699
0.23M
Blurry Image
Motion-ETR
BiT
SpkDeblurNNet =
SpkDeblurNet Vu=
SpkDeblurNet V
Ours VA=
Ours Va= 2
Ours Vy=
Figure 5: Visual comparison of our S-SDM against other methods on the GOPRO dataset.
stream based on the spike simulator [42]. To replicate real-world motion blur, synthesize each blurry input by averaging 97 frames from interpolated video sequences.
Real-world Data. We construct RGB-Spike binocular system and propose the first world Spike-guided Blur dataset (RSB) in this community. This system consists of a fixed-exposure RGB camera (Basler acA acA1120015000
and spike camera [10], enabling us to capture the blurry image and the corresponding spike stream simultaneously. Further details about our RSB and the temporlll calibration for binocular system are provided in the supplementary materials.
4.2 Experimental Results
We conduct both quantitative and qualitative comparisons of our S-SDM against state-of-the-art (SOTA) motion deblurring methods, including frame-basee Motion-ETR [40], LEVS [12], video- based BiT [46], event-based TRMD [4], REFID [25], RED [29] and spike- based SpkDeblurNet [8] on the GOPRO and RSB datasets. For event- event-based methods, we replace the event stream with the spike stream and adopt the same input representation in these methods [29, 25, 4). further cascade the image super-resolution technique DASR [26] as in [39] for the deblurred sequence to overcome the modality resolution inconsistency which not considered these methods. We reconstruct images from one blurry input for sequence restoration evaluation [39] listed in Tab. 1.

---

Table 2: Performance comparison between the SAN in GEM [39] and our designed LDN.
Methods
PSNR
SSIM
Params (M) 4
Flops (G) +
SAN [39]
27.283
0.773
2.36 0.234
107.84
LDN (Ours)
27.928
0.786
33.60
Results on GOPRO. To simulate the spike density domain gap depicted in Fig. 1, we train all supervised spike-based deblurring methods on the GOPRO dataset under spike threshold [42] ViA = and evaluate them on datasets with spike thresholds V. = 1, 1, 4. Quantitative comparison results are listed in Tab. and the visual comparison is demonstrated in Fig. 5. Given that the principal contribution of our S-SDM is self-supervised learning, it is foreseeable that our method might be slightly inferior to the supervised methods SpkdDeblurNet on the dataset with Ve2 = 1. While these supervised methods deteriorate on datasets with VAA * 1, method achieves great generalization benefiting from the self-supervision design. Specifically, SpkDeblurNet tends to produce darker and blurrier reconstructions on images with high thresholds as shown in Fig. 5. Besides, achieves better restoration performance than the self-super method RED due to our consideration of the pratial-resolution mismatch between two modalities and the designed teacher loss, which imposes a stronger constraint than the optical loss in RED.
method
Results on RSB. We further present visualizations of single frame and sequence reconstruction comparisons on the real-world RSB dataset as depicted in Fig. and Both frame-based and video- based approaches fail to replicate fine textures and detailed elements present in the blurry input. While SpkDeblurNet is capable of recovering structural details and the motion trajectory, it is deteriorated by significant noise and color distortion under conditions with lower spike density than those simulated in the GOPRO dataset. Similarly, in scenarios of higher spike stream densities, the restoration of SpkDeblurNet tends to exhibit ever-exposure, resulting in brightness inconsistency compared to the blurry imput. This over-exposune affects the dynamic range of the reconstructed image, ultimately compromising the overall perceptual quality and uniformity of the restored sequence. Our method addresses these challenges by finetuning on the RSB dataset, ensuring that the restored sequence aligns with the real-captured blurry input and spike stream. More comparative experiments and analyses are accessible in the supplementary material.
4.3 Ablation Study
We perform ablation experiments on the GOPRO and RSB datasets to evaluate the performance of each module within S-SDM, the validity of our designed network architecture, as well the overall effectiveness of our distillation learning framework. In this subsection, we evaluate the performance based on the single middle frame for simplicity.
Modules Cascading. Building upon the SDM, we sequentially cascade the BSN, the SR, and the LDN to evaluate their respective effecttiveness, with quantitative results GOPRO presented in Tab. 3. We employ bilinear interpolation as a substitute for the SR network in experiments I-1 and I-2 to align the spatial resolution of two modalities.
Qualitative ablation experiments are illustrated in Fig. 6. These comparisons reveal that while the SDM effectively removes motion blur, it struggles with significant noise and detail loss due to the spike noise and the low resolution of the spike stream. While the BSN mitigates noise and the SR network improves spatial resolution explicitly, the LDN trained via distillation learning further refines these enhancements, enabling the recognition of intricate textural features in the images, such as the license plate and the door number shown in Fig. 6.
Network Architecture. While our designed LDN mirrors the Scale-aware Network (SAN) proposed in the GEM [39], we replace the SAN with the LDN to compare the performance difference between the two architectures as depicted in Tab. 2. Despite the simple design of our LRN, which consists only of convolutional layers, ResBlocks, and basic modules such as CBAM [28], it outperforms SAN in both PSNR and SSIM while requiring fewer parameters and less computation, demonstrating that LDN is both sufficient and efficient for this task. This conclusion is consistent with our previous discussion, i.e, the restoration performance of the self-supervised learning framework primarily depends the quality of pseudo-labels rather than the network architecture.

---

Blur
240 0277
240 0212
Blurry Input
I-3
1-4
GT
Blurry Input
I1-5
11-6
Blur
1-1
1-2
Jl-2
11:3
JI-4
ABA
ABA
Blurry Input
Input
I1-5
11-6
Figure 6: Modules cascading comparisons GO-
Figure 7: Distillation learning comparisons on
PRO. Experiments ID can be viewed on Tab. 3.
RSB. Experiments ID can be viewed on Tab.
Table Distillation learning ablation on GOPRO.
Table 3: Modules cascading ablation on GOPRO.
ID
Lyea
Creblur
À
PSNR
SSIM
ID
BSN
SR
LDN
PSNR
SSIM
I[-1
X
10
23.102
0.441
H
23.012
0.486
II-2
/
26.563
0.723
1-2
V
24.634
0.661
11-3
V
V
10
27.345
0.762
I-3
V
V
26.144
0.708
II-4
50
27.742
0.778
I-4
V
27.928
0.786
II-5
100
27.928
0.786
I1-6
200
27.620
0.783
Distillation Learning. focus on analyzing the contribution the teacher loss Lin and the reblur loss Lieblur within the distillation framework, with quantitative results listed in Tab. 4. Without the teacher loss Lea the LDN tends toward learning identity mapping from blurry input. While under the guidance of the teacher model, the reblur loss Lieblur not only enforces motion consistency in the reconstructed sequence but also enriches the LDN with high-resolution details from the non-blurry regions of the input, thus improving the performance GOPRO listed in Tab. 4.
We further apply the LDN trained on GOPRO to the real-world dataset RSB, with qualitative visualization illustrated in Fig. 7. As observed in the figure, the absence of the reblur loss Leeblur leads to significant noise the recovered image, which predominantly arises from the disparity in spike density and generation mechanism between the simulated and real-captured spike stream. This discrepancy causes the LDN to overestimate the spike number, resulting black holes in regions with lower spike density than simulated, which reflects the drawback inherent in the supervised learning strategies as discussed in Sec. Increasing the weight of the reblur loss Lieblur allows the LDN to incorporate more information from the blurry input, thereby mitigating this issue. However, this adjustment also leads to the presence of blurry edges. We follow the parameters set in Experiment 11-6 and retrain the LDN the RSB dataset (referred effectively recovers the sharp edge of the calibration board and suppresses the spike noise in the background, validating the feasibility of our self-supervised framework in real-world scenarios.
Experiment I1-7). The retrained LDN
5 Conclusion
In this paper, we introduce a novel self-supervised spike-guided motion deblurring framework S-SDM, which reconstruets sequences of sharp images from real-world blurry inputs with the spike stream. Additionally, we construct an RGB-St Spike binocular system and propose the first spatially-tempore calibrated real-world dataset RSB in this community. Quantitative and qualitative experiments validate the superior generalization capabilities of our proposed S-SDM.
Linitation. The limitation of our S-SDM lies in its dependence strict spatial-temporal calibration. Misalignment will lead to color shifts and quality degradation in the deblurred sequence.
10

---

Acknowledgments
We sincerely appreciate Yuyan Chen (HUST) for her valuable suggestions and for polishing the figures. This work was supported by the National Natural Science Foundation of China (62176003, 62088 62306015), the China Postdoctoral Science Foundation (2023T160015), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC01), and the Beijing Nova Program (20230483362).
References
[1] Jaeseok Byun, Sungmin Cha, and Taesup Moon. Fbi-denoiser: Fast blind image denoiser for poisson-
gaussian noise. In CVPR, pages 5768 5768-5777, 5777, 2021.
[2] Yakun Chang. Chu Zhou, Yuchen Hong. Liwen Hu, Chao Xu, Tiejun Huang, and Boxin Shi. 1000 fps hdr video with spike-rgb hybrid camera. In CVPR, pages 22180 22190, 2023.
[3] Haoyu Chen, Minggui Teng, Boxin Shi, Yizhou Wang, and Tiejun Huang, A residual learning approach to deblur and generate high frame rate video with event camera. IEFE TMM, 2022.
[4] Kang Chen and Lei Yu. Motion deblur by learning residual from events. IEEE TMM, 2024.
[5] Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. IJCAI, 2022.
[6] Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In AAAI, volume 37, pages 350-358, 2023.
[7] Shiyan Chen, Jiyuan Zhang, Zhaofei Yu, and Tiejun Huang. Exploring asymmetric tunable blind-spots for celf-supervised denoising in real- world scenarios. arXiv preprint arXiv:2303. 16783, 2023.
[8] Shiyan Chen, Jiyuan Zhang, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Enhancing motion deblurring in nigh-speed scenes with spike streams. In NeurIPS, 2023.
[9] Guillermo Gallego, Tobi Delbrüick, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey.
IEEE TPAMI, 44(1):154 18044-8802200 2020.
[10] Tiejun Huang. Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, et al. 1000x faster camera and machine vision with ordinary devices. Engineering,
25:1 2023.
[11] Zhewei Huang. Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, pages 624-642. Springer, 2022.
[12] Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning to extract a video sequence from a single motion-blurred image. In CVPR, pages 1334-6342,22 2018.
[13] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self--upervised deep image denoising. NeurIPS, 32, 2019.
[14] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap- bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spol spot network. In CVPR, pages 17725 11734, 2022.
[15] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPR, pages 136-144, 2017.
[16] Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren. Learning event-driven video deblurring and interpolation. In ECCV, pages 695-710. Springer, 2020.
[17] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene dehlurring. In CVPR, July 2017.
[18] Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive high frame-rate with an event camera. In CVPR, pages 6820-6829, 2019.
[19] Bryan Peterson. Understanding exposure: how to shoot great photographs with any camera. AmPhoto books, 2016.
[20] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation In Medical Image Computing and Computer-Assister Intervention MICCAI 2015:
18th International Conference, Munich, Germany, October 5-9 2015, Proceedings. Part III 18, pages 234-241. Springer, 2015.
[21] Teresa
and Bernahé Linares-Barranco. 128 128 1.5% contrast sensitivity 0.9%
fpn 3 jss latency mw asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers. IEFE Journal of Solid-State Circuits, 48(3):827-838, 2013.
[22] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In ICCV, pages
14489-14499 14498, 2021.
[23] Chen Song. Qixing Huang, and Chandrajit Bajaj. E-cir: Event-enhanced continuous intensity recovery. In CVPR, pages 7803-7812, 2022.
[24] Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang. Peng Sun, Yaozu Ye, Kaiwei Wang. and Luc Van Gool, Event-based based fusion for motion deblurring with cross mooss attention. In ECCV, pages 412-428. Springer, 2022.
11

---

[25] Lei Sun, Christos Sakaridis, Jingyun Liang. Peng Sun. Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang. and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. In CVPR, pages 18043-18052.
2023.
[26] Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang. Wei An, and Yulan Guo.
Unsupervised degradation representation learning for blind super-resolution. In CVPR, pages 10581-10590,
2021.
[27] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image denoising with visible blind spots. In CVPR, pages 2027-2036, 2022.
[28]
Sanghyun Woo, Jongchan Park. Joon- Yoon-Yonn Lee, and In So Kweon. Cbam: Convolutional block attention
module. In ECCV, pages 3-19, 2018.
[29] Fang Xu. Lei Yu, Bishan Wang. Wen Yang. Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion deblurring with real events. In ICCV, pages 2583-2592, 2021.
[30] Jiyuan Zhang, Kang Chen, Shiyan Chen, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Spikegs: 3d gaussian splatting from spike streams with high-speed camera motion. arXiv preprint urXiv:2407.10022
2024.
[31] Jiyuan Zhang, Shiyan Chen, Yajing Zheng, Zhaofei Yu, and Tiejun Huang. Unveiling the potential of spike streams for foreground occlusion removal from densely continuous views. arXiv preprint arXiv:2307.00821, 2023.
[32] Jiyuan Zhang. Shiyan Chen, Yajing Zheng, Zhaofei Yu, and Tiejun Huang. Spike-guided motion deblurring
with unknown modal spatiotemporal alignment. In CVPR, pages 25047-25057, 2024.
[33] Jiyuan Zhang, Shanshan Jia, Zhaofei Yu, and Tiejun Huang. Learning cemporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI, volume 37, pages 137-147, 2023.
[34] Jiyuan Zhang. Lulu Tang, Zhaofei Yu, Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth
estimation for spiking camera. In ECCV, pages 34-52. Springer. 2022.
[35] Kaihao Zhang, Wenqi Ren, Wenhan Luo. Wei-Sheng Lai. Björn Stenger.
Yang, and Hongdong
Li. Deep image deblurring: A survey. IJC,, 130(9):2103-2130) 2022.
[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechuman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, pages 595, 2018.
[37] Weixia Zhang, Guangtao Zhai, Ying Wei. Xiaokang Yang. and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In CVPR, pages 14071-14081, 2023.
[38] Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. In CVPR, pages 17765-17774, 2022.
[39] Xiang Zhang, Lei Yu, Wen Yang. Jianzhuang Liu, and Gui-Song Xia. Generalizing event-based motion deblurring in real-world scenarios. In ICCV, pages 10734-10744, 2023.
[40] Youjian Zhang, Chaoyue Wang, Stephen Maybank, and Dacheng Tao. Exposure trajectory recovery from motion blur. PAMI, 44(11):7490-
2021.
[41] Jing Zhao, Ruigin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR, pages 12995-12005, 2021.
[42] Junwei Zhao, Shiliang Zhang. Lei Ma, Zhaofei Yu, and Tiejun Huang. Spikingsim: A hio-inspired spiking simulator. In ISCAS, pages 3003- 3007. IEEE, 2022.
[43] Rui Zhao, Ruiqin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. NewrlPS, (1779557722) 2022.
[44] Vajing Zheng, Zhaofei Yu, Song Wang. and Tiejun Huang. Spike-hased motion estimation for object
tracking through bio iio-inspired unsupervised learning. IEEE TIP, 32:335-3444 349, 2022.
[45] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. CVPR, pages 6358-6367, 2021.
[46] Zhihang Zhong, Mingdeng Cao, Xiang Ji, Yingiang Zheng, and Imari Sato. Blur interpolation transformer for real-worid motion from blur. In CVPR, pages 5713-5723, 2023.
[47] Lin Zhu, Siwei Dong. Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In ICME, pages 1432-1437. IEEF, 2019.
[48] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruc- tion via spiking neural model. In CVPR, pages 1438-1446, 2020.
12

---

Blu
Spike
SpkDeblurNet
Ours
Blu
BiT
SpkDeblurNet
Ours
Figure 8: Video comparison our method against the BiT and SpkDeblurNet on the RSB dataset under different luminance conditions, with the static visualization displayed in Fig. 13. It is recom- mended to view the pdf using Acrobat PDF reader and the gif demonstration with a higher frame rate is available in the supplementary zip file.
A
Appendix
This supplementary material provides a comprehensive elaboration on the methodologies and experi- ments in this paper. It organized into four distinct sections: Theory Analysis in appendix A.1, Net- work Settings in appendix A.2, RSB Dataset in appendix A.3, Experimental Details in appendix A.4 and Additional Figure in appendix A.5.
A.1 Theory Analysis
We define kB and kB as the ratios of the red channel the green and blue channels in the blurry input B respectively, ie.,
A B,/BBree
(15)
B,/B,
(16)
Since the gray image is the weighted sum of RGB channels, the ratio of the gray to the red image o formulated as:
0r Bg/B. B, = (w, B, + gre B + wb BB)/B,
(17)
(18)
w, Wore Bk wb/kB
(19)
where we denote the weight of channel E fr, gre, b} respectively. Similarly, we define (L, and 1:, T') as the fractions of the red channel relative to the green and blue channels in the short-exposure image E(t, T'), resulting in:
(t,
+ Wgre F T') + wh hh (t,
(20)
Given the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the short- exposure image E(t, T') are identical. This assumption implics that the intensity proportion among RGB channels in two images is approximately equivalent, i.e., kB xe & (t, T') and k. k (t, T'). Drawing from Eq. (19) and Eq. (20), We further deduce the following relation:
ar ~ E (t,T'),
(21)
which can be readily generalized across channels, leading to a.d ~ B (t,
13

---

Rotate
Concat
Cony
Transpose
BSN
(a)
CBAM
Predict
1 convolution, as 3x3 Convi
stride
LDN
Conv
(b)
stride
Figure 9: Network diagrams of the BSN (a) and our designed LDN (b).
A.2 Network Settings
A.2.1 Blind Spot Network
We construct our BSN based on the blind-spot strategy outlined in [13]. We first rotate the input image four times, then concatenate them into the U-Net [20] structure like denoising network. To prevent direct mapping from the input pixel to the output pixel, a single-pixel offset strategy is employed in the convolutional kernel to separate its receptive field from the central pixel. Finally, the denoising output is obtained by merging the results four branches via shown in Fig. (a))
A.2.2 Lightweight Deblur Network
The network structure is depicted in Fig. 9(b), where the encoder for blurry images consists two layers of down- down-sampling convolutions to align the spatial resolution of two modalities. In contrast to the intricate cross-attention mechanism outlined in [4, 8), we implement the fusion of two modalities by the simple operation of Concat(-). )), with the cascaded CBAM [28] and residual blocks for further feature fusion.
A.3
RSB Dataset
We detail the construction of our RGB-Spike hybrid system illustrated in Fig. 10. The system comprises a Spike Camera-001T-GGe with a resolution of 400 x 250 pixels, paired with a Basler McA1190-150ur RGB Camera, offering higher resolution of 1920 1200 pixels.
Calibration Board
(a) RGB Camera
(b)) RGB (Calibrated)
Spike Camera
400 x250
(c) Spike Camera
Figure 11: Calibration result.
Figure 10: RGB-Spike camera system.
14

---

Table 5: Quantitative comparison of the single frame task on the RSB dataset, where non-reference metric LIQE ranging from to 5 is employed. LIQE a positive metric denoted as f where higher
scores reflect better performance.
Methods
Board-L
Board-M M
Board-HF H
Face
Earphone
Average
LEVS [12]
1.0010
1.0342
1.0400
1.0029
1.1382
1.0433
Motion-ETR [40]
1.0153
1.0064
1.0279
1.0023
1.1724
1.0449
BiT [46]
1.0002
1.0069
1.0473
1.0138
1.6155
1.1367
SpkDeblurNet [8]
1.8493
1.3232
2.2698
1,2210
2.6606
1.8688
S-SDM (Ours)
2.0955
1.3872
2.4208
1.3814
2.7087
2.1987
Board-H foard-W Board-L
Face
Earphone
Blur
LEVS
Motion-ETR
BiT
SpkDeblurNet
Ours
Qualitative comparisons for single-frame restoration the RSB dataset illustrated, where "Board-L.", "Board-M"', and "Board-H" represent the board captured under low, middle and high lighting conditions.
Figure 12:
To achieve the spatial calibration of two cameras, we perform simultaneous captures of the calibration board using the RGB-Spike system as shown in Fig. 10. The RGB images are cropped to a resolution of 1600 x 1000, aligning them cropped RGB image to grayscale and take the TFP [47] image reconstruction result as the reference from the spike camera. We utilize the MATLAB calibration toolbox implement the calibration process, with the results detailed in Fig. 11.
fourfold the resolution of the spike camera. We convert the
Our RSB dataset contains 10 video sequences under different conditions, captured under varied conditions including scene brightness levels (e.g., Low, Middle, and High light) and motion patterns (e.g., camera shake and object motion), which introduce different types of motion blur. Besides, the RSB dataset comprises a large amount of blur-spike pairs with each blurry input corresponding to 400 spike frames.
A.4 Experimental Details
A.4.1 Comparison
To assess the performance of our method on the GOPRO dataset, we utilize the Peak Signal-to- Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as the quantitative metrics, which commonly used in motion deblurring tasks. In real-world datasets, where obtaining ground truth sharp sequence is challenging, we opt for the non-reference image quality assessment method Language- Image Quality Evaluator (LIQE) [37] reference. By assessing visual quality through
15

---

(a)
(b)
Blur
SpkDeblurNet [8]
(d)
Ours
Ours
(a)
Blur
SpkDeblurNet [8]
(d)
Ours
Ours
Figure 13: Qualitative comparison for the sequence reconstruction the RSB dataset. (b). denote results of "LEVS", "Moiton-ETR" "BiT" and "SpkDeblurNet" respectively. The upper panel depicts real-world scene with lower spike density than the simulation, whereas the lower image exhibits a higher spike density.
the computation of joint probabilities from visual-textus embeddings. LIQE adeptly identifies the clarity and blurriness of images independently of the ground truth, making ideally suited for our task.
Comparison on the RSB dataset. Qualitative and quantitative experiments of the single frame restoration task on the RSB dataset are shown in Fig. 12 and Tab. 5 respectively. The visual results coupled with the LIQE metrics demonstrate that our method outperforms other methods in handling the real-world RSB dataset. While the supervised SpkDeblurNet encounters substantial noise and overexposure challenges in both low-light and high-light environments, our approach demonstrates superior restoration performance, which is attributed to the designed self-supervised framework.
16

---

APA
VISHT
Blur
Motion-ETR
SpkDeblurNet VIA
SpkDeblurNet VAA 2
SpkDeblurNet 21
V==
Vu== 2
Vn=
Figure 14: Comparison of our S-SDM against other methods on GOPRO with Vaa = 2,
Blur
Motion-ETR
BIT
SpkDeblurNet V
SpkDeblurNet Vaa
SpkDeblurNet V
Ours Vu=
Ours Vn 2
Ours Vn
Figure 15: Comparison of our S-SDM against other methods on GOPRO with V.A 1, 1,2,,4 4.
Moreover, as depicted in Fig. 13 and Fig. 8, S-SDM exhibits outstanding performance in precisely retrieving luminance information and texture details while ensuring the motion consistency of reconstructed sequences. contrast, the video reconstruction of the BiT suffers from poor image quality and shows inadequate sequence continuity in the restored sequence. Besides, SpkDeblurNet encounters issues with color distortion, brightness inconsistency, and inaccurate texture restoration owing to the domain gap between synthetic and real-world datasets. These observations further highlight the superior performance of our S-SDM in real-world scenarios.
Comparison the GOPRO dataset. We provide additional visual comparison of our method against other SOTA methods on the GOPRO dataset as shown in Fig. 14 and 15.
A.4.2 Implementation
To augment the dataset and accelerate the training process, we randomly crop 512 x 512 image from each blurry frame, along with the 128 x 128 spike stream. We use PyTorch to build and train S-SDM using an NVIDIA GeForce GTX 4090 GPU and AMD EPYC 7742 64-Core Processor. The training of our LDN consumes about 4 hours the GOPRO. During the testing phase, feed the entire image and the spike stream into the network to assess performance.
We complete the training of BSN on the GOPRO dataset, employing an initial learning rate of 3e- and spanning 1000 epochs. The training uses the Adam optimizer with a cosine scheduler and sets the batch size to 8 for each epoch. Adopting the same settings as BSN, EDSR is trained on the blur-spike
17

---

Blur
11
Blur
Bluryy Impiat
Figure 16: Ablation study for evaluating modules of S-SDM on the GOPRO dataset. Experiments corresponding to the ID can be viewed through Tab.
Blurry Image
I[-1
II-2
II-3
II-6 Figure 17: Ablation study for evaluating the effectiveness of our distillation learning framework on the RSB dataset. The details corresponding the experiment ID can be viewed through the Tab. 4.
I1-4
11-5
II-7 (retrained II-6)
paired data for 70 epochs. Subsequently, LDN undergoes the training of 100 epochs with the learning rate adjusted to le -3
A.4.3 Ablation study
We provide additional ablation visualizations demonstrate the effectiveness of our designed modules and the distillation framework as shown in Fig. 16 and 17.
A.5
Additional Figure
RGB
D
Divide
5
Sum
Times
B
Spike
T,
N
(t,, 7')
T.
SDM
Figure 18: The schematic diagram of our designed SDM.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See lines 72-77.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See lines 298-299.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
This paper provided the theoretical analysis on the spike-guided motion
deblurring task.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Experiments reproduceable.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We have released the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.o cc/
polpiyllyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iip..c.aa./o wttpccccppppccccceccc.cccc.cccct httpppplpuuddsea.pppppppppppaaap /Coee/eodeimeeeeeeeeeeeeee /udie//esuieeeoo wwwisubmmssiiini ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, they are included in the supplementary materials. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We don't 't report it.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, they are included in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerpp//eeeeee/eee/ee cc httppl/c//cccccscdddeeeeeeee
Answer: [Yes]
Justification: Yes, follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, they are correctly used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcom thoooee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "fmOMiGM7lY", "reviewer": "Reviewer_G1PY", "summary": "This paper combines the RGB camera and the spike camera for image deblur. The key contributions consist a self-supervised learning framework for deblur and a real-world dataset RSB.", "questions": "See the weaknesses\n\n\n------------------------------------\nThe response addressed my concerns. I raised my score to 6.", "limitations": "See the weaknesses", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a novel self-supervised framework for image deblur with spike camera. The network design is interesting. The paper is well-written.", "weaknesses": "1 The originality is marginal. The whole framework is similar to [36]. Please clarify more on the difference between this paper and [36].\n\n2. The proposed LDN is also similar to the DCN in [36]. It would be better to see in the ablation whether LDN is better than DCN.\n\n3. It can be observed that smaller V_th leads to better performance. What if V_th == 0.5?\n\n4. The size and diversity of the RSB dataset is limited. It seems the RSB dataset only contains indoor scenes. Outdoor scenes with various objects are desired.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have taken to review our manuscript.\n\n***1. The originality is marginal. The whole framework is similar to [36]. Please clarify more on the difference between this paper and [36].***\n\nPaper \\[36\\] is an ICCV23 publication on a self-supervised evnet-based motion deblurring algorithm named GEM. Besides both being self-supervised algorithms, the content and framework designs of S-SDM and GEM are completely different. We explain the differences between the two in the following aspects:\n\n**A. Camera Principles**: GEM is designed for event cameras while our S-SDM is applied on spike cameras. These two neuromorphic cameras leverage different sampling techniques—differential sampling for event cameras and integral sampling for spike cameras. This difference results in substantial variation in the framework of the two self-supervised motion deblurring methods.\n\n**B. Number of Blurry Frames**: GEM relies on two blurry frames to provide mutual information for constraint when designing the self-supervised loss function, as shown in Fig. R2. However, S-SDM only requires a single blurry frame to complete the self-supervised task, making it more suitable for deployment in online real-time motion deblurring systems compared to GEM.\n\n**C. Self-Supervised Pipeline**: The core of GEM’s self-supervised algorithm is to explore the relationship between two different blurry images, Blur1 and Blur2, with the aid of the event stream to achieve self-supervised deblurring. In contrast, the core of S-SDM’s self-supervised approach is to enhance the quality of pseudo-labels using a teacher network consisting of the SDM, BSN, and EDSR, while exploring the relationship between the latent sharp image and the input blurry image to achieve self-supervision, as shown in Fig. R2.\n\n**D. Loss Functions**: GEM’s core loss function is the Blur2Blur loss, which trains the network to learn Blur2 from Blur1 with the aid of the event stream. S-SDM’s core loss function is the blur consistency reblur loss, which ensures that the deblurred images obtained at different times during the exposure time, when recombined, are consistent with the original blurry image, as shown in Fig. R2.\n\n**E. Teacher Model**: GEM uses the SAN network as the teacher model for generalizing it to scenes at different spatiotemporal scales, while S-SDM’s teacher model consists of SDM, BSN, and EDSR, aiming to provide high-quality pseudo-labels for the LDN network.\n\n**F. Theoretical Analysis**: GEM analyzes the relationship between varying degrees of blurriness and event streams. In contrast, S-SDM studies single-frame blurry images, spike streams, and sharp images. Due to the different sampling principles of spike cameras and event cameras, the theoretical analyses of GEM and S-SDM are entirely different.\n\n ***2. The proposed LDN is also similar to the DCN in [36]. It would be better to see in the ablation whether LDN is better than DCN.***\n\nWe should first clarify that the network framework in the GEM paper is the Scale-aware Network (SAN) rather than DCN. DCN is merely a feature extraction module used to enhance feature extraction in the image domain.\n\nRegarding the issue mentioned, we want to clarify that the core contribution of this paper lies in designing a self-supervised processing pipeline for deblurring with spike cameras. **The network design is not the core contribution. As stated in the paper, we use the encoder-fusion-decoder multimodal fusion framework in designing the network in line with previous studies.** This framework has been widely used in many previous studies. In the SAN network, the core design is not the encoder-fusion-decoder framework but rather its unique MSFF block.\n\n**Why didn’t we explore further in network architecture?** The reason is that for the self-supervised learning framework designed in this paper, improving network performance depends mainly on how to enhance the quality of pseudo-labels. In the supervised learning framework spk2deblurnet, since the supervision signal is direct and explicit, exploring the network architecture to better fit the relationships between blurry images, spike inputs, and sharp images can effectively improve performance. However, for a self-supervised pipeline, a lightweight and simple network is sufficient. Therefore, the network in this paper consists only of convolutional layers, ResBlocks, and simple modules like CBAM.\n\nFinally, to prove our statement, we compare our designed LDN network with the SAN network designed in GEM in terms of PSNR, SSIM, Params, and Flops on the single-frame motion deblurring task:\n\n**Table.R2: Comparison between the SAN network in GEM and our designed LDN.**\n| Methods | PSNR  | SSIM  | Params (M) | Flops (G)   |\n|:---------:|:-------:|:-------:|:------------:|:-------------:| \n| SAN [36]    | 27.283| 0.773 | 2.36   | 107.84| \n| LDN (Ours)    | **27.928**| **0.786**| **0.234**    | **33.60** |\n\nAs shown in Tab. R2, for the self-supervised deblurring task, our LDN network achieves better performance in terms of PSNR and SSIM while maintaining smaller model parameters and computational requirements. This demonstrates that for the self-supervised spike deblurring task presented in this paper, our designed simple network LDN is sufficient for this task while being parameters and computation lightweight.\n\n ***3. It can be observed that smaller V_th leads to better performance. What if V_th == 0.5?***\n\nPlease refer to the response in **To all reviewers**.\n\n ***4. The size and diversity of the RSB dataset is limited. It seems the RSB dataset only contains indoor scenes. Outdoor scenes  are desired.***\n\nOur RSB dataset comprises 10 video sequences, including 9 indoor scenes and 1 outdoor scene as shown in Fig. R4. Due to the time-consuming nature of time and space calibration across two modalities, we plan to collect more scenes and increase the diversity, which will be included in the submission version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5EsncfPESb", "reviewer": "Reviewer_VbQb", "summary": "This work proposes a spike-guided self-supervised image deblurring algorithm that combines the high spatial resolution of RGB cameras with the high temporal resolution of spike cameras to obtain sharp RGB images in real-world scenarios. The self-supervised network addresses performance degradation issues found in existing supervised spike deblurring algorithms. The spike-guided module considers noise and spatial resolution alignment between the two cameras. Both subjective and quantitative experimental results show that the proposed algorithm achieves excellent generalization performance in both real and simulated scenes, surpassing previous spike-guided deblurring algorithms.", "questions": "This method primarily targets optical flow motion estimation. For more complex motion In addition to addressing the performance degradation issues of some supervised models in real-world scenarios, what other advantages does the proposed unsupervised model offer?", "limitations": "The authors have stated the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed self-supervised spike-guided RGB deblurring algorithm effectively addresses the synthetic-real domain gap performance degradation of previous algorithms.\n2. The authors analyze the relationships between spike data, blurry RGB, and sharp RGB images, providing a theoretical foundation for the Spike-Guided Deblurring Model (SDM).\n3. The paper's clear writing and figures make it easy to follow, with comprehensive experiments demonstrating the RGB-Spike binocular system's spatiotemporal alignment and generalization in real scenarios.", "weaknesses": "1. While many existing image-based deblurring algorithms perform adequately, the introduction of spike cameras presents challenges such as aligning the two modalities. The authors have not sufficiently compared or explained the advantages of the spike-guided approach.\n2. The robustness of the model to noise lacks systematic analysis and discussion. Is this primarily influenced by the BSN Loss described in Equation 9?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for pointing out potential issues and improvements in our paper. \n\n***1. While many existing image-based deblurring algorithms perform adequately, the introduction of spike cameras presents challenges such as aligning the two modalities. The authors have not sufficiently compared or explained the advantages of the spike-guided approach.***\n\nWe aim to address this question from the following perspectives: \n\n**A. Theoretical Analysis**. Image-based methods for addressing motion blur often struggle to capture motion information accurately in real-world scenes because traditional cameras cannot precisely record motion details during the exposure. This limitation can lead to incorrect motion trajectories in real-world scenarios. For example, when using a conventional camera to capture a square object moving from left to right, the resulting blurry image does not inherently indicate whether the object is moving from left to right or right to left without prior information. Additionally, reconstructing the texture details of an object at any moment during the exposure from a single blurred image is difficult because it is an ill-posed problem with limited motion representation. However, spike streams, which contain rich motion and texture information, can effectively alleviate issues related to uncertain motion trajectories and missing texture details.  \n\n**B. Experimental Results.** On the synthetic dataset GOPRO, image-based methods like BiT have achieved promising results. This is because the motion patterns in the GOPRO dataset are relatively consistent, and the methods for synthesizing motion blur are also uniform. However, as seen in Fig. 3, 4, 11, and Tab. 4, in real-world scenarios where the motion patterns differ significantly from those in the GOPRO dataset, the BiT algorithm encounters severe image degradation and inaccuracies in motion trajectory recovery, which is evident in both quantitative performance results and qualitative visual outcomes.  \n\n**C. Ablation Study.** We conduct a simple ablation study using SpkDeblurNet on the GOPRO dataset in SpkDeblurNet to verify the contribution of spike streams in assisting deblurring. The comparison results are shown in the Tab. R3. It can be seen that the incorporation of the spike stream can improve about 5 db in PSNR for the single frame image deblurring task, which will behave better in the sequence reconstruction task since image-based methods suffer from severe motion ambiguity.\n\n**Table.R3: Ablation study on the effectiveness of the spike stream.**\n| Method        | PSNR  | SSIM  |\n|:---------------:|:-------:|:-------:|\n| Image         | 32.45 | 0.895 |\n| Image + Spike | **37.42** | **0.968** |\n\n\n ***2. The robustness of the model to noise lacks systematic analysis and discussion. Is this primarily influenced by the BSN Loss described in Equation 9?***\n\nPlease refer to the response in **To all reviewers**.\n\n ***3. For more complex motion In addition to addressing the performance degradation issues of some supervised models in real-world scenarios, what other advantages does the proposed unsupervised model offer?***\n\nThe core contribution of this paper is the introduction of a spike-based deblurring physical model, SDM, and a self-supervised spike-based deblurring model, S-SDM. The S-SDM model primarily addresses the noise and resolution mismatch issues present in the SDM model. In addition to overcoming the domain gap problem, the proposed unsupervised model has the following advantages:\n\n**A. Interpretability.** Our SDM model theoretically constructs the relationship between blurred images, spike streams, and sharp images. Compared to previous fully end-to-end spike deblurring networks like SpkDeblurNet, our method has a stronger theoretical foundation.\n\n**B. Implementability.** The SDM is a model-based motion deblurring method, which can be directly deployed in real-time systems and is relatively easy to implement.\n\n**C. Deployability.** Our S-SDM features a core motion deblurring network, LDN, with a very small parameter size (0.23M). Compared to the parameters of other supervised learning networks like SpkDeblurNet, the LDN network's parameters are only 2% of SpkDeblurNet's, making it more suitable for deployment in real-world motion deblurring scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "74bsTt1c7i", "reviewer": "Reviewer_RXBh", "summary": "The work focuses on improving image sharpness from blurry inputs using spike cameras with high-motion capture rates. Addressing limitations of supervised learning in real-world scenarios, the authors introduce a self-supervised framework for spike-guided motion deblurring. Validation through extensive experiments on both real-world and synthetic datasets confirms its superior performance and generalization ability.", "questions": "see the weakness", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a pioneering approach to spike-guided motion deblurring, addressing the challenge of recovering sharp images from blurry inputs captured by spike cameras. It demonstrates high-quality research with rigorous theoretical foundations and thorough experimental validations on synthetic and real-world datasets.", "weaknesses": "1，I'm curious about how the order of applying BSN (Blur to Sharp Network) and EDSR (Enhanced Deep Super-Resolution) influences model performance. Specifically, I wonder if this sequence could introduce additional artifacts during image restoration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time to provide constructive criticism and recommendations for our article.\n\n***I'm curious about how the order of applying BSN (Blur to Sharp Network) and EDSR (Enhanced Deep Super-Resolution) influences model performance. Specifically, I wonder if this sequence could introduce additional artifacts during image restoration.***\n\nBefore answering this question, we need to clarify why we need the BSN and EDSR networks. As stated in our paper, the SDM model has two main issues: \n\n1. Short-exposure spike frames contain **significant noise** due to limited information contained during this period. \n\n2. Two modalities have **inconsistent resolutions**, making it challenging to apply the SDM model directly to real-world scenarios. To address these issues, the self-supervised denoising network BSN and the super-resolution network EDSR are employed. The EDSR network is retrained based on the texture similarity between long-exposure spike frame and blurry input image.\n\nIn the following, we first explain the difference of BSN-EDSR and EDSR-BSN two orders from the following perspectives:\n\n***A. Theoretical Analysis***\n\n**EDSR-BSN:**  The advantage of this order is that it can recover as much information as possible from the low-resolution details, making it suitable for spike reconstructions that are rich in detail and have less noise. However, a drawback is that the super-resolution process may amplify the original noise in the image, making it difficult for subsequent denoising to completely remove the noise.\n\n**BSN-EDSR:**  The advantage of this order is that the denoised image provides a sharper baseline for super-resolution, helping the algorithm more accurately reconstruct image details and avoid amplifying noise. The drawback is that if the denoising is too aggressive, it may remove important details from the image, making it challenging for super-resolution to recover these details, especially in the image where details are relatively limited.\n\n***B. Quantitative Experiments***\n\nIn the GOPRO dataset, both scenarios mentioned above are present.  Through evaluating the performance of both orders in the entire dataset, we find that the denoising-first, super-resolution-second approach (BSN-EDSR) yields better results, so we adpot this processing baseline. \n\n**Table.R4: Ablation study on the order of the BSN and EDSR networks.**\n| Method  | PSNR   | SSIM  |\n|:---------:|:--------:|:-------:|\n| EDSR-BSN  | 25.914 | 0.692 |\n| BSN-EDSR  | **26.144** | **0.708** |\n\n\n***C. Visual Comparison***\n\nTo understand this intuitively, we conduct a visual comparison on the GOPRO dataset, as shown in Fig. R3. In the example (street scene) above Fig. R3, it can be seen that the EDSR-BSN method effectively recovers some detailed features of the floor compared to the BSN-EDSR method. The EDSR-BSN method cannot accurately recover these details because the BSN network first removes them, leaving little information for the EDSR network to super-resolve.\n\nIn the example below, regarding some background information, the EDSR-BSN method produces noticeable noise and image artifacts. (The magnified image regions use contrast enhancement to highlight detailed information.) This occurs because applying the EDSR to the reconstructed spike frame first enlarges the noise, resulting in a noisy final image. The subsequent BSN network cannot mitigate this due to the limited neighboring information available during the blind convolution process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Is3wMGCDfe", "reviewer": "Reviewer_1hhy", "summary": "The SpikeReveal paper uses a blurry RGB image and leverages pulse data captured by a spike camera of the corresponding scene to guide the image deblurring task. In terms of network design, the authors cascade a blind-spot network for denoising, a super-resolution network, and a deblurring network, employing a self-supervised approach to accomplish the deblurring task.", "questions": "[Definition of t_s]\nThere is an issue with the definition of t_s in line 109. When the first spike occurs, there is no previous spike to reference. Therefore, this definition needs to be revised.\n[Missing Ablation Study]\nTable 2: The module cascading ablation on GOPRO should include with BSN without SR with LDN and without BSN with SR with LDN. This would allow us to determine which component, BSN, SR, or LDN, contributes the most to the final performance.\n[Comparison with SpkDeblurNet]\nIn Table 1 and Figure 4, has SpkDeblurNet been retrained using the same simulation data as the authors' method? Why do the restoration results of SpkDeblurNet in Figure 4 appear noticeably darker? Additionally, why does SpkDeblurNet perform better than the authors' method at Vth=1V_{th}=1?", "limitations": "The authors could consider collecting paired datasets to better address the limitations of simulating pulse camera data. While the authors mention using real data, it serves only as input to the network without providing a genuine ground truth for comparison. Addressing this by capturing paired real-world data could significantly enhance the validity and applicability of their findings.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The SpikeReveal paper uses a blurry RGB image and leverages pulse data captured by a spike camera of the corresponding scene to guide the image deblurring task. In terms of network design, the authors cascade a blind-spot network for denoising, a super-resolution network, and a deblurring network, employing a self-supervised approach to accomplish the deblurring task.", "weaknesses": "The paper has several areas that could be improved to better achieve its stated goals.  Addressing these issues would improve the robustness and validity of the comparative analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your detailed feedback and suggestions, which have helped us identify key areas where our manuscript can be improved.\n\n***1.  [Definition of t_s] There is an issue with the definition of t_s in line 109. When the first spike occurs, there is no previous spike to reference. Therefore, this definition needs to be revised.***\n\nThank you for pointing out this issue. We will address the problem you mentioned in the final version.\n\n ***2. [Missing Ablation Study] Table 2: The module cascading ablation on GOPRO should include with BSN without SR with LDN and without BSN with SR with LDN. This would allow us to determine which component, BSN, SR, or LDN, contributes the most to the final performance.***\n\nWe have supplemented the ablation study based on your request, including the configurations for BSN + LDN and SR + LDN. The overall results of the ablation study are as follows:\n\n**Table.R5: Ablation study on the proposed different modules.**\n| ID           | BSN | SR  | LDN | PSNR   | SSIM  |\n|:--------------------:|:-------------------:|:-------------------:|:-----------:|:--------------:|:-------------:|\n| I-1          | ✗   | ✗   | ✗   | 23.012 | 0.486 |\n| I-2          | ✓   | ✗   | ✗   | 24.634 | 0.661 |\n| I-3          | ✓   | ✓   | ✗   | 26.144 | 0.708 |\n| I-4     | ✗   | ✓   | ✓   | 26.172 | 0.633 |\n| I-5   | ✓   | ✗   | ✓   | 26.662 | 0.745 |\n| I-6          | ✓   | ✓   | ✓   | **27.928** | **0.786** |\n\nThe table compares the performance of various configurations involving BSN, SR and LDN in terms of PSNR and SSIM. The baseline configuration (I-1), with no methods applied, shows the lowest image quality metrics. Adding BSN alone (I-2) significantly enhances both PSNR and SSIM, demonstrating effective noise reduction. When SR is added (I-3), further improvement is observed, reflecting enhanced detail recovery. The BSN + LDN setup (I-5) provides better results than SR + LDN (I-4), confirming BSN’s key role in enhancing quality over the SR network, which will amplify the image noise without the pre-processing of the BSN network thus degrading the image quality and reconstruction performance.\nFinally, the full combination of BSN, SR, and LDN (I-6) delivers the highest PSNR and SSIM scores, showcasing the synergistic effect of integrating all three methods for optimal image deblurring performance.\n\n***3. [Comparison with SpkDeblurNet] In Table 1 and Figure 4, has SpkDeblurNet been retrained using the same simulation data as the authors' method? Why does SpkDeblurNet perform better than the authors' method at Vth=1V_{th}=1?***\n\nFig. 4 presents the comparison of different methods in the sequence reconstruction task for real-world scenarios. In real scenes, the absence of ground truth sharp frames means that the SpkDeblurNet method cannot be retrained, leading to noise-related image degradation. In contrast, the S-SDM method effectively overcomes the domain gap and restores sharp texture details, benefiting from its self-supervised mechanism. Both S-SDM and SpkDeblurNet are trained under the same simulation dataset while S-SDM is further fine-tuned in real-world scenarios.\n\nIn Tab. 1, to mimic the domain gap between synthetic and real datasets and quantitatively represent this gap using PSNR and SSIM metrics, we designed our experiments such that all methods were trained with a scenario where $V_{th} = 1$ and evaluated across scenarios with different threshold values. The reason SpkDeblurNet performs better under the $V_{th} = 1$ condition is that it is constrained by strong supervision signal, which makes the network to learn the mapping from the blurry input and the spike stream to the sharp image. Under this specific setting, the strength of our self-supervised framework, S-SDM, to effectively overcome the domain gap is not fully realized.\n\n ***4. Why do the restoration results of SpkDeblurNet in Figure 4 appear noticeably darker?***\n\nPlease refer to the response in **To all reviewers**.\n\n ***5. The authors could consider collecting paired datasets to better address the limitations of simulating pulse camera data. While the authors mention using real data, it serves only as input to the network without providing a genuine ground truth for comparison. Addressing this by capturing paired real-world data could significantly enhance the validity and applicability of their findings.***\n\nSimultaneously capturing spike streams, blurry image inputs, and clear images requires precise spatiotemporal calibration of three different cameras. This process is time-consuming and labor-intensive, with the potential for calibration inaccuracies. Thank you for your feedback. We will consider further improving our camera system in future research to develop a dataset that includes RGB inputs, spike streams, and ground truth clear images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments and positive feedback. We are pleased that our paper has been recognized as "well-written" [G1PY,VbQb] with a "pioneering approach" [RXBh] and our network design found "interesting" [G1PY]. Our proposed S-SDM is acknowledged for effectively addressing the synthetic-real domain gap [VbQb], providing a strong theoretical foundation [VbQb], and demonstrating excellent experimental results [VbQb, RXBh]. We are also grateful for the recognition of our method to synergistically enhance the deblurring results in a self-supervised manner [1hhy]. The reviewers' insights are invaluable, and we will integrate all suggestions to refine our work further.

In this general response, we would like to address the crucial concerns regarding '***How $V_{th}$ influences the experiments.***'

> **[Reviewer G1PY]**: It can be observed that smaller V_th leads to better performance. What if V_th == 0.5? 

The threshold $V_{th}$ has a multifaceted impact in this task. When the spike threshold is high, the spike firing rate of the spike camera decreases, but the dark current and other noise in the spike camera simulation remain unchanged. This results in a lower signal-to-noise ratio  of the input information, leading to poorer deblurring performance. Conversely, when the spike threshold is too low, the spike camera's synchronous sampling mechanism causes issues. A lower threshold leads to multiple spikes being fired between intervals, but only one is read, resulting in less spike stream information during the readout time. 

Considering this problem from an extreme perspective: if the spike firing threshold is infinitely high, there will be no spike readout, only dark current noise. If the spike firing threshold is infinitely low, every sampling cycle will produce frames filled with ones, making it impossible to extract meaningful information.

We further evaluated the deblurring performance of the S-SDM model in a sequence recovery task under different spike threshold values of 0.25 and 0.5. The experimental results are as follows:

**Table.R1: Comparison of our method under different $V_{th}$.**
| $V_{th}$| 0.25   | 0.5    |    1     | 2      | 4      |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| PSNR   | 24.657 | 25.886 | 26.893 | 26.367 | 25.433 |
| SSIM   | 0.633  | 0.724  | 0.757  | 0.740  | 0.699  |

From the above table, it can be seen that $V_{th}= 1$ achieves the best performance, which aligns well with our previous analysis. Additionally, we provide visualization comparison of the SDM model at a threshold of $V_{th} = 0.5$ in Fig. R1. It can be observed that even though the short exposure spike frame suffers from texture damage due to truncation errors in most regions, the SDM model effectively retains the correct texture information by utilizing the blurry image $B$. SDM model, while removing motion blur from the blurry input, significantly mitigating truncation errors caused by the low threshold in the spike frames, which further demonstrates the robustness of our method.

> **[Reviewer VbQb]**: The robustness of the model to noise lacks systematic analysis and discussion. Is this primarily influenced by the BSN Loss described in Equation 9?

The proposed method effectively alleviates degradation issues such as noise, primarily due to the design of its self-supervised framework. This design grants it strong generalization capabilities in both real-world and synthetic datasets, as demonstrated in Fig. 1 of this paper. We agree with your view that the robustness of our method against noise is largely attributed to the BSN loss function. Next, we will explain the role of the BSN loss from the following perspectives:

**A. Working Mechanism.** The robustness of the BSN model against noise is primarily due to its use of a blind-spot convolution strategy. This approach employs self-supervision to eliminate noise in the short-exposure spike imaging frames. As a result, the BSN model can effectively remove noise of any type through retraining, offering strong generalization capabilities.

**B. Visual Comparison:** From the ablation studies in Figs. 5 and 17, we can see that I-2 effectively eliminates the substantial background noise present in I-1, such as the noise around license plates and letter signs. This demonstrates the effectiveness of the BSN self-supervised loss. This loss function is also capable of effectively removing dark current noise present in real-world scenarios, thus ensuring generalization to real scenes.

**C. Quantitative Analysis:** According to our latest ablation experiment results in Tab. R5, the BSN loss function effectively removes background noise both when used alone and when combined with SR and LDN. It improves PSNR by 1.6 dB and 1.8 dB, respectively, in these configurations.

> **[Reviewer 1hhy]** Why do the restoration results of SpkDeblurNet in Figure 4 appear noticeably darker? 

The reason SpkDeblurNet appears significantly darker in Fig. 4 is due to its nature as a supervised learning method. It has only learned to extract deblurring texture details from the spike streams with fixed firing rates in the synthetic GOPRO dataset ($V_{th} = 1$ under this condition). When applied to real-world scenarios, if the spike firing rate in the real scene differs from that of the synthetic dataset, SpkDeblurNet misestimates the spike density. This results in darker outputs when the spike density is lower than in the synthetic dataset, and overexposure in scenarios where the spike density is higher, as seen in Fig. 12.

Meta-review: The paper was rated by the reviewers with an Accept, 2 Weak Accept, and a Borderline Accept. The reviewers raised a number of concerns, but they were all addressed by the authors and the reviewers raised their initial ratings.
There is general consensus that the paper introduces a novel approach to deblur images by exploiting the spike camera.
The AC has read all the reviews, the rebuttal, and the discussions and agrees with the reviewers that the paper should be accepted.
Decision: Accept (spotlight)