SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams

Authors: kang  chen
shiyan  chen
jiyuan  zhang
baoyue  zhang
yajing  zheng
tiejun  huang
zhaofei  yu
𝑻𝑻  𝟎𝟎
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:43:59

Abstract: Reconstructing a sequence of sharp images from the blurry input is crucial for enhancing our insights into the captured scene and poses a significant challenge due to the limited temporal features embedded in the image. Spike cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Nonetheless, existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios that diverge from the synthetic training data domain. To address these challenges, we propose the first self-supervised framework for the task of spike-guided motion deblurring. Our approach begins with the formulation of a spike-guided deblurring model that explores the theoretical relationships among spike streams, blurry images, and their corresponding sharp sequences. We subsequently develop a self-supervised cascaded framework to alleviate the issues of spike noise and spatial-resolution mismatching encountered in the deblurring model. With knowledge distillation and re-blurring loss, we further design a lightweight deblur network to generate high-quality sequences with brightness and texture consistency with the original input. Quantitative and qualitative experiments conducted on our real-world and synthetic datasets with spikes validate the superior generalization of the proposed framework. Our code, data and trained models are available at \url{https://github.com/chenkang455/S-SDM}.

Main paper: 
 introduction: 
 Traditional cameras, constrained by their exposure-based imaging mechanism, often produce blurry images when capturing fast-moving objects or during camera movement throughout the exposure process [19, 35] . While these blurry images lose significant details, the ability to recover dynamic motion trajectories from the static blurry input becomes critically important. However, the inherent challenge lies in the limited motion features available within blurry frames, leading to potential ambiguities such as multiple motion trajectories corresponding to the same blurry input. This is exemplified by scenarios where two objects move along the same trajectory but in opposite directions [23, 24, 38] , rendering the task of motion deblurring ill-posed. Recent advancements in learningbased approaches [40, 12, 46] seek to address this challenge by establishing direct mappings from relationship between spike streams, blurry images, and sharp sequences, leading to the development of our Spike-guided Deblurring Model (SDM). We further construct a self-supervised processing pipeline by cascading the denoising network and the super-resolution network to reduce the sensitivity of the SDM to spike noise and its reliance on spatial-resolution matching between the two modalities. To reduce the computational cost and enhance the utilization of spatial-temporal spike information within this pipeline, we further design a Lightweight Deblurring Network (LDN) and train it based on pseudo-labels from the teacher model, i.e., the established self-supervised processing pipeline. Further introducing reblur loss during LDN training, we achieve better restoration performance and faster processing speed than the processing-lengthy and structure-complicated teacher model. To validate the performance of our S-SDM across various scenarios, we build an RGB-Spike binocular system and propose the first spatially-temporally calibrated Real-world Spike Blur (RSB) dataset in this community. Quantitative and qualitative experiments conducted on the real-world and synthetic datasets validate the superiority of our method. In summary, our key contributions are:• We develop a self-supervised spike-guided image deblurring framework, addressing the performance degradation due to the synthetic-real domain gap in supervised methods. • We perform an in-depth theoretical analysis of the fusion between the spike stream and blurry image, leading to the development of the SDM. • We propose a real-world dataset RSB and experiments on GOPRO and RSB datasets validate the superior generalization of our S-SDM. 
related work: 
 Spike Camera. The spike camera, inspired by the primate retina, stands apart from conventional cameras with its ability to generate synchronous spike streams for each pixel at extremely low latency. This distinct feature provides significant advantages in various applications such as highspeed imaging [47, 45, 48, 5, 41, 6, 33] , optical flow estimation [43] , object detection [44] , 3D reconstruction [30] , depth estimation [34] , motion deblurring [8, 32] , and occlusion removal [31] .Spike-guided Motion Deblurring. While the spike camera boasts an ultra-high temporal resolution, its development is currently impeded by the low spatial resolution. Additionally, the single-channel output from the spike camera restricts previous methods from recovering the image color information.To address these issues, a promising approach is establishing an RGB-Spike hybrid imaging system [2] .The binocular system achieves the multi-modality fusion of High-spatial/Low-temporal RGB blurry input and High-temporal/Low-spatial spike stream, thereby also serving as a spike-guided motion deblurring method [8] . However, to the best of our knowledge, existing spike-guided deblurring methods [2, 8] predominantly rely on supervised training on synthetic datasets. This reliance results in significant performance degradation when these methods are evaluated in real-world scenarios due to the domain discrepancies between synthetic and real datasets as illustrated in Fig. 1 .Event-based Motion Deblurring. Event camera [21] can asynchronously generate events that record log-intensity changes at the pixel level with minimal latency, which contains a rich set of motion features beneficial for motion deblurring tasks. Numerous supervised methods [16, 4, 3, 16, 23, 24] have been proposed to learn the mapping from the blurry input, events to the sharp outcome. Despite these advancements, a major hurdle remains in obtaining real blurry-sharp image pairs for training.To overcome the domain gap between synthetic and real-world datasets, recent methods [29, 38, 39] explored the mutual constraint between the blurry image and event stream, enabling the training of networks on real-world blur datasets.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See lines 298-299.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Spike Camera Mechanism. Consider L(t) to represent the latent sharp frame at time t. Each pixel p in the spike camera [10] has an integrator that accumulates the incoming photons at a high frequency.Once the cumulative intensity exceeds a predefined threshold C at time t e , pixel p emits a spike, and the accumulation of photons is reset to zero. This process can be mathematically described as follows:EQUATIONwhere t s denotes the firing time of the previous spike. While the spike camera is capable of generating asynchronous spike streams akin to that of event cameras, its effectiveness is constrained by the inherent limitations of its physical circuitry, which necessitates reading spikes at a predetermined sampling rate. We denote the generated spike stream as S ∈ {0, 1} K×1×H×W , where H and W signify the height and width of the image, and K represents the length of the spike sequence.Problem Formulation. In traditional photography, motion blur occurs when there is relative movement between the camera and the scene during the exposure period. According to the motion blur physical model [9] , the blurry image B can be represented as the average of the latent frame L(t) over the exposure T , i.e.:EQUATIONwhere T represents the exposure period. Despite the spike camera's superior temporal resolution, its spatial resolution remains comparatively low. This limitation is primarily attributed to the constraints in data transmission bandwidth and the challenges inherent in the manufacturing process. Here, we postulate that the spatial resolution of the spike camera is approximately one-quarter that of a conventional RGB camera.In this paper, we aim to enhance the High-spatial/Low-temporal resolution blurry input B ∈ R 1×3×H×W into a sequence of High-Quality images {L(t i )} K i=1 ∈ R K×3×H×W with the aiding of High-temporal/Low-spatial resolution spike stream S T ∈ {0, 1} K×1× H 4 × W 4 , which can be mathematically formulated as:EQUATIONIn Eq. ( 3), Deblur(•) represents the Spike-guided Deblur-Net as shown in Fig. 1 , i refers to the i-th frame in the spike stream S T , and t i is the timestamp associated with this frame.The spike camera, with its photodetector tailored to capture the single-channel light intensity, faces difficulties in obtaining the color information that the multi-channel RGB camera can effortlessly capture. Therefore, we modify the color intensity L(t) in Eq. ( 1) to the grayscale value L g (t) for further analysis:EQUATIONIn this formulation,L g (t) = w r • L r (t) + w gre • L gre (t) + w b • L b (t), where L c (t), w c denote the intensity and weight of channel c ∈ {r, gre, b} respectively.Given the blurry input B and its corresponding spike stream S T , we incorporate Eq. ( 4) into the motion blur model presented in Eq. ( 2). This integration formulates a link between the two modalities as outlined below:EQUATIONwhere B g denotes the grayscale version of the blurry input, and N T denotes the total number of spikes accumulated over the exposure period. The accumulation N T is calculated asN T = K i=1 S[i],with S[i] indicating the i-th frame of the spike stream.Within the exposure T , we consider a shorter spike sequence centered around the t moment S T ′ ∈ {0, 1} K ′ ×1×H×W , satisfying K ′ ≪ K, t ∈ T , and T ′ ⊂ T . Similar to Eq. ( 5), we can derive the relationship between the short-exposure gray image E g (t, T ′ ) and the short spike stream S T ′ as follows:EQUATIONwhere T ′ ≪ T represents the short exposure period. Given the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the short-exposure image E(t, T ′ ) are identical. This assumption implies that the intensity proportion among RGB channels in the blurry input α B c and the short-exposure imageα E c (t, T ′ ) is approximately equivalent, satisfying α B c = B g /B c and α E c (t, T ′ ) = E g (t, T ′ )/E c (t, T ′ ).More details can be found in the supplementary materials.Upon establishing it, we move forward to build a mathematical relation between the blurry image and the spike stream. By substituting the gray channel g with color channel c and dividing Eq. ( 5) by Eq. ( 6), we efficiently eliminate the unknown threshold C and weights α B c /α E c (t, T ′ ), leading to the following equation:EQUATIONBy applying Eq. ( 7) to RGB three channels, we explicitly establish the relationship between the color blurry input B, the color short-exposure image E(t, T ′ ) and the spike stream S T as shown in Fig. 18 . Since the exposure period T ′ is relatively short, it is reasonable to assume that the scene remains static. In this context, we interpret the short-exposure image E(t, T ′ ) as the latent sharp frame L(t), allowing us to modify Eq. ( 7) as follows:EQUATIONTo this end, we have conducted a comprehensive theoretical analysis of the spike-guided motion deblurring task, which is neglected in prior learning-based motion deblur methodologies [2, 8] . For further discussion readability, we refer to Eq. ( 8) as the Spike-guided Deblurring Model (SDM), which is analogous to the baseline motion deblur model EDI [18] in event camera.3.3 Self-supervised Spike-guided Deblurring ModelWhile SDM theoretically allows for the fusion of the blurry image and the spike stream, its practical deployment faces the following obstacles:• The deblurred image E(t, T ′ ) suffers from noise-related degradation due to the lack of adequate spike information during the short exposure T ′ .• The spatial resolution of the spike camera is approximately one-quarter of the RGB camera, rendering the SDM implementation impractical.To overcome these limitations, we further cascade the self-supervised denoising network to eliminate the spike noise in N T ′ and super-resolution network to match spatial resolutions of the blurry image and spike stream, with the processing pipeline illustrated in the bottom of Fig. 2 .Denoising Network. We leverage the Blind Spot Network (BSN) [13, 1, 27, 5, 14, 7] to predict the clean spike accumulation N T ′ from the input short-exposure spike stream S T ′ . The core idea of BSN is to design the blind-spot strategy that compels the convolutional layer to estimate the clean value of each pixel solely based on its surrounding pixels.Under the premise that the spike stochastic thermal noise is independent identically distributed [42] , the BSN is trained to deduce sharp spike frames from the input, with the loss function formulated as:EQUATIONwhere the denoised spike frame BSN(S T ′ ; Θ 1 ) is denoted by L bsn (t) for further analysis.Super-Resolution Network. In this task, we observe that the blurry input B g and the long-exposure spike frame N T exhibit the same texture features as shown in Eq. ( 5). This observation motivates us to train the Super-Resolution (SR) network based on pairs of the blurry images and the long-exposure spike frames.We leverage the well-explored Enhanced Deep Super-Resolution network (EDSR) [15] as the backbone of our SR network, with the loss function formulated as follows:EQUATIONWith the training of the SR network completed, we freeze its parameters and apply it to the denoised spike frame L bsn (t), yielding the resolution-enhanced spike frame L ↑ bsn (t).While the aforementioned processing pipeline achieves the multi-modality fusion of the blurry input and the spike stream, several aspects still need refinement:• The framework is lengthy and computationally demanding, which hinders its suitability for real-time system deployment. • The blind-spot strategy of the BSN limits the full utilization of the spatial information inherent in the spike stream. • The representation of the short-exposure image does not fully reflect the advantages of the high temporal resolution inherent in the spike stream.To improve them, we further build a knowledge distillation framework building upon the existing processing pipeline. This pipeline serves as the teacher model, providing the reconstructed sequence as pseudo-labels for the training of the student model LDN, as illustrated in Fig. 2 .Lightweight Deblur Network. LDN adheres to a similar input and output pattern as previous research [8] , i.e., taking the blurry input B and the short spike stream S T ′ centered around moment t as inputs, with the output being the reconstructed sharp image L ldn (t), mathematically formulated as follows:EQUATIONFull details regarding the LDN structure are available in the supplementary materials.To avoid the scenario where the LDN exactly replicates the mapping of the teacher model, we design the teacher loss L tea based on the LPIPS [36] loss and further introduce the blur reconstruction loss. The reblur loss L reblur measures the difference between the blurry input B and the re-synthesized blurry image B, satisfying:EQUATIONwhere L ldn (t m ) represents the m-th recovered image within the exposure period T and M is the total number of reconstructed images. Finally, we sum up two loss functions with the weighting parameter We conduct both quantitative and qualitative comparisons of our S-SDM against 316 state-of-the-art (SOTA) methods, including frame-based Motion-ETR [32] , LEVS 317 [12] , video-based BiT [38] and spike-based SpkDeblurNet [8] on the GOPRO bution of our method is self-supervised learning, it is foreseeable that our method 325 325 might be slightly inferior to the supervised method SpkDeblurNet on the syn- λ, and the final loss function is formulated as follows:EQUATION)4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
dataset: 
 Synthetic Data. For quantitative analysis of our spike-guided motion deblurring task, we construct the synthetic dataset based on the widely employed GOPRO [17] dataset. We initially utilize the interpolation algorithm XVFI [22] to augment the video frame by interpolating additional 7 frames between each pair of consecutive sharp images. To generate the spike stream that mimics reality closely, we downsample the interpolated video to the resolution of 320 × 180 and simulate the spike Ours 𝑉𝑉 𝑡𝑡𝑡 = 1 Motion-ETR SpkDeblurNet 𝑉𝑉 𝑡𝑡𝑡 = 2Ours 𝑉𝑉 𝑡𝑡𝑡 = 2SpkDeblurNet 𝑉𝑉 𝑡𝑡𝑡 = 4Ours 𝑉𝑉 𝑡𝑡𝑡 = 4 stream based on the spike simulator [42] . To replicate real-world motion blur, we synthesize each blurry input by averaging 97 frames from interpolated video sequences.Real-world Data. We construct an RGB-Spike binocular system and propose the first Real-world Spike-guided Blur dataset (RSB) in this community. This system consists of a fixed-exposure RGB camera (Basler acA1920-150uc) and a spike camera [10] , enabling us to capture the blurry image and the corresponding spike stream simultaneously. Further details about our RSB and the spatial-temporal calibration for our binocular system are provided in the supplementary materials.We conduct both quantitative and qualitative comparisons of our S-SDM against state-of-the-art (SOTA) motion deblurring methods, including frame-based Motion-ETR [40] , LEVS [12] , videobased BiT [46] , event-based TRMD [4] , REFID [25] , RED [29] and spike-based SpkDeblurNet [8] on the GOPRO and RSB datasets. For event-based methods, we replace the event stream with the spike stream and adopt the same input representation in these methods [29, 25, 4] . We further cascade the image super-resolution technique DASR [26] as in [39] for the deblurred sequence to overcome the modality resolution inconsistency which is not considered in these methods. We reconstruct 7 images from one blurry input for sequence restoration evaluation [39] as listed in Tab. 1. Results on GOPRO. To simulate the spike density domain gap as depicted in Fig. 1 , we train all supervised spike-based deblurring methods on the GOPRO dataset under spike threshold [42] V th = 1 and evaluate them on datasets with spike thresholds V th = 1, 2, 4. Quantitative comparison results are listed in Tab. 1 and the visual comparison is demonstrated in Fig. 5 . Given that the principal contribution of our S-SDM is self-supervised learning, it is foreseeable that our method might be slightly inferior to the supervised methods SpkdDeblurNet on the dataset with V th = 1. While these supervised methods deteriorate on datasets with V th ̸ = 1, our method achieves great generalization benefiting from the self-supervision design. Specifically, SpkDeblurNet tends to produce darker and blurrier reconstructions on images with high thresholds as shown in Fig. 5 . Besides, our method achieves better restoration performance than the self-supervised method RED due to our consideration of the spatial-resolution mismatch between two modalities and the designed teacher loss, which imposes a stronger constraint than the optical loss in RED.Results on RSB. We further present visualizations of single frame and sequence reconstruction comparisons on the real-world RSB dataset as depicted in Fig. 3 and 4 . Both frame-based and videobased approaches fail to replicate fine textures and detailed elements present in the blurry input. While SpkDeblurNet is capable of recovering structural details and the motion trajectory, it is deteriorated by significant noise and color distortion under conditions with lower spike density than those simulated in the GOPRO dataset. Similarly, in scenarios of higher spike stream densities, the restoration of SpkDeblurNet tends to exhibit over-exposure, resulting in brightness inconsistency compared to the blurry input. This over-exposure affects the dynamic range of the reconstructed image, ultimately compromising the overall perceptual quality and uniformity of the restored sequence. Our method addresses these challenges by finetuning on the RSB dataset, ensuring that the restored sequence aligns with the real-captured blurry input and spike stream. More comparative experiments and analyses are accessible in the supplementary material.We perform ablation experiments on the GOPRO and RSB datasets to evaluate the performance of each module within S-SDM, the validity of our designed network architecture, as well as the overall effectiveness of our distillation learning framework. In this subsection, we evaluate the performance based on the single middle frame for simplicity.Modules Cascading. Building upon the SDM, we sequentially cascade the BSN, the SR, and the LDN to evaluate their respective effectiveness, with quantitative results on GOPRO presented in Tab. 3. We employ bilinear interpolation as a substitute for the SR network in experiments I-1 and I-2 to align the spatial resolution of two modalities.Qualitative ablation experiments are illustrated in Fig. 6 . These comparisons reveal that while the SDM effectively removes motion blur, it struggles with significant noise and detail loss due to the spike noise and the low resolution of the spike stream. While the BSN mitigates noise and the SR network improves spatial resolution explicitly, the LDN trained via distillation learning further refines these enhancements, enabling the recognition of intricate textural features in the images, such as the license plate and the door number shown in Fig. 6 .Network Architecture. While our designed LDN mirrors the Scale-aware Network (SAN) proposed in the GEM [39] , we replace the SAN with the LDN to compare the performance difference between the two architectures as depicted in Tab. 2. In the self-supervised learning framework discussed in this paper, enhancing restoration performance primarily hinges on improving the quality of pseudolabels rather than the network architecture itself. Despite its simple design, which consists only of convolutional layers, ResBlocks, and basic modules such as CBAM [28] , LDN outperforms SAN in both PSNR and SSIM while requiring fewer parameters and less computation, demonstrating that LDN is both sufficient and efficient for the self-supervised spike-guided motion deblurring task. Distillation Learning. We focus on analyzing the contribution of the teacher loss L tea and the reblur loss L reblur within the distillation framework, with quantitative results listed in Tab. 4. Without the teacher loss L tea , the LDN tends toward learning identity mapping from the blurry input. While under the guidance of the teacher model, the reblur loss L reblur not only enforces motion consistency in the reconstructed sequence but also enriches the LDN with high-resolution details from the non-blurry regions of the input, thus improving the performance on GOPRO as listed in Tab. 4.We further apply the LDN trained on GOPRO to the real-world dataset RSB, with qualitative visualization illustrated in Fig. 7 . As observed in the figure, the absence of the reblur loss L reblur leads to significant noise in the recovered image, which predominantly arises from the disparity in spike density and generation mechanism between the simulated and real-captured spike stream. This discrepancy causes the LDN to overestimate the spike number, resulting in black holes in regions with lower spike density than simulated, which reflects the drawback inherent in the supervised learning strategies as discussed in Sec. 1. Increasing the weight of the reblur loss L reblur allows the LDN to incorporate more information from the blurry input, thereby mitigating this issue. However, this adjustment also leads to the presence of blurry edges. We follow the parameters set in Experiment II-6 and retrain the LDN on the RSB dataset (referred to as Experiment II-7). The retrained LDN effectively recovers the sharp edge of the calibration board and suppresses the spike noise in the background, validating the feasibility of our self-supervised framework in real-world scenarios. 
conclusion: 
 In this paper, we introduce a novel self-supervised spike-guided motion deblurring framework S-SDM, which reconstructs sequences of sharp images from real-world blurry inputs with the spike stream. Additionally, we construct an RGB-Spike binocular system and propose the first spatially-temporally calibrated real-world dataset RSB in this community. Quantitative and qualitative experiments validate the superior generalization capabilities of our proposed S-SDM.Limitation. The limitation of our S-SDM lies in its dependence on strict spatial-temporal calibration. Misalignment will lead to color shifts and quality degradation in the deblurred sequence.Figure 8 : Video comparison of our method against the BiT and SpkDeblurNet on the RSB dataset under different luminance conditions, with the static visualization displayed in Fig. 13 . It is recommended to view the pdf using Acrobat PDF reader and the gif demonstration with a higher frame rate is available in the supplementary zip file. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, they are correctly used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This supplementary material provides a comprehensive elaboration on the methodologies and experiments in this paper. It is organized into four distinct sections: Theory Analysis in appendix A. We define k B 1 and k B 2 as the ratios of the red channel to the green and blue channels in the blurry input B respectively, i.e.,EQUATIONEQUATIONSince the gray image is the weighted sum of RGB channels, the ratio of the gray to the red image α B r is formulated as:EQUATIONEQUATIONwhere w c denote the weight of channel c ∈ {r, gre, b} respectively.Similarly, we define k E 1 (t, T ′ ) and k E 2 (t, T ′ ) as the fractions of the red channel relative to the green and blue channels in the short-exposure image E(t, T ′ ), resulting in:EQUATIONGiven the observation that the color information of adjacent pixels often exhibits similarity under the premise of minor motion amplitude, we postulate that the colors of the blurry input B and the shortexposure image E(t, T ′ ) are identical. This assumption implies that the intensity proportion among RGB channels in two images is approximately equivalent, i.e., kB 1 ≈ k E 1 (t, T ′ ) and k B 2 ≈ k E 2 (t, T ′). Drawing from Eq. ( 19) and Eq. ( 20), We further deduce the following relation:EQUATIONwhich can be readily generalized across channels, leading to α B c ≈ α E c (t, T ′ ).We construct our BSN based on the blind-spot strategy outlined in [13] . We first rotate the input image four times, then concatenate them into the U-Net [20] structure like denoising network. To prevent direct mapping from the input pixel to the output pixel, a single-pixel offset strategy is employed in the convolutional kernel to separate its receptive field from the central pixel. Finally, the denoising output is obtained by merging the results of four branches via a 1 × 1 convolution, as shown in Fig. 9 (a).The network structure is depicted in Fig. 9(b) , where the encoder for blurry images consists of two layers of down-sampling convolutions to align the spatial resolution of two modalities. In contrast to the intricate cross-attention mechanism outlined in [4, 8] , we implement the fusion of two modalities by the simple operation of Concat(•), with the cascaded CBAM [28] and residual blocks for further feature fusion.We detail the construction of our RGB-Spike hybrid system as illustrated in Fig. 10 . The system comprises a Spike Camera-001T-Gen2 with a resolution of 400 × 250 pixels, paired with a Basler acA1920-150uc RGB Camera, offering a higher resolution of 1920 × 1200 pixels.pike CameraSpike Camera 400 ×250Calibration Board To achieve the spatial calibration of two cameras, we perform simultaneous captures of the calibration board using the RGB-Spike system as shown in Fig. 10 . The RGB images are cropped to a resolution of 1600 × 1000, aligning them to be fourfold the resolution of the spike camera. We convert the cropped RGB image to grayscale and take the TFP [47] image reconstruction result as the reference from the spike camera. We utilize the MATLAB calibration toolbox to implement the calibration process, with the results detailed in Fig. 11 .Our RSB dataset contains 10 video sequences under different conditions, captured under varied conditions including scene brightness levels (e.g., Low, Middle, and High light) and motion patterns (e.g., camera shake and object motion), which introduce different types of motion blur. Besides, the RSB dataset comprises a large amount of blur-spike pairs with each blurry input corresponding to 400 spike frames.To assess the performance of our method on the GOPRO dataset, we utilize the Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index (SSIM) as the quantitative metrics, which are commonly used in motion deblurring tasks. In real-world datasets, where obtaining ground truth sharp sequence is challenging, we opt for the non-reference image quality assessment method Language-Image Quality Evaluator (LIQE) [37] as a reference. By assessing visual quality through (c), (d ) denote results of "LEVS", "Moiton-ETR" , "BiT" and "SpkDeblurNet" respectively. The upper panel depicts a real-world scene with a lower spike density than the simulation, whereas the lower image exhibits a higher spike density. the computation of joint probabilities from visual-textual embeddings, LIQE adeptly identifies the clarity and blurriness of images independently of the ground truth, making it ideally suited for our task.Comparison on the RSB dataset. Qualitative and quantitative experiments of the single frame restoration task on the RSB dataset are shown in Fig. 12 and Tab. 5 respectively. The visual results coupled with the LIQE metrics demonstrate that our method outperforms other methods in handling the real-world RSB dataset. While the supervised SpkDeblurNet encounters substantial noise and overexposure challenges in both low-light and high-light environments, our approach demonstrates superior restoration performance, which is attributed to the designed self-supervised framework. Moreover, as depicted in Fig. 13 and Fig. 8 , S-SDM exhibits outstanding performance in precisely retrieving luminance information and texture details while ensuring the motion consistency of reconstructed sequences. In contrast, the video reconstruction of the BiT suffers from poor image quality and shows inadequate sequence continuity in the restored sequence. Besides, SpkDeblurNet encounters issues with color distortion, brightness inconsistency, and inaccurate texture restoration owing to the domain gap between synthetic and real-world datasets. These observations further highlight the superior performance of our S-SDM in real-world scenarios.Comparison on the GOPRO dataset. We provide additional visual comparison of our method against other SOTA methods on the GOPRO dataset as shown in Fig. 14 and 15 .To augment the dataset and accelerate the training process, we randomly crop 512 × 512 image from each blurry frame, along with the 128 × 128 spike stream. We use PyTorch to build and train our S-SDM using an NVIDIA GeForce GTX 4090 GPU and AMD EPYC 7742 64-Core Processor. The training of our LDN consumes about 4 hours on the GOPRO. During the testing phase, we feed the entire image and the spike stream into the network to assess performance.We complete the training of BSN on the GOPRO dataset, employing an initial learning rate of 3e -4 and spanning 1000 epochs. The training uses the Adam optimizer with a cosine scheduler and sets the batch size to 8 for each epoch. Adopting the same settings as BSN, EDSR is trained on the blur-spike Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper was rated by the reviewers with an Accept, 2 Weak Accept, and a Borderline Accept. The reviewers raised a number of concerns, but they were all addressed by the authors and the reviewers raised their initial ratings.
There is general consensus that the paper introduces a novel approach to deblur images by exploiting the spike camera.
The AC has read all the reviews, the rebuttal, and the discussions and agrees with the reviewers that the paper should be accepted.

Reviews: review: paper_summary: paper_summary: This paper combines the RGB camera and the spike camera for image deblur. The key contributions consist a self-supervised learning framework for deblur and a real-world dataset RSB.
strength_weakness: strength_weakness: 1 The originality is marginal. The whole framework is similar to [36]. Please clarify more on the difference between this paper and [36].

2. The proposed LDN is also similar to the DCN in [36]. It would be better to see in the ablation whether LDN is better than DCN.

3. It can be observed that smaller V_th leads to better performance. What if V_th == 0.5?

4. The size and diversity of the RSB dataset is limited. It seems the RSB dataset only contains indoor scenes. Outdoor scenes with various objects are desired.
questions: questions: See the weaknesses


------------------------------------
The response addressed my concerns. I raised my score to 6.
limitations: limitations: See the weaknesses
paper_summary: paper_summary: This work proposes a spike-guided self-supervised image deblurring algorithm that combines the high spatial resolution of RGB cameras with the high temporal resolution of spike cameras to obtain sharp RGB images in real-world scenarios. The self-supervised network addresses performance degradation issues found in existing supervised spike deblurring algorithms. The spike-guided module considers noise and spatial resolution alignment between the two cameras. Both subjective and quantitative experimental results show that the proposed algorithm achieves excellent generalization performance in both real and simulated scenes, surpassing previous spike-guided deblurring algorithms.
strength_weakness: strength_weakness: 1. While many existing image-based deblurring algorithms perform adequately, the introduction of spike cameras presents challenges such as aligning the two modalities. The authors have not sufficiently compared or explained the advantages of the spike-guided approach.
2. The robustness of the model to noise lacks systematic analysis and discussion. Is this primarily influenced by the BSN Loss described in Equation 9?
questions: questions: This method primarily targets optical flow motion estimation. For more complex motion In addition to addressing the performance degradation issues of some supervised models in real-world scenarios, what other advantages does the proposed unsupervised model offer?
limitations: limitations: The authors have stated the limitations.
paper_summary: paper_summary: The work focuses on improving image sharpness from blurry inputs using spike cameras with high-motion capture rates. Addressing limitations of supervised learning in real-world scenarios, the authors introduce a self-supervised framework for spike-guided motion deblurring. Validation through extensive experiments on both real-world and synthetic datasets confirms its superior performance and generalization ability.
strength_weakness: strength_weakness: 1，I'm curious about how the order of applying BSN (Blur to Sharp Network) and EDSR (Enhanced Deep Super-Resolution) influences model performance. Specifically, I wonder if this sequence could introduce additional artifacts during image restoration.
questions: questions: see the weakness
limitations: limitations: yes
paper_summary: paper_summary: The SpikeReveal paper uses a blurry RGB image and leverages pulse data captured by a spike camera of the corresponding scene to guide the image deblurring task. In terms of network design, the authors cascade a blind-spot network for denoising, a super-resolution network, and a deblurring network, employing a self-supervised approach to accomplish the deblurring task.
strength_weakness: strength_weakness: The paper has several areas that could be improved to better achieve its stated goals.  Addressing these issues would improve the robustness and validity of the comparative analysis.
questions: questions: [Definition of t_s]
There is an issue with the definition of t_s in line 109. When the first spike occurs, there is no previous spike to reference. Therefore, this definition needs to be revised.
[Missing Ablation Study]
Table 2: The module cascading ablation on GOPRO should include with BSN without SR with LDN and without BSN with SR with LDN. This would allow us to determine which component, BSN, SR, or LDN, contributes the most to the final performance.
[Comparison with SpkDeblurNet]
In Table 1 and Figure 4, has SpkDeblurNet been retrained using the same simulation data as the authors' method? Why do the restoration results of SpkDeblurNet in Figure 4 appear noticeably darker? Additionally, why does SpkDeblurNet perform better than the authors' method at Vth=1V_{th}=1?
limitations: limitations: The authors could consider collecting paired datasets to better address the limitations of simulating pulse camera data. While the authors mention using real data, it serves only as input to the network without providing a genuine ground truth for comparison. Addressing this by capturing paired real-world data could significantly enhance the validity and applicability of their findings.

Comments: title: We greatly appreciate your thoughtful feedback.
Thanks for your valuable time and increasing the score.
Look Forward to Feedbacks
comment: We're especially grateful for the increased score and the time you've dedicated to reviewing our paper. Your insights and support are truly meaningful to us. Thank you!
Thanks for the response.
I raised my score to 6.
Thank you sincerely for your insightful comments, valuable suggestions, kind appreciation of our work and increasing the score. Thanks a lot for your valuable time! Your time and input mean a lot to us.
Thanks to the authors for their response and the insights on the model's noise robustness, as well as its advantages over other image-based deblurring algorithms. The additional explanations provided by the authors regarding the S-SDM method from the perspectives of Interpretability, Implementability, and Deployability were also very clear. I have no further questions.
I raised my score to 7.
Dear Reviewers,

Thank you sincerely for your review. We would greatly appreciate it if you could inform us of any remaining questions or concerns that you may have, so that we can address them promptly prior to the deadline. Alternatively, if you feel that your initial concerns are addressed, we would appreciate updating your evaluation to reflect that. Thank you!