SpikeGen: Decoupled “Rods and Cones” Visual Representation Processing with Latent Generative Framework
Authors: Gaole Dai
Menghang Dong
Rongyu Zhang
Ruichuan An
Tiejun Huang
Shanghang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: bio-inspired
image representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: The process through which humans perceive and learn visual representations in dynamic environments is highly complex. From a structural perspective, the human eye decouples the functions of cone and rod cells: cones are primarily responsible for color perception, while rods are specialized in detecting motion, particularly variations in light intensity. These two distinct modalities of visual information are integrated and processed within the visual cortex, thereby enhancing the robustness of the human visual system. Inspired by this biological mechanism, modern hardware systems have evolved to include not only color-sensitive RGB cameras but also motion-sensitive Dynamic Visual Systems, such as spike cameras. Building upon these advancements, this study seeks to emulate the human visual system by integrating decomposed multi-modal visual inputs with modern latent-space generative frameworks. We named it ***SpikeGen***. We evaluate its performance across various spike-RGB tasks, including conditional image and video deblurring, dense frame reconstruction from spike streams, and high-speed scene novel-view synthesis. Supported by extensive experiments, we demonstrate that leveraging the latent space manipulation capabilities of generative models enables an effective synergistic enhancement of different visual modalities, addressing spatial sparsity in spike inputs and temporal sparsity in RGB inputs.

Published as a conference paper at ICLR 2026
SPIKEGEN: DECOUPLED "RODS AND CONES" VISUAL REPRESENTATIONS PROCESSING WITH LATENT GEN- ERATIVE FRAMEWORK
Gaole Dai
Menghang Dong"
Rongyu Zhang
Ruichuan An
Tiejun Huanges rjuuang @ @ku.edu.cn
Shanghang Zhangez shanghang @ pku.edu.cn
State Key Laboratory of Multimedia Information Processing, School of Computer Science,
Peking University
ABSTRACT
The process through which humans perceive and learn visual representations dynamic environments is highly complex. From structural perspective, the human eye decouples the functions cone and rod cells: cones are primarily responsible for color perception, while rods specialized in detecting motion, particularly variations in light intensity. These two distinct modalities of visual information are integrated and processed within the visual cortex, thereby enhancing the robustness of the human visual system. Inspired by this biological mechanism, modern hardware systems have evolved to include only color-sensitive RGB cameras but also motion- seniiiivee Dynamic Visual Systems, such as spike cameras. Building upon these advancements, this study seeks emulate the human visual system by integrating decomposed multi-modal visual inputs with modern latent-space generative frameworks. We named it SpikeGen. We evaluate performance across various spike-RGB tasks, including conditional image and video deblurring, dense frame reconstruction from spike streams, and high-speed scene novel-view synthesis. Supported by extensive experiments, demonstrate that leveraging the latent space manipulation capabilities of generative models enables an effective synergistic enhancement of different visual modalities, addressing spatial sparsity in spike inputs and temporal sparsity in RGB inputs.
Human Visual System
Downstream Tasks
5-S0M Neun
(NtartPS241
SolteGs (AMIDS
Rediaa
Conas (94)
collrtino info
Vasal Certer
Conditional
Inagetet 38
20.13
28.79
SSIN
6:51
0:49
Vrdge/Vdeee
SpikkeGen Pipeline
Spike
manitdda2021
MQF
5.31
BRISQUE
35.37
35-25
Spike Correra trigh Hil Hil
Synthaa
Deliur Outdder
28.20
28:34
SSNM
0.89
Figure 1: Overview of the motivation, task setups and main quantitative results of SpikeGen.
"Equal Contribution, #CCorresponding Author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Human vision is not the best any single metric bees can be tetrachromats (Kevan al. 2001 and many flies see flicker at far higher rates than we do (Borst et al., 2010) but it excels as a balanced, tightly coupled system that fuses rich low-level cues from different modalities with high-level semantics. This balance yields (i) broad color discrimination via trichromacy with fine color resolution (Horiguchi et al.) 2013). (ii) robust perception of moving targets with preserved spatial detail through active eye movements and temporally tuned pathways (Conner 1982), and (iii) powerful inference mechanism that "fills in" missing information under extreme conditions (e.g., low light, fast motion) (Chariker et al. 2016 2018).
The quest to replicate such capabilities has driven innovations in both imaging hardware and com- putational models. In hardware, RGB cameras directly emulate the trichromatic properties of cone cells (Peterson 2016), which are concentrated in the fovea to provide high spatial and color resolution but lack temporal sensitivity (Zhang et al. 2021a). Conversely, Dynamic Visual Systems (DVS) (Han) et al. 2020) like spike cameras draw inspiration from rod cells widely distributed outside the fovea utilizing their continuous integration mechanism achieve high temporal resolution for perceiving subtle light changes, albeit with trade-offs in color and spatial resolution (Marković et al.
2020). On the algorithmic front, Artificial Neural Networks (ANNs) have emerged as powerful tools for multimodal visual fusion. For instance, infrared imaging can detect heat sources in low-light but lacks color and texture details, while visible-light imaging provides high-resolution color information but falters in darkness. Fusion algorithms integrating these modalities have demonstrated significant utility in applications like nighttime security Yuuan et al. 2024).
Notably, the RGB and spike modalities are functionally decoupled. In the human visual system, the brain acts as a "model" that integrates information from both modalities while leveraging semantic knowledge to infer missing details. Inspired by this mechanism, our approach aims to develop multimodal model that emulates key characteristics of the human visual system. Reviewing current studies, when relying solely on the RGB modality, although achieving relative improvements over low - -quality inputs, often results in inherent information loss that introduces ambiguities in the output (Chen et al.) 2024 Chang et al., 2023) Chen et al., (023a). This phenomenon is known as the "sharpness trap" where overall contrast is enhanced without meaningful geometric recovery (i.e., preservation of fine structural features) (Jiang et al., (02T). Similarly, in tasks dominated by the spike modality - such as dense frame reconstruction - the outputs are typically grayscale and exhibit limited spatial resolution (c.g.. 250x400). As a result, incorporating priors from complementary modalities can significantly improve performance. Prior work, such as the Self-supervised Spike- guided Deblurring Model (S-SDM) (Chen et al. 2024). has demonstrated that texture cues captured by spike cameras can guide models to prioritize structural accuracy over superficial sharpness. Furthermore, we argue that even blurry RGB frames may provide valuable guidance for dense frame reconstruction. Since individual frames in the raw spike stream are spatially sparse, insufficient spike activity leads to spatial uncertainty (Zhu et al. 2019 2020 Zheng et al. 2023). In contrast, RGB frames preserve global spatial relationships among scene elements, thereby serving as coarse yet effective constraint to mitigate such uncertainty. In conclusion, cross- moos--modal processing exhibits greater potential, with recent successes extending to 3D vision tasks (e.g., multi-view spike stream for 3DGS (Guo et al. 2025
Dai et al. 2024)) further validating this trend.
Nevertheless, observed significant discrepancy between current approaches - such S- SDM and the human visual system. While existing methods rely on pixel-level self-supervised learning (SSL), the human visual system acquires information through latent representations without explicit pixel reconstruction. To bridge this gap, we propose SpikeGen, a model featuring a latent generative architecture, for the following reasons: (i) Since the emergence of Latent Diffusion Models (LDM) (Rombach et al. 2022). operations in latent space have become an efficient paradigm for model training. In SpikeGien we pre-train the spike encoder to align with the latent space of RGB Variational Autoencoder (VAE) (Kingma et al., 2013), achieving an 512x spatial-temporal downsampling. This substantially reduces computational overhead during training, particularly in terms of pixel-space loss computation, thereby enabling efficient pre-training on large-scale synthetic datasets (Deng et al.) 2009). (ii) Pre-traime tre-training in the latent space also improves adaptability to downstream tasks. Although level SSL methods such as Masked Autoencoders (MAE) (He et al. (2022) have proven effective, recent advances in DINO (Caron et al. 2021: Oquab et al. 2023) and Joint- Sont-Embedding Prediction Architectures (JEPA) (Assran et al. 2023 Drozdov et al. 2024) demonstrate superior generalization
2

---

Published as a conference paper at ICLR 2026
by operating in the embedding (i.e., latent) space. A key commonality among these approaches is their focus on capturing latent-level similarities rather than minimizing pixel-wise reconstruction errors, which helps mitigate issues such as the "sharpness trap." (iii) Most prior works are based on
deterministic modeling frameworks (Chen et al. 2024 2023a Chang et al. 2023 Fan et al. 2024). In contrast, SpikeGin adopts probabilistic framework by performing diffusion in the VAE latent space. This choice is motivated by that the scenarios we address inevitably involve information loss, whether due to blurring in RGB modalities or spatial sparsity in spike modalities. Diffusion models exhibit superior performance in similar cases, such as super-resolution (Moser et al., 2024 Gao ctt al. )023) and denoising (Ho al. 2020 Kawar al., 2022), attributed to their enhanced recovery accuracy, particularly for detailed content like textures and artificial structures.
In summary, our key contributions are as follows:
1. We systematically reviewed the merits of the human visual system, identifying the func- tionally decoupled nature of the human eye and cortical processing in latent space as fundamental to this advancement. Inspired by these biological mechanisms, we propose novel processing pipeline compatible with dual RGB-spike modality, named SpikeGen.
2. To the best of our knowledge, our work represents a pioneering effort leveraging a latent-based generative model for such tasks. The rationale stems from both efficiency and effectiveness considerations: latent-space operations enable an exceptional compression ratio (512 (512-fold) during training while circumventing common issues with pixel-space loss, such as the "sharpness trap," which undermines the model's generalization capability.
3. We validate SpikeGen across multiple downstream tasks, including conditional image/video deblurring, dense frame reconstruction from spike streams, and high-speed scene novel- view synthesis. This comprehensive evaluation encompasses all major tasks in visual spike stream & RGB processing, thereby strongly supporting our hypothesis regarding the challenges inherent to these tasks and the corresponding methodological designs.
2 RELATED WORKS
Visual Spike Stream Processing Spike cameras are bio-inspired vision sensors that capture local intensity changes asynchronously, emitting spike streams. Compared to traditional frame-based based cameras, they offer advantages such as high dynamic range (HDR), low power consumption, high temporal resolution, and inherent data compression, making them ideal for high-speed dynamics and challenging lighting conditions. Early research focused on texture recovery methods, like retina- inspired sampling by Zhu et al (Zhu al. 2019). More recently, deep learning has dominated the transformation of spike data into dense images. Zhao et al. introduced Spk2ImgNet (Zhao et al. 2021 for reconstructing dynamic scenes from spike streams. Spiking Neural Networks (SNNs) (Zhang et al. 2023b) have also been applied for biologically plausible processing, with Zhao et al. developing SSIR (Zhao et al. 2023) architectures. Fan et proposed STIR (Fan et al. 2024) for spatio-temporal interactive learning to improve reconstruction efficiency. In image deblurring, Chen et al. developed SpkDeblurNet (Chen et al. 2023b)and S-SDM (Chen et al.) 2024) to enhance motion deblurring using spike information. Spike cameras are also advancing 3D vision and scene perception, as shown by Dai et al (Dai et al. 2024) (SpikeNVS) and Guo et al. (Guo et al. 2025 (Spike(S)
Latent Generation Models Generative modelling has made significant progress, enabling high- fidelity image synthesis and other complex data modalities. Diffusion models, pioneered by Denoising Diffusion Probabilistic Models (DDPMs) Ho et al. (2020). achieve state-of-the-arr results by progres- sively denoising signals from Gaussian noise. To address slow sampling speeds, Song al. [Son et al. 2020) introduced Denoising Diffusion Implicit Models (DDIMs), which offer faster generation with comparable quality. Latent Diffusion Models (LDMss) Rombach et al. (2022) operate in a learned latent space, reducing computational demands. SDXL Podell et al. (2023) further improved LDMs for high-resolution image generation. Auto-regressive models also benefit from latent representa- tions; MaskGIT Ghan et al. (2022) uses transformers for masked generative image modelling with discrete latent codes. Recent research Li et al. (2024) explores autoregressive generation directly from continuous features, eliminating the need for quantization.
3

---

Published as a conference paper at ICLR 2026
Sele-Super Learning Self-Superrised Learning (SSL) has long been prominent research topic. The success of MAE (He et a (022) and SimMIM (Xie et al. 2022 demonstrated the effectiveness of Masked Image Modeling (MIM) in pixel space, prompting recent SSL approaches to investigate MIM in latent space as more efficient and effective alternative. Image Doint-Embedding Prediction Architectures (i-JEPA) (Assran al.
2023) represent a representative effort in this
direction. Concurrently, DINO (Caron et al. 2021) explored the potential of contrastive learning (CL) in latent space by aligning semantically consistent representations of the same image under different augmentations. iBot (Zhou et al. 2 integrated the strengths of latent space modeling, MIM, and CL into a unified framework, inspiring subsequent works such as DINOv2 (Oquab et al., (2023) and DINOv3 (Siméoni et al. 2025), which have scaled up latent space SSL to the cellion-parrmeter level terms of both data volume and model capacity.
METHODS
3.1 PRELIMINARY
Visual Spike Stream The imaging principle of spike cameras diverges from both the exposure- based mechanism of conventional RGB cameras and the differential approach of event cameras. A spike camera operates by integrating incoming light intensity until an accumulator reaches a predefined activation threshold, denoted as ViA- Upon reaching this threshold, a spike is emitted, and any surplus intensity / beyond the threshold is retained for the next integration cycle. If de represents the light intensity input at time step f, the value stored in the accumulator, A, evolves according to:
A, (A T+) mod Vin
spike for pixel p at coordinates (f..)) occurs when the accumulated value plus the current input binary spike value, indicating whether the brightness level was sufficient during the sampling interval. The formal definition is:
signal meets exceeds the threshold ViA. This determines the
1, if 1 Ven
Pi.it
0. otherwise
Reconstructing dense frames from these spike streams traditionally employs methods like Texture From Interval (TFI) or Texture From Playback (TFP) (Zhu et al.) (019). TFI excels at capturing texture contours. In contrast, TFP reconstructs textures across varied dynamic ranges by adaptively modifying the time window size based on contrast levels. Let di be the temporal latency, i.e.. the time elapsed since the last spike at pixel (i,j) before time t. Let w denote the size of the time window and Ni be the total number of spikes accumulated for pixel (i. within that window. The expressions for TFI and TFP reconstructed texture P at time are:
Veh
Ni W
TFI: Pt
TFP: P+FP
* #C
dz.j.²
where C' is constant scaling factor for TFP.
Latent Space Operation Latent Diffusion Models (LDMs) (Rombach et al. 2022) are a type of latent space generative model designed for high-resolution image synthesis while being computa- tionally efficient. Instead of operating directly in the high-dimensionnl pixel space like traditional Diffusion Models, LDMs apply the diffusion process within the lower-dimensional latent space learned by a powerful pre-trained autoencoder (e.g.. VAE (Kingma et al.) (013)). This autoencoder consists of an encoder E(-) that compresses the image x into latent representation a decoder (--) that reconstructs the image from the latent space ã D(z) (Rombach et al. 2022). The diffusion model eg is then trained solely within this latent space to denoise representations 2 at various noise levels t. This separation allows the computationally expensive training and sampling of the diffusion process to occur a much more manageable space, focusing on semantic information rather than imperceptible pixel details.
E(x)) and
The training objective for the Latent Diffusion Model is given by:
LLDM e= ll|e c0( t)
Here, é the noise sampled from a normal distribution, 2, is the noisy latent representation at timestep 1, and eo (= 1) is the neural network predicting the noise added to 2,-
4

---

Published as a conference paper at ICLR 2026
Blurry RGB
Mixed
Tokens
Processed RGB
VAE Encoder
sampling
pamma
Spike Stream
spite loss
Iggammaa
Decoder
VAE
Intensity
Spike Encoder
condition
difluse
panaa generator
diflciion Soids
Taken Type
Operation Type
RGB Latent VAE(Clear)
GT Spike
Pre-train Loss
Fine-tune Loss
temporally insuficiett
SDcomi
WIS
spatially Insufficiett
weights
Inputs
Spike
Separable Spike Encoder Configurable Modality Control
Figure The Overall Pipeline of SpikeGen. SpikeGen adopts a standard pre-training (self- supervised) and fine-tuning (task-dependent) pipeline. Specifically, visual information from two modalities is encoded, followed by the addition of the two latent representations. serves parameter control the effective weight and is randomly sampled from the interval [0, 1]. Notice that during the pre-trate pre-training phase, the diffusion loss is computed using the pre-extracted latent representation of the clear RGB image obtained via Variational Autoencoder (VAE) (Kingma et al., 2013). The spike stream loss during the fine-tuning phase is calculated based on the given ground truth and the synthetic spike stream generated from the predicted RGB output.
this process, y
3.2 SPIKEGEN: CONFIGURABLE DUAL MODALITY PRE-TRAIN
SpikeGen is developed by leveraging recent advancements latent generation models. Specifically, constructed modified version of the Masked Auto- Luto-Regressive Model (MAR) (Li et al. 2024 to better suit our tasks. Similar to MAR, we encode RGB visual information via a Variational Autoencoder (VAE) (Kingma et al., 2013 and generate the conditioning input using standard Vision Transformer (ViT) (Dosovitskiy et al. 2020) backbone. Subsequently, the conditioning input processed through compact Multi-l Perceptron (MIP), conducting per-token diffusion process (Song et al. 2020) for decoding the latent representation (see Figure
Diffusion with Decomposed Latent Condition From broader view, unlike the Masked Image Modelling (MIM) (He et al. 2022) strategies employed in MAR, we consider that both blurry RGB inputs (temporally insufficient) and spike inputs (spatially insufficient) represent forms degradation. Consequently, the VIT in SpikeGen receives complete tokens from two encoders and generates conditions for the per-token diffusion. Since we do not need to predict new tokens from void, we have further streamlined the auto-regressive process of MAR into generating all tokens simultaneously. This enhances the efficiency during both training and inference (see Appendix Figure 6). We also found that this design did not affect the overall performance (see Appendix Table
Temiprall Separable Spike latent Besides the Variational Autoencoder (VAE) utilized for encoding blurry RGB information, we developed a Spatial-Temporal Separable Spike (S3) Encoder. The S3 Encoder first applies series of 3D convolutional blocks to transform the input spike stream from dimensions [B, 1, T, H, W] to [B, Cout, T/8, 8, /8, W/8] (Cout = 512). These blocks progressively reduce the spatio-temporal resolution while increasing the channel depth, akin to the hierarchical structure employed in the UNet encoder. Following the temporal fusion stage, where the [B, Cout, T/8, H/8, W/8] features processed to fuse information along the temporal dimension explicitly, we generate temporal attention weights for the features using two consecutive 1xlx1 3D convolutions. These attention weights then multiplied element wise with the features.
5

---

Published as a conference paper at ICLR 2026
The resulting weighted features subsequently summed along the temporal dimension, collapsing it and producing a feature map of shape [B, Cout. H/8, W/8]. To further refine these spatially- resolved but temporally- funse features, 2D convolution is applied, followed by a LayerNorm operation and final LeakyRel.U activation. When temporal fusion is active, the ultimate output of the S3 Encoder is feature map of shape [B, Cout, H/8, W/8] (see Figure 2 and Appendix Table
Random Modality Dropout To enable a configurable modality control after pre-training, we randomly assign the addition ratio of RGB latent (extracted by VAE encoder) and the spike latent (extracted by S3 encoder). The random ratio is denoted as 7, which is sampled from a Gaussian distribution with a mean of μl = 0.5 and a variance of o² subsequently truncated to the interval [0, 1.. This can be formally expressed as Noo.1 (11 0.5, x² = 1). The mixed latent, Emized- can then be calculated by the formula Emized = -)) + "espikee We intend to simplify the mixing strategy to enable better downstream usage. However, this indicates we cannot directly use the latent extracted from the clear RGB images as learning objectives. Instead, we colour fade the clear RGB images based on the 7 value (higher fade intensity with increasing 7). Let Letar represent the original clear RGB image and gray be its corresponding grayscale version. The fade formula to obtain the faded image faded is as Faded = (1 - 7)- Lelear + 7 Lyray This ensures that when 1 (spike latent becomes dominant), saded tends towards graa.. and when y - 0 (RGB latent is dominant), faded remains close to Lalear- This enables, when spike latent becomes the main content (ry is high), the model to focus more texture reconstruction over precise colour prediction, as the learning objective [aded will have reduced colour information (see Figure 2
3.3 SPIKEGEN: TASK ADAPTATION
During pre-training, our loss computed solely as the diffusion loss between the clear RGB latent and the latent predicted by SpikeGen. This significantly reduces computational costs when training on large-scale datasets. However, this latent alignment can underperform during fine-tuning, particularly with limited data, as it may not sufficiently guide the model to capture fine- ane-grained details. For instance, the outdoor dataset (Ma et al.) 2022) for 3D scene reconstruction contains merely 34 images per scene. A common strategy mitigate this is to introduce pixel-space similarity measures during the fine-tuning phase; for example, models like SDXL (Podell et al.) (023) incorporate both MSE and perceptual losses calculated in the RGB pixel space. However, downstream tasks for SpikeGen could lack clear RGB ground truth, we propose a spike-alignment strategy instead (see Figure 2 Specifically, the latent generated by SpikeGen is decoded back into the pixel space, yielding predicted image Ipred- Based on the intensity of Ipred, we then generate a corresponding probability map Pored- This process starts with min-max normalization and the Inorm image is convolved with Gaussian kernel KG (parameterized by o..) to produce a smoothed version smooth Finally, smmotth undergoes gamma correction, P Ppeed (Ismooth)) "*r, with small amount of uniform random noise added. Predicted spike stream can then be generated by sampling Ppred. The aake-maligmment loss is computed by comparing the ground truth spike stream with the predicted spike stream.
EXPERIMENTS
We conducted comprehensive comparisons with more than 20 state-of-the-art baselines across major visual spike & RGB processing tasks (conditional image/video deblurring, dense frame reconstruction from spike streams, and high-speed scene novel-view synthesis) demonstrate the effectiveness and versatility of SpikeGen. We color-coded the performance in Table [ 1 3 with Red (1st), Blue (2nd). All the experiments follow the data usage and evaluation strategies used in S-SDM (Chen et al., 2024), STIR (Fan et al. 2024), and SpikeGS (Guo et al.) 2025). See Appendix A for more details.
4.1 PRE-TNAINING
We performed pre-training on the complete training set of ImageNet (Deng et al., 2009) utilizing 8 A800 GPUs We carefully tuned the hyperparameters to optimize the generation of synthetic data. Given the substantial volume of available data, we adopted a more aggressive blurring configurate configuration and sparser spike stream enhance the learning challenge for the model. Specifically, applied 40 x 40 blurring kernel and random sampled 8 from 64 generated spike frames per image. Through extensive pre-training, anticipate that SpikeGen will exhibit robust generalization capabilities and

---

Published as a conference paper at ICLR 2026
Blurry (RGB)
Spike (maaked on Roa)
SpikeGen (wo Spike]
SpikeGen Wwo ROB)
SpikeGen [Spike
Clear (RGB)
Magnified
Figure 3: Conditional Image Deblurring on Synthetic RGB-Spike Data. For all experiments, the imput visual spike streams for SpikeGen are in binary format (0/1) without colour information. Here, for better visualization, row 2 (from top) of the left panel demonstrates the cut-out result of the RGB channels using 3 spike frames (results row used frames as input). We also magnified a few results with highlighted detail for better comparison of the structural correctness (right panel).
successfully achieve the configurable modality conditioning discussed in Section 3.2 In Figure 3
illustrate the generalization performance of SpikeGen the ImageNet test Overall, even when RGB images are significantly blurred and spike information is relatively sparse, SpikeGen can effectively reconstruct artificial geometric shapes of objects (column 1, right) and natural fine structures (column 2. right). Furthermore, by controlling the injection of modalities, we showcased single-modal generalization capability (rows and 4, left). When dual modalities collaborate, the model leverages their complementary strengths to achieve optimal results (row 5, left). We provided more visualization Appendix C.2
Furthermore, in terms of efficiency. we aim to evaluate whether the latent generation backbone selection SpikeGen, combined with a non-autoregressive design (see Section (.2). can reduce computational overhead. As shown in Table 7 our approach achieves a significant reduction in processing time without compromising performance.
4.2 FINE-TUNING GENERALIZATION
Given the extensive range of downstream experiments we have conducted, this section presents only the most concise and comparative results. For additional details, please refer to Appendix C
We adhered to the same experimental setups as described in S-SDM (Chen et al. (202)) for this task. S-SDM generated paired blurry RGB-spike stream data based on the GOPRO dataset (Nah et al. 2017). Each blurry RGB input was converted by the SpikingSim simulator (Zhao et al. 2022) into an average of 98 spike frames (we use 8) with a spike threshold Via during training and Va. 1/2/4 during validation. As shown in the quantitative results presented in Table I SpikeGen significantly outperformed all baselines across all Vea configurations. Specifically, the relative improvement in PSNR increased from approximately to approximately as the spike guidance became sparser (higher Vo.). We attribute this phenomenon to two primary factors. First, the pre-training to fine-tuning strategy we adopted enabled us to leverage the diversity of data encountered during self-supervised learning in latent space. Second, the random modality dropout applied during pre-training enhanced robustness when dealing with limited spike guidance. Additionally, the use of fewer spike frames at this stage further strengthened this capability (see Appendix C.3 for ablation studies using pure RGB inputs). This observation is also corroborated by the qualitative results depicted in Figure
Conditional Image/Video Deblurring
7

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative Results of Conditional Video Deblur Task. The column colour reflects
different thresholds when generating spike frames (code and value from S-SDM (Chen et al. 2024 Higher thresholds produce spike frames with higher sparsity.
Dataset: GOPRO
Nah et al.
2017
Via
Via
Via
Methods
Dual Modality
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
LEVS (CVPRI8)
0.60
Motion-ETK (TPAMI2TT
Thong ot
21.96
BIT (CVPR23) Zhong et
23.64
0.70
TRMD (TMM24) Chen Yu
27.32
21.20
0.60
18.57
RED (ICCV21) el 2021
24.46
0.74
23.18
0.67
21.94
0.61
REFID (CVPR23) (Sun et al. 2023
28.12
0.82
15.29
0.34
13.62
0.27
SpäDeblunNer (NIPS23) Chen et ml
2023b
28.31
0.83
14.41
0.30
11.62
0.20
S-SDM (NTPS24) Chen al 2024
26.89
0.76
26.37
0.74
25.43
0.70
SpikeGen (RGB&&Spike)
29.30
0.84
28.78
0.82
28.07
0.81
Table Quantitative Results of Dense Frame Reconstruction Task. The row color indicates method type, with red denoting training- free methods, yellow representing event-based methods, and blue corresponding to spike-based methods.
Dataset: SREDS
2023
Datased: Vidur202)
Methods
PSNK
SSIM f
LPIPS 1
NIQE
BRISQUE J
NIQE
BKISQUE
ZFP (ICME19)
Zha
2019
25.35
0.68
11.26
5.97
43.07
9.34
45.20
7F1 (1CME)9))
Zha ml.
2019
18.50
0.64
0.26
4.52
44.93
10,10
58.31
TF37P (CNPRI))
Zheng
64.57
E7-Ner (ICCV21)
6,51
F2VID (TIP
Frem
X124
36.37
0.95
0.05
3.13
16.77
6.306
17.02
SS/R (TCSVT25) 2hin d
32.61
0.92
0.05
3.47
15.66
5.75
25.34
Spl2leggNer (CVPR11)
WGSSE (AAAH23
16.15
STRR (NIPS24)
Fin 10.4
38.79
0.97
0.02
2.92
14.84
5.39
15.85
SplkeGow (TPPASSpio)
39.25
0.98
0.01
2.83
14.99
5:33
1599
Table 3: Quantitative Results of Novel View Synthesis Task. The column colour varies according to the types datasets.
Dataset: Blender
Objects Mildenhall et al.
2021
Out-Doon (M al.
2022
Average
Methods
Modality
LPIPS
3DGS
Kurbl
33.31
0.07
3DGS (Blur) Kerbl
2023
X
26.95
0.88
0.12
23.38
0.69
0.45
25.16
0.78
0.28
DeMlrNNRR (CVPR22) Ma 2022
23.71
0.84
0.19
28.77
0.86
26.24
0.85
0.17
DeblurGS (PACMCGT124)
Chen Liu. 2024
28.47
0.91
23.55
0.70
26.01
0.81
0.25
SpikeNeRF (ICME24)
Goo 2024
28.24
0.92
0.08
18.14
0.62
0.41
23.19
0.77
0.25
SpikeGS (AAAI25)
Guo et 2025
30.
0.93
28.00
0.83
29.12
0.88
0.13
SpikeGen (3DGS)
31.81
0.94
0.07
28.26
0.89
0.13
30.04
0.92
0.10
8

---

Published as a conference paper at ICLR 2026
Blurry RGB
Clear GT
Spike Frame
Clear
Motion-ETR (TPMN))
(CVPR23)
TFP (merge spike freams)
(AA123))
S-SDM (NIPS2A)
SpikeGen (RGB-Spike)
STIR (NIP524)
SpikeGen (TrP
Conditional Video Deblur GOPRO Dataset
Dense Frame Reconstruction SREDS Dataset
Figure 4: Mutual Guidance with Dual Modality Inputs. The top panel presents the results obtained from both RGB-based deblurring methods and spike-RGB-based approaches. SpikeGen demonstrated superior performance compared to all competitors in terms of visual fidelity. The bottom panel illustrates the outcomes various methods when only limited number spike frames (here, 16 frames) are available. SpikeGen addresses spatial ambiguity caused by spike sparsity by leveraging the merged result of spike frames (i.e., TFP) as pseudo-dense modality.
Chair
LEGO
Hot-Dog
Drums
Blam RGB
Debavos THE
Hanry Bick
Iilarr
Daloros
spike Frone
Aushind (ARAON)
Prane
clear
CWeWr RGB
clear RDB
spirisn
Clry RGR
Spieeee (0091)
Ficus
Ship
Microphone
Materials
BOB
RGc
Elury RGB
Spike
Juskeds (AAAS!)
Prane
SONSOFIAMIZ
SplicTrune
Spil
Joskeds (AARE!
cleariico
SpsheGen
Sprkedee (555)
Clear REB
SpsleGend1GS
CleerfRGS
Spikeee (002
Figure 5: Qualitative Results of Novel View Synthesis Task.
Dense Frame Reconstruction As elaborated in Section we posit that the spatial relationships embedded within dense RGB data, when blurred, address the sparsity inherent in spike streams. Consequently, validating this hypothesis on the dense frame reconstruction task holds merit. Following the experimental setups outlined in STIR (Pan et al. 2024), we utilized data from REDS (Xu et al. 2021) to synthesize spike- dataset (SREDS). Specifically, SREDS converts each RGB frame REDS into 64 corresponding spike frames (we use 16) and further crops them into patches of size 96 96, resulting in total of 21,840 patches. Additionally, the model was evaluated on the real-world dataset mom't VidarReal2021 (Zhu al., (020). As demonstrated in Table[]aand Figure SpikeGin exhibits superior performance, both in referenced metrics (PSNR, SSIM, and LPIPS /Zhang et al. 2018) and no-reference metrics (NIQE (Mittal et al. 2012b) and BRISQUE Mittal et al.) 201 2a)). A pivotal factor contributing to these results is the integration of Texture from Playback (TFP) (Zhu et al. 2019) as pseudo grayscale image. TEP aggregates all spike frames within fixed temporal window, akin simulating shutter timing in conventional RGB cameras for spike streams. This approach enhances spatial richness at the expense of temporal resolution, thereby introducing blur. SpikeGen mitigates this limitation by reprocessing the spike

---

Published as a conference paper at ICLR 2026
streams to achieve high-quality reconstructions (see Appendix C.4 for ablation studies without TFP
priors).
High-speed Scene Novel- View Synthesis Finally, evaluated SpikeGen's 's performance on the recently emerging task of high-speed scene novel-view synthesis. Our experimental setups were
based on SpikeGiS (Guo et al. 2025), which transformed the synthetic Blender datasets introduced in
NeRF (Mildenhall et al. 2021) (for object-centric scenes) and DeblurNeRF (Ma et al.) (022) (for
outdoor scenarios) into dual-modality pairs. As illustrated Figure | while DeblurGS (Chen
Liu 2024) demonstrates overall sharpness, exhibits blurry details due to its reliance solely on the
RGB modality. SpikeGS successfully recovers finer details; however, its optimization prioritizes texture repair using the binary spike stream at the expense of accurate color saturation matching to the ground truth RGB. In contrast, SpikeGen achieves an excellent qualitative balance and further validates its adaptability through quantitative results (see Table 3D. Notably, despite being two-stage method, SpikeGen outperforms other approaches, as evidenced by extra experiments comparing various two-stage methods (see Appendix C.5
4.3 CONCLUSION
We propose SpikeGen, the first latent generative framework specifically designed for decoupled visual representation processing. We successfully pre-trained the model on more than one million images and demonstrated its broad applicability across diverse tasks, covering all major domains of joint spike stream and RGB visual processing. We believe this can serve as foundation for advancing latent generative modeling in neuromorphic vision, closely mirroring the human capacity integrate perception and imagination for multimodal visual processing in dynamic environments.
ETHICS STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the computational experiments, we have provided the code in Detailed information regarding the se- lected baselines, data usage, and codebases is presented in the Appendix.
https: com/ wwwp:/wwwwweeeeeeeeeeee
ACKNOWLEDGEMENTS
G.D. was supported by the National Natural Science Foundation China (W2442028).
R.Z. supported by the National Natural Science Foundation of China (625B2090).
S.Z. was supported by the National Natural Seience Foundation of China (62476011), and by the Beijing Natural Science Foundation (L252060).
REFERENCES
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-emb predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Alexander Borst, Juergen Haag, and Dierk Reiff. Fly motion vision. Annual review of neuroscience, (3(1).99770, 2010.
10

---

Published as a conference paper at ICLR 2026
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1 11315-11323 2022.
Yakun Chang, Chu Zhou, Yuchen Hong, Liwen Hu, Chao Xu, Tiejun Huang, and Boxin Shi. 1000 fps hdr video with a spike-rgb hybrid camera. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Logan Chariker, Robert Shapley, and Lai-Sang Young. Orientation selectivity from very sparse Ign inputs in comprehensive model of macaque v1 cortex. Journal of Neuroscience, 36(49): 12368-12384, 2016.
2016.
Logan Chariker, Robert Shapley, and Lai-Sang Young. Rhythm and synchrony in cortical network model. Journal of Neuroscience, 38(40))8621-8633 2018.
Kang Chen and Lei Yu. Motion deblur by learning residual from events. IEEE Transactions on Multimedia, 26:6632-6647, 2024.
Kang Chen, Shiyan Chen, Jiyuan Zhang, Baoyue Zhang, Yajing Zheng, Tiejun Huang. and Zhaofei Yu. Spikereveal: Unlocking temporal sequences from real blurry inputs with spike streams. Advances in Neural Information Processing Systems, 37:62673-62696, 2024.
Shiyan Chen, Jiyuan Zhang, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Enhancing motion deblurring in high-speed scenes with spike streams. Advances in Neural Information Processing Systems, 01770279-70299 2023a.
Shiyan Chen, Jiyuan Zhang, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Enhancing motion deblurring in high-speed scenes with spike streams. Advances in Neural Information Processing Systems, (6.70279-70292. 2023b.
Wenbo Chen and Ligang Liu. Deblur-gs: 3d gaussian splatting from camera motion blurred images. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 7(1):1-15, 2024.
JD Conner. The temporal properties of rod vision. The Journal of Physiology, 332(():139-11)) 1982.
Gaole Dai, Zhenyu Wang, Qinwen Xu, Ming Lu, Wen Chen, Boxin Shi, Shanghang Zhang, and Tiejun Huang. Spikenvs: Enhancing novel view synthesis from blurry images via spike camera. arXiv preprint arXiv:2404.06711 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Katrina Drozdov, Ravid Shwartz-Ziv, and Yann LeCun. Video representation learning with joint- embedding predictive architectures. arXiv preprint 2=XXx:2442.10923 2024.
Burak Ercan, Onur Eker, Canberk Saglam, Aykut Erdem, and Erkut Erdem. Hypere2vid: Improving event-based video reconstruction via hypemntworks. IEEE Transactions on Image Processing, 2024.
Bin Fan, Jiaoyang Yin. Yuchao Dai, Chao Xu, Tiejun Huang, and Boxin Shi. Spatio-temporal interactive learning for efficient image reconstruction of spiking cameras. Advances in Neural Information Processing Systemx, 37:21401-21427 2024.

---

Published as a conference paper at ICLR 2026
Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10021-10030, 2023.
Yijia Guo, Yuanxi Bai, Liwen Hu, Mianzhi Liu, Ziyi Guo, Lei Ma, and Tiejun Huang. Spike- nerf: Neural radiance field based on spike camera. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2024.
Yijia Guo, Liwen Hu, Yuanxi Bai, Jiawei Yao, Lei Ma, and Tiejun Huang. Spikegs: Reconstruct 3d scene captured by fast-moving bio-inspired camera. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 3293-3301, 2025.
Jin Han, Chu Zhou, Peiqi Duan, Yehui Tang, Chang Xu, Chao Xu, Tiejun Huang, and Boxin Shi. Neuromorphic camera guided high dynamic range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1730-1739, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6857 2020.
Hiroshi Horiguchi, Jonathan Winawer, Robert Dougherty, and Brian A Wandell. Human trichromacy revisited. Proceedings of the National Academy of Sciences, 1106)):220-E220 2013.
Junjun Jiang, Chenyang Wang, Xianming Liu, and Jiayi Ma. Deep learning- baased based face super- resolution: A survey, ACM Computing Surveys (CSUR). 55(1):1-36, 2021.
Meiguang Jin, Givi Meishvili, and Paolo Favaro. Learning to extract video sequence from single motion-blurred image. Proceedings of the IEEE Conference on Computer Vision and Pattern 2018.
Recognition, pp.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 55:23593-236605 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Gruph., 42(4):139-1, 2023.
Peter G Kevan, Lars Chittka, and Adrian G Dyer. Limits to the salience of ultraviolet: lessons from colour vision bees and birds. Journal of Experimental Biology, 204( 204(14):2511-258) 2001.
Diederik P Kingma. Max Welling, et al. Auto-encoding variational bayes, 2013.
Tianhong Li. Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 06424-56445 2024.
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro Sander. Deblur- nerf: Neural radiance fields from blurry images. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12861-12870, 2022.
Danijela Marković, Alice Mizrahi, Damien Querlioz, and Julie Grollier. Physics for neuromorphic computing. Nature Reviews Physics, 2(9):499 510, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality as- sessment in the spatial domain. IEEE Transactions on image processing, (((2)):495547000 2012a.
12

---

Published as a conference paper at ICLR 2026
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):00--12,, 2012b.
Brian B Moser, Arundhati S Shanbhag, Federico Raue, Stanislav Frolov, Sebastian Palacio, and Andreas Dengel. Diffusion models, image super-resolution, and everything: A survey. IEEE Transactions on Neural Nerworks and Learning Systems, 2024.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3883-3891, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 2023.
Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6820-6829, 2019.
Bryan Peterson. Understanding exposure: how to shoot great photographs with any camera. AmPhoto books, 2016
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01953: 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 14499. 2021.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Lei Sun, Christos Sakaridis, Jingyun Liang, Peng Sun, Jiezhang Cao, Kai Zhang, Qi Jiang, Kaiwei Wang, and Luc Van Gool. Event-based frame interpolation with ad-hoc deblurring. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18043-18052, 2023.
Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Event-based video reconstruction using transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2563-2572, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653-9663, 2022.
Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion deblurring with real events. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 2583-2592 2021.
Maoxun Yuan, Xiaorong Shi, Nan Wang, Yinyan Wang, and Xingxing Wei. Improving rgb-infrared object detection with cascade alignment-guiie guided transformer. Information Fusion, 105:102246, 2024.
13

---

Published as a conference paper at ICLR 2026
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming Hsuan Yang, and Ling Shao. Mtlti-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14821-14831 2021.
Jiyuan Zhang, Shanshan Jia, Zhaofei Yu, and Tiejun Huang. Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 137-147, 2023a.
Richard Zhang. Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Youjian Zhang, Chaoyue Wang, Stephen J Maybank, and Dacheng Tao. Exposure trajectory recovery from motion blur. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11): 7490-7504, 2021a.
Youjian Zhang, Chaoyue Wang, Stephen J Maybank, and Dacheng Tao. Exposure trajectory recovery from motion blur. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11): 7490-7504, 2021b.
Yuan Zhang, Jian Cao, Jue Chen, Wenyu Sun, and Yuan Wang. Razor snn: efficient spiking neural network with temporal embeddings. In International Conference on Artificial Neural Networks, pp. 411-422. Springer, 2023b.
Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11996-12005, 2021.
Junwei Zhao, Shiliang Zhang, Lei Ma, Zhaofei Yu, and Tiejun Huang. Spikingsim: A bio-inspired spiking simulator. 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 3003-3007. IEEF, 2022.
Rui Zhao, Ruiqin Xiong, Jian Zhang, Zhaofei Yu, Shuyuan Zhu, Lei Ma, and Tiejun Huang. Spike camera image reconstruction using deep spiking neural networks. IEEE Transactions on Circuits and Systems for Video Technology, 34(6):5207-521 2023.
Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High- speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6358-6367, 2021.
Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Tiejun Huang, and Song Wang. Capture the moment: High-speed imaging with spiking cameras through short-term plasticity. IEEE Transactions Pattern Analysis and Machine Intelligence, (5(7):8127-8144 2023.
Zhihang Zhong, Mingdeng Cao, Xiang Ji, Yinqiang Zheng, and Imari Sato. Blur interpolation transformer for real-world motion from blur. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5713-5723, 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert tre-training with online tokenizer. arXiv preprint arX00:2111.07832, 2021.
Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference Multimedia and Expo (ICME), pp. 1432-1437. IEEE, 2019.
Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438-1446, 2020.
14

---

Published as a conference paper at ICLR 2026
A EXPERIMENT SETUPS
A.1 BASELINES
Conditional Video Deblurring
Jin al. (Jin et al., 2018) proposed LEVS, which tackled the extraction of video sequence from a single motion-blurred image by proposing deep learning scheme that sequentially reconstructs pairs of frames using loss functions invariant to their temporal order.
Zhang et (Zhang et al., propoeed MotionETR, focusing on recovering dense, potentially non-lineat exposure trajectories from motion-blurred image by proposing novel motion offset estimation framework to model pixel-wise displacements over time.
Zhong et al. (Zhong et al., 2023) addressed real-world motion recovery from blur (joint deblurring and interpolation) by introducing Blur Interpolation Transformer (BiT) that utilizes multi-scale residual Swin Transformer blocks and temporal supervision strategies.
Sun et al. (Sun et al., 2023) worked on event-based video frame interpolation with the
capability of ad-hoc deblurring for both sharp and blurry input frames, using a bidirectional recurrent network (REFID) that adaptively fuses image and event information.
Xu et al. (Xu et al. 2021 tackled motion deblurring using real-world event data through self sepersupere learning framework (RED-Net) that leverages event-predictts optical flow for blur and photometric consistency constraints.
Chen and Yu (Chen & Yu 2024) addressed event-based motion deblurring by proposing Ttoestage Residual-based Motion Deblurring (TRMD) framework, which first estimates an intensity residual sequence from events and then uses it to reconstruct sharp frames from the blurry image.
Chen et al. (Chen et al. 2023b) aimed at enhancing motion deblurring in high-speed scenes by proposing SpkDeblurNet, model that leverages spike streams as auxiliary cues and employs content-awarr motion magnitude attention and transposed cross-attention fusion for RGB-spike data integration.
Chen et al. (Chen al., 2024) (SpikeReveal) focused on unlocking temporal sequences of sharp images from real blurry inputs assisted by spike streams, using a self-supervised spike-guider deblurring model (S-SDM) that explores theoretical relationships between the modalities.
Dense Frame Reconstruction
Zhu et al. Zhu et al. (2019) addressed visual texture reconstruction from spike streams by proposing retina-ine -insiirdd sampling method (TFP and TFI) and associated spike stream decoding techniques.
Zheng et al. Zheng et al. 2021) tackled high-speed image reconstruction from spiking cameras by introducing novel models (TFSTP and TFMDSTP) based on the short-term plasticity (STP) mechanism of the brain.
Zhang et al. Zhang et al. (2023a) focused on learning effective temporal-ordered representa- tions for spike streams by proposing Wavelet-Guided Spike Enhancing (WGSE) paradigm that utilizes discrete wavelet transforms.
Zhao et al. Zhao et al. (202T) developed Spk2ImgNet, a spike-to- to--g neural network, learn the reconstruction of dynamic scenes from continuous spike streams generated by spiking cameras.
Zhao et al. Zhao al. 2023) proposed SSIR to addressed spike camera image reconstruction by employing deep Spiking Neural Networks (SNNs), aiming for comparable performance state-of-the-ee art methods but with lower computational costs.
Fan et al. Fan et al. (2024) tackled efficient image reconstruction for spiking cameras by proposing a spatio-temport interactive reconstruction network (STIR) that jointly performs inter-frame feature alignment and intra-frame feature filtering in coarse-to to-fine manner.
15

---

Published as a conference paper at ICLR 2026
Ercan et al. EEran al. (2024) aimed to improve event- video reconstruction by proposing HyperE2VID, a dynamic neural network architecture that uses hypemnetorrs to generate per-pixel adaptive filters guided by a context fusion module.
Weng et al. [Wng al. (2021) addressed event-based video reconstruction by proposing ET- Net, a hybrid CNN- TENN-Trasfrrmer framework designed to leverage both fine local information from CNNs and global context from Transformers.
Novel-View Synthesis
Kerbl et al. Kerbl al. (202k) introduced 3D Gaussian Splatting for real-time, high-quality radiance field rendering by representing scenes as a collection of 3D Gaussians optimized from sparse points and rasterized efficiently.
Ma al. Ma al. (2022) addressed the reconstruction Neural Radiance Fields (NeRFs) from blurry images by proposing Deblur-NeRI, which simulates the blurring process using a deformable sparse kernel module within an mualysi--b--syttenn framework.
Chen and Liu Chen & Liu (2024) tackled 3D scene reconstruction using Gaussian Splat- ting from camera motion-blurres images (Deblur-GS) by jointly optimizing 3D Gaussian parameters and the camera's 's motion trajectory during exposure time.
Guo et al. Guo et al. (2024) proposed Spike-NeRF, the first Neural Radiance Field based method for 3D reconstruction and novel view synthesis of high-speed scenes using continu- ous spike streams from a moving spike camera, incorporating spike masks and a specialized loss function.
Guo et al. Guo et al. 2025)) developed SpikeGS to reconstruct 3D scenes from Bayer-pattern spike streams captured by fast-moving bio- bio-inspired camera by integrating spike data into the 3D Gaussian Splatting pipeline using accumulation rasterization and interval supervision.
A.2 DATASETS
Pre-Training We utilize the complete training set from ImageNet (Deng et al., 2009) to synthesize spike-RGB pairs. ImageNet comprises 1,000 categories, with each category containing approximately 1,000 RGB images, resulting in total of roughly 1 million images. For each clear RGB image, we transform into blurred image and an 8-frame spike stream. The RGB blur kernel configured as 40 40, and the coverage rate of the spike stream is set to 0.1. The Dataset and Dataloader code are provided in the supplementary materials.
Conditional Video Deblurring To facilitate quantitative analysis of our spike-assisted motion deblurring performance, constructed synthetic dataset leveraging the commonly adopted GO- PRO Nah et al. 2017) dataset. The process commenced with the application of the XVFI (Sim) et al. 2021) interpolation algorithm, which served augment the video data by generating seven additional intermediate frames between every pair of successive sharp images. For the creation of spike streams that faithfully mimic real-world sensor data, the interpolated video sequences were first spatially downscaled to a 320 x 180 resolution, followed by simulation using a dedicated spike simulator (Zhao al.) 2022). To replicate the effects of real-world motion blur, each blurry input frame was then synthesized through the averaging of 97 consecutive frames from these upsampled video sequences.
Dense Frame Reconstruction Following STIR (Fan al. 2024). the SREDS dataset, recently introduced collection that was synthesized from the established REDS (Xu et al. 202T) dataset. This dataset organized into 240 distinct scenes for training and separate scenes for testing. Each scene consists of 24 sequential frames, with every frame being accompanied by a corresponding spike stream (N=20) that centrally aligned to it. The spatial resolution for this data is 1280 x 720 pixels. In preparation for training, crop each scene into non-overlapping patches of 96x96 pixels, which yields a total of 21,840 patches. To evaluate our model's effectiveness, we employ real-world captured data. This validation set includes the "mom VidarReal20211" (Zhu et al.) 2020) publicly accessible dataset, which has a resolution of 400 x 250 and features significant object and camera motion high speeds (this dataset has also been previously employed in other studies).
16

---

Published as a conference paper at ICLR 2026
Novel- View Synthesis We use the same dataset used in SpikeGS (uuo et al. 2025). It first contains the Blender dataset presented by NeRF (Mildenhall et al. (021). this dataset comprises path- aath-traced images of eight objects characterized by intricate geometry and realistic non-Lambertian materials. For six of these objects, viewpoints were sampled across the upper hemisphere, while the other two were rendered from viewpoints covering full sphere. For every scene in this dataset, 100 views are rendered to serve as input and 200 views for testing purposes, all at an 800 x 800 pixel resolution. The second one is the Blender dataset presented by DeblurNeRR (Ma al. 2022). This dataset synthesized five distinct scenes, with multi-view cameras manually positioned to simulate realistic data capture conditions. To create images with camera motion blur, it first introduced random perturbations to the camera poses and then performed linear interpolation between the original and perturbed poses for each view; images rendered from these interpolated poses were subsequently blended in linear RGB space to form the final blurry images.
A.3 METRICS
Several metrics are commonly used to evaluate the quality of images, differing in whether they require reference (FulleReference, FR) operate without one (No- Roo-Reference, NR).
Peak Signal-to-Noie Ratio (PSNR) is an FR metric that measures image fidelity based on the Mcan Squared Error (MSE) between reference image and processed image K, both of size m A higher PSNR generally indicates better reconstruction quality. It is defined as:
MSE
(T(..) - k)i
(4)
MAX?
PSNR = 10 1og
(5)
MSE
where MAX, is the maximum possible pixel value of the image (e.g., 255 for 8-bit images).
Structural Similarity Index Measure (SSIM) another FR metric designed to better align with human perception of image quality by considering changes in structural information, luminance, and contrast. For two image windows x and y, SSIM is calculated as:
(2 + ci (20 + (2) SSTM (z. v) (n3 + 12 (0) + o 02)
(6)
where Mai My are the local means, o o² are the local variances, Gny is the local covariance, and C1, C2 are small constants to stabilize the division. The overall SSIM is typically the average of local SSIM values. Higher SSIM values (closer to 1) indicate greater similarity.
Learned Perceptual Image Patch Similarity (LPIPS), proposed by Zhang et al. ZZang et al. 2018), is an FR metric that aims to better reflect human perceptual judgments by comparing deep features extracted from images using pre-trained convolutional neural networks. The distance between reference image 2o and a distorted image x is computed by summing the L2 distances of their feature activations j' ,,16 (normalized and channel-wise scaled by wr) across different layers of network:
d(z, = Σ H.W:
Σ 1 3 (Dhm - 1l
b,o0
Lower LPIPS scores indicate higher perceptual similarity.
Natural Image Quality Evaluator (NIQE), introduced by Mittal et al. (Mittal et al.) (0122)). an NR (blind) IQA metric. It operates by constructing a model based on natural scene statistics (NSS) extracted from a corpus of pristine natural images. The quality of a test image is then measured by the distance between the NSS features extracted from it and the pristine model. NIQE does not require training on human opinion scores distorted images. Lower NIQE scores suggest better perceptual quality, closer to natural image statistics. The core idea involves fitting multivariate Gaussian model to features derived from patches of natural images and then measuring the deviation of test image patch features from this model.
Reind/Referenceeess Image Spatial Quality Evaluator (BRISQUE), developed by Mittal et al. (Mit tal et al. 2012a), is another NR IQA model that operates in the spatial domain. It quantifies the loss
17

---

Published as a conference paper at ICLR 2026
of "naturalness" in image by analyzing deviations in its locally normalized luminance coefficients (specifically, Mean Subtracted Contrast Normalized - MSCN coefficients) from statistical models of natural images. Features derived from the MSCN cocfficient distributions used to train Support Vector Regressor (SVR) to predict subjective quality score. Lower BRISQUE scores generally indicate better quality.
A.4 CODE-BASSE -ASEE
Pre-Training Our code is based on the implementation of MAR (Li et al. 2024
L http:/hhb..oom//////ee.ea
Conditional Video Deblurring
Our code is based the implementation of S-SDM (Chen et al.
2024).
SDM
Dense Frame Reconstruction
Our code is based on the implementation of STIR (Fan et al. 2024).
Novel-View Synthesis Our code is based on the implementation of SpikeGS (Guo et al. 2025
18

---

Published as a conference paper at ICLR 2026
B TRAINING CONFIGURATIONS
All experiments were carried out on eight A800 GPUs, utilizing Adam as the optimizer with weight decay rate of 0.05, The training schedule was configured include warm-up phase (spanning 10% of the total epochs), during which the learning rate increased from 1e-8 to the target value, followed by cosine decay to minimum learning rate of 1e-6.
B.1 MODEL
All the results are conducted with the same model hyperparameters listed in Table
Table 4: Hyperparameters for SpikeGen model
Parameter
Value
encoder encoder_embed _eceer__meee__ee dim
768
encoder_depti
12
encoder_num_headde
12
decoder_embed_de dim
768
decoder_depth
12
decoder_rrun__eads
12
mlp_ratio
4
norm norm_layer
partial (nn. LayerNorm, eps=10-")
2048
RoPE base
10000
Table 5: Detailed Specifications of the S3 Encoder Spatial-Temporal Separable Spike Encoder)
Perfow por ducay evceriigg cemers
pail qult
Tixe epade perp-commeall doecuam satpl fecouse dirtedt tiiph
unde-- publingtt
Dokcaang B
Scced sinpoail diveaurgg digg atlaroo Ganara cxpresiic
demonve play oiteren
(A AU OIN
Caurdd
Caooaa suup ungles bi gloipe aqgar and
Teige Erspeniiionee Mabkie
ont We Dincarie
Apply i mumna mara irupural conpesil
27t 2014
2% 1A.01 (000
copieil
0.512 05.
Eaywheree here Todettl 198 (98
10.112 903)
B.2 TRAINING
We listed the hyperparameters for pre-training and fine-tuning in Table covering the necessary hyperparameters which not explicitly outlined in our codebases.
Table 6: Training Hyperparameters
Parameter
ImageNet
GOPRO
SREDS
Blender
Learning Rate
1.00 x 10-1 1.00 x 10
1.00 10
1.00 x 10-5
Epoch
10
100
150
500
Batch Size
64
32
64
32
Image Resolution
256
256
768 x 384
96 x 96
800 x 800
Spike Frame #
8
24
8
19

---

Published as a conference paper at ICLR 2026
Clear
Masked Modeling
Focus more on High-level (e.g., Text Image)
Auto- Regressive
Blurny acaa
Focus more on Low-level Tasks Deblur)
Modality
Spilee
Figure Modified SpikeGen Pipeline for Low-level Tasks
C ADDITIONAL RESULTS
C.I EFFICIENT LATENT PROCESSING WITHOUT AUTO-REGRESSION
As outlined in Section 3 our model emphasizes the integration and restoration of existing visual tokens rather than generating new tokens from void. In light of this focus, and further enhance computational efficiency, we eliminated the autoregressive component that was present in the original MAR framework. It is important to note that this architectural modification does compromise the model's performance, as demonstrated in Table
Table 7: Efficiency Comparison: SpikeGen vs. Diffusion Baselines
Dataset: ImageNet Device: 1*NVIDIA A100 Resolution: x56x 256
Backbone
time/256 images
PSNR
SSIM
LPIPS
DIF-B (100 VIT stops) x 256 tokens
2min19s
18.11
0.59
0.28
MAR-B (61 x VIT steps 100 x MLP steps) gradually to 256 tokens
1min32s
18.79
0.65
0.21
SpikeGen [1 VIT steps 100 MI.P steps) 256 tokens
2.7s
19.08
0.62
0.19
C.2 GENERALIZATION ON IMAGENET
We present extra visualization results as an extended support for the discussion in Section 4.1
C.3 CONDITIONAL VIDEO DEBLURRING
As the conclusion elaborated detail in the main text, introducing the spike modality can greatly alleviate the ambiguity of deblurring images, thus avoiding falling into the "sharpness trap" of single RGB modality deblurring. In Figure 8 we demonstrate that this property can successfully transferred to downstream tasks. We can observe that, regardless of the sparsity of the introduced spikes (controlled by VoA), the final deblurring results are superior to those relying on a single RGB modality. Moreover, as ViA decreases (indicating denser spike information), this improvement becomes more pronounced. From the visualization results, we can also see that although the overall texture is similar, introducing the spike modality can significantly enhance the restoration details.
C.4 DENSE FRAME RECONSTRUCTION
We mentioned in the main text that using the results of TFP as pseudo-dense frames can significantly improve the performance the model, and we further demonstrated this in Table When relying solely spike streams, the model has difficulty directly fusing spatially sparse binary spike streams. Especially when the number of spike frames is insufficient. However, simply increasing the number of spike frames means that the model needs to multiply the inference consumption. Therefore, using the results of TFP as pseudo-dense frames a good balancing operation. When using 1/4 of the spike frames (we use 16 frames while STTR (Fan et al. 2024 uses 64 frames), have improved
20

---

Published as a conference paper at ICLR 2026
Spike (masked on RGB)
Blurry
SpikeGen (w/o Spike]
SpikeGen (w/o RGB)
SpikeGen
(Spike & RGB)
Clear
Spike (masked
Blurry (RGB)
SpikeGen Spike]
SpikeGen (w/o RGB)
SpikeGen (Spike RGB)
Clear (RGB)
Figure Extra Experiments: In-domain Generalization for Conditional Image Deblurring
Table 8: Ablation Study: Pseudo Dense Frame Control on SpikeGen
Dataset: SREDS
(Synthetic)
Zhao et
2023
Dataset: mone Vider (Keal)
Zha
2020
Mathods
PSNK
SSIM
LPIPS
NIQE
BRISQUE 1
NIQE
BRISQUB
(ICME19)
(CME19)
7hn
Spiketien (Spike)
33.62
0.94
0.05
3.11
15.37
5.70
22.03
Spiketee (TFL&&ppike)
37.19
0.96
0.44
3.34
15.92
5.60
16.76
SpikeGen (LPP&SSpkc))
39.25
0.98
283
14.99
533
15.97
21

---

Published as a conference paper at ICLR 2026
201
-111
+0.07
0.7
2340
Viel
Whe
Weed
vhat
Weel
atoe gaia
100 onl
Spiiee
G
Frames
Blumy
ROB
SREE
G
SpikeGen
Figure Ablation Study: Modality Control on SpikeGen with GOPRO Dataset
Spike Frames
STIR
SpikeGen (Psappial
Figure 9: Ablation Study: Stress Test under Limited Spike Information.
22

---

Published as a conference paper at ICLR 2026
Table 9: Extra Experiments: Comparison Other Two-Stage Methods
Dutuset: Blender
Objects
Mildenhall et al.
2021
Out-IDoor
Ma al.
2022
Average
Methods
Modality
PSNR
SSIM
LPPPS
PSNR
SSIM
LPIPS
PSNK
SSIM
LPIPS
3DGS Kerbl
0.10
0.07
3DGS (Blur)
Korbl et al.
2023
26.95
0.88
0.12
23.38
0.69
0.45
25.16
0.78
0.28
MPR (3DGS)
27.01
0.40
0.19
YPI (SDGS) (CVPR21)
Zhong et al. 2021
29.97
0.92
0.11
26,96
0.82
0.25
29.92
0.84
0.14
(ICME19)
Zhu
0.36
0.22
KDM (3103) (CMPR 19)
Pan of al.
2019
29.45
0.92
0.11
27.88
0.79
0.32
29.73
0.84
0.12
(3DGS) (NeurIPS24)
SpikeGen (3DGS)
31.11
0.94
0.07
28.26
0.89
0.13
30,04
0.92
0.10
Chair
LEGO
Hot Dog
Drums
Ficus
Ship
Microphone
Materials
5.905.9000
S4T7 15
SOMCE
FGNR 5S0M
PSHR:31 55
PSNR SSIM
PSHR:34
Pool
Wime
Tanabata
Factory
Room
Bluny
Blurry
MAY MU
SpikeGen (3D65)
SpakeGen
MAY
Clear RGB
PSNR:31.88 SSIM:0.8880
551770.8977
SSIM:0.8921
0.8480
Figure 10: Extra Experiments: Category Evaluation of SpikeGen on Blender Dataset
the performance of SpikeGen to the state-of-the-art level, even though the focus our work not on single-modal tasks. To further verify our conclusion, we conducted a more extreme comparison. In Figure [ we further reduced the number of spike frames to In this case, STIR is unable to restore the spatial details of the object, while our method can still maintain certain level of detail restoration, such as the texture of the floor tiles.
C.5 NOVEL- VIEW SYNTHESIS
We mentioned in the main that our method is not an end-to-end model similar SpikeGS Guo ct al. (2025). However, we believe that this two-stage method also has potential advantages. For example, can better leverage pre-trained priors and decouple the complexity of optimizing texture and perceptual fidelity. Therefore, we also compared the effects of using other spike processors for two-stage reconstruction. As shown in Table although this two-stage method is feasible, not easy to achieve stable excellence. In the Object data, since the background (pure black) and objects are relatively simple, other two-stage methods also perform well. However, outdoor scenes, due to the complex background and tiny components, the performance of the other two-stage methods is greatly affected, while SpikeGen can still maintain a relatively stable performance. We further presented all the test contents on the datasets and the specific metrics of method in Figure 10
23

---

Published as a conference paper at ICLR 2026
D THE USE OF LARGE LANGUAGE MODELS (LLMS)
This paper solely employs Large Language Models to refine written content, encompassing grammar correction, tone adjustment, and formatting.
24


Reviews:
[{"review_id": "LMRED8pkvu", "reviewer": "Reviewer_146W", "summary": "This paper presents SpikeGen, a biologically inspired framework that mimics the rods–cones decoupling mechanism in human vision.\nThe model fuses spike streams (representing temporal luminance information) and RGB images (representing spatial–chromatic information) in a shared latent diffusion space, thereby achieving joint visual representation learning.", "questions": "1. How are the spike encoder’s temporal windows determined across different frame rates or datasets?  \n   Is the encoder adaptive to spike density variations?\n\n2. For the deblurring task, does SpikeGen explicitly model the exposure time or motion trajectory, or rely solely on spike event accumulation?\n\n3. How is the latent diffusion conditioned on spike features?  \n   Is it concatenation, cross-attention, or a learned fusion layer?\n\n4. How does the model generalize to **real spike data** (e.g., Vidar)?  \n   Have the authors tested domain adaptation or noise robustness?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 The paper's multimodal fusion in the latent space rather than the pixel space is elegant and computationally efficient. The approach avoids the need for spatially precise alignment between asynchronous spikes and frames.\n\n2 The paper includes quantitative and visual comparisons on multiple datasets and modalities, covering both synthetic and real-world-like settings.", "weaknesses": "(A) Task–Method Mismatch\n\nThe model’s design is static latent diffusion, but some tasks (e.g., motion deblurring) inherently require explicit temporal modeling (e.g., flow, exposure trajectory).\nSpikeGen’s temporal encoder uses only 3D convolutions, which might not capture fine-grained dynamics.\n\nIn novel-view synthesis, the model lacks geometric consistency constraints (e.g., ray-based volumetric modeling).\nThe improved perceptual quality may not correspond to true 3D structure preservation.\n\n(B) Evaluation and Fairness Issues\n\nSpikeGen is compared against single-modality baselines (e.g., SpkDeblurNet, DeblurGS) while itself using RGB+Spike dual input — an unfair comparison unless baselines are also dual-modality.\n\nThe absence of clear ablation studies (e.g., removing diffusion, removing spike input, removing γ-fusion) makes it difficult to identify the true source of performance gain.\n\nTraining details for each task (e.g., hyperparameters, dataset scale, latent dimensionality) are insufficient for reproducibility.\n\n(C) Data Authenticity and Realism\n\nMost spike data are synthetic (via SpikingSim) and do not represent real sensor noise, asynchronous pixel behavior, or refractory effects.\n\nFor each task, the paper lacks quantitative evaluation on real spike-camera datasets.\n\n(D) Overly Perfect Results / Missing Uncertainty\n\nSpikeGen outperforms all prior works on all metrics across all tasks — an unlikely scenario that raises concerns about overfitting or inconsistent training conditions.\n\nThe diffusion framework, by nature, introduces randomness and perceptual diversity; yet, the results show unrealistically consistent sharpness and color balance without variance analysis.\n\n(E) Efficiency and Practicality\n\nDespite operating in latent space, diffusion models remain computationally expensive.\nThe paper does not report inference speed or energy consumption, which are crucial in neuromorphic vision research that emphasizes efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JKF77XbvcQ", "reviewer": "Reviewer_xKds", "summary": "The paper proposes SpikeGen, a novel latent generative framework for decoupled visual representation learning between RGB and spike modalities. It performs diffusion modeling in the latent space, combining VAE-encoded representations with a per-token diffusion mechanism to balance efficiency and effectiveness. The framework incorporates a configurable dual-modality latent pre-training mechanism and a spatio-temporal separable spike encoder for efficiently extracting temporal–spatial features from spike streams. Experiments conducted on multiple benchmark datasets, including REDS, GOPRO, VidarReal, and Blender-NeRF, show that SpikeGen surpasses existing methods across multiple metrics, demonstrating superior generalization and robustness.", "questions": "1. During the pre-training phase on ImageNet, the model randomly samples only 8 frames from 64 generated spike frames. How do the authors ensure that such sparse temporal sampling can still effectively capture spike information? Have experiments with different sampling numbers (e.g., 16 or 32 frames) been conducted to verify that the model indeed utilizes spike information rather than primarily relying on RGB features?\n\n2. In Section 3.2, the authors mention that the model generates temporal attention weights through two consecutive 1×1×1 3D convolutional layers. Could the authors supplement this part by explaining how these weights are computed and applied, to clarify their role in the feature fusion process?\n\n3. The model is pre-trained on ImageNet using 8 A800 GPUs. Does the model's outstanding performance stem from the retraining advantage gained through ample computational resources, or from model innovation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces latent variables into such tasks for the first time, enabling the model to achieve higher-level feature modeling while maintaining both efficiency and effectiveness. The proposed method demonstrates strong originality, reliable and comprehensive experimental results, and clear exposition, making it an contribution to multimodal visual representation learning.", "weaknesses": "1. The training cost of SpikeGen’s latent diffusion process is relatively expensive and requires substantial pre-training resources, which limits its reproducibility and practical deployability.\n\n2. During the pre-training phase on the ImageNet dataset, the spike frame configuration involves randomly sampling 8 frames from 64 generated spike frames for each image. This setup may prevent the model from fully leveraging information from the spike modality, causing it to rely primarily on RGB inputs and thus partially degrade into a single-modality model.\n\n3. Certain details in the paper are described ambiguously. In Section 3.2 (Spatial–Temporal Separable Spike Latent), the authors mention that the model generates temporal attention weights through two consecutive 1×1×1 3D convolutional layers to model the temporal dimension explicitly. However, this process lacks formal equations or explicit computational explanations, which affects the interpretability and reproducibility of the method.\n\n4. Although the model achieves impressive results on synthetic and benchmark datasets, its validation on real-world event-based data remains limited, making it insufficient to fully demonstrate the model’s generalization capability under complex real-world conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NrvH55NmA", "reviewer": "Reviewer_QTzd", "summary": "This paper proposes a framework called SpikeGen, which takes a blurry image and the corresponding spike stream as input to generate a clear image. The authors evaluated the method on three downstream task datasets, demonstrating its effectiveness.", "questions": "1. In lines 276 to 280, the authors describe the calculation process for the faded image $I_{faded}$. However, I could not seem to find where this result is applied within the method and Figure 2.\n\n2. In line 312, the authors claim their model was pre-trained on ImageNet. However, that dataset only includes clear images. I am curious how the blurry images and the spike streams were obtained, especially the latter.\n\n3. Based on the formula in line 270, the hyperparameter $\\gamma$ takes a value of 0 or 1 with high probability (>60%). Does such a modality drop rate seem too high? This seems to indicate that the model mainly receives single-modality input during the training process.\n\n4. How are the pixel-space similarity measures performed during fine-tuning with limited data?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript is well-structured, the figures are clear, the layout is appropriate, and the language flows smoothly.\n\n2. The approach is bio-inspired and well-justified.\n\n3. The experimental evaluation is sufficient and the results demonstrated are promising.", "weaknesses": "1. The paper's citation format needs to be revised.\n\n2. In Figure 2, I believe the blurry RGB latent and the Clear RGB latent should show differences (or be visually distinct), instead of being represented by the same shape and color.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEfNwujoZ5", "reviewer": "Reviewer_CRQ3", "summary": "Mimicking the decoupling of color and light intensity perception of human into cone and rod cells, the authors proposed SpikeGen, which outperforms previous state-of-the-art methods in multiple spike-RGB tasks: 1) conditional image/video deblurring, 2) dense frame reconstruction and 3) scene novel-view sythesis. \n\nSpikeGen follows the general setup of a Masked Auto-regressive mode (MAR) pretrained with diffusion loss. It uses a frozen VAE encoder and a trainable spike encoder to encode blurry RGB input and spike stream into latent representations, respectively. \n\nSpikeGen has two training stages, a pretraining stage and a finetuning stage. During the pretraining, the loss used is the per-token diffusion loss between the faded clear RGB latent and the latent predicted by the ViT followed by a compact MLP. The spike stream loss is used during the finetuning stage.\n\nThe experiment results reported in the paper show that SpikeGen beats all other benchmarks on all three tasks.", "questions": "1. Have you tried to linearly interpolate gamma values for testing? How does gamma values affect performances?\n\n2. Please see the weaknesses section for additional questions and suggestions", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The empirical results reported by the authors on the three tasks show the superior performances of SpikeGen over all previous methods compared in the paper. The superiority of the proposed method and training pipeline is consistent across various tasks, metrics and threshold settings. \n\n2. The figures presented in the paper are clear, well-organized, and informative, especially Figure 2. It effectively and clearly conveys the overall training pipeline while also giving details on the architecture of the Spatial–Temporal Separable Spike Encoder. This visualization greatly improves the reader’s understanding of the specific modifications made to the original MAR pipeline.\n\n3. The adoption of the latent diffusion training for a dual modality spike-RGB self-supervised training method is both innovative and effective. The introduction of a random gamma parameter during training enables controllable modality dominance, allowing adaptation to different downstream tasks with minimal effort.", "weaknesses": "1. Although the authors briefly mention at line 475 that SpikeGen’s novel-view synthesis pipeline is two-stage, the main text provides insufficient explanation. It remains unclear to me 1) what the two stages are specifically (deblurring + vanilla 3DGS?),  2) how they differ from other benchmarks reported in Table 3, and 3) how time efficient (or inefficient) is SpikeGen compared to other single-stage methods. A more detailed description and comparison would greatly improve the paper’s completeness.\n\n2. Table 9 in the appendix provides comparisons with other two-stage NVS methods. However, it only includes two relatively old, training-free approaches. The authors are encouraged to include other two-stage methods, e.g. those listed in Table 2, to provide a more complete and up-to-date comparison.\n\n3. An ablation study replacing MAR with a regular conditional DiT diffusion model would provide a much stronger basis for SpikeGen’s architectural design choices. \n\n4. Similarly, an ablation on design choices of the S3 encoder would also further validate the architecture's effectiveness. Nevertheless, considering the page limit, this omission is understandable and does not substantially affect my overall evaluation.\n\n5. At lines 69–70, the phrase “By reviewing current studies” appears to be a typo or editing error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)