Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras
Authors: Bin Fan
Jiaoyang Yin
Yuchao Dai
Chao Xu
Tiejun Huang
Boxin Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking camera
Image reconstruction
High-speed motion
Spatio-temporal interaction
Coarse-to-fine
Primary area: machine_vision

Abstract: The spiking camera is an emerging neuromorphic vision sensor that records high-speed motion scenes by asynchronously firing continuous binary spike streams. Prevailing image reconstruction methods, generating intermediate frames from these spike streams, often rely on complex step-by-step network architectures that overlook the intrinsic collaboration of spatio-temporal complementary information. In this paper, we propose an efficient spatio-temporal interactive reconstruction network to jointly perform inter-frame feature alignment and intra-frame feature filtering in a coarse-to-fine manner. Specifically, it starts by extracting hierarchical features from a concise hybrid spike representation, then refines the motion fields and target frames scale-by-scale, ultimately obtaining a full-resolution output. Meanwhile, we introduce a symmetric interactive attention block and a multi-motion field estimation block to further enhance the interaction capability of the overall network. Experiments on synthetic and real-captured data show that our approach exhibits excellent performance while maintaining low model complexity.

Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras
Bin Fan Jiaoyang 2.3+ Yuchao Dai Chao Xu'
Tiejun Huang223 Boxin Sh
Nat'll Key Lab of General AI, School of Intelligence Science and Technology, Peking University State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University Nat'l Eng. Research Ctr. of Visual Technology. School of Computer Science, Peking University Sccooo of Electronics and Information, Northwestern Polytechnical University {binfan, shiboxin jjhuang) @du edu. cn, yinjiaoyangâstu. pku. edu. cn, xuchao@cis pku. edu. en, daiyuchao@nspu. edu. cn
Abstract
The spiking camera is emerging neuromorphic vision sensor that records high- speed motion scenes by asynchronously firing continuous binary spike streams. Prevailing image reconstruction methods, generating intermediate frames from these spike streams, often rely on complex step-by-step network architectures that overlook the intrinsic collaboration of spatio-temporal complementary information. In this paper, we propose an efficient spatio-tempont interactive reconstruction network to jointly perform inter-frame feature alignment and intra- frame feature filtering in a coarse-to-fine manner. Specifically, it starts by extracting hierarchical features from a concise hybrid spike representation, then refines the motion fields and target frames scale-by-scale, ultimately obtaining full-resolution output. Meanwhile, we introduce a symmetric interactive attention block and multi- motion field estimation block to further enhance the interaction capability of the overall network. Experiments on synthetic and real-captured data show that our approach exhibits excellent performance while maintaining low model complexity. The code is available at hhtps:: //tpt/ggtttt/tt/////e/t wwt./oom/////// CitCVEb/STTSRR
Introduction
High-speed imaging has become high-profile topic fields such as autonomous driving, industrial monitoring, and robotics, due to its ability to precisely capture the continuous light intensity behaviour in scene. Conventional digital cameras often rely on expensive specialized sensors when capturing fast-moving objects, the trade-off between frame rate and cost has limited the widespread adoption and further development of high-speed cameras. In recent years, neuromorphic cameras, especially event cameras [3 40 19 and spiking cameras 7 226 have emerged innovative vision sensors. They possess characteristics such as high temporal resolution, high dynamic range, and low latency, opening up new possibilities for high-speed imaging of consumer- grade cameras.
The spiking camera achieves integral sampling with 40,000Hz by emulating the central fovea's Each photoreceptive unit continuously and independently captures photons, and asynchronously fires spikes once the accumulated intensity exceeds given threshold. Unlike event cameras that only record relative changes in light intensity (i.e., differential sampling), spiking cameras have the ability to encode the absolute light intensity because the spike firing rate is proportional to the scene brightness. Consequently, the spiking camera can preserve
sampling mechanism in the retina 34 47
'Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

[a] step network architecture
39
STIRIOurs)
Hyper
WGSE [AAA1023]]
31-
(TI22224)
pike Streams
Temporal Mition
Estimation
Spabal intensityy
Reconstructon
Spk2lingee
38-
Spk2lmgNet
[b)] oint matoo-intneetyyy learning
34-
ET-Net (CCV2023)
* Params (M1
HISER
32
(TSVVT2023)
1M
10M
35M
Spate- Temporal
Ours
20
33
40
50
1.00
120
140
100
Interactive Refinement
Runtime imsi
Figure 1: Different paradigms of spike-to-image Figure 2: Model comparison of PSNR, run- time, and model size. The PSNR calculated on the SREDS dataset [57]. The runtime is tested using an RTX 3090 GPU on real-captured data [66] with spatial resolution of 400 x 250.
reconstruction. (a) Prevailing step-by-step network architecture (e.g.. Spk2ImgNet (55]). (b) Our pro- posed joint paationinneesstyy learning framework. A simple yet effective hybrid spike embedding repre- sentation (HSER) is also proposed as a link between the binary spikes and the deep model.
Our model achieves favorable results in terms of accuracy and efficiency.
more sufficient scene texture information, making it highly promising for pixel-level tasks. such 8) depth estimation [53 46). semantic segmentation [52 64],
as image reconstruction [65, 55
However, spiking cameras solely record dense binary time-sequence information, making it difficult to directly apply existing vision algorithms designed for conventional frame-based cameras. To reconstruct dynamic scene content from asynchronous spike streams, traditional methods either exploit the temporal statistical characteristics [65], e.g., texture from playback (TFP) and texture from inter-spike-intervals (TFI), or mimic the human physiological mechanisms, egg., retina-like visual imaging [66] and short-term plasticity [63 62] Nonetheless, noise and motion blur frequently present a tricky trade-off throughout the dynamic scene reconstruction process, which could lead to less than ideal reconstruction results. In contrast, deep learning-bast based methods [55] (7), with their powerful representation capabilities mine latent apatio-tempord cues from spike streams through end-to-end learning, offer more promising way to address the dynamic scene reconstruction problem of spiking cameras.
and optical flow estimation 23
59
49
Deep leariingbassed methods usually cascade three independent modules: spike embedding represen- tation (SER), temporal motion estimation, and spatial intensity recovery, as illustrated in Fig. (a). typically extracts time-series information from the spike stream, serving an essential bridge between the binary spikes and the deep model. The temporal motion estimation module either explicitly estimates the motion field [56 9 or implicitly establishes temporal motion correlations (e.g., deformable convolution 60), attention 4 5).. aiming to align context in the feature space. Following this, an additional spatial intensity recovery module [56] 4 60 added to reconstruct the intermediate frame from the aligned feature representations. Although this design paradigm of first estimating motion and then reconstructing images has achieved reasonably good results, hinders the information interaction and joint optimization time and space, creating a bottleneck for further improving the image reconstruction quality of spiking cameras. On the one hand, motion estimation and intensity recovery are inherently a "chicken-ande and-egg""" problem: more accurate motion modeling will lead to better intermediate frame reconstruction, and vice versa. On the other hand, this step-by-step combination tends to reduce inference efficiency, which is detrimental to efficient deployment in real-world applications.
The first module [56] 52
60) 59
58
49
In this paper, we point out that temporal motion estimation and spatial intensity recovery can be mutually reinforcing, as shown in Fig. [ (b). To this end, we design an efficient Interactive Reconstruction network, termed STIR. Specifically, we first deliver concise hybrid spike embedding representation (HSER) into a hierarchical feature encoder to obtain pyramid features different granularities. Then, a spatio- opatio-temporal interactive decoder is proposed to enable the joint refinement of spatio-temporal complementary information from coarse to fine. In particular, inter-frame feature alignment and intra-frame feature filtering can be performed simultaneously. The former mainly focuses on temporal motion cues to complete warping-based feature registration, while the latter progressively maintains purer image features through synthesis. addition, we integrate symmetric interactive attention block at the top-level pyramid and introduce multi-motion

---

field estimation block at the bottom-leve pyramid, further upgrading the network's s apatio-tempora interaction ability. To fully harness the network's potential, a simple yet effective HSER module is also devised, which incorporates the common advantages of explicit spike representation based on internal statisties (with better certainty and explainability) and implicit spike representation based on neural networks (with stronger expressive power). Extensive experimental results on synthetic and real-captured data demonstrate that our approach significantly outperforms state-of-the-ar (SOTA) image reconstruction methods, with a 1.35dB improvement in PSNR while also enjoying fast inference speed, as shown in Fig.
The main contributions of this paper can be summarized as follows:
1) We propose STIR, an efficient and flexible framework for image reconstruction of spiking cameras, which facilitates joint learning of complementary motion and intensity information.
2) We design symmetric interactive attention block that enhances the bilateral correlation between the intermediate frame and temporal contextual features.
3) We develop a simple yet effective hybrid spike embedding representation module with both good interpretability and strong expressive power.
2 Related Works
Neuromorphic Cameras. Neuromorphic cameras mimic neurobiological structures and functionali- ties of the retina. Different from conventional frame-based cameras, they operate asynchronously at the pixel level, allowing each pixel to act independently. Two main types of neuromorphic cameras in- clude event cameras, e.g., DVS BT), DAVIS (37), ATIS [40], CeleX [3I] and spiking cameras 26]. Event cameras utilize differential sampling approach, triggering events only when changes in illuminance surpass a specific logarithmic threshold. Conversely, spiking cameras follow an integral sampling method, where photon accumulation leads to spike firing once given threshold is reached. Therefore, event cameras produce sparser outputs, while spiking cameras provide a more regular input format for reconstructing absolute light intensity.
Event-to-image to-image Reconstruction. Deep learning- based methods for reconstructing intensity images from events have demonstrated significant progress. E2VID 41 a seminal work for this purpose by using a recurrent fully convolutional network. Following E2VID, numerous studies have augmented from various angles, including FireNet 42), E2VID++ 4S). FireNet++ 43. Also, SPADE layers and Transformer were integrated into E2VID in 2 48 which enhanced the quality but at higher cost. HyperE2VID 10 used hypernetworks to generate per-pixel adaptive filters and adopted a dynamic neural network architecture. However, since event cameras solely record changes in relative light intensity, they struggle to reconstruct the texture details of visual scenes.
Spike-to-image Reconstruction. In the task of spike-to-image reconstruction, traditional methods usually leverage the temporal statistical properties of spiking cameras. Zhu et al. [65] explored the spike generation principle and proposed two basic methods, a...a., TFP and TFI. Zhao et al. (56) hierarchically merged short- and long-term filtering. Another line work focuses on mimicking human physiological mechanisms [66] 65] 63 62
Deep learning techniques have also propelled advancements in this challenging task. Spk21mgNet (55) was the first CNN-based architecture and achieved impressive results. The wavelet transform was combined with CNN-based learnable modules in [52]. Recently, an energy-efficient scheme was developed [57] based on the spiking neural network (SNN). tat--ddnami-rrangg and high sech-frame-raa rate videos were generated in [3] by 18 [2]. Furthermore, several have also been developed to alleviate the dependence on synthetic
introducing the rolling readout mechanism [13] II 16 14 15
17
self-supervised CNNs datasets. However, due to the step-by-step paradigm, the above CNN-based architectures inevitably have higher model complexity, blocking them from mobile and real-time applications. contrast, our STIR model jointly considers temporal motion estimation and spatial intensity recovery, thus facilitating the intrinsic collaboration of spatio-temporal complementary information.
Preliminaries
3.1 Working Mechanism of the Spiking Camera
The spiking camera employs an "integrate-and-fir" mechanism. Each pixel independently and continuously receives photons from the scene and converts them into photoelectrons, which are then
3

---

(a) verall Architecture of
DenseBlock
5%
S
S
I
(b) Motion-intensity Interactive Block
Explich
Imoliett
Nottonintenssly Interactive Blact
Layertum
V
F
Conv2D
F.F.
Mubi M Motonn Ficid Estimation
K
Saftmax
Q
Bi
Actantion Map
FL,F2.F2
Modon-intens Interactive Block
P
ConvBlock
If X34
Q+
Attention Map
K
Interactive Block
R
V.
ConvBlock
Motion-intens Interactive Block
(c) Symmetric Interactive Attention Block
Motion-intes Interective Block
(ML MA
Concetenation
ConvBlock
X Multiplicaiion
16. 3t
3 Reshape
Symmetric
Warping
Interactive Attention
Downsample
G grouns of metion fields
Upsample
(d) Multi-Motion Field stimatiion Block
Figure 3: Overview of STIR framework (a) and details of the key components (b), (c), and (d). In our joint learning architecture, spatio-temporal features are refined progressively from coarse to fine, where warping-based inter-frame feature alignment (Orange line) and synthesis-based intra-frame feature filtering (Purple dashed line) are simultaneously performed in (b). Integrating (c) and (d) at the top and bottom pyramids, respectively, can boost the spatio-temporal interaction of the network.
accumulated via integrator. When the accumulated photoelectrons exceed the predetermined threshold, the spiking camera asynchronously fires a spike, while clearing the pixel's photoelectrons to start new accumulation cycle. The working mechanism can be formulated as [56 26 61]
A,(x) = -/[ aI,(x)dr mod 0,
(1)
where (x(x) reflects the number of photoelectrons accumulated pixel x (x(x) represents the light intensity at pixel x at timestamp т. The photoelectric conversion rate denoted as and the firing threshold is set to 0. Assume that (in microsecond level) is used to quantify a spike accumulation cycle, the spiking camera can output dense binary spike plane with spatial resolution of H x W at timestamp nó, N. As result, during "integrate andee-nd--iie fire" process with a temporal length of N, the spatio-temporal resolution the spike stream S.¹ will reach H W N, where denotes the central timestamp of S.N.
(x, =
in the integrator:
3.2 Problem Statement
Given a continuous binary spike stream SAN {0. Hx.13 with spatio-temporal resolution of x WA x 3N, centered at timestamp L1, similar to 5) we divide evenly into three non- overlapping spike sub-streams S S. and S. in chronological order, centered at timestamps fo. 41 and (2, respectively. This paper aims to reconstruct an intermediate intensity frame I,, corresponding to timestamp f1. Notably, under such problem setting, in order recover It, successfully, S can be utilized to model the intrinsic representation of spatial features corresponding f1, while S.N and S. can be exploited to complement the contextual information in the temporal domain.

---

4 Methodology
4.1
Overview
The overall network architecture is depicted in Fig. We first propose a hybrid spike embedding representation module in Sec. 4.2 which characterizes the three spike sub-streams S S and S. as feature maps Ft. Fe,, and Ftz corresponding to timestamps f.. f1, and 122 respectively. Subsequently, to enable spatio-temporal interactions more granularity, they are adopted to produce multi-scale pyramid features, including intermediate features {Fr ) and temporal contextual
features { ( M/ 1 through weight-sharing hierarchical feature encoder in Sec. 4.3 Here, I indicates the number of pyramid levels. Then, a compact symmetric interactive attention block is leveraged in Sec. to initially model the bilateral correlations between the top-level pyramid features S,N and {
1o J Finally, warping-based inter-frame feature alignment and synthesis- based intra-frame feature filtering are jointly executed across the spatio-temporal interactive decoder in Sec. [4.5] Therefore, progressive motion-intensity collaboration is achieved in a single encoder- decoder. Additionally, we estimate G groups of motion fields the bottom-level pyramid, which helps to improve the performance and robustness of the whole model.
4.2 Hybrid Spike Embedding Representation
The spike embedding representation is dedicated to mining time-series information from the input spike stream, serving crucial link between the spike stream and the deep network model. To encode the corresponding light intensity features from the spike stream, a simple strategy is to utilize explicit spike representation approaches based internal statistics, such as TFP (65) TFI (65) This approach builds on the temporal statistical characteristics of spiking cameras, which can physically provide good interpretability and relatively stable intensity frames for spike-to-image reconstruction tasks like video frame interpolation tasks (30 29 45). Nevertheless, this strategy often struggles to balance noise and motion blur, resulting in limited feature expression capabilities. Another more effective way is to implicitly engineer more robust features via CNNs [22 55 5
4
49 58, 60).
However, due to the lack of certainty, the spike embedding features obtained in this manner will change when the network parameters are updated during training, liritting the efficient alignment of context in the temporal motion estimation process. In summary, we hope to seek a tractable spike embedding representation method that not only offers good certainty and strong expressive capability but also maintains low computational cost.
To this end, we propose HSER to combine the advantages of explicit and implicit spike representations. Specifically, for the input spike sub-stream S.N. i {0, 1, 2). we first obtain multiple explicit spike representations using the widely-used TFP method [65] based on varying temporal windows. This is inspired by [55 because short windows give better details but bring noise, while long windows can suppress noise but easily introduce blur. At the same time, we also feed S into a ResNet [21] block for implicit modeling. Finally, these resulting features are concatenated along the channel dimension, and then the spike embedding feature Fr, is generated through 2D convolution.
4.3 Hierarchical Feature Encoder
After obtaining the spike embedding features Fio. Fr., and Fr, corresponding to the three continuous spike sub-streams, we set them as the bottom-level features F) F), and F of the feature pyramid. On this basis, a hicrarchical feature encoder is designed to build an L-level feature pyramid. such that gratulgganullarity feature representations { {Pt, _ and (4%)) are extracted from the spike streams. Note that the network parameters are shared across the three spike sub-streams. At each level of the pyramid, we use a x 2D convolution with stride of 2 for feature downsampling. followed by residual block (21). Additionally, PReLU activation 200 is appended after each 2D convolution. The number of feature channels at the l-level pyramid is C, the following, (1% ) {F), 1 and {Ft, H will facilitate inter- and intra mntra-frame interactive learning from coarse to fine.
4.4 Symmetric Interactive Attention
Recently, the transformer has demonstrated its capability to model long-range correlations between features in various visual tasks 32
53
33
39
48
To inject prior motion-intensity guidance
5

---

into the subsequent interactive decoder, we present an effective and efficient symmetric interactive attention block. It leverages the multi-head cross-attention mechanism at the top-level pyramid to symmetrically capture the mutual dependencies between F/ and { Ft.. Especially, we utilize the intermediate feature F7 as the query, while making temporal contextual features Ft and F/ as key/value separately, to ensure symmetric interaction with the query.
As illustrated in Fig. C),, we linearly project each component (i.e., query, key, and value) by applying layer normalization. In this way, the intermediate feature F is projected into two queries, Qto and Qta respectively. At the same time, the temporal contextual features FL and F2 are projected into two keys, Kt, and Ktz, as well as two values, Vt, and Vi2, respectively. The attention-based bilateral correlations can be symmetrically computed as follows:
K7.Q0 (K
Attention, -6, = Softmax
Ve, = 0,2,
(2)
where o; denotes the learnable scaling parameter used to control the magnitude of the dot product. Similar to [51] [5], we perform multi-head query-key feature interaction along the channel rather than spatial dimensions, which can effectively enhance computational efficiency due to linear com- plexity instead of quadratic. By aggregating local and non-local contexts, the final interaction feature x RR "/2"" XW/2"-1 "x3C can be yielded as:
Conv (|Attention EY (ttentionn_+ta)]
where [] denotes channel- wise concatenation operation and Conv is point-wise convolution layer Note that the intrinsic intermediate feature F? is preserved through skip connections.
4.5 Tempool Interactive Decoder
Instead of using step-by-step network architecture like [
5 60
propose to jointly and
progressively perform temporal motion estimation and spatial intensity recovery (ef-, Fig. [ (b)), thereby maximizing their complementary advantages across spatio-temporal contextual features. Specifically, we develop an efficient motion-intensity interactive block to simultaneously accomplish warping-based inter-frame feature alignment and synthesis-base intra-frame feature filtering, which is inspired by well-established event-based video frame interpolation methods [45) 44 27]. Notably, warping can integrate light intensity information over the time series, while synthesis can mitigate the influence of spike fluctuations. By cascading multiple motion-intensity interactive blocks from coarse to fine granularity, the intermediate frame can be progressively decoded. Moreover, at the bottom-level pyramid, we propose to predict multiple motion fields to aggregate more comprehensive temporal contexts, which is beneficial to further improve image reconstruction quality.
Motion-Intensity Interactive Block, The network details are depicted in Fig. 3 (a) and (b). At the top-level pyramid, x is input to the motion-intensin interactive block, which simultaneously and synthesizes the intermediate intensity frame I! using a dense block [25]. Subsequently, at the L - -1 level of the pyramid, M and Mi are upsampled backward warp the temporal contextual features 126-1, _.--111 thereby registering the spatio-temporal information around timestamps in and i2 to the intermediate timestamp tj. This process is referred to inter-frame feature alignment, where feature warping is expressed as:
estimates the motion fields Mi Mf;
pレ-1= WW ( 1M i =0
where indicates the upsampled variables, ---1 represents the warped feature candidate at the I T-1 1 level. Meanwhile, we bilinearly upsample 1f, and then concatenate it with the corresponding interme- diate feature F4-1 followed by a 2D convolution reduce the influence of spike fluctuations. This process essentially implements intra-frame feature filtering through the synthesis of the upsampled frame 11,; and the intermediate feature Fl!-!. Formally,
S Conv2D (114, +4,1))
(5)
Note that motion-based based warping is more effective in handling significant pixel displacements but is less robust occlusions. Conversely, intensity- based synthesis exhibits better robustness to occlusions and inconsistent brightness but may degrade image quality in short-time spike sub-streams. To this

---

Table 1: Quantitative comparisons against SOTA methods on the synthetic SREDS dataset [57
and real-captured dataset [66]. Best and second- best results are boldfaced and underlined, respec- tively. Thanks to the spatio-temporal interaction, our approach consistently demonstrates optimal reconstruction performance, along with excellent parameter size, GPU memory usage, and FLOPss
Params Memory FLOPPs
Synthetic Dataset
Real Dataset
Method
(M)
(G)
(T)
PSNRT SSIM T.PIPS. NIQE] BRISQUE,
NIQE,
BRISQUE]
TFP 665
25.35
0.690
0.2547
5,970
43.074
9.342
45.202
TFT [65
18.50
0.638
0.2590
4.518
44.933
10.09
58.309
TFSTP 63
20.68
0.618
0.2761
5.348
51.697
10.92
64.566
SSIR [57
0.38
10.4
0.24
32.61
0.919
0.0500
3,467
15,664
5.750
25.341
ET-Net 48
16.7
17.7
0.52
34.57
0.938
0.0535
3,400
17.155
6.512
17.393
HyperE22VID
10
10.7
6.87
0.43
36.37
0.947
0.0506
3.126
16.774
6.306
17.020
Spk2ImgNet [55
3.76
14.6
9.17
36.13
0,950
0.0294
3.084
15.348
5.662
16,518
WGSE [22]
3.85
19.7
3.93
37,44
0.958
0.0241
3.032
15,555
5.620
16,154
STIR (Ours)
5.11
9.20
0.42
38.79
0.966
0.0183
2.915
14.835
5.394
15.854
end, we further purchase dense block to merge the complementary advantages of warping-based and synthesis-basee features, i.e.,
1 M M DenseBlock ([s%- 1 tMM M
Therefore, information sharing and mutual collaboration of spatio-temporal features be achieved by progressively refining the motion-intensity interactive blocks from L-level to 1-level pyramids.
Multi-Motion Field Estimation Block. Multi-motion field estimation has been proven to be feasible strategy to improve reconstruction quality in video frame interpolation tasks [3] 24, 29]. Inspired by this, we simply amplify the output channels in the bottom-level pyramid to estimate
G' groups of motion fields [M M I g E [1, G.) Hence, G groups of warped feature candidates can be appended with diversity at full resolution, as shown in Fig. (d). This is beneficial for compensating additional details when local inaccuracies occur in single group of motion fields,
thereby enhancing spatio-temporal interaction capabilities. The analyses are detailed Sec.
4.6 Loss Function
We employ the combination of reconstruction loss Lrees perceptual loss Eper, and multi-scale consis- tency loss Luse as the total loss function L to train our network, namely,
L = Lree + AxperCrer + Linee
where we empirically set Aper to 0.2. The & distance between the final predicted image and the ground truth image is measured in Lrec i.e.
Lre
-
(8)
HW 11
We also introduce Sper to mitigate the blurry effect and preserve more details, that is,
Sper HW Voover (fth) Orzz (1,
where uugg the feature extractor of the pre-trained VGG-Net. Furthermore, we propose Luse to force the multi-scale intermediate intensity frames {7t { synthesized from the 2-level to L-level pyramids, to be consistent with the ground truth. The & distance can be formulated as follows:
Lmse HW 1)
1
Σ
1
1
1
(10)
2
1=2
Experiments
5.1 Implementation Details
Datasets. We adopt the recently released SREDS dataset [57), which is synthesized based on the REDS dataset [38], for network training. It is divided into 240 training scenes and 30 testing scenes.
7

---

Binary Sikes
TFP
TFSTP
S5IR
Hypert 2V11
Spk2ImgNet
WGSE
Ground Truth
Binary S pices
TFP
TFSTP
SSIR
ET-Net
HyperE 2V
Spk2imghet et
WGSE
Our
Figure 4: Visual comparison on synthetic 57 (top) and real-captured [6]] (bottom) data. Our method reconstructs precise boundaries of fast-moving objects with higher fidelity. Zoom in for more details.
Each scene contains 24 consecutive frames, with a corresponding spike stream of = 20 centered around each frame. The spatial resolution is 1280 720. During training, each scene is cropped non- overlappingly to 96 x 96, yielding a total of 21,840 patches. We evaluate our model using real-captured data, including: 1) The publicly available "mom VidarReal2021" [6]] and "recVidar ReaarReal2019"" [66) datasets (with 400 x 250 resolution, containing high-speed motion of objects and cameras, and also used in (62 52 2) Real spike data (1000 x 1000) collected by ourselves using a spiking camera.
Training Details. Our model is trained using the Adam optimizer (2 for 150 epochs with a batch size of 8. The initial learning rate is 0.0001 and decays by factor of 0.7 every 50 epochs. The temporal length of the input spike stream is 60, i.e., 5, ie.. L 5. Note that the reconstruction loss Lrecs perceptual loss Lper and multi-scale consistency loss Lise are used together to train our network. In our HSER module, we construct a 5-channel TFP-based explicit representation with scaling step of 4, well implicit representation, for each spike sub-stream. Thus, the number of feature channels 16, 24, 32, 64, and 96, respectively. Besides, 3 groups of motion fields are estimated the bottom-level pyramid, i.e., G = 3. Spikes and ground truth images are randomly flipped vertically as well as rotated 90", 180" 270" during training. All models are trained and tested on single NVIDIA RTX 3090 GPU.
20. The number of pyramid levels set to
11-channel ResNet-bbsed
Evaluation Metrics. We apply standard PSNR and SSIM metrics and learned perceptual metric LPIPS [54] to measure the visual quality quantitatively. Moreover, two non-reference image quality assessment metrics NIQE 36) and BRISQUE [35] are employed. higher PSNR/SSIM (f) or lower LPIPS/NIQE/BRISQUE (L) score indicates better performance.
Comparison Methods. We compare our method with following four types of baselines. 1)
Traditional methods: TFP 655 TFI 665,, and TESTP [63] 2) SNN-based: SSIR [57), designed

---

Table 3: Ablation studies the SREDS dataset [57]. Underlining indicates full model.
(a) Feature pyramid levels. The more granular
(b) Motion-intensity interaction. Combining warping-
hierarchical features can promote superior results due finer feature alignment and refinement.
based and synthesis- based features coarse-to- corrse-to-fine refine- ment can significantly improve reconstruction quality.
PSNR
SSIM
#Paras
TFLOPs
PSNR SSIM LPIPS #Paras TFLOPs
3-velel
37.99
0.960
0.832
0.377
w/o warping
37.74 0.960
0.023 4.872
0.263
4-1evel
38.02
0.962
2.119
0,403
w/o synthesis
12.19 0.414 0.685
5.088
0.416
5-Jevel
38,79
0.966
5.107
0.419
Full model
38.79
0.966 0.018
5.107 0.419
(c) Symmetric interactive attention. Removing it or re- placing with independent cross cross- attention mechanisms both result lower reconstruction accuracy.
(d) Multi-motion field estimation. We investigate different groups of motion ficlds. More motion field
favours compensation for additional image details.
PSNR
SSIM
#Paras
TFLOPs
PSNR
SSIM
LPIPS
#Paras
TFLOPs
Removing 38.09
0.962
4.967
0.4186
37.89
0.961
0.0211
4.944
0.273
Independent
38.23
0.963
5.318
0.4218
38.79
0.966
0.0183
5.107
0.419
Interactive
38.79
0.966
5.107
0.4194
38.37
0.964
0.0207
5.366
0.654
(e) Model capacity. xN denotes the width multiplier for the feature channel. Our method offers good flexibility
(I) Loss function. Using the full loss term greatly contributes to the best results. Lee loss is crucial to training an effective spike-to-image model.
and the performance better with larger model capacity.
PSNR
SSIM
#Paras
TFLOPs
PSNR
SSIM
LPIPS
x0.5
38.09
0.962
1.554
0.325
w/o Lie w/o Lper
36.43
0.957
0,0227
x1.0
38.79
0.966
5.107
0.419
37.3
0.954
0,0446
X x1.5
38.80
0.967
10.94
0.568
w/o Luse
38.54
0.964
0.0194
x2.0
38,86
0.967
19.06
0.770
Total loss
38.79
0.966
0.0183
for energy-efficient ppike-to-imagee reconstruction. Note that, except for SSIR, the other comparison methods adopt CNNs. 3) Event-based: ET-Net [48] and HyperE2VID [10), where proposed HSER is cascaded with the classical event-to-image reconstruction architectures. 4) CNN-based: Spk2ImgNet (55) and WGSE [52] which are SOTA spike-to-image reconstruction methods.
5.2 Comparison with SOTA Methods
As shown in T our approach significantly outperforms SOTA methods in terms reconstruction accuracy on both synthetic and real datasets. Apart from SNN-based SSIR (57], which has limited performance despite being computationally efficient, our method enjoys the lowest model complexity and competitive model size. CNN-based spike-to-image methods incur high inference costs due to their step-by-step paradigm. Notably, on average, our model 11x x faster than Spk2ImgNet (55) and 5x faster than WGSE [52). Moreover, event-based architectures have limited adaptability. Traditional methods show unsatisfactory reconstruction quality due to restricted modeling power.
The qualitative results are presented in Fig. We can see that our method produces perceptually more pleasing and higher-fideltty images, especially for the edges of fast- fast-moving objects. For instance, in cases involving pedestrians, drones, and small balls, our method achieves sharper edges, less noise, and fewer blurring and aliasing artifacts. Note that our method also ensures fast reconstruction of intensity frames (cf., Fig. 2). which further highlights its potential for practical applications.
5.3 Ablation Studies To verify the effectiveness of the proposed method, we conduct series of ablation studies from the perspective of network architecture and loss function on the SREDS dataset [57]
Ablation on Spike Embedding Represen- tation. We implement various spike em- bedding representation methods, including explicit, implicit, and combined. The multi- dilated representation [49] stacks multi- ple dilated convolutions for larger recep- tive field, while the hierarchical spatial- temporal (HiST) representation [58) inte- grates multi-scale 3D convolutions for fea- ture fusion, both of which have been applied for optical flow estimation. As shown in Table 2 despite
Table 2: Ablation on spike embedding representation.
Explicat TFP
Implicit
Muli-dilmted
HIST
ReaNet
PSNR
SSIMI
LPIPSI
36.39
0.951
36.71
0.953
0,0305
38.01
0.962
0.0201
37.60
0.959
0.0225
06
0.962
0.02116
38.79
0.966
0.0183

---

proves to be a relatively more workable spike representation method. Our HSER organically combines TFP and ResNet, which takes full advantage of both explicit and implicit representations, thus achieving the best result. Note that, even using only the simplest TFP [65), our method demonstrates competitive performance, which also demonstrates the effectiveness of our spatio-temporal interactive learning architecture.
being straightforward, ResNet 21
Ablation on Feature Pyramid Level. We investigate the influence of varying hierarchical features. As shown in Table 3a even with just a 3-1evel pyramid, our method significantly outperforms existing CNN-based methods in terms of model size and computational efficiency, while still guaranteeing leading performance. As the pyramid level increases, will introduce finer-grained spatio-temport interaction, which is conducive to achieving better image reconstruction quality.
removing the warping- based inter-frame feature alignment results in sub-optimal performance, indicating that temporal contextual information is beneficial for intermediate frame recovery. Notably, the overall performance is severely degraded when the synthesis- -assed intra-feature filtering is removed, demonstrating the essential role of intermediate features in reconstructing the target frame. When the motion- inteeniit interaction is performed simultaneously from coarse to fine, a superior performance is obtained.
Ablation on Hotion-Intensity Interactive Block. As reported in Table
Ablation on Symmetric Interactive Attention Block. We either feed the top-level pyramid features directly into the decoder or use a standard cross-attention beforehand that independently models unilateral feature correlations. Due the symmetric bilateral feature interaction, which facilitates more accurate context awareness, our method achieves superior performance, as shown in Table 3c] Also, our interactive attention has small parameters and FLOPs, ensuring lightweight network design.
Ablation Multi-Motion Field Estimation Block. We propose estimating multiple motion fields at the bottom-level pyramid to compensate for more contextual details. As shown in Table 33 using multiple groups of motion fields yields higher reconstruction quality, consistent with [0]24]. As the number of motion fields increases, the model exhibits minor performance fluctuations. Still, it achieves gains over models based on a single group of motion fields.
A Ablation on Model Capacity, We apply a width multiplier (22] to the feature channels based on
presents that increasing the model capacity has a positive effect, indicating that our architecture is highly flexible. Particularly, the parameter size and computational cost of x0.5 are ahead of SOTA methods, and it also has commendable reconstruction capability.
the current configuration.
A Ablation on Loss Function. We evaluate the impact of different loss terms in Table 3i is evident that our total loss function is effective, as it performs the best when all loss terms are included.
6 Conclusion
In this paper, we proposed an efficient spike-to-image reconstruction method based on spatio-temporal interactive learning. In particular, joint motion-intensit learning architecture was designed to perform inter-frame feature alignment and intra-frame feature filtering progressively. Moreover, we introduced symmetric interactive attention block and a multi-motion field estimation block for bilateral correlation modeling and context detail compensation. Extensive experiments on synthetic and real data have demonstrated that our approach has excellent performance in reconstruction quality and inference speed, while also enjoying good flexibility and applicability.
Limitations, Our proposed HSER, similar to other spike embedding representation methods [56] 52 60 59 58 49), implicitly assumes that the scene has sufficient illumination, such that the image reconstruction can be achieved based on spike streams with fixed temporal length. However, in extremely low-light scenarios, the limited accumulated light intensity within a fixed temporal length results in darker reconstructed images and increased noise, adversely affecting the visual experience. Note that this is a common problem for current spike-to-image reconstruction methods [56] 52 65 We plan to extend our model to handle these issues in future work.
63 [5].
Acknowledgments. This work was supported by National Science and Technology Major Project (20212D0199803), Beijing Natural Science Foundation (L233024), Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park (7241 10000352401)), and National Natural Science Foundation of China (620881 102088102, 621 62136001, 62276007, 62401021). Bin Fan was also supported by China National Postdoctoral Program for Innovative Talents (BX20230013) and China Postdoctoral Science Foundation (2024M750101).
10

---

References
[1] Christian Brandli, Raphael Berner. Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240x 180 130 db jss lateney global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits (JSSC), (9(10):2333-234) 2014.
[2] Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang. SPADE-E2VID: Spatially-adaptive denormalization for event-hased video reconstruction. IEFE Transactions on Image Processing (TIP), 30:2488-2500, 2021.
[3] Yakun Chang, Yeliduosi Xiaokaiti, Yujia Liu, Bin Fan, Zhaojun Huang, Tiejun Huang. and Boxin Shi. Towards HDR and HFR video from rolling-mixed-bit spikings. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[4] Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI), page 2859-2866, 2022.
[5] Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI). pages 350-358, 2023.
[6] Shiyan Chen, Jiyuan Zhang, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Enhancing motion deblurring in high-speed scenes with spike streams. Proceedings of the Advances Neural Information Processing Systems (NeurIPS), 36, 2023.
Siwei Dong, Tiejun Huang, and Yonghong Tian. Spike camera and its coding methods. In Proceedings of the Data Compression Conference (DCC), pages 437-437, 2017.
[8] Yanchen Dong, Ruiqin Xiong, Jing Zhao, Jian Zhang, Xiaopeng Fan, Shuyuan Zhu, and Tiejun Huang. Joint demosaicing and denoising for spike camera. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 1582-1590, 2024.
[9] Yanchen Dong, Jing Zhao, Ruiqin Xiong, and Tiejun Huang. High-speed scene reconstruction from low-light spike streams. In Proceedings of the IEEE International Conference on Visual Communications and Image Processing (VCIP), pages 1 1-5, IEEE, 2022.
[10] Burak Erean, Onur Eker, Canberk Saglam, Aykut Erdem, and Erkut Erdem. HyperE2VID: Improving event-based video reconstruction via hypernetworks. IEEE Transactions on Image Processing (TIP), 2024.
[11] Bin Fan and Yuchao Dai. Inverting a rolling shutter camera: Bring rolling shutter images to high framerate global shutter video. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4228 4237, 2021.
[12] Bin Fan, Yuchao Dai, and Mingyi He. SUNet: Symmetric undistortion network for rolling shutter correction. In Proceedings the IEEE International Conference Computer Vision (ICCV), pages 4541-4550, 2021.
[13] Bin Fan, Yuchao Dai, and Mingyi He. Rolling shutter camera: Modeling, optimization and learning. Machine Intelligence Research, 20(6):733-798, 2023.
[14] Bin Fan, Yuchao Dai, and Hongdong Li. Rolling shutter inversion: Bring rolling shutter images to high framerate global shutter video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):614-6256) 2022.
[15] Bin Fan, Yuchao Dai, and Hongdong Li. Learning bilateral cost volume for rolling shutter temporal super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):382-3379,, 2024.
[16] Bin Fan, Yuchao Dai, Zhiyuan Zhang, Qi Liu, and Mingyi He. Context-aware video reconstruction for rolling shutter cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17572- 17582, 2022.
[17] Bin Fan, Ying Guo, Yuchao Dai, Chao Xu, and Boxin Shi. Self-supervised learning for rolling shutter temporal super-resolution. IEEE Transactions Circuits and Systems for Video Technology, 2024.
[18] Bin Fan, Yuxin Mao, Yuchao Dai, Zhexiong Wan, and Qi Liu. Joint appearance and motion learning for efficient rolling shutter conrection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition ( (CVPR), pages 5671-5681, 2023.

---

[19] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Couradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence (T-PAMI). 44(0):15--80, 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the I.F. International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[22] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint Toxiv:7704.04880 2017.
[23] Liwen Hu, Rui Zhao, Ziluo Ding, Lei Ma, Boxin Shi. Ruiqin Xiong, and Tiejun Huang. Optical flow estimation for spiking camera. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 17844-17853, 2022.
[24] Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko. Many-to-many splatting for efficient video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3553-3562, 2022.
[25] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4700-4708, 2017.
[26] Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, Jiaming Li, Shanshan Jia, Yihua Fu, Boxin Shi, Si Wu, and Yonghong Tian. 1000 faster camera and machine vision with ordinary devices. Engineering, 25:1 111110119,, 119, 2022.
[27] Taewoo Kim, Yujeong Chae, Hyun-Kurl Jang, and Kuk-Jin Yoon. Event-based video frame interpolation with cross-modal asymmetric bidirectional motion fields. In Proceedings of the IEFE Conference on Computer Vision and Puttern Recognition (CVPR), pages 18032-18042, 2023.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[29] Li, Ye Zhu, Ruoteng Li, Xintao Wang. Yue Luo, and Ying Shan. Hybrid warping fusion for video frame interpolation. International Journal of Computer Vision (IJCV), (30(12)22980- 2990029999299 2022.
[30] Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. AMT: All All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9801-9818 10, 2023.
[31] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128x 128 120 db 15 latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits (JSSC), 43(2):666-576, 2008.
[32] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10002, 2021.
[33] Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3532- 3542, 2022.
[34] Richard H Masland. The neuronal organization of the retina. Neuron, 76(2):266-280 2012.
[35] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing (TIP), (122):4695555 2012.
[36] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters (SPL), 20(3):209-22) 2012.
[37] Diederik Paul Moeys. Federico Corradi, Chenghan Li, Simeon A Bamford, Luca Longinotti, Fabian F Voigt, Stewart Berry, Gemma Taverni, Fritjof Helmchen, and Tobi Delbruck. A sensitive dynamic and active pixel vision sensor for color or neural imaging applications. IEEE Transactions on Biomedical Circuits and Systems, 12(1):123 106.,11622200 2017.
12

---

[38] Seungjun Nah. Sungyong Baik, Seokil Hong. Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. NTIRE 2019 challenge video deblurring and super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pages 1996-2005 2019.
[39] Junheum Park, Jintae Kim, and Chang-Su Kim. BiFormer: Learning bilateral motion estimation via bilateral transformer for 4K video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1568-1577, 2023.
[40] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt. A QVGA 143 db dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS. IEEE Journal of Solid-State Circuits (JSSC), 46(1):259 275, 2010.
[41] Henri Rebeeq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modemn computer vision to event cameras. In Proceedings the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3857-3866, 2019.
[42] Cedric Scheerlinck, Henri Rehecq, Daniel Gehrig, Nick Barnes, Rohert Mahony, and Davide Scaramuzza. Fast image reconstruction with an event camera. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 156-163, 2020.
[43] Timo Stoffregen, Cedric Schecrlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Klee- man, and Robert Mahony. Reducing the sim-to-real gap for event cameras, In Proceedings of the European Conference on Computer Vision (ECCV), pages 534-549. Springer, 2020.
[44] Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios Georgoulis, Yuanyou Li, and Davide Scaramuzza. Time Lens++: Event-based frame interpolation with paramctric non-lincar flow and multi- scale fusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17755-17764, 2022.
[45] Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis. Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza. Time Lens: Event-based video frame interpolation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16155-16164, 2021.
[46] Yixuan Wang, Jianing Li, Lin Zhu, Xijie Xiang. Tiejun Huang, and Yonghong Tian. Learning stereo depth estimation with bio-inspired spike cameras. In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), pages 1 6. IEEE, 2022.
[47] Heinz Wässle. Parallel processing in the mammalian retina. Nature Reviews Neuroscience, Neunoxiinne,,10),77777555 2004.
[48] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Event-based video reconstruction using transformer. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2563 2572, 2021.
[49] Lujie Xia, Ziluo Ding, Rui Zhao, Jiyuan Zhang, Lei Ma, Zhaofei Yu, Tiejun Huang. and Ruiqin Xiong. Unsupervised optical flow estimation with dynamic timing representation for spike camera. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
[50] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. MANIQA: Multi-dimennson attention network for no-reference image quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1191-1200, 2022.
[51] Syed Waqus Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5728-5739, 2022.
[52] Jiyuan Zhang, Shanshan Jia, Zhaofei Yu, and Tiejun Huang. Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 137-147, 2023.
[53] Jiyuan Zhang. Lulu Tang, Zhaofei Yu. Jiwen Lu, and Tiejun Huang. Spike transformer: Monocular depth estimation for spiking camera. Proceedings of the European Conference on Computer Vision (ECCV), pages 34-52. Springer, 2022.
[54] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595 2018
13

---

[55] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2lmgNet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 11996-12005, 2021.
[56] Jing Zhao, Ruiqin Xiong, Jiyu Xie, Boxin Shi, Zhaofei Yu. Wen Gao, and Tiejun Huang. Reconstructing clear image high-speed motion scene with a iiina-inspired spike camera. IEEE Transactions on Computational Imaging (TCI), 8:12 2.1 2021.
[57] Rui Zhao, Ruiqin Xiong. Jian Zhang, Zhaofei Yu, Shuyuan Zhu, Lei Ma, and Tiejun Huang. Spike camera image reconstruction using deep spiking neural networks. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT). 2023.
[58] Rui Zhao, Ruiqin Xiong, Jian Zhang, Xinfeng Zhang, Zhaofei Yu, and Tiejun Huang. Optical flow for spike camera with hierarchical apatial-tempon spike fusion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages
2024.
[59] Rui Zhao, Ruigin Xiong, Jing Zhao, Zhaofei Yu, Xiaopeng Fan, and Tiejun Huang. Learning optical flow from continuous spike streams. Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 3577905-7920, 2022.
[60] Rui Zhao, Ruiqin Xiong. Jing Zhao, Jian Zhang, Xiaopeng Fan, Zhaofei Yu, and Tiejun Huang. Boosting spike camera image reconstruction from a perspective of dealing with spike fluctuations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[61] Yajing Zheng, Jiyuan Zhang, Rui Zhao, Jianhao Ding, Shiyan Chen, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang. SpikeCV: Open a continuous computer vision era. arXiv preprint ar Xiv:2303. 1684, 2023.
[62] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Tiejun Huang, and Song Wang. Capture the moment: High- speed imaging with spiking cameras through short-term plasticity. IEFE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI),
2023.
[63] Yajing Zheng, Lingxiao Zheng, Zhaofci Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6358-6367, 2021.
[64] Lin Zhu, Xianzhang Chen, Xiao Wang, and Hua Huang. Finding visual saliency in continuous spike stream. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), pages 7757-7765, 2024.
[65] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In Proceedings of the IEFF International Conference on Multimedia & Expo (ICME), pages 1432-1437, 2019.
[66] Lin Zhu, Siwci Dong, Jianing Li, Tiejun Huang. and Yonghong Tian. Retina-like visual image reconstruc- tion via spiking neural model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1438-1444 1448,146,200 2020.
14

---

A More Visual Results
We additionally present more spike-to-image reconstruction results. Figs. 5, 6 and 8 show qualitative comparisons on "mom VidarRea/2021" [61]. "SREDS" [57]. and "rec VidarReal2019 [66] datasets, respectively. It can be observed that compared to the baseline methods, our approach consistently and robustly produces the most satisfactory image reconstruction results, with fewer aliasing artifacts and clearer object outlines, e.g., the intricate structures of distant buildings, the tight and dense keycaps of keyboard, and the steel cables and railings of the bridge shot from a high-speed train (350km/h). etc. In particular, the image reconstruction results on our real-captured spike data are exemplified in Fig. | One can see that our approach also effectively recovers finer and more accurate image details with less noise, providing better visual experience. These experiments also fully validate the excellent generalization ability of our proposed method.
B Further Analysis of Hybrid Spike Embedding Representation
In Table 2 of the main manuscript, have demonstrated the simplicity and effectiveness of the proposed hybrid spike embedding representation (HSER). Here, we further validate the superiority of HSER by integrating different spike embedding representations into HyperE2VID [T0)
(i.e., replacing the event voxel grid with our HSER). Note that MlypeEE2VID [TO] is a recently proposed SOTA event-to-image reconstruction method. As reported Table 4 using the explicit TFP cS alone yields seemingly feasible result. Notably, in the -ase- implicit spike embedding representations (e.g., [49 (8)) ResNet [21 remains more effective strategy even as a regular tool. In contrast, when the vanilla TFP is organically combined with ResNet (i.e., our HSER), the best reconstruction accuracy is achieved. This is mainly because our HSER efficiently merges the certainty of explicit representation with the strong expressive power of implicit representation. Note that our HSER also ensures faster inference speed for the overall network. Additionally, our HSER can better adapt the event-to-image reconstruction model to spike-to-image reconstruction, which is good indication that the key reconstruction module transfcrable by adjusting the frontmost embedding representation. In conclusion, HSER is concise and effective spike embedding representation paradigm, which can be used to further enhance the performance of off-the-shelf image reconstruction methods.
C Visualization of Ablation Results
In To network architecture and loss function. We recognize that the primary innovation of our joint learning architecture lies in the interactive and joint perspective to handle temporal and spatial information simultaneously. It is this holistic approach that sets our method apart and yields superior performance, as opposed to depending solely on nigh-performing individual components. We also understand that our model is handy to scale to fit into diverse scenarios. For instance, in scenarios that demand high precision and have abundant computational resources, larger model is preferred. Conversely, for mobile or real-time applications, a simpler model can be employed.
of the main manuscript, we have quantitatively evaluated the effectiveness of the proposed
Here, we further visually present more ablation results based on the Vica dataset 666, as shown in Fig. 7 can be seen that removing the warping-basee inter-frame feature alignment as well as estimating only single group of motion fields at the bottom-level pyramid hinders the complementary exploitation objects, resulting in unpleasant visual artifacts. Note that removing the synthetic-based intra-frame feature filtering leads to catastrophic failure of the model. Furthermore, removing the symmetric interactive attention block impedes the construction of bilateral correlations, which adversely affects the subsequent joint motion motion-intensity refinement. Lastly, removing the reconstruction loss Leee is also detrimental to the reconstruction of high- images. In contrast, thanks to joint learning architecture design, our full model can promote more efficient spatio-tempon interaction, thereby reconstructing higher-fidelity intermediate images.
temporal contextual information, especially for fast-moving
D Effectiveness Validation of High Frame Rate Video Reconstruction
When inputting continuous spike stream, our approach is capable of reconstructing consecutive video frames sequentially. In Fig. I0, we further illustrate the high frame rate video reconstruction
15

---

Table 4: Quantitative comparisons of HyperE2VID [00 under different spike embedding repre- sentations on the SREDS dataset [57] The runtime is tested using single RTX 3090 GPU on
real-captured data 666 with a spatial resolution of 400 250, similar to Fig. in the main manuseript.
Explicit
Implicit
PSNRT
SSIM
I.PIPS
Runtime (ms)
TFP 65
Multi-dilated
49
HIST 58
ResNet 21
31.48
0.866
0.1523
6.14
35.62
0.940
0.0611
7.56
34.92
0,938
0.0616
14.6
36.18
0.946
0.0520
6.47
35.58
0.937
0.0692
8.19
35.19
0,940
0.0599
14.9
36.37
0.947
0.0506
6.84
Binary Spicss
TFP
TFI
TESTP
SSIR
ET-Net
HyperE2VID
5ok2ImaN
WGSE
Ours
TFSTP
SS
et
HyperE
5k22lnngiee
WGSE
Ours
Figure 5: More qualitative comparison on the real-captured "mom VidarReal2021" dataset [61]. Our reconstructed images exhibit sharper and clearer edge detail on objects like keyboards and footballs. Please zoom in for more details.
results based on our real-collected spike data, including filming rapidly spinning fan well as recording the instantaneous process when water balloon bursts. It evident that our approach can generate smooth and consistent consecutive image sequences faithfully and accurately, in which rich image content is restored, such as fan leaves and water splashes, demonstrating its effectiveness in practical applications, such capturing remarkable high-speed motion moments.
E
Broader Impact
The method proposed herein provides efficient solution for dynamic scene reconstruction by leveraging continuous spike streams with high temporal resolution. Our proposed technique may hold potential benefits for variety of real-world applications and users, especially in scenarios involving high-speed motion. As the goal of spike-to-image reconstruction to reproduce real scene details, our method may not pose negative ethical implications if we do not discuss specific scene content.
16

---

Binary Spikes
TFP
TFSTP
SSIR
ET Ne-Net
HyperEzvID
Spk2lmgNet
WGSE
Our
Ground Truth
Spikes
TFP
TFSTP
5SIR
ET-Net
HyperE 2VID
Spk2imghiet
WGSE
Our
Grundd Truth
Figure 6: More qualitative comparison on the synthetic "SREDS" dataset Our method shows excellent reconstruction results for both the complex structures of distant buildings and the backpacks of nearby pedestrians. Please zoom in for more details.
C
Interactive attention
L.se
warping
Single-motion fleld
model
w/o Interactive attention
w/o
w/l warping
Single-motion
model
Visualization of ablation results based on the "rec Vidaareeee/2 dataset [66], From left to right, show the removal of symmetric interactive attention block, the removal of reconstruction loss L the removal warping- inter-frame feature alignment, and the estimation of single- motion field. Our full model reconstructs higher-fidelity images with fewer artifacts, Please zoom in for more details.
Figure 7:
17

---

Binary Spices
TFSTP
HVDD
5pi2linghet
Sinary Spices
TFP
TFSTP
SSIR
Sok2lmnNeet
WGSE
Binary's S
TFSTP
ET-Net
HyperE 2VID
5ok2imaket
WGSE
Ours
Figure 8: Qualitative comparison on the real-captured "rec Vidarreeeal2 dataset [66] Whether
photographing high-speed rotating fan (2600 rpm) and fast-moving car (100 km/h), or shooting from high-speed train (350 km/h), our method recovers more image details and more accurate structure. Please zoom in for more details.
18

---

Binary Splas
TFP
TFI
TFSTP
SSIR
R
x
3
4
33
3
33
x
R4
7
8
2
ET-Net
HyperE2VID 2VID
Spk21mgNet
WGSE
Dur
Binary Spices
TFSTP
ET.Net
HyperE2VID
Sak2lmgNeet
WG5E
nary Spikes
SSIR
ET-Net
HyperEZVID
Spk2imgNet
WGSE
Ours
Figure Qualitative comparison on our real-captured spike data. Our method can suppress noise and restore more accurate details more efficiently overall. Yellow dashed boxes and red arrows indicate these regions. Please zoom in for more details.
19

---

1
16
21
26
36
46
61
66
71
76
81
96
1
3
10
11
12
13
14
16
17
18
Figure 10: High frame rate video reconstruction results on real spike data we captured with spiking camera. The temporal sequence of 18 intensity frames is visualized, including two high-speed scenes, ie.., a bursting water balloon and rapidly spinning fan (~~500 rpm). We add yellow dot to indicate the rotation of the fan leaves.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have claimed the contribution and scope of this paper in the abstract and introduction, while specifically summarizing the contribution of this paper at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We have analyzed the limitations of our method in the paper, such as the impact on image reconstruction quality in extremely low-light scenarios.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
21

---

Answer: [No]
Justification: We present the experimental results of traditional methods TFP 165 TFI 665
and TFSTP [3], where the corresponding references are given. Our method employs CNNs to build the spike-to-image reconstruction model, and the corresponding formulas are numbered.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
we have provided clear details about the methodology and implementation, which will help reproduce our experimental results. Additionally, we will release our code and models at htttpee//ggtabb..cc//////////// com/GitCVEb/STIII for repro- ducibility.
Justification: In Sec. 4 and Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
22

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our code and models on htttps://github.co htt.:ttthwu.mmmimtt//cc...
STIR
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/publll wwtpc/ppppppg..d....s..pppppppee http://uuddeee/occuuubbiiiiaee wwwoo//Swwiiiiiooooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The training and test details have been presented in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not provide emror bars. Similar to comparison methods, we quantitatively evaluated the performance of all algorithms directly on the test set under the same experimental conditions, which can guarantee fairness. Moreover, we count the average inference time multiple sets of data to serve as the final reported runtime.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have pointed out that all models are trained and tested on single NVIDIA RTX 3090 GPU. In addition, have reported the parameter size, GPU memory usage, and FLOPs for each method in Table
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https: http://eeeii....eeeee hwtparips.cccpulli//BBeeee cc//pubbii//Ethiccciiiddlliii1
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
The work in this paper satisfies the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

We have discussed the potential impact of our work in the last part of the appendix.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not pose such risks. Moreover, the real spike data collected was conducted indoors and did not face safety risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For the datasets and code used in this paper, we have included the necessary citations, complying with the respective licenses and usage terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The evaluation, license, limitations, and the other details of the code will be provided on otc.pppe///ggaapooooo///o/////tr This paper does not introduce other new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
26

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "qwRzHjRym9", "reviewer": "Reviewer_1ZaT", "summary": "This paper proposes a new efficient spatio-temporal interactive reconstruction network that enhances image reconstruction by jointly optimizing inter-frame feature alignment and intra-frame feature filtering in a coarse-to-fine approach. The network leverages a hybrid spike embedding representation and introduces novel components like a symmetric interactive attention block and a multi-motion field estimation block to refine the motion fields and target frames progressively. Tested on both synthetic and real-world data, this approach significantly outperforms existing methods.", "questions": "1. How does the model handle image reconstruction tasks under extreme motion or lighting conditions?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clarity: The paper is clearly written and easy to read.\n\n2. Comprehensive experiments: The author conducts extensive experiments and ablation studies to demonstrate the proposed method's effectiveness.", "weaknesses": "1. Generalization: the paper lacks the verification of generalization ability under unknown or broader conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your precious time and recognition of our work. We first list your advice and questions, then give our detailed answers.\n\n> *W1*: Generalization ability under unknown or broader conditions.\n\nThe commonly used generalization test approach of current spike-to-image reconstruction methods is to train on simulated data and then perform generalization tests on real-world datasets. We adopted this approach as well, but tested it on a wider range of real-world data, including “momVidarReal2021”, “recVidarReal2019” (also used in [1, 2]), and our real-captured data, which covers high-speed camera/object motion scenarios under complex indoor and outdoor conditions and is **sufficient to demonstrate the generalization performance of our method**. \n\nAs for the more unknown or broader conditions, they remain to be further explored. But we will consider involving more diverse scenarios and building datasets under more challenging conditions to tap into the potential of our models in future work. Thanks for your advice.\n\n> *Q1*: How does the model handle image reconstruction tasks under extreme motion or lighting conditions?\n\nWith a sampling rate of 40,000 Hz, the spiking camera is inherently capable of handling ultra-high-speed motion effectively. Even in scenarios that exceed the response speed of the human eye, our model still shows excellent performance. As illustrated in **Figure 10** in the Appendix, our method successfully reconstructed **the instantaneous process of a water balloon bursting in great detail**.\n\nIn Section Limitations, we have discussed model performance in extremely low-light scenarios. Empirical experiments have shown that the limited accumulated light intensity often leads to darker images and increased noise. However, this limitation is common to current spike-to-image reconstruction methods [3, 4, 5] and can be further explored in our future work.\n\n---\n\n[1] Capture the moment: High-speed imaging with spiking cameras through short-term plasticity. In IEEE TPAMI 2023.\n\n[2] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.\n\n[3] Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR 2021.\n\n[4] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.\n\n[5] Spike camera image reconstruction using deep spiking neural networks. In IEEE TCSVT 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKd6fmblEQ", "reviewer": "Reviewer_GueT", "summary": "In this paper, authors propose a novel method for reconstructing images from spiking camera representations. The approach involves constructing a spiking embedding representation followed by a complex network of sub-networks, the importance of each respective block being evaluated through ablation studies. Notably, the proposed model achieves state-of-the-art results with relatively few parameters while providing detailed explanations of the intricate methodology (available in supplementary material).", "questions": "It would be beneficial to investigate whether this method can be directly applied to more common event-based cameras found in the market, given analogies between spiking and traditional event-based cameras. This could involve evaluating the proposed method on different benchmarks using data obtained from these cameras.", "limitations": "While the model achieves impressive results for the problem statement, it is challenging to determine which information contributes most to success. Further investigation of this point can be done by applying the methodology to various sources of inputs and observing how the model adapts (e.g., autonomous driving vs drone-taken scenes). This will provide valuable insights into the effectiveness of the proposed approach in different contexts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main contribution of this paper lies in its ability to achieve exceptional performance on the task at hand. The importance of each sub-network within the complex network is accurately evaluated through ablation studies, demonstrating the effectiveness of the proposed approach. Furthermore, the detailed explanation of the methodology provided in supplementary material enhances our understanding of the intricate process.", "weaknesses": "One limitation of this paper is that it relies on previous models such as ResNet or transformers without providing a mathematical justification for these choices beyond their ability to efficiently solve the task at hand. This may hinder the reproducibility and interpretability of the model, which are crucial aspects in machine learning research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your precious time and recognition of our work. We first list your advice and questions, then give our detailed answers.\n\n> *W1*: Using previous models without mathematical justification.\n\n(1) **Mathematical justification**. We deeply relate to your concerns about the mathematical foundation of a machine learning model. However, rather than merely copying the model or relying on them to achieve the state-of-the-art performance, we choose them by **matching their functionalities with our needs**. \n\n- **ResNet** excels at robust feature extraction by allowing gradients to flow more easily during backpropagation, and it has been tested in all sorts of tasks for its exceptional modeling ability. Its modularity and flexibility also allow us to integrate it easily into the spike embedding representation module. Combined with explicit representations (TFP), we designed a hybrid spike embedding representation (HSER). After trying different methods, empirical validation in Table 2 not only demonstrated the effectiveness of using ResNet compared with Multi-dilated [1] and HiST [2] representations, but also the effectiveness of the proposed hybrid scheme.\n\n- **Cross-attention** improves the model’s ability to understand and utilize contextual relationships and facilitates the alignment of features between sequences. In our setting (Section 3.2), we use three non-overlapping spike sub-streams $S_{t_0}^{N}$, $S_{t_1}^{N}$, and $S_{t_2}^{N}$ to reconstruct the intermediate intensity frame $I_{t_1}$. The center is the reconstruction objective, while the sides serve as auxiliary components. In order to better incorporate contextual information from $S_{t_0}^{N}$ and $S_{t_2}^{N}$ into the intermediate time $t_1$, we adopted the idea of cross-attention and presented a symmetric interactive attention block. This symmetric design collaboratively enhances the bilateral correlation between the intermediate feature $F_{t_1}^{L}$ and temporal contextual features $F_{t_0}^{L}, F_{t_2}^{L}$ and also injects prior motion-intensity guidance into the subsequent interactive decoder, which boosts the model performance as shown in Table 3(c). \n\n(2) **Reproducibility**. The model structure and experiment details are clearly illustrated and described in the paper, which facilitates reproduction. Moreover, **we have sent our anonymous open-source code link to the AC for reviewers’ inspection as requested by NeurIPS**.\n\n(3) **Interpretability**. We have justified above the functionalities of previous models, the needs of our method, and the modifications we make to match the two, which may help in understanding the designs intuitively.\n\n> *Q1*: It would be beneficial to explore whether this method can be applied to event-based cameras. \n\nThanks a lot for your insightful idea. We have demonstrated in Table 4 that our HSER can better adapt the event-to-image reconstruction model to spike-to-image reconstruction, which is a good indication that the key reconstruction module is transferable by adjusting the frontmost embedding representation. Moreover, the core contribution of this paper lies in the idea of spatial-temporal joint learning. Applying this design philosophy to event-based reconstruction **holds significant promise**, given analogies between spiking and event cameras.\n\nHowever, **the reconstruction of event cameras might pose more challenges and need further investigation** considering the different working mechanisms of event cameras and spiking cameras. Event cameras utilize a differential sampling approach and record the change in light intensity. Whereas spiking cameras follow an integral sampling method, thus preserving the absolute value of light intensity. Though this presents a slight digression from our current work, we look forward to inventing an **input-agnostic** image reconstruction method that unifies the input types for neuromorphic cameras in the future. \n\n> *Limit1*: It is challenging to determine which information contributes most to success. Further investigation can be done by applying the methodology to various sources of input.\n\nThe key contribution is that we adopted **an interactive and joint perspective** to address temporal and spatial information simultaneously. It was under the guidance of this idea that we proposed a **single-stage** architecture and multiple customized modules, all of which **synergistically** enhanced the performance to the state-of-the-art. Among them, **synthesis-based intra-feature filtering acts as the foundation** since it is devoted to reconstructing the intermediate intensity frame (as shown in Table 3 (b)), indicating that *“spatial” information* is essential for intermediate frame reconstruction. In contrast, other modules improve reconstruction quality to varying degrees, e.g., warping-based inter-frame feature alignment helps to aggregate contextual *“temporal” information* to achieve higher quality reconstruction. So it is crucial to note that all modules are designed **with an overarching purpose** instead of being assembled haphazardly.\n\nAs for further investigation, we appreciate you a lot for your insight. At present, spike datasets in more scenarios like autonomous driving and drone-taken scenes are not available. In the future, we will take these scenarios into consideration and validate the model’s effectiveness and generalization across different contexts.\n\n---\n\n[1] Unsupervised optical flow estimation with dynamic timing representation for spike camera. In NeurIPS 2023.\n\n[2] Optical flow for spike camera with hierarchical spatial-temporal spike fusion. In AAAI 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cNwGFO4rP", "reviewer": "Reviewer_94R6", "summary": "This paper proposes a new method for reconstructing images from spiking camera data called STIR (Spatio-Temporal Interactive Reconstruction network).", "questions": "1.Can you explain the concept of the \"hybrid spike embedding representation\" (HSER) module and how it balances interpretability with expressive power?\n\n2. The paper discusses ablation studies on various components of the model. Which component seemed to have the most significant impact on the model's performance?\n\n3. The paper mentions that the architecture is flexible and can be scaled. How is this scaling achieved, and what are the trade-offs involved?\n\n4.Can you explain the significance of using the intermediate feature FL t1 as the query and the temporal contextual features FL t0 and FL t2 as key/value in the attention mechanism?", "limitations": "1. While the method performs well on the tested datasets, its performance on a wider range of real-world scenarios is not fully explored.\n\n2. Although faster than some existing methods, the approach still requires significant computational resources, which may limit its applicability in some real-time or resource-constrained settings.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The joint motion-intensity learning architecture is innovative and addresses limitations of previous step-by-step methods.\n2. Faster inference speed compared to many existing methods, with competitive model size and complexity.", "weaknesses": "1. The overall architecture is quite complex, which may make it challenging to implement or adapt.\n2. While the empirical results are strong, there's limited theoretical justification for why this approach works better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and insightful comments. We address each concern below.\n\n> *W1 & Limit2 & Q3*: The overall architecture is too complex to implement or adapt. The approach still requires significant computational resources. How is the scaling achieved, and what are the trade-offs involved?\n\n(1) **Model Complexity**. We understand your concerns. But if we take a **holistic perspective**, it becomes clear that the model follows a typical **encoder-decoder architecture**. More importantly, since our motivation is to *jointly address temporal motion estimation and spatial intensity recovery in a single-stage manner*, we actually **simplified the whole reconstruction process**. Our method has achieved real-time inference on an NVIDIA RTX 3090 GPU and 400×250 inputs (see Fig. 2), which cannot be done for existing CNN-based spike-to-image reconstruction methods. In addition to FLOPs presented in Table 1 (note that *our method has the lowest FLOPs among CNN-based architectures*), we further profile the computational complexity on an NVIDIA RTX 3090 GPU and 1280×720 inputs, i.e.,\n\n| Model | SSIR | Spk2ImgNet | WGSE | **Ours** |\n|:---|:---|:---|:---|:---|\n|GPU memory usage (MB)| 10612 | 14956 | 20180 | **9424**|\n\nHence, we respectfully contend that, compared with other existing methods, our method has already achieved excellent performance with low model complexity and low computational resources. In the future, we will explore ways to enhance the efficiency of our approach on resource-constrained platforms. \n\n(2) **How the model adapts**. It is clearly stated in Section 5.3 that **feature pyramid levels**, **model capacity** (the width multiplier for the feature channel), and **number of multi-motion fields** can be adjusted. In the implementation, you just need to change the corresponding parameters to achieve scaling, e.g., when the number of pyramid levels is 3, our method maintains the state-of-the-art performance with the smallest number of parameters (**0.832M**).\n\n(3) There is **a trade-off between computational complexity and model performance**. From Table 3, the increase in model size improves the reconstruction quality but leads to more parameters and computations. Our model is **handy to scale** to fit into diverse scenarios. For instance, in scenarios that demand high precision and have abundant computational resources, a larger model is preferred. Conversely, for mobile or real-time applications, a simpler model can be employed.\n\n> *W2*: Limited theoretical justification for why this approach works better.\n\nWe deeply relate to your concerns and acknowledge the importance of theoretical analysis. Yet, our primary focus was to demonstrate the **practical effectiveness of our motivation**, i.e., temporal motion estimation and spatial intensity recovery can be mutually reinforcing. Therefore, in the paper, we **prioritized extensive experimental validation to show its feasibility**. Particularly, the sub-models we used (like ResNet and cross-attention) to meet our needs are widely tested for their exceptional modeling ability in academic research and have **a solid mathematical foundation**. We have made modifications to fit into our model, whose effectiveness was justified by extensive ablation studies.\n\n> *Q1*: The concept of the \"hybrid spike embedding representation\" (HSER) module and how it balance interpretability with expressive power?\n\nHSER is composed of two parts: **explicit and implicit**. Explicit representations using TFP provide an anchor for image reconstruction since the results of TFP can be viewed as low-quality reconstructed images that have **good interpretability**. Implicit representations obtained from residual blocks have **strong expressive capability** to map original spike data to features. Previous methods have relied on only one side without integrating the power of both. Besides, HSER is very **light-weight**, which can balance interpretability with expressive power in an efficient way. **All of these are clearly described in Section 4.2**.\n\n> *Q2*: Which component seemed to have the most significant impact on the model’s performance?\n\nAblation studies have shown that each module improves reconstruction quality in its own right and is equally important. Yet Table 3 (b) underscores the **foundational role of synthesis-based intra-feature filtering** since it reconstructs the intermediate intensity frame. Rather than being assembled haphazardly, all modules are designed **with an overarching purpose**: to address temporal and spatial information simultaneously **in an interactive and joint manner**. Under the guidance of this idea, we proposed a **single-stage** architecture and multiple customized modules, all of which **synergistically** enhanced the performance to the state-of-the-art.\n\n> *Q4*: The significance of using $F_{t_2}^{L}$ as the query and using $F_{t_0}^{L}$ and $F_{t_2}^{L}$ as key/value in the attention mechanism.\n\nThe center $F_{t_1}^{L}$ corresponds to the final reconstruction objective, while the sides $F_{t_0}^{L}$ and $F_{t_2}^{L}$ serve as auxiliary components. In doing so, we followed the standard formula of cross-attention and designed a symmetric interaction strategy to collaboratively incorporate contextual information from both sides into the center, thus improving the overall performance as demonstrated in Table 3(c).\n\n> *Limit1*: The performance on a wider range of real-world scenarios.\n\nTo the best of our knowledge, we have covered all the publicly available real-world datasets for testing, i.e., “momVidarReal2021” and “recVidarReal2019”. (Note: You might find some other real-world datasets, but their scenarios are either already covered by these two datasets or involve motion that is too slow to effectively demonstrate the model’s performance.) It is also worth mentioning that we tested it on our real-captured spike data, which exhibits excellent results (see Figs. 9 and 10).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KzUu9j6WxU", "reviewer": "Reviewer_1mUp", "summary": "Spiking cameras are sensors that capture high-speed motion by firing continuous binary spike streams asynchronously. Current image reconstruction methods from these spike streams use complex architectures that overlook the collaboration of spatio-temporal information. This paper proposes an efficient spatio-temporal interactive reconstruction network that aligns inter-frame features and filters intra-frame features progressively. The network refines motion fields and target frames scale-by-scale, utilizing a symmetric interactive attention block and a multi-motion field estimation block to enhance interaction capabilities. Experiments on both synthetic and real data show the method's high performance and low model complexity.", "questions": "1.\tIn Section 1: “Our codes and model weights will be open source.” Is it possible to provide the codes and model weights in the supplementary material for reviewers’ inspection?\n2.\tThe experiments have been conducted only on the SREDS dataset. Can the results be generalized with a larger variety of benchmarks?", "limitations": "The limitations have been discussed in Appendix G.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe tackled problem is relevant to NeurIPS.\n2.\tThe preliminaries in Section 3 are described clearly.\n3.\tThe results show that the proposed approach outperforms related works.\n4.\tSeveral ablation studies have been conducted.", "weaknesses": "1.\tIn Section 2, the paragraph describing Spike-Based Image Reconstruction is confusing. In particular, there is confusion when discussing CNNs and SNNs. Please clarify.\n2.\tIn Section 4, it is not clear what are the limitations of the related work and how these issues are addressed in the proposed methodology. It would be useful to describe the proposed methodology more in detail through a detailed top-level algorithm describing all the operations involved.\n3.\tIn Section 5.1, the experiment details are not described with sufficient detail to allow reproducibility. Please describe all the tools used and provide values of all the parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious time and insightful comments. We first list your advice and questions, then give our detailed answers.\n\n> *W1*: A little confusion in Spike-Based Image Reconstruction.\n\nThank you for your question that has brought our attention to a potential point of confusion. We would like to clarify two things in Related Works: \n\n(1) “Spike-Based Image Reconstruction” refers to image reconstruction using spiking cameras (a type of neuromorphic camera), which is not related to the spiking neural network (SNN, a type of network architecture). \n\n(2) When introducing deep learning techniques in this part, we first introduce **supervised** methods, which can then be divided into CNN-based [1, 2] and SNN-based [3]. Then what follows are the **self-supervised** CNN-based methods [4, 5]. \n\nTo avoid ambiguity and aid understanding, we will revise the subtitle of Related Works from “Spike-Based Image Reconstruction” to “**Spike-to-image Reconstruction**” and Lines 109-111 in the manuscript as follows:\n\nOriginal Lines 109-111: “Furthermore, several self-supervised CNNs have also been developed. However, due to the step-by-step paradigm, the above CNN-based architectures inevitably have higher model complexity, blocking them from mobile and real-time applications.”\n\nRevised Lines: “***The above three are supervised methods, whereas several self-supervised CNNs have also been developed.*** However, due to the step-by-step paradigm, the above CNN-based architectures inevitably have higher model complexity, blocking them from mobile and real-time applications. ***In contrast, our single-stage model jointly considers temporal motion estimation and spatial intensity recovery, thus facilitating the intrinsic collaboration of spatio-temporal complementary information.***”\n\n> *W2*: The limitations of the related work and our improvement.\n\nThe limitations of related work:\n\n(1) SSIR [3] (the SNN-based method), though energy-efficient, performs largely below the ideal level. While CNN-based methods [1,2] achieve promising results, the step-by-step paradigm inevitably leads to higher model complexity.\n\n(2) In spike embedding representation, previous methods relied on either explicit or implicit representation. But relying on each side leads to the drawback of not being able to balance interpretability and strong expressiveness.\n\nOur improvement:\n\n(1) Our single-stage architecture targets the previous step-by-step paradigm and addresses temporal motion estimation and spatial intensity recovery jointly, therefore exhibiting excellent performance while maintaining low model complexity (see Table 1 and Figure 1).\n\n(2) We developed a hybrid spike embedding representation (HSER) to offer good certainty and strong expressive capability simultaneously while maintaining low computational cost.\n\n**We have provided an overview of the workflow of our method in Section 4.1**. Combined with Figure 2, it becomes easy to understand the mechanism and role of each module.\n\n> *W3*: More training details to allow reproducibility.\n\nDue to page limitations, we put more training details and loss functions in Appendices A and B, which are sufficient to reproduce. We resonate with your concern for reproducibility and will put all the training details into the main text.\n\n> *Q1*: Is it possible to provide the codes and model weights in the supplementary material for reviewers’ inspection?\n\nIn accordance with NeurIPS requirements, we have sent **the anonymous open-source code link** to the AC for reviewers’ inspection.\n\n> *Q2*: Experiments only on the SREDS dataset. Can the results be generalized with a larger variety of benchmarks?\n\nCurrently, there are two available benchmark datasets for the spike-to-image reconstruction task: SREDS and Spike800.\n- The Spike800 training set was introduced in Spk2ImgNet [1] in 2021. The spatial resolution of images is 400x250. Each scene contains 5 GT images, with 1 GT corresponding to 13 spike planes. There are 240,000 GT images and 13×240,000=3,120,000 spike planes.\n- The SREDS training set was introduced in SSIR [3] in 2023. The spatial resolution of images is **1280x720**. Each scene contains 24 GT images, with 1 GT corresponding to 20 spike planes. There are **524,160** GT images and 20×524,160=**10,483,200** spike planes.\n\nBoth are synthesized based on the REDS dataset [6] and by the same simulator as that in Spk2ImgNet [1]. Considering the higher resolution images and larger number of training samples in SREDS, SREDS can be considered an **upgraded version** of Spike800. So we ultimately chose the latest SREDS dataset.\n\nThe commonly used generalization test approach of current spike-to-image reconstruction methods is to train on simulated data and then perform generalization tests on real-world datasets. We not only did experiments on the synthesized SREDS dataset but also on **a variety of real-captured datasets**, including “momVidarReal2021”, “recVidarReal2019” and our newly collected spike data, which covers high-speed camera/object motion scenarios under complex indoor and outdoor conditions and is sufficient to demonstrate the generalization performance of our method. Our method significantly outperforms existing methods in terms of reconstruction accuracy on both synthetic and real datasets.\n\n---\n\n[1] Spk2ImgNet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR 2021.\n\n[2] Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In AAAI 2023.\n\n[3] Spike camera image reconstruction using deep spiking neural networks. In IEEE TCSVT 2023.\n\n[4] Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI 2022.\n\n[5] Self-supervised joint dynamic scene reconstruction and optical flow estimation for spiking camera. In AAAI 2023.\n\n[6] Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPRW 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for reconstructing images from spiking camera data, called STIR (Spatio-Temporal Interactive Reconstruction network). 

In general this was a borderline paper. Two of the reviewers were initially leaning towards rejecting the paper. After the rebuttal, one of the reviewers was satisfied sufficiently by the author responses, to raise the ranking to a borderline accept. 

Among the drawbacks of the paper, as indicated by the reviewers, is that this is quite an expensive architecture, which might limit its applicability in various problems. However the authors argue that it is feasible to reduce the number of layers used if someone wants to make it less expensive. It is not clear how the quality of the generated images would be affected if the number of layers were reduced. 

Another concern expressed by the reviewers is that it is not clear why this approach performs so well in terms of the quality of the generated images, compared to state-of-the-art models, since state-of-the-art models also use ResNet and cross-attention. The authors have indicated that they believe this is due to the fusion of motion and intensity data, and they will include some extra discussion on this in the paper. The authors indicated that they will release all their source code, making the work easily reproducible, and feasible for someone to better study the algorithm and understand it.

After reading the paper carefully, I agree for the most part that it is well written, and the quality of the results seems superior to the other methods presented, qualitatively and quantitatively. The algorithm is not terribly novel, and I agree that it is quite an expensive algorithm, limiting its applicability for various problems. Nevertheless I felt that the positives outweighed the negatives so I leaned towards accepting the paper.

The authors need to make sure that their source code is released, and they include detailed instructions on how someone can reproduce their results.
Decision: Accept (poster)