Spike4DGS: Towards High-Speed Dynamic Scene Rendering with 4D Gaussian Splatting via a Spike Camera Array
Authors: Qinghong Ye
Yiqian Chang
Jianing Li
Haoran Xu
Xuan Wang
Wei Zhang
Yonghong Tian
Peixi Peng
Venue: NeurIPS 2025 poster ()
Keywords: High-speed scene; Novel view synthesis; Multiple views; Spike camera
Primary area: other
TL;DR: A novel framework for spike camera reconstruction

Abstract: Spike camera with high temporal resolution offers a new perspective on high-speed dynamic scene rendering. Most existing rendering methods rely on Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) for static scenes using a monocular spike camera. However, these methods struggle with dynamic motion, while a single camera suffers from limited spatial coverage, making it challenging to reconstruct fine details in high-speed scenes. To address these problems, we propose Spike4DGS, the first high-speed dynamic scene rendering framework with 4D Gaussian Splatting using spike camera arrays. Technically, we first build a multi-view spike camera array to validate our solution, then establish both synthetic and real-world multi-view spike-based reconstruction datasets. Then, we design a multi-view spike-based dense initialization module that obtains dense point clouds and camera poses from continuous spike streams. Finally, we propose a spike-pixel synergy constraint supervision to optimize Spike4DGS, incorporating both rendered image quality loss and dynamic spatiotemporal spike loss. The results show that our Spike4DGS outperforms state-of-the-art methods in terms of novel view rendering quality on both synthetic and real-world datasets. More details are available at https://github.com/Qinghongye/Spike4DGS.

Spike4DGS: Towards High-Speed Dynamic Scene Recontruction with 4D Gaussian Splatting via a Spike Camera Array
Qinghong Y, Yiqian Chang Jianing Li Haoran Xu Xuan Wang",
Wei Zhang Yonghong Tian 1,3,4.f, Peixi Peng 1,3,f
Shenzhen Graduate School of Peking University
2 Shenzhen Campus of Harbin Institute of Technology SPeng Cheng Laboratory
School of Computer Science, Peking University
5 Shenzhen Campus of Sun Yat-sen University
Abstract
Spike camera with high temporal resolution offers a new perspective on high- speed dynamic scene rendering. Most existing rendering methods rely Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) for static scenes using a monocular spike camera. However, these methods struggle with dynamic motion, while a single camera suffers from limited spatial coverage, making it challenging reconstruct fine details in high-speed scenes. To address these problems, we propose Spike4DGS, the first high-speed dynamic scene rendering framework with 4D Gaussian Splatting using spike camera arrays. Technically, we first build a multi-view spike camera array to validate our solution, then establish both synthetic and real-world multi-vier spike-based reconstruction datasets. Then, we design a multi-view spike-based dense initialization module that obtains dense point clouds and camera poses from continuous spike streams. Finally, we propose spike- pixel synergy constraint supervision to optimize Spike4DGiS, incorporating both rendered image quality loss and dynamic spatiotemporal spike loss. The results show that our Spike4DGS outperforms state-of-the-ar methods in terms of novel view rendering quality on both synthetic and real-world datasets. More details are available at the project page
Introduction
Novel view synthesis [16] is cornerstone of many cutting-edge applications, enabling the creation of precise novel views from ideal image sequences. However, conventional cameras struggle in high-speed motion scenarios, where rapid movement causes motion blur and significantly degrades reconstruction quality [39] While some approaches [30 44 attempt to improve reconstruction from motion-blurred images, they remain fundamentally limited by the sampling rates of RGB cameras. As result, the use of new vision sensors for high-quality rendering in high-speed scenes has garnered increasing attention.
20 48 with high temporal resolution offer new perspective high-speed scene rendering. Unlike conventional cameras, they asynchronously encode absolute light intensity into continuous spike streams rates of up to 20k Hz. This unique property makes them particularly
Spike cameras 9
"Qinghong Ye and Yiqian Chang contributed equally to the paper. Peixi Peng, Yonghong Tian( Tian([pxpeng (pxpeng.yhtian @ pku.edu.cn and Wei Zhang/2hangwei121335522 126.com) are corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

effective for preserving high-speed scene textures with greater detail. Existing studies 64 65 24 67 66
47 have demonstrated their capability for fine-grained 2D reconstruction. The advantages of spike cameras also indicate their immense potential for advancing 3D scene reconstruction and novel view synthesis.
62
60
58
57
46
59
Extensive works [15 6 have explored neuromorphic cameras for rendering in high-speed scenarios. 31 52 36] utilize Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) for scene representation and novel view synthesis using event cameras. However, event cameras capture only light changes rather than absolute brightness, making them challenging for fine-grained 3D reconstruction. Other efforts [69 27 51 [54] have explored 3DGS and NeRF with alternative spike cameras overcome these limitations and enhance rendering quality. For example, pioneering works with SpikeNeRF [69] and Spike-NeRF [88] have demonstrated the feasibility of using spike streams to reconstruct 3D scenes. Nevertheless, NeRF-based methods suffer from time-consuming training and inference processes, and their implicit representations limit scene editing capabilities. In contrast, Gaussian Splatting offers a compelling alternative, providing high accuracy and fast inference speeds. For instance, SpikeGS [51 [54, SpikeNVS (7]. and USP-Gaussian [S] achieve impressive results in photorealistic 3D reconstruction using spike cameras. However, these IDGS-based methods may face difficulties in handling high-speed dynamic scenes. Additionally, relying solely on monocular cameras may present 3D reconstruction challenges areas with weak textures when using static cameras. In fact, camera arrays could enhance texture information through multi-view perspectives, enabling dynamic scene rendering even without camera motion.
Some studies [19 38 21 28
To address the aforementioned challenges, we propose Spike4DGSS, the first high-speed dynamic scene rendering framework utilizing 4D Gaussian Splatting with spike camera arrays. We aim at overcoming the following challenges: (i) Camera setup and dataser How could we establish a multi-view spike camera array and build high-quality spike-based reconstruction datasets? (ii) Effective model How could we design efficient dynamic scene rendering model that directly processes multi-view spike streams with 4D Gaussian splatting44GGS)
To be specific, we first build mulli-view spike camera array and then establish both synthetic and real-world spike-based reconstruction datasets. Then, we design a novel multi-view spike-based dense initialization module to generate dense point clouds and camera poses from continuous spike streams. Finally, we propose pixel-spike synergy supervision strategy to optimize Spike4DGss which incorporates both reconstructed image quality loss and dynamic spatiotemporal spike loss. Experimental results show that our Spike4DGS outperforms state-of-the-ar methods terms of ren- dering quality both synthetic and real-world datasets. We further verify that spike cameras achieve higher rendering quality than event cameras and RGB cameras in high-speed scenes. Meanwhile, rendering quality improves as the number of spike cameras increases.
The main contributions of this work are summarized as:
We introduce Spike4DGS, the first framework that combines spike camera arrays with 4DGS, enabling novel view synthesis in high-speed dynamic scenarios.
We present Spike-Pixel Synergy Supervision strategy to optimize the parameters of our Spike4DGS for enhanced rendering quality.
We build a spike camera array, along with highly realistic synthetic and real-world datasets that contain multi-view spike streams. We believe two standardized datasets open up opportunities for research in this novel problem.
Related Work
2.1 NVS for Dynamic Scenes
Novel View Synthesis (NVS) tasks aim to generate unknown views of an object or scene from a set of images of known views. Representative papers include Neural Radiance Fields (NeRF) [32 and 3D
Gaussian Splatting (3DGS) (23. Recently, large number of static 3DGS- based techniques 4 53
have been proposed due to their high quality and real-time rendering without using neural networks like NeRF. However, the assumption of static scenes prevents application to real-world scenarios
with moving objects. Therefore, several works extend the 3DGS dynamic scenes 45 50
26
These methods are usually divided into two main lines. For instance, De3DGS [50] and 4DGS
45

---

model spatial-temporal deformation with an implicit deformation field as the first line. On the other hand, the second line is based on the idea that scenes' motion could be encoded into the 3D Gaussian representation straightly, such as STG [26] and D3DGS [29] which represents changes in 3D) Gaussian over time through a temporal opacity and a polynomial function for each Gaussian. The above approaches could perform well on synthetic datasets and simple real-world datasets. However, when there are some high-speed objects in the scene, traditional dynamic reconstruction methods using RGB image data may suffer motion blur since most standard RGB cameras have limited frame rates. This motivates us to use the neuromorphic cameras to avoid this problem.
2.2 Neuromorphic Cameras on 3DGS
There are two types of bio-inspired sensors: event cameras [15] and spike cameras [9] Event
cameras are based on the temporal contrast sampling method and generate events asynchronously
when pixel brightness changes exceed threshold. Early event-based reconstruction approaches [28
31
38 21 42 have been proposed to derive neural radiance fields directly from event streams.
E2nerf [6 and evagaussian 52 achieved sharp reconstruction from blurry images. E-4DGS [I3]
achieved high-fidelity dynamic reconstruction from the multi-view event cameras. GS2E [25] has
introduced an effective event stream generator by gaussian splatting. Another type of neuromorphic camera is called the spike camera. Spike cameras record the absolute light intensity at fairly high frame rate and provide more explicit input format for detailed reconstruction. Some nerf- based methods [69 27 [8] have verified the feasibility of reconstruction with spike, but they suffer suboptimal training and rendering speeds due to the complex spike simulation network. Some 3DGS- based spike reconstruction methods have emerged to optimize this defect. Yu's SpikeGS[3T] reconstructed view synthesis results from a continuous spike stream captured by moving spike camera. In a harder setting, Zhang's 's SpikeGS[5A reconstructed scenes via a single spike stream with monocular high-speed camera motion. However, there is established spike-based method for addressing the challenge of rendering high-speed dynamic scenes using multi-view spike streams. On this basis, our work aims to overcome the limitations and construct a spike-based 3D Gaussian Splatting model for high-speed dynamic scenes via a spike camera array.
3 Methodology
3.1 Preliminaries
Spike Camera. Spike camera is a bio-inspired sensor which records and converts the absolute light intensity at fairly high frame rate (up to 20kHz) into accumulated voltage through photoreceptors [64]
8. If the accumulated voltage V reaches the scheduling threshold θ, a spike will be triggered and V
reset zero, mathematically formulated follows:
v v(I)= J. L(t)dt mode,
(1)
where Ltt) represents the instant light intensity at time t, t, is the moment when the previous spike was emitted, and the constant photoelectric conversion coefficient.
DUSt3R Initialization. DUSt3R 41 is a dense initialization method used for 3D reconstruction. Compared with COLMAP Initialization [40), DUSt3R provides more accurate point clouds under low-quality image input with less time, which is more suitable for high-speed scenes. Specifically, given a pair of images (7, 12), DUSt3R utilizes ViT for the encoder and decoder [1]] and DPT head 337 for estimating point clouds PC:
PC= DPT T2).
(2)
Although dUST3r and its improved versions 49 55) could generate dense point clouds based on multi-view images, however, the quality of point clouds depends on the quality of input images.
4D Gaussian Splatting. 4D Gaussian Splatting (4DGS) 45 is used for rendering dynamic scenes. It proposes a network that learns the Gaussian deformation field to predict the deformation of each 3D Gaussian. For input 3D Gaussian & and time I, a spapial-temporal structure encoder H and multi-head Gaussian deformation decoder D are used for calculating the deformations AG:
AG (G)
(3)
3

---

Rotae
Spike-based Dease
Rendering Images
Score
Inilialization
Hicplaaz
VIT Eis
Poin Cloud
Deñer Gaaaiizea
Shibal latae
Care
b De-HHortcc
Sylke
Fais
Rerdernng
Camera
Array
Sylke Sottures
Mulliicw
Mutiview 4DGS
Insae Supervisiz
1ight
Spike
Intasaly
Spiko Supersiioon
Salae
Standari Destation
Spike- Symergy Supervision
Figure I: The Framework of our Spike4DGS, we establish the connection between the real-world spike streams and the dynamic scene rendering images. The input multi-view spike streams sent to Multi-view Spike-based Dense Initialization to estimate point cloud and camera poses. Based on the initialization, 4DGS with Spike- Synergy Supervision consisting of Pixel Loss and Spike Variation Loss is utilized for rendering. Our Spike4DGS could reconstruct high-speed dynamic scenes with delicate motion and texture details.
3.2 Problem Fomulation
We aim to reconstruct high-speed dynamic scenes with delicate motion and texture details. To reach this goal, we build a multi-view spike camera array to capture the figh-speed motion and propose novel method called Spike4DGS for rendering based on continuous spike streams. The problem could be denoted as:
Spike4DGS: : {(S1, S2, 11 VN I
where S1, S2. SN-1 are the spike streams captured from N-1 views of spike cameras for training, I is the rendering image of novel view VN (the view of N-th spike camera) at time t.
To solve this problem, Spike4DGS first builds end-to-end Multi-view Spike-based Dense Initial- ization (MSDI) method to estimate the dense point cloud and camera poses from input spike streams, detailed in Sec. 3.3] Then, the initial point cloud and camera poses sent to 4D Gaussian Splatting to generate novel view rendering results. To get more delicate motion and texture details, Spike-Pixel Synergy Supervision (SPSS) is proposed for the constraint of 4D Gaussian Splatting in The total framework of Spike4DGS is shown in Fig.
Sec. 3.4
3.3 Multi-view Spike-based Dense Initialization
Previous rendering tasks based on spike cameras, such as SpikeNeRF initialization method. They first use spike-to-imate methods such TFI (64). TFP [64] and spk2img [6] to get images, and then utilize COLMAP [40 or DUSt3R 41 on these images for the scene initialization. However, this two-step method is complex and requires high-quality spike- to-image results. When dealing with high-speed dynamic scenes, the lack of texture details in converted images may influence the final point cloud estimation. In contrast, we propose end- to-end framework, called Multi-view Spike-basee based Dense Initialization (MSDI), which consists of a spike feature extractor, point cloud and pose estimator, and image generator. Given a series of spike streams from N spike cameras, MSDI aims to estimate 3D point clouds, camera poses, and their corresponding images:
69 employ a two-step
MSDI (SI, SN) (PC, ([R[T] [R[]])) (I,...,
(5)
where PC the estimated point cloud, [R[T], 1, [ N are the camera rotation and translation parameters of N spike cameras, I1,..., IN the images generated from input spike streams. As

---

an end-to-ent and network, MSDI is fine-tuned together with pre-trained weights on multi-view spike streams captured from the Carla [T] simulator.
Spike Feature Extractor. Firstly, we build a feature extractor for the multi-view spike streams. Assuming r2 is time interval around frame time f. For input spike stream (T) which lasts for time interval I' and is captured from the i-th camera view, MSDI rotates Sr, four times to obtain complete receptive field in four directions and concentrate them together:
(T).(T)) (AT 0))@ E {0", 90" 180", 270"),
(6)
where CAT and Rot mean concatenation and rotation operations respectively. Then, we utilize shift-based 3 x convolution layer from [54] to extract features:
(T,) = (T((S
where M the shift-based convolution layer, fr, the extracted spike features for input spike streams (Tz)) Replacing the input (T,), to N views of spike streams, we can get series of extracted features
Point Cloud and Pose Estimator. estimate the point cloud and camera poses, the problem could be formulated as:
Estimator: (N, + PC, (".,
(8)
To solve the problem, MSDI builds an estimator consisting of N ViT encoders 11 number of views) and fusion transformer. For N ViT encoders, each encoder ViT, handles camera view with shared weights:
(equals to the
F((T)) = ViT, (I,
(9)
For the fusion transformer Fusion TF, it 24-layer transformer which is the same as
49
GT G2(T)...., GN (T,) = Fusion F,To),,.,
(10)
where GT(T,) +) is the temporal feature of i-th camera view. This operation generates temporal features with global contextual understanding from all views.
Then, a DPT 37 decoding head is utilized for decoding the temporal features into a point cloud PC and its corresponding confidence map Epc:
Epc) = DPT(( GETT,,, Gx
(11)
Finally, we utilize the global point cloud PC to estimate camera rotations and translations ([R[T),, (..,, x)) via RANSAC-PnP [1
addition, MSDI also generates discrete images from continuous spike streams. For the time interval H, around frame time I, the image at time & in i-th camera view could be generated from:
Image Generator.
T0(1 = (CT)) E (I,..,,N),
(12)
where (t( is the generated image, BCONV is a network consisting of three 1 x convolutions followed from BSN (6).
Compared with the previous two-step initialization methods like TFI 64 CCOLMAP 40 our end-
to-end MSDI method could avoid the errors of the final estimations which occur from the low-quality images generated in the intermediate steps.
3.4 4DGS with Spike-Pixel Synergy Supervision
After MSDI, we could get the initial point cloud and camera poses. Then the initial 3D Gaussian G could be obtained from them. Thus, we could generate the deformed Gaussians from 4DGS 45 and
render an image i(t) at i-th view p.:
(())) Render Render(4DGS(C.
(4DGS(G, p.).
(13)
5

---

According to 4DGS, the rendered image loss could be formulated as follows to offer our Spike4DGS pixel supervision:
CT = ||,,(t) I(t)]lI.
(14)
where (t) is the rendered image at the time t, [t( is the generated images from the above MSDI. However, this pixel loss concentrates only on image similarity but ignores the texture and motion details, which are contained in spike streams. To take advantage of the spike characteristics, we propose a spike variation loss which first translates the rendered images into spike streams and then compares the variation between generated and real spike streams.
Translate from Rendered Image to Spike Stream. Let us denote the intensity values of the pixel (xr, y) rendered images at time [1 y. t). After getting the real light of the scene, we convert the scene light intensity into spike streams using an anngrate--nd-Fiin (IF) [1 [2] mechanism. Following intensity translation method 688
we could establish the following relationship:
(15) where n( :) is the deviation matrix corresponding to the response nonuniformity noise which could be obtained by capturing a uniform light scene and recording the intensity. IF(-) is a IF neuron, and S; (x., y, t) is the predicted spike values of the pixel (zz v) at time t.
S (x. y. = IF (I, v, n(r,y)),
Spike Variation Supervision. Different from static scenes, objects in high-speed moving scenes constantly changing, thus the segment of spikes ST(T,) 2) in a time interval Pe around / is a continuously changing sequence. For high-speed moving objects, naive LI L2 loss treats each frame separately, without considering how these objects move over time. In contrast, we propose spike variation supervision,in order to concentrate on the dynamic changes of objects. Since high-speed motions are continuous, spike streams in a short time interval around time t could be generated from single spike at time t. Thus, we simply design one-layer MLP network é. to map single spike S; (f) to spike sequence (T,), in time interval r, around t:
(, = (.
(16)
where the shape of S; (4) is (H, W, 1) and the shape of Si(T.) is (H, W,,,). H and W are the height and width of the spikes. Then, caleulate the average value and the standard deviation of this spike sequence (T', and compare them with ground truth. Therefore, the Spike Variation Loss could be presented as: Capkee (S.g(S avg(S (T.))lla
+ [[std(( (T.)))l
sta (T.))
(17)
Combining this regularization with the original Pixel Loss, we get the final synergy training loss for our Spike4DGS:
toota
Lt
papike
cpixel
(18)
4 Experiment
4.1 Datasets
To verify the validity of our proposed Spike4DGS, we create two datasets. The first dataset is real-world object dataset collected by the aforementioned spike camera array. The second dataset is a high-speed synthetic outdoor dataset generated by the CARLA simulator [10] All our experiments are conducted on these two self-made datasets. We achieve the best results on both two datasets, which demonstrates the superiority of our Spike4DGS.
Real-world Object Dataset. As shown in Fig. 2, the spike array consists of 9 spike cam- eras that are capable of capturing spike streams with spatial resolution of 250 x 400 and tem- poral resolution of 20k Hz. During the data col- lection process, synchronized recordings were made from all cameras, ensuring that motion was consistently represented across different
Figure 2: Our spike camera array.

---

Table 1: Quantitative evaluation on synthetic outdoor dataset. PSNR-dB 1, SSSIM 1 LPIPS 4.
Jaywalk SSIM LPIPS
Bicycle SSIM
Mothod
PSNR
PSNR
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
TF +DDDGS
20.65
75.7
0.384
21.73
77.7
0.385
19.98
76.2
0.377
20.09
76.2
0.387
19.24
75.3
0.402
TFP 67 +D
20.88
76.5
0.357
21.78
78.9
0.346
19.85
76.0
0.372
20.07
76.6
0.361
19.20
76.4
0,389
Spk2img 57] +D3DGS
76.6
0.304
21.68
77.5
20.12
0
76.3
0.389
18.93
TF1 6T1 +STG
24.48
84.2
0.224
24.52
84.1
0.221
24.47
84.3
0.227
23.50
84.0
0.223
26.55
88.7
0.201
TFP
67 +ST M6
24.45
84,0
0.220
24.50
84.2
0.225
24.53
84.1
0.222
24.60
84.3
0.226
26.48
89.0
0.205
Spk22mg 57]
+TGG 26
24.72
84.6
0
24.75
84.5
0.220
24.70
84.7
0.223
23.73
84.8
0.219
25.94
88.4
0,202
TF 64 +DDGS 45
26.88
87.7
0.213
27.02
89.7
0.202
25.52
90.5
0.198
25.38
86.6
0.219
25.21
86.1
0.214
TFP 67 45
27.96
91.2
0.192
27.15
90.4
0.196
26.57
88.3
0.206
25.01
86.5
0.213
25,27
87.0
0.220
Spk2img
57 +4DGSS
27.04
90.9
0.208
26.17
88.6
0.219
26.35
88.9
0.218
24.75
87.8
0.229
24.87
87.9
0.227
Dy Sp)
23.04
0.852
0.243
22.17
23.35
0.857
0.238
22.75
0.849
0.244
22.87
Dy-SpikeNeRF h
17:04
0.784
0.372
16.17
0,773
0.384
16.35
0.776
0.379
14.75
0.759
0.395
15.87
0.768
0.388
Our Spike4DGS
28.29
93.1
0.189
27.69
27.92
0.193
27.74
0.199
27.13
0.197
Spike
TFI+D3DGS
TFI-STG
TFI-4DGS
Ours
Ground Truth
Car
POLIE
ROLICE
yan
Jaywalk
Figure 3: Quantitative comparison with other methods on the dataset on our synthetic outdoor dataset. We mainly compare our method with some SOTA approaches. In contrast, method delivers both superior outlines and clear details.
views. When capturing spike streams, fix the position of our spike camera array and place the high-speed dynamic objects front of the cameras. we record spike streams of approxi- mately 0.5 seconds for each real-world scene with high-speed objects. Firstly, we choose high-speed dynamic objects such as "the collapse of bricks" (Bricks) and "the spin of a turntable" "(Tunttaee) spike camera aeconSye.,,,y, we minimize noise by providing the spike
and put them before camera with ideal light intensity and obtaining multiple ideal spike streams. These spike streams could be converted to images by our MSDI module in the training process, and use them image supervision.
Synthetic Outdoor Dataset. To quantitatively analyze our superiority, we create a synthetic outdoor dataset using the Carla (I0) simulator, which includes scenarios like Jaywalk, Bicycle, Van, Car, and Motor. These scenarios feature objects of varying sizes and speeds, with objects appearing from the left side of the image and moving to the right. An array of 9 camera sets with an overhead view was set up to capture different views of the objects. Each view setting consists of spike camera at 20k FPS for training and an RGB camera at 1k FPS for evaluation.
4.2 Experimental Setup
Competitors. Due to the relative lack of methods for dynamic novel view synthesis based on spike cameras, we completed the comparison using some two-stage rendering approaches. First, we choose some direct spike-to-image approaches: TF TFP and spk2img (5)), and then combine them .For more comprehensive experiment, we also manually integrated a deformation network from 4DGS into SpikeGS and SpikeNeRF (denoted as Dy-SpikeGS and Dy-SpikeNeRF) as comparison. Those comparisons are initialized by
with previous multiview dynamic NVS methods 45
26
29
7

---

Table 2: Quantitative evaluation on real-world object dataset. Unit: Brisque J. NIQE 4. MetalQA 1-
Bricks NIQE
Chips NIQE
Turntable NIQE
Method
Brisque
MetalQA
Brisque
MetalQA
Brisque
MetalQA
Brisque
NIQE
MetalQA
[F DDDDGS 29
57.35
16.71
0.114
61.32
16.43
0.124
56.53
15.34
0.117
57.87
14.87
0.1
TFP 67 +D3DGS 29
56.87
15.37
0.127
60.15
15.01
0.131
55.57
14.93
0.123
56.37
14.53
0.126
Spk22mng HDDDGS 23
58.78
0.133
59.27
0.135
56.94
15.87
0.129
57.14
14.65
0.137
TH. 64 +STG 26
45.33
13.88
0.143
43.77
13.93
0.149
36.45
10.53
0.150
33.32
12.64
0.143
TFP
+STG
20
44.83
13.21
0,148
44.46
11.08
0.150
37.94
10.87
0.161
34.09
9.98
0.155
Spk22ing +STG 26
45.61
13.63
0.147
44.67
12.08
0.152
38.01
10.92
0.157
34,02
9.27
0.149
TP
64
++DGS
[5
39.56
10.23
0.165
45.15
10.01
0.162
37.57
9.93
0.170
33.37
9.53
0.163
TF.
67
+IDDS
45
40.58
10.42
0.164
44.46
11.08
0.150
37.94
10.87
0.161
34.09
9.98
0.155
Spk22mn
57 +4DGS
45
40.53
10.55
0.167
44.67
12.08
0.152
38.01
10.92
0.157
34.02
9.27
0.149
Dy-SpikeG5(51
13.84
0.139
47.92
13.57
0.133
46.
13:22
0.137
Dy-SpikeNoRF 69
61.84
17.25
0.116
62.17
16.93
0.113
60.58
16.71
0.118
59.74
16.35
0.115
Spike4DGS
34.29
33.52
0.183
26.86
0.179
23,86
8.36
0.180
Spike
TFI-D3DGS
TFH+STG
TF1+4DGS
Ours
Real Scene
Bricks
Chips
Turntable
Figure 4: Qualitative comparison on real-world object datasets with SOTA rendering methods. We present the RGB photos of real scenes in the last column due to the lack of ground truth.
DUS(3R 41 to obtain point clouds. In our framework, the training data are multiple spike streams for both synthetic and real datasets.
the MSDI in Sec. 3.3 estimate initialize point clouds, camera poses, and generate images. We fine-tune this end-to-end framework together with pre-trained weights using multi-view spike streams captured from dynamic scenes in the Carla simulator. The fine-tuning adopts L.1 loss supervised by Carla images and confidence-awarc pointmap regression loss in DUSt3R [4T]. We utilize Adam optimizer with learning rate of 0.0001 effectively minimize both losses and ensure stable convergence during training. Secondly, for 4DGS with Spike-Pixee Synergy supervision in Sec. 3.4 the learning rate is the same 4DGS [45). At each optimization iteration, we randomly sample batch of views from the same time t. The total experiments are conducted on a single NVIDIA GTX 4090 with PyTorch and the optimization for single scene typically takes about 20 minutes to converge and 40 FPS when rendering. For the metrics of the synthetic outdoor dataset, we employ three widely-used image quality assessment metrics, PSNR (45). SSIM [43] and LPIPS (56) For the real dataset, which lacks corresponding ground truth images, we employ NIQE 355 BRISQUE 34 and MetalQA 63 as no-reference image quality evaluation
Implementation Details. Firstly,
metrics, the same as ofte-of-the-art works 69 51
4.3 Performance Comparison
4.3.1 Synthetic Outdoor Data Experiments
Quantitative Performance. We present a detailed comparison of our method against some two-stage rendering approaches, such as TFI [64)+TTG (26]. TF D3DGS 44G) (45), on our synthetic outdoor dataset. As demonstrated in Table I our method outperforms the SOTA two-stage rendering methods across all five distinct scenes. Note that our Spike4DGS improves more on the higher-speed scenes like "Car", which proves the effect in high-speed scenarios.
8

---

Qualitative Performance. We also present the qualitative results in Fig. where: (I) THI 664
+D3DGS
29 almost failed to reconstruct the outlines and details of each scene. (II)
TFI 64 +4DGS
45 could only reconstruct the outlines with texture details missing. (iii) In contrast, our method demonstrates superior performance, generating clearer and more detailed novel views, especially in challenging high-speed scenes like "Van".
4.3.2 Real-world Object Data Experiments
Quantitative Performance. As shown in Table 2 our method achieves superior performance compared those SOTA rendering approaches. Our method achieves an average improvement in BRISQUE [33 34 NIQE (35] and MetalQA 66 by 15,4%, 4.6% and 6.7% % respectively,
indicating higher-qualit, novel view rendering.
Qualitative Performance. In qualitative experiments, we focus on the generated texture details. Fig- 4 presents that method could generate high-quality texture details. For example, in the third line of Fig. 4 our rendering could generate clear alphabets "A, B, C..." on the Turntable dataset while others could not.
4.3.3 Analysis
Performance Analysis. The quantitative and qualitative results above show that our Spike4DGS could significantly surpass the SOTA approaches. Moreover, on the synthetic outdoor dataset, our method proves the ability to render the outdoor high-speed scenes. While on the real-world object dataset, our method could also reconstruct indoor high-speed objects. This highlights the robustness and comprehensive improvements of our Spike4DGS different high-speed dynamic scenes.
4.4 Ablation Study
Contribution of Each Component. We con- duct ablation study assess the contribu- tion of each component in our SDike4DGS. As shown 3 we combine 4DGS with some initialization frameworks. The results demonstrate that our MSDI initialization achieves the best performance, and the SPSS module consistently improves reconstruction quality under different initialization frame- works. Results the last row demonstrate that the full model (4DGS+MSDI-SPSS) achieves the best performance, validating the effectiveness of our design for spike-based 3D reconstruction.
Table 3: The contribution of each component.
Methods
PSNR1
SSIM,
I.PPS.J.
TFI
COLMAP
5D
21.08
84.9
0.253
TFI +DUSBR
25.66
89.3
0.205
MSDI
26.66
90.3
0.198
-COLMAP +SPSS
0.233
TFI 64 DDUSSBR
4 +PSSS
26.34
90.2
0.201
MSDI+SPSS(Ful model)
27.74
91.2
0.190
Table 4: Ablation supervision strategy-
Supervision Swaregy
PSNR*
SSIM
LPIPS.
PiselLoss 3
21.22
85.0
0.251
SpikeLoss
76.3
PixelLoss
45 LopikeLoss
69
26.69
89.5
0.203
Ablation on Supervision. To evaluate the ef- fect of our supervision strategy, we compare our SPSS in Sec. [3.4 with pixel loss in vanilla 4DGS 45), spike loss in SpikeNerf (69) and the combination of the above. The quantita- tive results novel view synthesis are listed in T Our SPSS, consisting of pixel loss and spike variation loss, obtains the highest PSNR performance.
(Ours)
1CE:18
N12E:11.45
Ablation on View Numbers. In this part, we investigate the relationship between view numbers and performance. As shown in Fig.5l we present both quantitative and qual- itative comparisons. For the quantitative experiments, we use Brisque 33 34 and NIQE (35] as the evaluation metric. Our Spike4DGS achieves the best performance
M0E16667
NG2I SU
5 views
7 views
views
Figure 5: Quantitative and qualitative ablation of view numbers on real-world object datasets.

---

Table 5: Comparison of different methods under varying numbers of views. (Brisque/NIQE)
Methods
3 Views
4 Views
Views
6 Views
7 Views
8 Views
TFI+D3DGS (29
83,4/23.87
73.6/21.67
65.2/20.81
62.9/18.60
59.8/16.20
56.53/15.33 3.34
TFI+STG [26
71.0/22.80
57.0/16.70 61.2/17.50
42.0/13.10 52.1/14.60
36.45/10.53
TFI+4DGS 45
70.3/22.70
60.8/18.40
56.4/16.50
43.6/13.90 51.8/14.40
37.57/9.99 93
Ours
63.43/18.54
53.62/15.67
45.23/13.81
32.91/11.67
29.84/10.20
26.86/9.86
both visual quality and quantitative score. The quantitative comparisons across different methods are summarized in Table It proves that more view numbers will lead to higher rendering quality. This means that our method could be extended more views in the future.
The effect of MSDI's Generator To validate the effectiveness of our MSDI module for spike2image initialization, conducted quantitative comparisons with representative spike-to-image methods, including TFI, TFP, and spk2img. On the synthetic dataset where ground truth images are available, evaluated the spike-to-img quality using average PSNR and SSIM. The results are summarized in T Note the reconstruction images in test have the same view with the training images, hence the improvements more clearly than NVS.
Table Quantitative comparison on average quality.
Metric
TF1 [3]
TFP [3]
TVS [4]
Spike2img [5]
MSDI's images
Avg PSNR1
24.51
25.37
22.43
30.76
34.90
Avg SSIM
0.850
0.852
0.821
0.904
0.961
The effect of Whole MSDI. This part provides a more detailed evaluation of the module MSDI Specifically, feed the MSDI outputs (i.e., reconstructed images, point clouds, and poses) into the standard 4DGS procedure, respectively, to validate the contribution of MSDI. addition, the traditional spike-to- -to-image methods (e.g.. TFI) and MSDI+SPSS also listed for comparison. The experiments are conducted on the synthetic dataset, and the average performances are reported the following Tab.
7
It indicates MSDI's contribution positively.
Table 7: Focus Comparison MSDI without SPSS.
Method
Avg PSNR
Avg SSIM
Avg LPIPS}
TFI + COLMAP + 4DGS
21.08
0,849
0,253
TFI DUST3R 4DGS
25.66
0.893
0.205
MSDI IMAGE + COLMAP+4DCG
22.43
0.854
0.233
MSDI IMAGE DUST3R 4DGS
25.76
0.898
0.203
MSDI + 4DGS
26.66
0.903
0.198
MSDI + SPSS (Ours)
27.74
0.912
0.190
5
Conclusions
This paper introduces Spike4DGS, a novel framework that seamlessly integrates multiple spike streams captured by spike array into 4DGS training, effectively addressing the challenges of reconstructing high-speed dynamic scenes. Spike4DGS designs a novel Multi-view Spike-based Dense Initialization module to obtain dense point clouds from continuous spike streams and Spike- Pixel Synergy Supervision strategy to optimize the parameters for enhanced rendering quality. We contribute two novel datasets and conduct comprehensive evaluations. The results the datasets demonstrate that our Spike4DGS surpasses previous SOTA dynamic reconstruction approaches in high-speed dynamic scenes, with almost sacrifice in training cost and rendering FPS.
Limitation. While our spike camera array is functional, the approach has limitations which we will solve in our future works. The images rendered by our Spike4DGS are grayscale due to the lack of RGB information. Additionally, the spike array not portable, which limits its use in mobile or field-based applications.
10

---

Acknowledgements The study was funded by the National Natural Science Foundation of China under contracts Shenzhen Science and Technology Progamn(KQTD2024022912222 62422602, No. 62425101, No. 62332002, No. 62372010, No. 62027804, No. 62088 102, No. 62206281, 1, and the major key project of the Peng Cheng Laboratory (PCI.2021A13 and PCI.2024AS204). Computing support was provided by Pengcheng Cloudbrain.
No.
References
[1] Alex M Andrew. Multiple view geometry in computer vision. Kybernetes, 30/9/10) 2001.
[2] Jeongmin Bae, Seoha Kim, Youngsik Yun, Hahyun Lee, Gun Bang. and Youngjung Uh. Per Per-gaussian embedding- based deformation for deformable 3d gaussian splatting, arXiv preprint arXiv:2404.0361. 2024.
[3] Anish Bhattacharya, Ratnesh Madaan, Fernando Cladera, Sai Vemprala, Rogerio Bonatti, Kostas Danilidis, Ashish Kapoor, Vijay Kumar, Nikolai Matni, and Jayesh K Gupta. Evdnerf: Reconstructing event data with dynamic neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5846-5855 2024.
[4] David Charatan, Sizhe Lester Li. Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition. pages 19457-19467, 2024.
[5] Kang Chen, Jiyuan Zhang, Zecheng Hao, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. ssp-gaussian: Unifying spike-based image reconstruction, pose correction and gaussian splatting. arXiv, 2024.
[6] Shiyan Chen, Chaoteng Duan, Zhaofei Yu, Ruiqin Xiong, and Tiejun Huang. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In IJCAI, pages 2859-2866, 2022.
[7] Gaole Dai. Zhenyu Wang. Qinwen Xu, Ming Lu, Wen Chen, Boxin Shi, Shanghang Zhang, and Tiejun Huang. Spikenvs: Enhancing novel view synthesis from blurry images via spike camera. arXiv, 2024.
[8] Siwei Dong, Lin Zhu, Daoyuan Xu, Yonghong Tian, and Tiejun Huang. An efficient coding method for spike camera using inter-spike intervals. arXiv preprint arXiv: 1912.09669, 2019.
[9] Siwei Dong, Tiejun Huang, and Yonghong Tian. Spike camera and its coding methods. arXiv, 2021.
[10] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010. 10..009999999 2020.
[12] Wei Fang. Zhaofei Yu, Yangi Chen, Tiejun Huang, Timothée Masquelier. and Yonghong Tian. Deep residual learning spiking neural networks. In Advances in Neural Information Processing Systems, pages 21056-21069. Curran Associates, Inc., 2021.
[13] Chaoran Feng, Zhenyu Tang, Wangbo Yu, Yatian Pang, Yian Zhao, Jianbin Zhao, Li Yuan, and Yonghong Tian. E-4dgs: High-fidelity dynamic reconstruction from the multi-view event cameras. ACM MM 2025, 2025.
[14] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6))381-39) -395, 1981.
[15] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: survey, IEEE transactions on pattern analysis and machine intelligence, 14411)154-180, 2020.
[16] Kyle Gao, Yina Gao, Hongjie Hc, Dening Lu, Linlin Xu, and Jonathan Li. Nerf: Neural radiance field in 3d vision, a comprehensive review, arXiv preprint XXX.:2210.003777 2022.
[17] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[18] Yijia Guo, Yuanxi Bai, Liwen Hu, Mianzhi Liu, Ziyi Guo, Lei Ma, and Tiejun Huang. Spike-nerf: Neural radiance field based on spike camera. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, IEEE, 2024.

---

[19] Haiqian Han, Jianing Li. Henglu Wei, and Xiangyang Ji. Event-3dgs: Event-based 3d reconstruction using 3d gaussian splatting. In Proceedings of the Advances in Neural Information Processing Systems, pages 128139-128159. 2025.
[20] Tiejun Huang, Yajing Zheng, Zhaofei Yu, Rui Chen, Yuan Li, Ruiqin Xiong. Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, et al. 1000x faster camera and machine vision with ordinary devices. Engineering. 25: 110-119; 2023.
[21] Inwoo Hwang, Junho Kim, and Young Min Kim. Ey-nerf: Event based neural radiance field. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 837-847, 2023.
[22] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5322-5332. 2024.
[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trunsactions on Gruphicx, 42(4):1-14, 2023.
[24] Jianing Li, Xiao Wang, Lin Zhu, Jia Li, Tiejun Huang, and Yonghong Tian. Retinomorphic object detection in asynchronous visual streams. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1332-1340, 2022.
[25] Yuchen Li*, Chaoran Feng", Zhenyu Tang, Kaiyuan Deng, Wangho Yu, Yonghong Tian, and Li Yuan. Gs2c: Gaussian splatting is an effective data generator for event stream generation. arXiv preprint 1522 2025.
[26] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8508-8520, 2024.
[27] Zhanfeng Liao, Yan Liu, Qian Zheng, and Gang Pan. Spiking nerf: Representing the real-world geometry by a discontinuous representation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13790-13799 2024.
[28] Weng Fei Low and Gim Hee Lee. Robust e-nerf: Nerf from sparse & noisy events under non-uniform motion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 18335- 18346, 2023.
[29] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809. IEEE, 2024.
[30] Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V Sander. Deblur-nerf: Neural radiance fields from blurry images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12861- 2022.
[31] Qi Ma, Danda Pani Paudel, Ajad Chhatkuli, and Luc Van Gool. Deformable neural radiance fields using rgb and event cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3590-3600, 2023.
[32] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1)999-106, 2021.
[33] Anish Mittal, Anush K Moorthy, and Alan C Bovik. Blindtreferenceless image spatial quality evaluator. In 2011 conference record of the forty fifth asilomar conference on signals, systems and computers (ASILOMAR), pages 723-727. IEEE, 2011.
[34] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEFE Transactions on image processing, 21112)44695-4708, 2012.
[35] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):209-21 202, 2012.
[36] Yunshan Qi, Lin Zhu, Yu Zhang, and Jia Li. E2nerf: Event enhanced neural radiance fields from blurry images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13254- 13264, 2023.
12

---

[37] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision. pages 12179-12188, 2021.
[38] Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, and Vladislav Golyanik. Eventnerf: Neural radiance fields from single colour event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4992-5002, 2023.
[39] Ryusuke Sagawa, Ryo Furukawa, and Hiroshi Kawasaki. Dense 3d reconstruction from high frame-rate video using static grid pattern. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(9): 1733-1747, 2014.
[40] Johannes Lutz Schönberger and Jan-Michael Frahm. sanucture-fro-mmoiiin revisited. In Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[41] Shuzhe Wang. Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric
3d vision made easy. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697-207000 2024.
[42] Shuaixian Wang, Haoran Xu, Yaokun Li, Jiwei Chen, and Guang Tan. le-nerf: Exploring transient mask inpainting to enhance neural radiance fields in the wild. Neurocomputing, 618:129112 12, 2025.
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4))600-61 612, 2004.
[44] Yuchen Weng, Zhengwen Shen, Ruofan Chen, Qi Wang, and Jun Wang. Fadeblur-gs: Event assisted 3d deblur reconstruction with gaussian splatting. arXiv. 2024.
[45] Guanjun Wu, Taoran Yi, Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-timo dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20310 10120000 2024.
[46] Xijie Xiang, Lin Zhu, Jianing Li, Yixuan Wang, Tiejun Huang, and Yonghong Tian. Learning super- resolution reconstruction for high temporal resolution spike stream. IEEE Transactions on Circuits and Systems for Video Technology, 33(1): 16-29, 2021.
[47] Jiangtao Xu, Jiawei Zou, Zhiyuan Gao, and Jianguo Ma. Analysis input-dependent noise in self-timed reset dynamic vision sensor and its impact on data quality. IEEE Sensors Journal, 19(15):6240-625 2019.
[48] Jiangtao Xu, Liang Xu, Zhiyuan Gao, Peng Lin, and Kaiming Nie. A denoising method based on pulse interval compensation for high-speed baske image sensor. IEEE Transactions on Circuits and Systems for Video Technology, 31(8):2966 2116)299662280 2020.
[49] Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint Xiv:22501. 13928. 2025.
[50] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-20341, 2024.
[51] Jinze Yu, Xin Peng. Zhengda Lu, Laurent Kneip, and Yiqun Wang. Spikegs: Learning 3d gaussian fields from continuous spike stream. Proceedings of the Asian Conference on Computer Vision, pages 4280-4298, 2024.
[52] Wangbo Yu, Chaoran Feng, Jiye Tang, Xu Jia, Yuan, and Yonghong Tian. Evagaussians: Event stream assisted gaussian splatting from blurry images. arXiv preprint arXiv:2405.2022. 2024.
[53] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatring, In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 19447-19456, 2024.
[54] Jiyuan Zhang, Kang Chen, Shiyan Chen, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Spikegs: 3d gaussian splatting from spike streams with high-speed camera motion. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9194-9203, 2024.
[55] Junyi Zhang, Charles Hermmann Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan arXiv preprint 2110022410 2024. 2024.
Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion.
13

---

[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 586-595, 2018.
[57] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11996-12005. 2021.
[58] Jing Zhao, Ruiqin Xiong. Jian Zhang, Rui Zhao, Hangfan Liu, and Tiejun Huang. Learning to super-rese resolve dynamic scenes for neuromorphic spike camera. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1379-3587 2023.
[59] Rui Zhao, Ruiqin Xiong, Jian Zhang, Zhaofei Yu, Shuyuan Zhu, Lei Ma, and Tiejun Huang. Spike camera image reconstruction using deep spiking neural networks. IEEE Transactions on Circuits and Systems for Video Technology, 34(6):5207-5211 2023.
[60] Rui Zhao, Ruiqin Xiong, Jing Zhao, Jian Zhang, Xiaopeng Fan, Zhaofei Yu, and Tiejun Huang. Boosting spike camera image reconstruction from a perspective of dealing with spike fluctuations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24955-24965, 2024.
[61] Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang. Deep learning for event-based vision: A comprehensive survey and benchmarks. arXiv, 2023.
[62] Yajing Zheng, Lingxiao Zheng, Zhaofei Yu, Boxin Shi, Yonghong Tian, and Tiejun Huang. High-speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6358-6367, 2021.
[63] Hancheng Zhu, Leida Li, Jinjian Wu. Weisheng Dong. and Guangming Shi. Metaiqa: Deep meta-learning for no-reference image quality assessment. In Proceedings of the IEFE//VF conference on computer vision and pattern recognition, pages 14143-14152, 2020.
[64] Lin Zhu, Siwei Dong, Tiejun Huang. and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432-1437. IEEF, 2019.
[65] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image recon- struction via spiking neural model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1438-1446, 2020.
[66] Lin Zhu, Jianing Li, Xiao Wang. Tiejun Huang, and Yonghong Tian. Neuspike-net: High speed video reconstruction via bio-inspired neuromorphic cameras. In Proceedings of the IEEF/CVP International Conference on Computer Vision, pages 2400-2409, 2021.
[67] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Ultra-high temporal resolution visual reconstruction from fovea-like spike camera via spiking neuron model. IEEE Transactions on Pattern 2022.
Analysis and Machine Intelligence, 45(1):1233-1
[68] Lin Zhu, Yunlong Zheng, Mengyue Geng, Lizhi Wang, and Hua Huang, Recurrent spike-based image restoration under general illumination. In Proceedings of the 31st ACM International Conference on Multimedia, pages 8251-8260, 2023.
[69] Lin Zhu, Kangmin Jia, Yifan Zhao, Yunshan Qi, Lizhi Wang, and Hua Huang. Spikenerf: learning neural radiance fields from continuous spike stream. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition, pages 6285-6295, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the contributions of this work the end of the Introduction, and elaborate on how we achieve them the Methodology.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification: We present the assumption and its proof in our Methodology section, along with clear statements and proper references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed model designs and experimental settings in the main text.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
The dataset and code used in this work will be released shortly after simple organization.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpollliigggddde/oooooeo Soweeeeee/ooeSuubiiieoo ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The detailed experimental settings are presented in Section 4.2 and the supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We use statistical significance in the evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources are described Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
There are no ethical concerns involved in this work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, and models) used in the paper are properly credited, and the license and terms of explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21


Reviews:
[{"review_id": "GfqP43tIw5", "reviewer": "Reviewer_KHNA", "summary": "This paper proposes Spike4DGS, the first high-speed dynamic scene rendering framework with 4D Gaussian Splatting using spike camera arrays. 3DGS from conventional cameras is fundamentally limited by framerate, and therefore struggle with fast-moving scenes. Spike cameras asynchronously encode absolute light intensity into continuous spike streams at rates of up to 20k Hz. However, prior works with spike cameras are limited to 3DGS: thereby suffering in dynamic scenes. \n\nTo solve this, the input multi-view spike streams are sent to a Multi-view Spike-based Dense Initialization to estimate point cloud and camera poses. MSDI consists of ViT based encoders which are fused to generate a point cloud initialization and pose estimates. The features are also used to generate discrete images from the spike stream. Using these as inputs to a 4DGS generation pipeline, a 4DGS model may be generated. To better supervise 4DGS training, in addition to the pixel level loss, rendered images are also converted to spike streams, which can be supervised against the original spike stream.\n\nThe authors propose both a synthetic and real multi-view dataset of spike streams. On this dataset, the proposed method outperforms comparison methods in terms of novel view rendering quality on both synthetic and real-world datasets. Ablation studies show the benefit of various components in the method.", "questions": "1. The text in the figure 4 results seems flipped. Why is this?\n2. Can we have some sense of training time, and how it compares with baseline methods?\n3. Is there any reason why all the cameras were arranged in a planar manner?", "limitations": "Yes, with some suggestions in the answers above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is written well, and it is reasonably straightforward to understand the method and contributions\n2. The motivation for the method, in terms of capturing high-dynamic scenes using spike cameras, is well described\n3. This paper proposes the first 4DGS method for spike cameras\n4. Results show superior performance against compared baselines\n5. Ablations motivate the need for various method components well\n\nWeaknesses:\n1. Since this is a 4DGS method, I think it is imperative to see several novel views for a given scene to show that the generated 3D representation is indeed robust, which I could not find in the paper.\n2. Overall, the method seems largely sequential. First, spike camera views are converted into traditional RGB camera representations (images, point clouds, camera poses etc.), and these are passed through an existing 4DGS method. So it seems the major novelty is the conversion of spike camera views to RGB views.\n3. The training camera views are all planar and therefore cannot truly represent the entire 3D scene, meaning the 3D representation quality on faraway camera views cannot be tested. \n4. It is not clear what type of scenes this method is good at representing vs what types of scenes it is not good for: the limitations section should incorporate this.", "was_revised": true, "final_justification": "I would like to thank the authors for the response and the discussion. After reading the comments from other reviewers, and the authors responses, I would like to weight towards acceptance and lean towards a 'weak accept' score, in context of the strengths and weaknesses of this work.", "rebuttal": "Thank you for your positive and thoughtful comments. We are delighted that you find our idea very important and the results great success and impressive. We would like to address your concerns and answer your questions in the following.\n\n# Respond to W1\n>Since this is a 4DGS method, I think it is imperative to see several novel views for a given scene to show that the generated 3D representation is indeed robust, which I could not find in the paper.\n\nIn fact, for both the real-world and synthetic datasets, the images shown in our Figures 3, 4, and 5 comparisons are rendered from a novel view. Specifically, for the real-world dataset, we employ a 3×3 spike camera array. For the synthetic dataset, we simulate the same 3×3 camera grid. In both cases, we select the central view as the novel test view and other 8 views as training views. As shown in Figures 3, 4, and 5,  the novel view performances demonstrate that our method maintains reasonable robustness, consistent with the goals of a 4DGS framework.\n\n# Respond to W2\n>Overall, the method seems largely sequential. First, spike camera views are converted into traditional RGB camera representations (images, point clouds, camera poses, etc.), and these are passed through an existing 4DGS method. So it seems the major novelty is the conversion of spike camera views to RGB views.\n\nWe appreciate the reviewer’s observation. While our pipeline does leverage the strength of existing 4DGS techniques for scene representation and rendering, we emphasize that our contribution goes far beyond a simple conversion from spike views to RGB representations.\n\n**Introduction of spike variation supervision**\nApart from MSDI, we also propose a Spike-Pixel Synergy Supervision, which introduces a spike variation supervision.\nThis loss encourages local consistency in the temporal dimension, effectively reducing noise in the spike stream without excessively blurring the true motion.  It measures how much the values change on average, and how much they vary. This helps make the output more stable and smooth, but still allows sharp motion when needed.\nTo further validate its effectiveness, we conducted comparative experiments on the synthetic dataset, evaluating the average PSNR and SSIM. We compare our spike total variation loss (TV loss) with naive L1 loss in spikes, as shown in Table.R1. Our spike TV loss in SPSS gets the SOTA result.\n\n#### **Table.R1 Impact of Spike Loss Function Choice in SPSS.**\n||Avg PSNR ↑ |Avg SSIM ↑ |\n|---------------------|--------|--------|\n| Ours(TV loss)| **27.74**| **0.912**  |\n| w/o TV loss,use L1 loss|26.87| 0.906  |\n| w/o any spike loss | 26.66  | 0.903 |\n\n**Introduction of MSDI**\nWe introduce MSDI, a framework tailored for spike-based 3D vision, which is fundamentally different from conventional RGB pipelines such as Dust3R. MSDI includes not only a module for converting spike data into images, but also designs for camera pose estimation and 3D geometry reconstruction directly from spike modality. We quantitatively evaluate the camera pose estimation accuracy of MSDI in comparison with Colmap and Dust3R [1] on the synthesis dataset. The ground truth camera poses could be obtained from the Carla simulator. The Rotation Error (°)  and Translation Error are computed as same as Dust3R [1]. The results are shown in the table below Table.R2:\n\n#### **Table.R2 Comparison of Camera Pose Estimation results.**\n| Method           | Rotation Error (°) ↓ | Translation Error (%) ↓ | Time  ↓ |\n|------------------|----------------------|--------------------------|------------|\n| TFI+COLMAP | 2.72                 | 9.8                      | 30.3s       |\n| TFI+Dust3R   | 1.27                 | 3.5                      | 9.6s       |\n| Ours (MSDI)      | **0.94**             | **2.1**                  | **9.1s**    |\n\n# Respond to Q1\n> The text in Figure 4 results seems flipped. Why is this?\n\nThe observed image flip arises from differences in the data acquisition devices. The raw spike data is collected using a spike camera array, whereas the last column of RGB images is captured by a handheld RGB mobile phone, solely for the purpose of illustrating the real-world scenes. These RGB images are not used in model training or in the rendering pipeline.\n\nThe flip is not caused by our model or the rendering process. Instead, it results from a transpose operation applied during the preprocessing of spike data. In contrast, the RGB images are captured directly without any such operation. This discrepancy has no impact on the reconstruction results.\n\n# Respond to Q2\n> Can we have some sense of training time, and how it compares with baseline methods?\n\nFor this question, we compiled a comparison table of the training time of our method and the baseline methods on our synthesis dataset. We report the training time and the average PSNR of each method on one Nvidia RTX 4090, as shown in Table R.3.\n\n#### **Table.R3 Training Time and the Average PSNR of Each Method**\n| Methods  | TFI+STG[2]  | TFI+4DGS [3] | TFI+D3DGS[4]  | Ours |\n|----------|-------|-------|-------|-----------|\n| Avg time | 24min | 22min | 19min | 25min     |\n| Avg PSNR     | 24.70   | 26.00   | 20.34 |  **27.74** |\n\nWhile our method achieves similar training efficiency, its key contribution lies in the MSDI module and dual-modality supervision design. Both of them significantly improve the reconstruction results.\n# Respond to Q3\n> Is there any reason why all the cameras were arranged in a planar manner?\n\nIn practice, we used a 3×3 camera array and ensured that the object was centered directly in front of the middle camera. All surrounding cameras were tilted inward and oriented toward the center camera, forming a conical viewing region with significant overlap among the fields of view. This configuration ensures that each scene is well covered from multiple angles, which is reasonable for successful 3D reconstruction. The dataset will be released with the publication of the paper. \n\n***\n##### [1] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.\n##### [2] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508–8520, 2024.\n\n##### [3] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310–20320, 2024.\n\n##### [4] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800–809. IEEE, 2024."}, {"review_id": "A6OUW7OCLA", "reviewer": "Reviewer_xVW6", "summary": "This paper presents a novel framework for high-speed dynamic scene rendering. It uses an array of spike cameras to capture dynamic scenes at very high temporal resolution to reduce motion blur, and then applies 4D Gaussian Splatting (4DGS) for Novel View Synthesis (NVS). The core components include a Multi-view Spike-based Dense Initialization (MSDI) module, which converts spike camera streams into 4DGS readable formats, and a Spike-Pixel Synergy Supervision (SPSS) strategy, which enables direct 4DGS supervision using spike signals.", "questions": "The selling points of the paper seem to be the MSDI and SPSS modules, but I am having a hard time assessing their claimed effectiveness. So, my questions are around the method design and evaluation:\n1.\tEquation (17): Can you explain why you chose the total variation loss rather than an L1 or L2? A brief justification of this design choice would help clarify its role.\n2.\tMSDI is very interesting to me. It would be helpful if the authors could provide a more detailed evaluation of this module—for example, by feeding the MSDI outputs (point clouds, reconstructed images, and poses) directly into 4DGS without SPSS. This would help identify whether the gains come primarily from better pointclouds/poses (versus DUSt3R) or better images (versus spk2img).\n3.\tCan you also try replacing only the initialization pointcloud with one from DUSt3R or COLMAP while keeping everything else the same? This would help isolate the contribution of your initialization strategy as any Gaussian Splatting based rendering method is greatly reliant on the initial pointcloud input.\n4.\tCould you showcase some examples in more realistic settings—such as a car driving down a road in real lighting conditions? I understand that your pipeline assumes uniform lighting, but seeing examples in real-world scenarios would help reviewers better assess its effectiveness and generalizability. I may reconsider my final rating depending on these results.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper thoroughly explains the current limitations in dynamic scene rendering, particularly the motion blur introduced by conventional low-speed cameras.\n2. The idea of using a spike camera array to alleviate motion blur and enable multi-view NVS is intuitive, reasonable, and novel.\n3. The proposed MSDI and SPSS modules, if fully evaluated, have the potential to support many 3DGS-related downstream tasks.\n4. Quantitatively, although there is no prior work with an identical setup, the reported metrics surpass the SOTA in dynamic NVS tasks by a significant margin.\n\nWeaknesses:\n1. The Methodology section lacks clarity. The authors simply list a series of equations without explaining the intuition or reasoning behind these design choices, making the section very difficult to follow.\n2. There is little to no analysis in the Experiments section. It is unclear to the reviewer which components actually contribute to the large performance gains, raising questions about the setup for other methods.\n3. Limited generalizability: The real-data experiments only cover small-scale, toy examples. The authors also mention providing ideal lighting conditions when collecting real data. Taken together, these factors raise questions about the pipeline’s ability to handle datasets outside their own collection.\n4. Too many typos: The paper contains way too many typos and grammatically incorrect sentences, making it quite distracting to read.", "was_revised": true, "final_justification": "The author provided an excellent rebuttal that addressed most of my concerns. I would like to raise my rating and suggest that the author add Q3 and Q4 to the revision if the paper is accepted.", "rebuttal": "We sincerely appreciate your valuable comments on the weaknesses in our manuscript's writing issues. Due to space limitations, we were unable to provide sufficient details in the method and experiment sections. Here, we provide detailed explanations and clarifications in response to your questions Q1–Q4.\n***\n# Response to Q1\n> Equation (17): Can you explain why you chose the total variation loss rather than an L1 or L2? A brief justification of this design choice would help clarify its role. \n>The Methodology section lacks clarity. The authors simply list a series of equations without explaining the intuition or reasoning behind these design choices, making the section very difficult to follow.\n\n**Reason for loss choice**\nDifferent from static scenes, objects in high-speed moving scenes are constantly changing, thus the segment of spikes in a time interval is a continuously changing sequence. L1 or L2 loss treats each frame separately, without considering how things move over time. In contrast, TV loss helps keep motion smooth and consistent over time. TV loss looks at the differences between frames. It measures how much the values change on average, and how much they vary. This helps make the output more stable and smooth, but still allows sharp motion when needed.\n\n**Comparison Experiments between losses**\nTo further validate our motivation, we conducted comparative experiments on the synthetic dataset, evaluating the average PSNR and SSIM under L1, L2, and temporal TV losses, as shown in Table.R1.\n\n#### **Table.R1 Impact of Spike Loss Function Choice in SPSS.**\n|Metric|L1|L2|Our TV Loss|\n|:----:|:--:|:--:|:--------:|\n|Avg PSNR↑|26.87|26.75|**27.74**|\n|Avg SSIM↑|0.906|0.905|**0.912**|\n***\n# Response to Q2\n>MSDI is very interesting to me. It would be helpful if the authors could provide a more detailed evaluation of this module—for example, by feeding the MSDI outputs (point clouds, reconstructed images, and poses) directly into 4DGS without SPSS. This would help identify whether the gains come primarily from better pointclouds/poses (versus DUSt3R) or better images (versus spk2img).\n\nThank you for your attention to MSDI! Regarding your second question, we have actually reported the effect of MSDI combined with 4DGS without SPSS in Table 3. The results demonstrate that, as an initialization module, MSDI outperforms TFI+colmap and DUSt3R in our Spike4DGS pipeline, and the SPSS module further improves the performance of each pipeline.\n\nIn light of your insightful comment, **we conducted additional experiments to investigate this aspect**. Specifically, we feed the MSDI outputs (i.e., reconstructed images, point clouds and poses) into standard 4DGS procedure respectively to validate the contribution of MSDI.  In addition,  the traditional spike-to-image methods (e.g., TFI) and MSDI+SPSS are also listed for comparisons. The experiments are conducted on the synthetic dataset and the average performances are reported in the following Table.R2. It indicates both the image and point clouds/ camera poses of MSDI contribute positively. \n\n#### **Table.R3 Comparison of dynamic reconstruction methods with different image sources(PSNR/SSIM) without SPSS**\n| Method |Avg PSNR↑  |Avg SSIM↑ |\n|---------------------------------|-------|-------|\n| TFI + COLMAP + 4DGS             | 21.08 | 0.849 |\n| TFI + DUST3R + 4DGS             | 25.66 | 0.893 |\n| MSDI IMAGE + COLMAP + 4DGS      | 22.43 | 0.854  |\n| MSDI IMAGE + DUST3R + 4DGS      | 25.76 | 0.898|\n| MSDI (IMAGE + POINTCLOUD/POSE) + 4DGS       | **26.66** | **0.903** |\n| MSDI (IMAGE + POINTCLOUD/POSE) + SPSS (Ours) | **27.74** | **0.912**|\n***\n# Response to Q3\n>Can you also try replacing only the initialization pointcloud with one from DUSt3R[4] or COLMAP[5] while keeping everything else the same? This would help isolate the contribution of your initialization strategy, as any Gaussian Splatting-based rendering method is greatly reliant on the initial point cloud input.\n\nWe fully agree that this is an important experiment to assess the impact of the initialization point cloud quality on the final reconstruction results. To this end, **we conducted additional experiments where we replaced only the initialization point cloud and camera poses**. \nBased on this setting, we conduct the experiments on the synthetic dataset, and the average performances are reported in the following Table.R4.\n\n#### **Table.R4 Our full Spike4DGS on different initialization point clouds.**\n|Methods|Avg PSNR↑|Avg SSIM↑| \n|--------|-------|----|\n|MSDI IMAGE + COLMAP+SPSS|23.68|0.867|\n|MSDI IMAGE + Dust3R+SPSS|26.92|0.907|\n|MSDI +SPSS (Full Model)|**27.74**|**0.912**|\n***\n# Response to Q4\n>Could you show case some examples in more realistic settings—such as a car driving down a road in real lighting conditions? I understand that your pipeline assumes uniform lighting, but seeing examples in real-world scenarios would help reviewers better assess its effectiveness and generalizability. I may reconsider my final rating depending on these results.\n\nWe thank the reviewer for the helpful suggestion. **In response, we collected a new scene dataset in an outdoor environment under natural lighting.** We used the dataset to reconstruct some motion examples in real daylight. This new dataset is similar to the scenes in the synthesis data, such as a moving car and a running pedestrian. But the lighting is uneven, and the background is not controlled. These examples add to our previous real-data results. They show that our method can handle complex scenes with changing light and outdoor noise.\n\nDue to rebuttal policies, we are unable to include images. Instead, we provide a table summarizing avg BRISQUE and NIQE on those realistic scenes collected outdoors under natural lighting conditions in Table.R5. \n\n#### **Table.R5 Quantitative Results (BRISQUE↓ / NIQE↓) on More Real Spike-Captured Dynamic Scenes.**\n| Method             | Moving Car  | Running Pedestrian | Moving Bicycle  |\n|--------------------|----------------------------------------|-----------------------------------------------|--------------------------------------------|\n| TFI + D3DGS  | 60.12 / 17.32                  | 58.78 / 15.98                         | 59.27 / 15.04                       |\n| TFI + STG    | 43.26 / 13.20                  | 46.86 / 13.76                        | 43.52 / 13.15                       |\n| TFI + 4DGS   | 35.12 / 11.24                 | 35.37 / 12.05                         | 33.86 / 12.14                       |\n| Spike4DGS          | **30.63 / 10.32**                  | **31.12 / 9.89**                          |**29.95 / 10.01**                       |\n\nThese results demonstrate that our method remains robust even without assuming ideal lighting conditions. In future work, we will continue to explore reconstruction performance under more realistic scenarios.\n\n***\n##### [1] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432–1437. IEEE, 2019.\n##### [2] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1438–1446, 2020.\n##### [3] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11996–12005, 2021.\n##### [4] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.\n##### [5] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.\n##### [6] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310–20320, 2024."}, {"review_id": "sKuo3KdhYs", "reviewer": "Reviewer_Q7d2", "summary": "The paper proposes Spike4DGS, the first method for reconstructing high-speed dynamic scenes using a spike camera array. The approach begins with a Multi-view Spike-based dense initialization to generate an initial point cloud, estimate camera poses, and synthesize RGB images from the spike streams. These synthesized images and estimated poses are then used to optimize 4D Gaussian Splatting (4DGS). To further enhance reconstruction quality, the method incorporates a novel Spike Variation Supervision mechanism that leverages the unique characteristics of spike data. Additionally, the authors create two benchmark datasets, one synthetic and one real-world, to evaluate the method’s performance. Extensive experiments demonstrate that Spike4DGS achieves superior results compared to existing baselines.", "questions": "See weakness 1, 2, 3.", "limitations": "Discussed in supplmentary section I.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper introduces Multi-view Spike-based Dense Initialization (MSDI) to jointly initialize the point cloud and estimate camera poses directly from spike streams, while simultaneously recovering images. This unified approach replaces the traditional two-step pipeline: first recovering images and then performing point cloud initialization, thereby reducing reliance on spike-to-image conversion methods, which may produce suboptimal results.\n2. The authors introduce one synthetic and one real-world dataset as benchmarks for evaluating high-speed dynamic scene reconstruction from spike streams. These datasets may serve as valuable resources for the community and help advance future research in spike-based reconstruction.\n3. A spike variation loss is introduced to exploit the rich texture and motion cues inherent in the spike stream, thereby enhancing the quality of the reconstructed scenes.\n4. The paper is well-written and easy to follow.\n\n\nWeakness\n1. All baseline methods follow a two-stage pipeline of spike-to-image conversion followed by dynamic reconstruction. Would it be more appropriate to consider a stronger baseline by directly modifying existing spike-based methods such as SpikeGS or SpikeNeRF to support dynamic rendering—e.g., by integrating a deformation field? This could provide a more direct and fair comparison.\n2. A visual and quantitative comparison between the RGB images generated by spike-to-image methods (e.g., TFI, TFP, spk2img) and those produced by the proposed MSDI module would better validate the effectiveness of the spike-based initialization.\n3. In Figure 5, the rendering quality degrades noticeably as the number of input views decreases. It would be helpful for the authors to clarify how many input views were used in the comparisons with baseline methods. Additionally, providing a comparison with baselines under varying numbers of input views would offer valuable insight into the robustness of the proposed method in sparse-view settings.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding the comparison with Dynamic Spike-GS and Dynamic Spike-NeRF, the quality comparison between the proposed MSDI and traditional spike-to-image methods, as well as the clarification of the training view setting. Thus I will maintain my positive score.", "rebuttal": "# Response to W1\n>All baseline methods follow a two-stage pipeline of spike-to-image conversion followed by dynamic reconstruction. Would it be more appropriate to consider a stronger baseline by directly modifying existing spike-based methods such as SpikeGS or SpikeNeRF to support dynamic rendering—e.g., by integrating a deformation field? This could provide a more direct and fair comparison.\n\nWe thank the reviewer for his insightful suggestion. **Both SpikeGS [1] and SpikeNeRF [2] focus on reconstructing static scenes and are not applicable to dynamic scenarios involving object motion.** \n\nFollowing your suggestions, we conducted additional experiments. As there are currently no publicly available dynamic extensions for these methods, we manually integrated a deformation network from 4DGS into SpikeGS and SpikeNeRF (denoted as Dy-SpikeGS and Dy-SpikeNeRF) and adapted their data reader pipelines accordingly.  Experiments are conducted on the synthesis dataset, and the average performances are reported in Table R1.  As shown in Table R1, both original and modified dynamic versions of SpikeGS and SpikeNeRF have poor performance. **Note Dy-SpikeGS and Dy-SpikeNeRF are both re-designed and re-implemented, their results are just for reference.**\n\n#### **Table.R1 Comparison with manually Modified SpikeGS and SpikeNeRF.**\n\n| Method        |Avg PSNR(dB)↑ |Avg SSIM↑ \n|---------------|-----------|-------|\n| Original SpikeNeRF| 14.43|0.576\n| Original SpikeGS | 20.03|0.808\n| Dy-SpikeNeRF| 16.70     | 0.628\n| Dy-SpikeGS | 23.31     | 0.835 \n| Spike4DGS   | **27.74**     | **0.912**\n\n\n***\n# Response to W2\n>A visual and quantitative comparison between the RGB images generated by spike-to-image methods (e.g., TFI, TFP, spk2img) and those produced by the proposed MSDI module would better validate the effectiveness of the spike-based initialization.\n\nTo validate the effectiveness of our MSDI module for spike-to-image initialization, **we conducted quantitative comparisons with representative spike-to-image methods, including TFI, TFP, and spk2img.**\n\nOn the synthetic dataset where ground truth images are available, we evaluated the spike-to-img quality using average PSNR and SSIM.  The results are summarized in Table.R2. Note the reconstruction images in test have the same view with the training images, hence the improvements are more clearly than NVS.\n\n#### **Table.R2 Comparison of spike-to-img quantitative results on synthesis dataset.**\n\n|   Metric   |  TFI [3] |  TFP[3]  |  TVS[4]  | Spike2img[5] |   MSDI's images   |\n|:------:|:------:|:-----:|:-----:|:---------:|:---------:|\n|Avg  PSNR↑  |  24.51 | 25.37 | 22.43 |   30.76   | **34.90** |\n|Avg  SSIM↑  | 0.850  | 0.852 | 0.821 |   0.904   | **0.961** |\n\nOn real-world data, we adopted the no-reference Brisque↓/NIQE↓ metric. The results are summarized in Table.R3. Our MSDI module consistently achieves superior performance across all metrics, demonstrating its ability to produce more accurate and perceptually coherent images from raw spike inputs.\n\n#### **Table.R3 Comparison of spike-to-img Brisque↓/NIQE↓ (↓) on real-world dataset.**\n|   Scene   |  TFI [3] |  TFP[3]  |  TVS[4]  | Spike2img[5] |   MSDI's images   |\n|:---------:|:-----:|:-----:|:-----:|:---------:|:--------:|\n| Avg.Brisque↓ |36.98|33.45|40.73|30.23|20.34\n|Avg.NIQE↓|14.92 |13.31|17.43|10.13|**7.68**\n***\n# Response to W3\n> In Figure 5, the rendering quality degrades noticeably as the number of input views decreases. It would be helpful for the authors to clarify how many input views were used in the comparisons with baseline methods. Additionally, providing a comparison with baselines under varying numbers of input views would offer valuable insight into the robustness of the proposed method in sparse-view settings.\n\n**Camera Settings**\nIn all experiments reported, we used the full 3×3 spike camera array, including 8 training views and 1 testing view, to ensure fair and consistent comparison with baseline methods. The testing view is always the center camera in the array. In Figure 5, the first column corresponds to 4 training views and the test view. The second column corresponds to 6 training views and the test view. \n\n**Comparison Experiments**\nFollowing your suggestion, we have additionally conducted experiments on baseline methods with varying numbers of training views to provide a more comprehensive comparison. All conditional experiments are conducted on our real-world \"Turntable\" dataset and shown in Table.R4, and the number of test views is kept to 1 central view.\n\n#### **Table.R4 Impact of Number of Total Training Views for Different Reconstruction Methods.(Brisque↓/NIQE↓)**\n| Methods       | 3 Views       | 4 Views       | 5 Views       | 6 Views       | 7 Views       | 8 Views       |\n|--------------|---------------|---------------|---------------|---------------|---------------|---------------|\n| TFI+D3DGS[6] | 83.4/23.87    | 73.6/21.67    | 65.2/20.81    | 62.9/18.6     | 59.8/16.20    | 56.53/ 15.34   |\n| TFI+STG[6]   | 71.0/22.8     | 61.2/17.5     | 57.0/16.7     | 52.1/14.6     | 42.0/13.1     | 36.45/10.53     |\n| TFI+4DGS[7]  | 70.3/22.7     | 60.8/18.4     | 56.4/16.5     | 51.8/14.4     | 43.6/13.9     | 37.57/9.93     |\n| Ours         | **63.43/18.54** | **53.62/15.67** | **45.23/13.81** | **32.91/11.67** | **29.84/10.2** | **26.86/9.86** |\n\nOur total view configuration experiments follow the same setup as in the main paper, consistently using the center camera of the 3×3 array as the test view. In summary, our method demonstrates strong reconstruction capability and high image quality across different input view densities, maintaining advantages even under sparse view conditions, which reflects the robustness and effectiveness of the approach.\n\n***\n##### [1] Jinze Yu, Xin Peng, Zhengda Lu, Laurent Kneip, and Yiqun Wang. Spikegs: Learning 3d gaussian fields from continuous spike stream. In Proceedings of the Asian Conference on Computer Vision, pages 4280–4298, 2024.\n##### [2] Lin Zhu, Kangmin Jia, Yifan Zhao, Yunshan Qi, Lizhi Wang, and Hua Huang. Spikenerf: Learning neural radiance fields from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6285–6295, 2024.\n##### [3] Lin Zhu, Siwei Dong, Tiejun Huang, and Yonghong Tian. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432–1437. IEEE, 2019.\n##### [4] Lin Zhu, Siwei Dong, Jianing Li, Tiejun Huang, and Yonghong Tian. Retina-like visual image reconstruction via spiking neural model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1438–1446, 2020.\n##### [5] Jing Zhao, Ruiqin Xiong, Hangfan Liu, Jian Zhang, and Tiejun Huang. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11996–12005, 2021.\n##### [6] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8508–8520, 2024.\n##### [7] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20310–20320, 2024."}, {"review_id": "cgv5LJCnpF", "reviewer": "Reviewer_mUPw", "summary": "This paper constructs an array of spike cameras and uses it to capture multiview images of a dynamic scene. With this novel dataset in hand, the authors then develop and apply 4D Gaussian splatting reconstruction framework to reconstruct a dynamic volumetric reconstruction of the scene. The proposed reconstruction algorithm has a novel initialization scheme and loss function. The initialization scheme estimates camera poses and an initial point cloud from an input spike stream using a vision transfomer. The loss function consists of (1) a spike loss between the capture data and spikes rendered by the reconstruction and (2) a pixel loss between images reconstructed directly from the spike cameras and those rendered by the reconstruction. The proposed method outperforms existing methods on real and simulated data.", "questions": "Could the authors provide qualitative evidence that the method is successfully reconstructing 3D geometry? E.g., a visualization of the 3DGS depth map or views rendered from a sequence of perspectives that were not originally captured. This is my major criticism of the current manuscript---it claims 3D reconstruction but doesn't convincingly demonstrate it.\n\nAre Figures 3, 4, and 5 rendered from novel viewpoints or viewpoints that were already captured? How close are the rendered views' perspectives to an existing view's?\n\nWill the datasets be publicly released? The paper lists the dataset as a contribution, suggesting it will be, but this should be made more explicit.\n\nDid the cameras have overlapping view points? (In the main text they are positioned s.t. their views likely wouldn't overlap.) If the viewpoints are not overlapping and the camera is not moving, how is 3D reconstruction being performed?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\nTo my knowledge, the proposed method and camera array are novel\n\nThe dataset, assuming it is publicly released, would be useful\n\nThe proposed method significantly outperforms existing methods.\n\n## Weaknesses:\n\nMissing Evidence: The paper provides limited evidence that 3D reconstruction was successful. While tables 2-4 in the supplement report novel view reconstruction accuracy on simulated data, there are no \"fly-through\" in the supplement, depth maps, or other qualitative evidence showing that accurate 3D geometry was recovered. The demo videos are rendered from a fixed perspective and fail to demonstrate any 3D geometry was successfully recovered. It's unclear if the results shown in the paper are rendered from a novel view point or one that was already captured by one of the cameras.\n\nClarity and missing information: The main text shows the cameras positioned in such as way that their viewpoints wouldn't overlap and 3D reconstruction would be extremely challenging. The supplement suggests these cameras were positioned with overlapping views. Please provide information about the position, orientation, FoV, etc. of each camera. \n\nIn the abstract and title, the paper uses the word \"rendering\" when what is really meant is \"reconstruction\".", "was_revised": true, "final_justification": "The authors clarified that their reported results were actually testing novel view synthesis capabilities.", "rebuttal": "Thank you for your positive and thoughtful comments. We would like to address your concerns and answer your questions in the following.\n\n# Response to Q1 and W1\n>Could the authors provide qualitative evidence that the method is successfully reconstructing 3D geometry? E.g., a visualization of the 3DGS depth map or views rendered from a sequence of perspectives that were not originally captured. This is my major criticism of the current manuscript---it claims 3D reconstruction but doesn't convincingly demonstrate it.\n\n>The paper provides limited evidence that 3D reconstruction was successful. While tables 2-4 in the supplement report novel view reconstruction accuracy on simulated data, there are no \"fly-through\" in the supplement, depth maps, or other qualitative evidence showing that accurate 3D geometry was recovered. The demo videos are rendered from a fixed perspective and fail to demonstrate any 3D geometry was successfully recovered. It's unclear if the results shown in the paper are rendered from a novel view point or one that was already captured by one of the cameras.\n\nWe have conducted fly-through visualizations to demonstrate the 3D stereo effect. Due to the rebuttal policy, we are currently unable to provide videos here. Upon public release, we will provide videos showcasing this stereo visualization. We appreciate the reviewer highlighting the importance of this aspect.\n\nTo further verify the reconstruction capability of our method, we have computed depth maps from our model's outputs. We extracted depth information from our synthesis dataset as ground-truth and used rendering depth images from our methods and baselines to perform quantitative comparisons. The results reported in Table.R1 are averaged across five different scenes from our synthetic dataset. We follow two metrics (Abs Rel and δ < 1.25) of Dust3R[1] to evaluate the depth results.\n\n#### **Table.R1 Evaluation of depth estimation accuracy on Avg Abs Rel and δ < 1.25.**\n| Method              |Avg Abs Rel ↓ |Avg δ < 1.25 ↑ |\n|---------------------|-----------|------------|\n| Spike2img + D3DGS   | 0.523     | 0.512     |\n| Spike2img + STG     | 0.340     | 0.538      |\n| Spike2img + 4DGS    | 0.342     | 0.536      |\n| Our Spike4DGS  | **0.195** | **0.565**  |\n\nAbs Rel (Average Absolute Relative Error) quantifies the mean absolute discrepancy between rendering and ground truth depth values, normalized by the ground truth depth. Lower Abs Rel values correspond to higher accuracy, indicating that the predicted depths closely approximate the true depths relative to their magnitude.\n\nδ < 1.25 (Threshold Accuracy at 1.25) denotes the fraction of rendering depth estimates for which the ratio between prediction and ground truth lies within the threshold of 1.25. A higher δ value means a greater proportion of rendering depth falling within an acceptable error margin, reflecting superior depth estimation fidelity. \n\n# Response to Q2 \n\n> Q2: Are Figures 3, 4, and 5 rendered from novel viewpoints or viewpoints that were already captured? How close are the rendered views' perspectives to an existing view's?\n\nWe apologize for the ambiguity. In fact, for both the real-world and synthetic datasets, the images shown in our main text Figures 3, 4, and 5 comparisons are rendered from **a novel view**. Specifically, for the real-world dataset, we use a 3×3 spike camera array and select the **central view** as the novel test view for evaluation. Similarly, for the synthetic dataset, we simulate the same 3×3 camera grid and also select the **central view** as the test viewpoint. All experiments in our main paper follow this setting. **We did not use training views for testing, strictly following the experimental protocol of the Novel View Synthetic task.**\n\n# Response to Q3\n\n>Will the datasets be publicly released? The paper lists the dataset as a contribution, suggesting it will be, but this should be made more explicit.\n\nWe appreciate the reviewer’s attention to our dataset contribution. \nFor the real-world dataset, we use purely spike data from 9 views. For the synthesis dataset, we additionally include clean RGB images and RGB images with motion blur from corresponding views. These RGB images are incorporated specifically to support evaluation metrics and qualitative comparisons.\n\nDue to the double-blind review policy, we are currently unable to provide direct dataset links. We confirm that the dataset, along with all related resources, will be made publicly available with the publication of the paper.\n\n# Response to W2  and Q4\n> Clarity and missing information: The main text shows the cameras positioned in such as way that their viewpoints wouldn't overlap and 3D reconstruction would be extremely challenging. The supplement suggests these cameras were positioned with overlapping views. Please provide information about the position, orientation, FoV, etc. of each camera.\n\n>Did the cameras have overlapping view points? (In the main text they are positioned s.t. their views likely wouldn't overlap.) If the viewpoints are not overlapping and the camera is not moving, how is 3D reconstruction being performed?\n\nThe camera arrangement illustrated in Figure 2 of the main text is for demonstration purposes only and does not exactly represent the actual setup used during data collection.\n\n**For our real-world dataset**, we used a 3×3 camera array from a square layout and ensured that the object was centered directly in front of the middle camera. The distance between neighboring cameras' positions is around 20 cm. During data collection, all surrounding cameras were tilted inward and oriented toward the center camera, forming a conical viewing region with significant overlap among the fields of view. This configuration ensures that each scene is well covered from multiple angles, which is reasonable for successful 3D reconstruction.\n\n# Response to W3\n>In the abstract and title, the paper uses the word \"rendering\" when what is really meant is \"reconstruction\".\n\nThank you for pointing out the misuse of the term \"rendering\" in the abstract and title. The  \"reconstruction\"  and  \"rendering\"  corresponds to the 3D scene and novel view synthesis, respectively. We will carefully check the manuscript  to avoid any confusion.\n\n***\n#### [1]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024."}, {"review_id": "N842c8kHJ1", "reviewer": "Reviewer_TNTX", "summary": "This paper proposes using a multi-view spike camera array to synthesize novel views of dynamic scenes. The authors first construct both synthetic and real-world multi-view spike datasets and introduce a GS initialization module that directly estimates dense point clouds and camera poses from spike streams, without relying on existing SfM tools like COLMAP or DUST3R. The framework further integrates 4D Gaussian Splatting for high-speed dynamic scene modeling.", "questions": "How does the spike-based dense initialization compare to initialization based on original images (e.g., COLMAP)? Table 3 only compares against spike-reconstructed images, which are of low quality.\n\nWhile some inaccuracy in point cloud is acceptable for initialization, it would be informative to provide some evaluations of point clouds and camera poses, especially compared to COLMAP and DUSt3R.\n\nWhen using MSDI for initialization, are the poses further refined in 4DGS stage? Is the pose accuracy comparable to COLMAP-based pipelines?\n\nWhat is the computational cost of the proposed method?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nA new dataset with multi-view spike camera data covering both synthetic and real-world scenes is introduced, which is valuable for further research.\n\nCompared with traditional two-stage methods that first reconstruct images before applying 4DGS, the proposed approach shows better performance, demonstrating the benefit of joint modeling.\n\nWeaknesses\n\nAlthough spike cameras offer high temporal resolution, this advantage is neither fully exploited in the framework design nor clearly reflected in the experimental results. \n\nOn the synthetic dataset shown in Figure 3, all methods achieve suboptimal results. It raises the question of whether image-based NVS methods (e.g., COLMAP + 4DGS) suffer from similar artifacts on this data. \n\nThe authors claim that traditional methods rely on high-quality image reconstruction (Line 140), but the proposed method also requires reconstructed images (Line 170), and the deformed Gaussians in Section 3.4 are highly dependent on image quality. The spike signals only serve as auxiliary loss. \n\nWhile several spike-based NVS methods are mentioned in Section 2.2, they are not directly compared in experiments. Can those methods be adapted to the multi-view setting? This deserves further discussion. \n\nThe proposed multi-view spike-based dense initialization lacks clarity in what makes it unique to spike data. If the input were replaced with multi-view images, the module might still function. \n\nIt remains unclear what the core advantage over the \"spike-to-image + image 4D NVS\" pipeline is: better initial point clouds and poses? Higher-quality reconstructed images? Or improved supervision through spike constraints?", "was_revised": true, "final_justification": "Based on the first rebuttal, I am willing to raise the score to borderline accept. However, I am not convinced by the author's subsequent clarifications, particularly as I believe the amount of spatiotemporal data of spikes significantly exceeds that of sharp image frames, so I am unable to give a more positive rating.", "rebuttal": "# Response to W1\n>Although spike cameras offer high temporal resolution, this advantage is neither fully exploited in the framework design nor clearly reflected in the experimental results.\n\nThanks for the reviewer’s insightful comment on underusing the spike camera’s high temporal resolution. To address this, we performed an ablation study by downsampling the spike data from 20,000 Hz to lower temporal resolutions while keeping the rest unchanged. This was done by removing frames at fixed intervals to simulate lower frame rates. Results are shown in Table.R1 on our synthesis dataset.\n\n#### **Table.R1 Performance of Spike4DGS under Varying Temporal Resolutions of Spike Input.**\n|Hz|20000|10000|5000|2000|1000|500|200|100|\n|--|--|--|--|--|--|--|--|--|\n|PSNR↑|**27.74**|25.43|22.71|21.88|18.32|16.87|16.22|15.54|\n|SSIM↑|**0.912**|0.885|0.842|0.810|0.745|0.685|0.650|0.625|\n\nThe reconstruction quality degrades significantly as the temporal resolution decreases. These results demonstrate that our method implicitly leverages and benefits from the high temporal resolution provided by the spike camera. \n\n# Response to Q1 and W2\n>How does the spike-based dense initialization compare to initialization based on original images (e.g., COLMAP)? Table 3 only compares against spike-reconstructed images, which are of low quality.\n\n>On the synthetic dataset shown in Figure 3, all methods achieve suboptimal results. It raises the question of whether image-based NVS methods (e.g., COLMAP + 4DGS) suffer from similar artifacts on this data.\n\nThank you for pointing this out. In response to your question, we conducted additional experiments on synthetic data. Specifically,  two types of images are extracted from the CARLA simulator:\n1) CI(Clean images): The clean RGB images without  motion blur, which is an ideal case to validate the upper limit of RGB-based methods;\n2) BI(Blurred images): The RGB images with motion blur, which are closer to realistic situations.\n\n Based on CI and BI, we experiment on the synthetic dataset, and the average performances are reported. Since Dust3R often has better results than COLMAP, Dust3R is used for the initialization of CI and BI. For fair comparison, all rendering images are evaluated in gray style.  The results on the synthesis dataset are shown in Table.R2.\n\n#### **Table.R2 3D Reconstruction Performance Comparison from Different Input Data**\n\n|Method|PSNR↑|SSIM↑|\n|--|--|--|\n|CI+Dust3R+4DGS(clean)|30.31|0.942|\n|BI+Dust3R+4DGS(blur)|22.72|0.861|\n|Ours Spike4DGS|27.74|0.912|\n\nThe results show that, in an ideal case where the motion blur doesn't exist, the clean RGB image could achieve good performance. However, considering the realistic case with motion blur, the performance of the RGB-based method is limited. This is also our main motivation to introduce spike cameras. \n\nWe will add this statement in the revision.\n\n# Response to W3\n>The authors claim that traditional methods rely on high-quality image reconstruction (Line 140), but the proposed method also requires reconstructed images (Line 170), and the deformed Gaussians in Section 3.4 are highly dependent on image quality. The spike signals only serve as auxiliary loss.\n\nWe acknowledge that our previous wording may have been ambiguous.\n\nFor the statement of Line 140 and Line 170:  The \"traditional methods\" mentioned in Line 140 refer specifically to image-based initialization frameworks such as COLMAP or Dust3R. These methods require generating images first, and then performing COLMAP or Dust3R on the images; hence, the quality of image reconstruction directly influences the point clouds and camera poses. In contrast, our MSDI predicts the image, point clouds, and camera poses end-to-end. Hence, the quality of image reconstruction wouldn't influence the point clouds and camera poses.\n\nFor the statement of Section 3.4:  Our method is indeed dependent on image quality, but the loss of spike signals also plays an important role in SPSS. Spike signals are high-temporal-resolution streams that can capture subtle dynamic changes in the scene, such as motion edges and rapid variations. Our spike total variation loss(TV loss) in SPSS helps keep motion smooth and consistent over time. As shown in Table.R3, we validate the effectiveness of the spikeloss through empirical evaluation:\n\n#### **Table.R3 the Effectiveness of the Spikeloss**\n|Supervision|PSNR↑|SSIM↑|\n|------|-----|-----|\n|Ours spike TV loss|**27.74**|**0.912**|\n|w/o TV loss, use L1 spikeloss|26.87|0.906|\n|w/o any spikeloss|26.66|0.903|\n\n# Response to Q2\n> While some inaccuracy in point cloud is acceptable for initialization, it would be informative to provide some evaluations of point clouds and camera poses, especially compared to COLMAP and DUSt3R.\n\nThank you for your suggestion for the additional experience. We quantitatively evaluate the camera pose estimation accuracy of MSDI in comparison with Colmap and Dust3R[1] on the synthesis dataset. The ground truth camera poses could be obtained from the Carla simulator. The results are shown in the table.R4 below:\n\n#### **Table.R4 Comparison of Camera Pose Estimation Results.**\n|Method|Rotation Error (°) ↓|Translation Error (%) ↓\n|---|---|---|\n|TFI+COLMAP|2.72|9.8|\n|TFI+Dust3R|1.27|3.5|\n|MSID IMAGE+COLMAP|2.62|9.6|\n|MSDI IMAGE+Dust3R|1.19|3.2|\n|Ours (MSDI)|**0.94**|**2.1**|\n\nThis demonstrates that even when using the same spike data, our MSDI method outperforms the TFI + image-based initialization framework, as MSDI is not limited by image quality.\n\n# Response to Q3\n>When using MSDI for initialization, are the poses further refined in 4DGS stage? Is the pose accuracy comparable to COLMAP-based pipelines?\n\nTo save computational resources, we did not further optimize the camera poses during the 4DGS stage as same as original. This is because our initialization via MSDI already yields higher pose accuracy compared to COLMAP-based pipelines from the results in Table.R4. \n\n# Response to Q4\n>What is the computational cost of the proposed method?\n\nThe computational cost of our method consists of two main parts:\n\n1) The MSDI module is trained using 4 NVIDIA RTX A6000 GPUs in parallel and takes approximately 40 hours. Its inference usually takes about 10s, compared with several minutes in Colmap.\n2) The subsequent reconstruction stage runs on a single RTX 4090 GPU and takes around 20+ minutes, depending on the number of frames and views. The time is a little longer than the original 4DGS.\n\n# Response to W4\n>While several spike-based NVS methods are mentioned in Section 2.2, they are not directly compared in experiments. Can those methods be adapted to the multi-view setting? This deserves further discussion.\n\nThank you for the valuable comment. The methods mentioned in Section 2.2 are indeed designed for static scene reconstruction, even though they also utilize neuromorphic (spike) cameras. To the best of our knowledge, our work is the first to extend neuromorphic vision into the dynamic 4D reconstruction setting. **These static methods are adapted to the multi-view setting but are not directly applicable to dynamic scenes.** Thus, our proposed framework fills this gap by enabling temporally consistent 4D reconstruction from asynchronous spike data in dynamic environments.\n\nFollowing your suggestions, we conducted additional experiments. We manually integrated a deformation network from 4DGS into SpikeGS and SpikeNeRF (denoted as Dy-SpikeGS and Dy-SpikeNeRF).  Experiments are conducted on the synthesis dataset, and the average performances are reported in Table R5.  As shown in Table R5, both original and modified dynamic versions of SpikeGS and SpikeNeRF have poor performance. **Note Dy-SpikeGS and Dy-SpikeNeRF are both re-designed and re-implemented, their results are just for reference.**\n\n#### **Table.R5 Comparison with manually Modified SpikeGS and SpikeNeRF.**\n\n| Method        |Avg PSNR(dB)↑ |Avg SSIM↑ \n|---------------|-----------|-------|\n| Original SpikeNeRF| 14.43|0.576\n| Original SpikeGS | 20.03|0.808\n| Dy-SpikeNeRF| 16.70     | 0.628\n| Dy-SpikeGS | 23.31     | 0.835 \n| Spike4DGS   | **27.74**     | **0.912**\n\n# Response to W5\n>The proposed multi-view spike-based dense initialization lacks clarity in what makes it unique to spike data. If the input were replaced with multi-view images, the module might still function.\nThe MSDI is a framework tailored for spike-based 3D vision, which is trained by spike data. So the deal input of the module is spike data, not images like Colmap's input. The module couldn't function if the input were replaced with multi-view images.\n\n# Response to W6\n>It remains unclear what the core advantage over the \"spike-to-image + image 4D NVS\" pipeline is: better initial point clouds and poses? Higher-quality reconstructed images? Or improved supervision through spike constraints?\n\nWe first compare the MSDI images with some spike-to-image methods, including TFI, TFP, and spk2img. On the synthesis dataset, where ground truth images are available, we evaluated the spike-to-image quality using PSNR and SSIM. The results are summarized in Table.R6.\n\n#### **Table.R6 Spike-to-image Quality**\n|Metric|TFI|TFP|TVS|Spike2img|MSDI'simages|\n|------|------|------|------|-------------|-------------|\n|Avg PSNR|24.51|25.37|22.43|30.76|**34.90**|\n|Avg SSIM|0.850|0.852|0.821|0.904|**0.961**|\n\nBased on the results in Table.R6, we acknowledge that a little of the final improvement comes from the enhanced image quality produced by MSDI, but not all. We conducted an additional ablation study in Table.R7:\n\n**Table.R7. Ablation on Each Module**\n|Method|PSNR↑|SSIM↑|\n|------|-----|-----|\n|TFI+DUST3R+4DGS|25.66|0.893|\n|MSDI Image+DUST3R+4DGS|25.76|0.898|\n|MSDI(Image and pointclouds/poses)+4DGS|26.66|0.903|\n|MSDI(Image and pointclouds/poses)+4DGS+SPSS|27.74|0.912|\n\nIt can be observed that point clouds/poses and images from MSDI all improve performance, and the SPSS module further enhances the utilization of spike data, leading to better results."}]

Author rebuttal:

Meta-review: This paper presents Spike4DGS, a framework based on 4DGS for reconstructing dynamic scenes using a spike camera array. The method employs MSDI (Multi-view Spike-based Dense Initialization) to jointly estimate images, point clouds, and poses, and introduces SPSS (Spike-Pixel Synergy Supervision) to leverage the high temporal resolution of spike signals in the training loss. The authors also construct both synthetic and real-world datasets, which they plan to release.

In the rebuttal, the authors provided detailed validations of MSDI and SPSS, additional evaluations on more practical real-world scenes under natural illumination, quantitative evaluations of geometric reconstruction accuracy, and comparisons with further baselines. These clarifications and experiments largely addressed the reviewers’ concerns, and all reviewers converged on positive ratings.

Overall, this paper demonstrates the effective application of spike camera sensing to dynamic 4DGS. The planned dataset release will likely serve as a valuable resource for future research. I recommend acceptance.

The AC encourages the authors to include the additional experiments provided in the rebuttal, including the visual analyses, in the camera-ready version.
Decision: Accept (poster)