Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting
Authors: Yiming Wang
Lucy Chai
Xuan Luo
Michael Niemeyer
Manuel Lagunas
Stephen Lombardi
Siyu Tang
Tiancheng Sun
Venue: NeurIPS 2025 poster ()
Keywords: Feed-forward Models
3D Gaussian Splatting
Novel View Synthesis
Streaming Reconstruction
Primary area: applications
TL;DR: We propose a hybrid Splat-Voxel representation that fuses and refines Gaussian Splatting, improving static scene reconstruction and enabling history-aware streaming reconstruction in a zero-shot manner.

Abstract: Recent advances in feed-forward 3D Gaussian Splatting have led to rapid improvements in efficient scene reconstruction from sparse views. However, most existing approaches construct Gaussian primitives directly aligned with the pixels in one or more of the input images. This leads to redundancies in the representation when input views overlap and constrains the position of the primitives to lie along the input rays without full flexibility in 3D space. Moreover, these pixel-aligned approaches do not naturally generalize to dynamic scenes, where effectively leveraging temporal information requires resolving both redundant and newly appearing content across frames. To address these limitations, we introduce a novel Fuse-and-Refine module that enhances existing feed-forward models by merging and refining the primitives in a canonical 3D space. At the core of our method is an efficient hybrid Splat-Voxel representation – from an initial set of pixel-aligned Gaussian primitives, we aggregate local features into a coarse-to-fine voxel hierarchy, and then use a sparse voxel transformer to process these voxel features and generate refined Gaussian primitives. By fusing and refining an arbitrary number of inputs into a consistent set of primitives, our representation effectively reduces redundancy and naturally adapts to temporal frames, enabling history-aware online reconstruction of dynamic scenes. Trained on large-scale static scene datasets, our model learns an effective global strategy to process around 200k primitives within 15ms and significantly enhances reconstruction quality compared to pixel-aligned reconstruction approaches. Without additional training, our model generalizes to video by fusing primitives across time, yielding a more temporally coherent result compared to baseline methods with graceful handling of occluded content. Our approach achieves state-of-the-art performance in both static and streaming scene reconstructions while running at interactive rates (15 fps with 350ms delay) on a single H100 GPU.

Learning Efficient Fuse-and-Refiee for Feed-Forwazd
3D Gaussian Splatting
Yiming Wang ETH Zurich
Lucy Chai Google
Xuan Luo Google
Michael Niemeyer Google
Manuel Lagunas Google
Stephen Lombardi Google
Siyu Tang ETH Zurich
Tiancheng Sun Google
Abstract
Recent advances in feed-forward 3D Gaussian Splatting have led rapid improve- ments in efficient scene reconstruction from sparse views. However, most existing approaches construct Gaussian primitives directly aligned with the pixels in one more of the input images. This leads to redundancies in the representation when input views overlap and constrains the position of the primitives lie along the input rays without full flexibility in 3D space. Moreover, these pixel-aligned approaches do not naturally generalize dynamic scenes, where effectively lever- aging temporal information requires resolving both redundant and newly appearing content across frames. To address these limitations, we introduce novel Fuse- and-Refine module that enhances existing feed-forward models by merging and refining the primitives in a canonical 3D space. At the core our method is an efficient hybrid Splat- Voxel representation from an initial set of pixel-aligned Gaussian primitives, we aggregate local features into a coarse-to-fine voxel hierar- chy, and then generate refined Gaussian primitives. By fusing and refining an arbitrary number of inputs into consistent set of primitives, redundancy and naturally adapts to temporal frames, enabling history-awant online reconstruction of dynamic scenes. Trained large-scale static scene datasets, model learns an effective global strategy to process around 200k primitives within 15ms and significantly enhances reconstruction quality compared to pixel-aligned reconstruction approaches. Without additional training, our model generalizes to video by fusing primitives across time, yielding more temporally coherent result compared to baseline methods with graceful handling of occluded content. Our approach achieves state-of-the-ar performance both static and streaming scene reconstructions while running at interactive rates (15 fps with 350ms delay) on a single H100 GPU.
sparse voxel transformer to process these voxel features and
representation effectively reduces
Introduction
Novel view synthesis from sparse views is problem for augmented and virtual reality (AR/VR) applications, where practical constraints require the reconstruction to be fast and accurate for positive user experience. Recent feed-forward 3D Gaussian Splatting methods 78 84 trained with large-scale scene datasets offer efficient solutions for the sparse-view reconstruction problem. The key idea these approaches learn large reedfforward network. typically Transformer-based. to predict Gaussian primitives and use splatting as fast renderer 22
The common approach taken by these methods is regress per-pixel Gaussian primitives that lie along the input camera rays. However, this leads to redundancy in the representation when input
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

views have overlapping content, and can also lead to incomplete geometry with occluded content not visible from the input views. This per-pixel representation also does naturally generalize to dynamic scenes, where redundant primitives between frames should be merged and new primitives should emerge to capture newly appearing content.
To address these limitations, we introduce novel Fuse-and- module that processes and merges Gaussian primitives in a canonical 3D space. Our approach enables more flexible primitive placement and better density control, leading to improved reconstruction quality in static scenes. It also greatly facilitates streaming reconstruction dynamic scenes by avoiding error accumulation and redundant primitives over time, while better handling temporal occlusions and disocclusions through the use of historical information. We note that past approaches to merging primitives have relied on heuristics [23] that typically degrade reconstruction quality and require additional time-consuming gradient descent optimization steps. In contrast, our learning-basse baaedd solution leverages Transformer trained on large-scale scene datasets combined with the structured nature of voxels as an intermediate scaffold to efficiently fuse and refine Gaussian primitives.
We design a hybrid Splat- Voxel model that proceeds as follows: given a set of initial Gaussian primitives from existing pixel-aligned feed-forward models, we first perform Splat-to- Voxel Transfer to aggregate local features from Gaussian primitives into a voxel representation, and then apply a Transformer generate refined Gaussian primitives from voxel features. Our voxel representation is specifically designed to ensure both efficiency and high-quality output. We use a coarse-to consseeto-fiin to-fine voxel hierarchy to balance detail-preservatie transformer on features at the coarse resolution while refining the primitives at high resolution. Our auee-nnd-Refin module can merge 200K Gaussian primitives from existing feed-forward 31) Gaussian models in just 15 ms, while delivering a PSNR improvement of around 2 dB.
and computational overhead, operating a sparse voxel
We train our model exclusively on static datasets, but our model also enables history-aware streaming reconstruction of sparse-view dynamic scenes without additional training or modifications. This is advantageous as multiview video datasets remain limited in both size and diversity, prohibiting large- scale training. For dynamic scenes, we first warp primitives from past frame to the current frame, filtering out those with large warping errors, and deposit primitives from both the past and current frame onto our Splat- Voxel representation. The resulting refined primitives significantly improve the temporal consistency of existing feed-forward models, while avoiding redundant primitives and mitigating error accumulation. Our method achieves state-of-the-ae performance on several multi-view video datasets under the sparse camera setup, and can reach interactive rates (15fps with 350ms delay) under single H100 GPU.
In summary, the main contributions of our paper are follows:
We introduce novel Fuse-and-Refine module that enhances existing feed-forward 3D Gaussian Splatting methods and extends them to history-aware streaming reconstruction in a zero-shot manner.
We propose hybrid Splat- Moxel representation that combines coarse-to to-nin voxel hier- archy with a sparse voxel Transformer, enabling effective learning of the Fuse-and- Refine module on large-scale static scene datasets.
Extensive experimental results demonstrate that our approach achieves state-of-the-arr perfor- mance on both static and streaming based dynamic scene reconstruction, while maintaining interactive runtimes.
Related Work
Novel View Synthesis A wide range of methods have been developed over the past decades for novel view synthesis (NVS). A key design decision is the choice of 3D scene representation to establish correspondences and capture structure. Popular representations include multi-plane
images [6 3 66 73 30), neural fields [1 57, voxel grids 62 42 50 and point clouds [77
22 In this work, we adopt 3D Gaussian Splatting
22 as final rendering representation due to its ability to achieve high-quality and real-time consistent novel view synthesis.
Existing NVS methods can be broadly divided into those that optimize scene representation at test time and those that directly predict through a feed-forward network [68
60
53 82
55 54 20
The combination of Transformers and 3D Gaussian Splatting has emerged an efficient approach to

---

predict pixel-aligned Gaussian splatting in data- However, pixel-aligned prediction design often results in suboptimal scene reconstruction due to limited flexibility in accurately positioning primitives in 3D space. To address this limitation, several works have leveraged feed-forward networks operating on 3D representations, such as graph [86], voxel 33 52 5] or point cloud [7], to process 3D Gaussian Splatting. Yet, extending these representations to dynamic scenes remains challenging and largely unexplored. To this end, we propose hybrid point-voxel representation that generalize to dynamic scenes during inference while trained only on static scenes, Our design shares a similar spirit with voxel-based point cloud processing networks [88 36 38). which leverage voxel structures for efficient point cloud processing. However, existing methods B1 typically rely on autonomous deiisdddiiiiiieeeeeee
manner [84 6 83 64 71].
projections (e.g.. sieee view) to handle large-scale scenes. In contrast, we construct cost volume suitable for general unbounded scenes and introduce a coarse-to- voxel hierarchy that enables global attention the coarsest level.
Reconstructing Dynamic Scenes
Handling dynamic motions in novel view synthesis (NVS) is challenging due to occlusions and the need for temporal consistency. Dynamic scene reconstruction
is typically addressed in multi-view setups with fixed cameras (79 58 74 9 377 40) or monocular
settings where both the camera and scene move [44] 28 75 29
69]. Most methods optimize on the entire video sequence, relying learned or hand-crafted priors such as optical flow [65| (4),
tracking M. and depth estimation 44 80 49 48 for supervision [28] 75 Our method leverages pretrained models for point tracking and optical flow as motion priors and operates causally and online, using only previously received frames without requiring access to the full video.
Streaming reconstruction [ presents a significant challenge in dynamic scene reconstruction, aiming to enable online novel view synthesis from potentially unbounded video streams for real-world have explored the use of historical information to accelerate optimization, but remain computationally expensive and depend on dense multi-view inputs. Feed-forwatt models are well-suited for sparse multi-view streaming scene reconstruction. However, many of these methods can only take a single frame as input [15] (3] temporal information and consequently suffering from flickering and occlusion artifacts. Moreover, the scarcity of real-world multi-view dynamic datasets hinders the ability to train feed-forward dynamic models [51 32] for real-world streaming applications. By generalizing from static scene training to dynamic video inference without requiring additional temporal data, our method enables efficient, history- aware novel view synthesis for streaming scenarios.
applications. Several methods 770 26 63
51 84 4], ignoring
Preliminaries: Feed-forward 3D Gaussian Splatting
3DGS represents scene as a set of anisotropic 3D Gaussian primitives, each defined by its position μ, E R. opacity a; E R, quatermion rotation r, E R scale s, E R-3, and spherical harmonic coefficients c, E R where S denotes the number of coefficients used for modeling view-dependent color. The rendered color at pixel denoted as C(p),, is computed via splatting-batt based rasterization, where each 3D Gaussian projected to 2D screen space and blended using front-to-badd alpha compositing:
3D Gaussian Splatting (3DGS) 22
C(p) Σ Go.
P (p)
(P)
II o, (107 (p) j=1
(1)
Here, (p) denotes the 2D projection of the i-th 3D Gaussian at pixel p.
Feed-forward Reconstruction Model To predict the 3D Gaussian primitives in a feed-forward manner from sparse-view images, we adopt the multi-view Transformer architecture from GS- LRM Given N RGB images of resolution H W with known camera intrinsics and extrinsics. we extract per-pixel features by concatenating RGB values with Plücker ray embeddings [46]. These features are grouped into non-overlapping p x p image patches [2 and encoded using a shallow CNN to obtain patch features of channel dimension C. The patch features from all N views are then flattened into sequence of N x HW tokens, denoted as (X). These tokens are processed by p²
a multi-view Transformer 67 to jointly model geometric structure and appearance across views, producing latent features {Z]:
3

---

Initial Gaussian Primitives
Fused and Refined Primitives
Fuse-and- Refine Module
High-res
Skip connection
Decoder MLP
Deposit
Sparsify
Same topologyi
Sparse-voxel Transformer
Low
Splat-to-Vowel Tranater
Our Lane-and-Refinee module takes input Gaussian primitives produced by existing feed-forward models, either from the current frame warped from previous reconstructions, and produces fused and refined primitives that improve scene reconstruction. The input primitives are first deposited into a nigh-resolution voxel grid with a splat-to-voxel Transfer strategy, which is then adaptively sparsified to construct hearse-to-fine voxel hierarchy. A sparse voxel transformer is applied at the coarse level to capture global context, and new primitives are subsequently generated the high-resolution level.
Figure I: Method Overview.
Z Tr ((X))
(2)
Finally, linear projection is applied each latent token produce an initial of Gaussian primitives and associated feature vectors:
(Ga, Fx)- Linear( ((Z))
Here, G2 represents the parameters of a 3D Gaussian primitive aligned with the k-th image patch, which can be directly rendered using Gaussian Splatting [22]. while F2 denotes an additional feature vector used for the subsequent learning-basee fusion and refinement.
4 Learning to Fuse and Refine
Given an arbitrary set of initial Gaussian primitives, for instance those from multi-view images (as in Sec. 3 from previous time steps (as will describe in Sec. 5). aim fuse nearby primitives and produce refined primitives that improve novel view synthesis results.
Challenges Preserving the fidelity of millions of Gaussian primitives while merging them highly non-trivial, with several factors contributing to the challenge. First, maintaining the original 3D Gaussian distributions without divergence fundamentally infeasible when combining multiple kernels. This issue is further exacerbated in the context of sparse-view novel view synthesis, where the fusion problem is inherently ill ill-posed. In addition to merging the Gaussian distribution determined by primitive mean and covariance, other rendering attributes like color, opacity, scale, and rotation must also be reconciled with limited supervision from sparse 2D observations. As a result, heuristic approaches [33 often lead to degraded reconstruction quality.
Solution Instead, we propose an efficient learning-based approach that not only merges 3D prim- itives, but also further refines them through feed beed-forward Transformer. Trained on large-scale scale scene data, our model learns a global strategy to fuse input primitives into refined of splats in just 15 ms, significantly improving reconstruction performance. The proposed learning framework adopts hybrid Splat-Voxel
Voxel representation, where an intermediate voxel grid serves as a structured spatial proxy for aggregating and distributing splats. A voxel Transformer operates in latent space to fuse information in a canonical 3D space and produce refined splats, overview the proposed Fusc-and Refinee module is illustrated in Fig.

---

4.1 Splat-to- Splat-to-Voxel Transfer
Starting with the predicted Gaussian primitives G2 and their associated image features F; from the multi-view Transformer (Eqn.2) 2) we first operate a Splat-to- Voxel Transfer procedure to convert the point-based representation structured voxel representation. Implementation details, including the choice of kernel function K. and other parameters, are provided in the supplementary material (Sec.
Splat Depositing Given a dense high-resolution voxel grid V, we deposit each Gaussian primitive Gj to the of {M&] of nearest voxels based on the position μ4 using distance kemnel K. That is, for voxel Vj, its weight associated with G2 is:
E K(aa-xy)) if (M+).
Kija x,)
Wik
otherwise.
where xj represents the center of the j-th voxel. We choose the adjacent 8 voxels for each splat in our implementation.
Splat Fusion Each voxel V then accumulates the deposited splat attributes using opacity as weighting. The corresponding voxel attribute E, at voxel Vg is then the normalized weighted sum of the splat features Fy in the voxel:
1 E, W
Σ (M,),
WILCR Fai GR]. W
Σ kE(M,)
Wik 0g
(5)
where [Fa; GR] denotes the concatenation of the feature vector F5. and Gaussian attributes G We use the notation k E M2 to
The interpolation weight wisk is computed as described in Eqn. [. indicate that indexes over the set of all Gaussian primitives, each associated with weight, Gaussian attributes, and a feature vector.
Coarse-to-fine voxel hierarchy Once the splat features are deposited onto the voxels, construct a coarse-to-fine voxel hierarchy. Specifically, the high-resolution voxels are downsampled by factor of [d. h, w] to generate a low-resolution voxel grid that serves the coarse scene representation. Sharing the same spatial topology, each coarse-level voxel directly corresponds to a [d, h, w| block fine
of fine-level voxels and can be restored by subdivision. We denote N; as the set of d voxels associated with the i-th coarse voxel. Using the previously computed voxel features E, as the fine voxel features, we now compute coarse voxel features E;
We derive coarse voxel features using a shallow multi-layer perceptron (MLP) on the concatenation of the fine features and coarse voxel weights as the sum of the associated fine voxel weights:
E
MLP course (E, 1 E N:l), W: Σ W
(6)
JEN;
Voxel Sparsification We leverage geometric cues from coarse voxel weights W, which encapsulate the spatial distribution and opacity of Gaussian primitives within each voxel, to efficiently cull empty and insignificant regions of the scene. We sort the coarse-level voxels by their voxel weights and retain only the top 20% with the largest weights as a sparse voxel representation of the scene. This sparsification is crucial for the downstream voxel transformer, and reduces the input token count to fewer than 10K, which allows the voxel transformer to run in 15 ms. Note that the high-resolution fine-level voxels also sparsified accordingly if their corresponding coarse-level voxels culled.
4.2 Sparse Voxel Transformer
From coarse voxel features, we treat each voxel feature token and reshape them to ID vectors. We train a transformer to process the set of coarse voxel features E
(0*) Transformer (E*})
The processed latent features (0r) are then replicated from the coarse voxel grids within their corresponding fine voxel grids, producing latent features } at the fine voxel resolution. A shallow
5

---

Table 1: Quantitative Comparisons on the RealEstate10K Dataset. Our method achieves state-of-the-an per-
Table 2: Quantitative Comparisons the DL3DV Dataset. We build our Voxel Transformer on GS- LRM (35] and present ablation studies model design. For fair comparison, we use 12 multi-view Transformer layers with our Voxel Transformer to match the size and runtime of GS-LRM, which uses 24 layers.
formance among recent feed-forward
3D Gaussian Splatting methods.
Method
PSNR1
SSIMT
LPIPS
pixelSplat
26.09
0.863
0.136
Method
PSNR1 SSIM LPIPS), Time(ms)
MVSplat 6
26.39
0.869
0.128
GS-LRM
28.59
0.925
0.063
52.8
TranSplat
83
26.69
0.875
0.125
Non-learning Fusion
12.57
0.357
0.741
33.8
HiSplat 64
27.21
0.881
0.117
Ours (w/o Coarse-to-fine)
20.62
0.587
0.247
48.6
OmniScene 71
26.19
0.865
0.131
Ours (w/o Sparse Voxel)
29.69
0.926
0.060
72.0
DepthSplat 76
27.47
0.889
0.114
+ Ours (w/ 3D CNN)
29.44
0.922
0.061
82.1
GS-LRM
x4
28.10
0.892
0.114
+ Ours (w/ Splat Feature)
29.40
0,924
0.061
49.2
Ours
28.47
0.907
0.078
Ours
30.61
0.935
0.052
52.5
MLP subsequently generates refined Gaussian primitives by integrating the initial fine-level voxel attributes E, with the fine-level transformer latents O,.
G'j MTP fine (IO,, E,))
where G I represents the predicted Gaussian primitives around the sparsified fine-level voxel grids, with denoting the number of rendering parameters.
4.3 Training
We train the Splat feed-forward networks on multi-view static scene datasets 87 photometric loss combining Mean Squared Error (MSE) and perceptual I.PIPS 85
35
using a
Cf I) + XCrriss (I,
where i denotes ground-truth target image and I the corresponding rendered target image from the predicted Gaussian primitives, which come from either the multi-view transformer or voxel transformer. We set À to 0.5 for the multi-viee view transformer and 4.0 for the voxel transformer. We train our full-scale model for cross-dataset generalization on the DL3DV dataset [35] with a batch size of 128 for total of 300K iterations using two-stage training strategy. In the first stage, train the multi-view Transformer backbone for 200K iterations, followed by joint fine-tuning of both the multi-view and voxel Transformers for additional 100K iterations. Network architecture and
further training details are provided in the supplementary material (Sec. A. and Sec. B
5 Zero-shot Streaming Fusion
In this section, demonstrate an application of the Sparse Voxel Transformer, trained solely on static scenes, to enable zero-shot history-aware novel view streaming without requiring any training on dynamic scenes. Please refer to the Appendix (Sec. A.3) 1 for full implementation details.
3D Warping Given Gaussian primitive with an initial position fer at a previous frame t', we first estimate its corresponding 3D position μ4 at the current frame & by performing triangulation on the 2D correspondence obtained by a pre-trained 2D point tracking model [1]. detail, we first compute its 2D projection (Pğ i I...N} across N input views. We then run 2D point tracking on the projected points to obtain their corresponding 2D pixel position Pi in the current frame. The 3D position subsequently recovered by finding the closest point the N camera rays originating from the 2D projections Pi- We adopt embedded deformation graph [6] 43 25 to propagate reliable motion estimates from sparse set of anchor points to the entire scene. Anchor points are selected via farthest point sampling [7],, and an embedded deformation graph is constructed by connecting each splat to its K nearest anchor points, with blend weights computed based on spatial distances. For efficiency, we only maintain primitives from a set of past keyframes, and the contribution of these primitives are smoothly adjusted as older keyframes are replaced by newer ones to ensure seamless temporal transitions.

---

Table Quantitative Comparison Under Varying Input Views the DL3DV Dataset. Trained
with 4 input views, our method consistently outperforms GS-LRM across different input-view settings inference.
2 Views
4 Views
8 Views
16 Views
Method
PSNR SSIM LPIPS|
PSNRT SSIMT LPIPS| PSNR
SSIM LPIPS PSNR
SSIM LPIPS J
GS-LRM
18.99
0.810
0.135
28.59
0.925
0.063
22.51
0.879
0.106
20.12
0.792
0.177
Ours
26.32
0.876
0.093
30.34
0.934
0.054
28.98
0.930
0.060
26.06
0.889
0.093
Inputs
GT
GS-LRM
Ours
Figure 2: Static Scene Reconstruction. Qualitative comparison of our method with GS-LRM 84 on RealEstate [87]. Our method preserves high fidelity reconstructions and sharp details.
During streaming reconstruction, we incorporate both historical Gaussian primitives warped from keyframes and Gaussian primitives predicted from multi view observa- tions at the current frame. All of the collected Gaussian primitives are deposited to voxels (Sec. 4.1 and then processed by the voxel transformer (Sec. 4.2), producing refined, history-aware Gaussian primitives. To mitigate artifacts resulting from warping inaccuracies historical primitives (e.g. warping error, appearance variation, during splat fusion based on the per-pixel eтror between the past and current frame splats when rendered to the input views.
Error-aware Fusion
objects),
adaptive fusion weighting mechanism
6 Experiments
In this section, we present evaluations of feed-forward novel view synthesis on both static and dynamic scenes, along with ablation studies on our splat-voxel representation designs.
6.1 Feed-forward Novel View Synthesis
We benchmark our method two widely used datasets, RealEstate 10K cover both indoor scenes and unbounded large-scale environments.
87 and DL3DV B3), which
Following the training and testing protocol of 4. our method achieves state-of-the-att performance on RealEstate 10K, as shown in Table [ As illustrated in Fig. 2 our method reconstructs finer details such thin structures (c.g., chairs) and reflections compared to the strongest bascline, GS-LRM (84)
Additional qualitative comparisons and training details are provided in the Appendix (Sec. and Sec.
We then demonstrate the ability of our method to enhance existing feed-forwart 3D Gaussian Splat- ting approaches through comparison on the DL3DV dataset [35] with GS-LRM [84). Table demonstrates that our method outperforms GS-LRM by approximately 2 PSNR, while main- taining a comparable number of network parameters, similar inference time, and the same training configuration. Our method also demonstrates strong generalization capability across varying input views. Trained with 4 input views on the DL3DV dataset, method consistently outperforms

---

Table 4: Novel View Streaming Metrics. We compare reconstruction quality of our model to prior
methods on two multi-view video datasets with sparse-view input. We report the average running time of processing single frame on Neural3DVideo at the resolution of 320 x 240. The best and second-best results are highlighted for clarity.
Veudel3DVideo [2
Long Volumetrie Video
[7]
Method
Time (s)
PSNRT
SSIM
LPIPS
Flicker,,
PSNRT
SSIM
LPIPS|
Flicker]
3DGS 22
9.8
21.02
0.6234
0.4989
43.51
18.56
0.6368
0.4334
61.69
3DGStream 63
3.8
14.24
0.3542
0.6016
6.554
16.97
0.4405
0.5040
12.44
4DGS 74
6.0
23.16
0.7812
0.2079
2.746
18.75
0.6361
0.3554
6.528
GS-LRM 84
0.04
21.80
0.8488
0.1278
5.714
25.49
0.8558
0.1377
8.463
0.07 (non-keyframe)
Ours
27.41
0.8863
0.1040
2.916
25.56
0.8645
0.1242
7.782
0.35 (keyframe)
GT
3DGS
3DGStreum
4DGS
GS-LRM
Ours
Figure 3: Qualitative Comparison Novel View Streaming. We compare against per-frame methods (3DGS and GS-LRM) and temporal methods (4DGS and 3DGStream) on Long Volumet- Video 779 with 4 input views and Neural3DVideo (27) with 2 input views.
GS-LRM across awide range of input configurations as shown in the Table These results further highlight the strong generalization ability of our method for long- long-sequence, large-scale scene recon- struction. A compairson with Gaussian Graph Network [86] is provided in the appendix Table [3] which state-of-the-art method for handling diverse input views.
6.2 Novel View Streaming
We conduct cross-dataset evaluation of our method on dynamic reconstruction, demonstrating its generalization capability in enhancing reconstruction quality and improving temporal coherence. We train both GS-LRM and our model on DL3DV, and compare method with baselines on the novel view streaming task on two dynamic scene datasets, Neural3DVideo
[27] and Long Volumet- For
input views for Neural3DVideo and 4 views for Long Volumetric quantitative evaluation of temporal coherence, we introduce flicker metric that complements the per-frame novel view reconstruction scores. Temporal flicker is quantified by measuring feature-space variations over time. For predicted frames L, and ground truth Ij, we compute
VicVideo 79]. We
Flicker, = D: D;|.
where D, ((((4) 6(4-1)l12 and Dj [lb(1]) (((11))12 denote temporal feature differences extracted from a pretrained VGG network [56]. The final flicker score computed by averaging Flicker, over the entire sequence. Lower values indicate smoother temporal consistency.
As presented Table our method achieves the highest framf novel view synthesis performance across all three evaluation metrics on both datasets with the second-best flicker scores. In comparison to the state-of-the-ar per-frame structs geometry and appearance with greater accuracy and finer details (Fig. 6.1 Our method reduces
method GS-LRMM, our approach qualitatively recon-
8

---

Table Comparison of GS-LRM with Simple 3D Warping and Fusion Strategy. Incorporating
historical information for streaming reconstruction remains challenging due to accumulated errors, whereas our Bass-and-Refing network learns to effectively mitigate this limitation.
Method
PSNR
SSIM
LPIPS}
Flicker.
GS-LRM
21.80
0.8488
0.1278
5.714
GS-LRM 3D Warping
21.40
0.8250
0.1419
2.163
GS-LRM + Warping Fusion (Concat + Dropout)
19.52
0.7429
0.2502
3,983
GS-LRM + 3D Warping + Ours (Fuse-and and Refine)
27.41
0.8863
0.1040
2.916
Reference
GT time slice
GS-LRM
Ours
Figure 4: Temporal Slice Comparison. We extract a horizonal line in the ground truth and generated videos to visualize temporal stability. Applying GS-LRM independently each frame results in noticeable flickering, whereas our method achieves greater temporal consistency and improved occlusion robustness through temporal fusion and refinement.
flickering through history-aware modeling, visualized with temporal slices (Fig. 4) Although 4D Gaussian Splatting yields slightly better flicker scores, it produces significantly poorer novel views and its costly temporal optimization is impractical for online processing of long sequences.
We further apply 2D tracking and 3D warping to GS-LRM demonstrate that directly warping past reconstructions introduces significant errors. As shown in T GS-LRM + 3D Warping fails to handle new content and accumulates warping errors over time. A simple fusion baseline (Concat + Dropout) mitigates primitive growth by discarding 50% per frame but lacks refinement and often removes accurate primitives.
For the reported running time in Table 3 GS-LRM and ours are tested on a single H100 card. 3DGS, 3DGStream, and 4DGS are tested on single A A100 GPU due to limitations with the H100 card. While the efficiency these optimization-basee methods can be enhanced, they still do not achieve frame rate of FPS. Running the pre-trained 2D tracking model at keyframes every 5 frames increases the runtime from 0.07s to 0.35s. A detailed runtime breakdown our method for both static and dynamic scenes is presented in Table
6.3 Ablation Studies
Ablation on Sparse Voxel Transformer. We show model ablations DL3DV in Table [ Naive non-learning fusion of Gaussian attributes in each voxel significantly degrades results. The absence of a coarse-to-fine setup limits the capacity of the output Splats, and without sparsification, we downsampled the grid to 0.75x its original resolution to avoid exceeding memory capacity for the voxel transformer, both leading poorer reconstruction quality. We also experiment with using 3D CNN to downsample and upsample the voxel grid instead of our designed coarse-to- to-fine voxel hierarchy. This approach yields results approximately dB worse than our final model and is significantly slower. We also find splat features helpful, operating on splat parameters directly leads to ~1 dB drop in PSNR.
blation on Temporal Components. We conducted ablation studies on the Neural3DVideo dataset to evaluate our streaming reconstruction components, shown in Table 7 Removing historical splats

---

Table 6: Inference Runtime Breakdown of Our Method for Static and Dynamic Reconstructions.
Static Scenes (256x256 Resolution)
Multiview Transformer Splat-to-Voxe Transfer
Voxel Transformer
Rendering
Time (s)
0.026
0.006
0.015
0.003
Dynamic Scenes (320x240 Resolution)
Multiview Transformer 2D Tracking
3D Warping
Fuse-and- anseennd-Reeine Rendering
Time (8)
0.030
0.028
0.018
0.021
0.003
Table 7: Ablation on Temporal Components. Incor- porating historical information and our proposed mech- anisms for leveraging it are crucial for achieving consis- tent and high-quality novel-view streaming results.
Table 8: Comparison Between Two-stage and Joint training. Jointly training GS- LRM and our method achieves better per- formance than the sequential two-stage strategy that pre-trains GS-LRM.
Method
PSNRT SSIM LPIPS.) Flicker].
Method
PSNR SSIM LPIPS.
w/o History
27.17
0.890
0.107
5.91
w/o 3D Warping
27.12
0.884
0.107
6.09
GS-LRM
28.59
0.925
0.063
w/o Error-aware Fusion
26.73
0.874
0.109
3.01
Ours (Two-Stage)
30.34
0,934
0.054
Full Model
27.41
0.886
0.104
2.92
Ours (Joint)
30.61
0,935
0.052
for fusion failed to resolve temporal flicker. Disabling 3D warping caused abrupt keyframe changes and higher reconstruction errors. Removing error-aware fusion degraded quality due to warping artifacts and other errors.
Ablation on Training Schemes. Our framework supports both the two-stage training strategy described Sec. 4.3 and joint training the multi-view and voxel Transformers from scratch. As shown in Table g
the joint training scheme can achieve better performance than the two-stage
approach.
7 Conclusion
In summary, we introduce novel Tuse-and-Refing module that can efficiently consolidate Gaussian primitives from multiple sources into canonical and consistent representation. It enhances existing feed-forward models, and the design of the module naturally allows for reusing primitive information from past frames for better temporal coherence streaming setups. Extensive evaluations show that our method achieves state-of- saate-of-the-all the-art art performance on both static and dynamic datasets under sparse camera setups, delivering sharper reconstructions and reduced temporal flicker compared previous feed-forward and optimization- based methods, all at interactive frame rates single GPU.
While the results of our approach are promising, it does have limitations. First, depends on reason- ably good initial primitives. For instance, in large-bascline setups such as the Long Volumetric Video dataset, even the current best feed-forward model, GS-LRM, experiences a performance drop. As a result, our method yields limited improvements based on its input in such challenging scenarios. potential solution is to train on larger-scal scale dataset that covers broader range of baseline configurations. Second, temporal artifacts still persist, particularly in regions with dynamic objects. The observed artifacts primarily arise from our current simple 3D warping strategy. Lifting 2D tracking into 3D from sparse views remains inherently error-prone and presents a persistent challenge in this ongoing field. A promising future direction is to investigate approaches that avoid explicit tracking or adopt adaptive online reconstruction strategies, potentially prioritizing regions with higher reconstruction errors.
would like to express our gratitude to John Flynn, Kathryn Heal, Lynn Tsai, Srinivas Kaza, Qin Han, and Tooba Imtiaz for their helpful discussions, well as Clément Godard, Jason Lawrence, Michael Broxton, Peter Hedman, Ricardo Martin-Brualla, and Ryan Overbeck for their valuable comments.
Acknowledgement.
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 2016.
arXiv preprint
[2] James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[3] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with layered mesh representation. ACM Transactions on Graphics (TOG), 39(4)886 2020.
[4] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19457-19467, 2024.
[5] Anpei Chen, Haofei Xu, Stefano Esposito, Siyu Tang, and Andreas Geiger. Lara: Efficient large-baseline radiance fields. In European Conference on Computer Vision (ECCV), 2024.
[6] Yuedong Chen, Haofei Xu, Chuanxia Zheng. Bohan Zhuang. Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pages 370-386. Springer, 2024.
[7] Yutong Chen, Marko Mihajlovic, Xiyi Chen, Yiming Wang. Sergey Prokudin, and Siyu Tang. Splatformer: Point transformer for robust 3d gaussian splatting, 2024.
[8] Jacyoung Chung, Jeongtack Oh, and Kyoung Mu Lec. Depth-regularized optimization for 3d gaussian splatting in few-shot images. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 811-820, 2024.
[9] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe. Adam Kirk, and Steve Sullivan. High-quality streamable frec-viewpoint video. 34(4), 2015.
[10] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344- 2022.
[11] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10061-10072, 2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010. 11929, 2020.
[13] Yilun Du Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs, 2023.
[14] John Flynn, Michael Broxton, Paul Dehevec, Matthew Du Vall, Graham Fyffe, Ryan Overheck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceddings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367-2376. 2019.
[15] John Flynn, Michael Broxton, Lukas Murmann, Lucy Chai, Matthew Vall, Clément Godard, Kathryn Heal, Srinivas Kaza, Stephen Lombardi, Xuan Luo, et al. Quark: Real-time, high-resolution, and general neural view synthesis. ACM Transactions on Graphics (TOG), (3(6:1-2 2024.
[16] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5515 5524, 2016.
[17] Qiankun Gao, Jiarui Meng, Chengxiang Wen, Jie Chen, and Jian Zhang. Hicom: Hierarchical coherent motion for dynamic streamable scenes with 3d gaussian splatting. Advances in Neural Information Processing Systems, 37:80609 80633,0033330 2024.
[18] Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.6666111.. 15, 2016.
[19] Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXi:22010.0425,, 2020.

---

[20] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli. Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint urXiv:2311.044
2023.
[21] Haian Jin. Hanwen Jiang, Hao Tan. Kai Zhang. Sai Bi, Tianyuan Zhang. Fujun Luan, Noah Snavely. and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint urXiv:2410.1242.. 2024.
[22] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler. and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(():133- 2023.
[23] Bernhard Kerbl, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3d gaussian representation for real-time rendering of very large datasets. ACM Transactions on Graphics, 43)).. July 2024.
[24] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[25] Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Damiilidis, Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.174D1, 2024.
[26] Lingzhi Li, Zhen Shen, Zhongshu Wang. Li Shen, and Ping Tan. Streaming radiance fields for 3d video synthesis. Advances Neural Information Processing Systems, 35:13485--13498, 2022.
[27] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[28] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.
[29] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[30] Zhenggi Li. Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In Computer Vision-ECCV 2020: 16th European Conference, Glaxgow, UK. August 23-28, 2020, Proceedings. Part I 16. pages 178-196. Springer, 2020.
[31] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie. Chonghao Sima. Tong Lu. Qiao Yu, and Jifeng Dai. Bevformer: learning bird's e-ye--ieww representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[32] Hanxue Liang. Jiawei Ren, Ashkan Mirzaei, Antonio Torralba, Ziwei Liu, Igor Gilitschenski, Sanja Fidler. Cengiz Oztireli, Huan Ling, Zan Gojeic, et al. freddforwwrrd bullet-time reconstruction of dynamic scenes from monocular videos. arXiv preprint XXv::2412.03526 2024.
[33] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive viewviin video. In SIGGRAPH Asia 2022 Conference Papers, pages 1 1-9. 2022.
[34] Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, and Feng Xu. Occlusionfusion: Occlusion-aware motion estimation for real-time dynamic 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1736 1745, 2022.
[35] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. 10k: A large-scale scene dataset for deep lcarning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[36] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel can for efficient 3d deep learning. Advances in neural information processing systems, 32, 2019.
[37] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.0777 07751, 2019.
[38] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng. Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Proceedings of the IEEE/CVF international conference computer vision, pages 1173, 2021.
12

---

[39] Sheng Miao. Jiaxin Huang, Dongfeng Bai, Xu Yan. Hongyu Zhou, Yue Wang. Bingbing Liu, Andreas Geiger, and Yiyi Liao. Evolsplat: Efficient volume- based gaussian splatting for urban view synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 11286-1 1226, 2025.
[40] Marko Mihajlovic, Sergey Prokudin, Siyu Tang, Robert Maier, Federica Bogo, Tony Tung. and Edmond Boyer. Splatfields: Neural gaussian splats for sparse 3d and 4d reconstruction. In European Conference on Computer Vision (ECCV). Springer, 2024.
[41] Ben Mildenhall. Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[42] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM transactions on graphics (TOG), 41(4):1 155, 2022.
[43] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 143-352, 2015.
[44] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo -ruaia- Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF international conference computer vision, pages
2021.
[45] Luigi Piccinelli, Yung-Hsu Yang. Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10106-10116, 2024.
[46] Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London, pages 725-791, 1865.
[47] Charles Ruizhongtai Qi, Li Yi, Hao Su, Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[48] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179-12188, 2021.
[49] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE trunsactions on pattern analysis and machine intelligence, 44()):123316377, 2020.
[50] Christian Reiser, Stephan Garbin, Pratul Srinivasan, Dor Verbin, Richard Szeliski. Ben Mildenhall, Jonathan Barron, Peter Hedman, and Andreas Geiger. Binary opacity grids: Capturing fine geometric detail for mesh-based view synthesis. ACM Transactions on Gruphics (TOG), 43(4):1-14, 2024.
[51] Jiawei Ren, Cheng Xie, Ashkan Mirzaei, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, Huan Ling, et al. L4gm: Large 4d gaussian reconstruction model. Advances in Neural Information Processing Systems. 37:56828-5666
2025.
[52] Xuanchi Ren, Yifan Lu, Hanxue Liang, Zhangjie Wu, Huan Ling, Mike Chen, Sanja Fidler, Francis Williams, and Jiahui Huang. Scube: Instant large-scale scene reconstruction using voxsplats, Advances in Neural Information Processing Systems, 37:97670-97667 2024.
[53] Gernot Riegler and Vladlen Koltun, Free view synthesis. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23 28, 2020, Proceedings, Part XIX 16, pages 623 640. Springer, 2020.
[54] Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry-free view synthesis: Transformers and no 3d priors. In Proceedings of the IEEF/CVF International Conference Computer Vision pages 14356-14364 2021.
[55] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora. Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent seene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1556, 2014.
[57] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d structure- aware neural scene representations. Advances in neural information processing systems, 32, 2019.
13

---

[58] Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari. Gordon Wetzstein. and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[59] Michael Steffen, Robert M Kirby, and Martin Berzins. Analysis and reduction quadrature errors in the material point method (mpm). International journal for numerical methods in engineering, 76(6))922-944) 2008.
journaa
[60] Mohammed Suhail. Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision. pages 156-174. Springer. 2022.
[61] Robert W Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM siggraph 2007 papers, pages 80-es, 2007.
[62] Cheng Sun, Min Sun, and Hwann-Tzong Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5459-5469, 2022.
[63] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 20675-20685, 2024.
[64] Shengji Tang, Weicai Ye, Peng Ye, Weihao Lin, Yang Zhou, Tao Chen, and Wanli Ouyang. Hisplat: Hierar- chical 3d gaussian splatting for generalizable sparse-vicw reconstruction. arXiv preprint 2024.
[65] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 402-419. Springer, 2020.
[66] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 55 551-560, 2020.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
[68] Qiangian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin- Bruulla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-basce rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2021.
[69] Qiangian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 2024. 2024.
4d reconstruction from single video. arXiv preprint ar 21v7 13764,
[70] Yiming Wang, Qin Han, Mare Habermann, Kostas Danillidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[71] Dongxu Wei, Zhiqi Li, and Peidong Liu. Omni-scene: Omni- mani--aussiinn representation for ago-centric sparse- view scene reconstruction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22317-22327, 2025.
[72] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from single image. In Proceedings of the IEFE/CVE conference on computer vision and pattern recognition, pages 7467-7477 2020.
[73] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yemphraphai, and Supasom Suwajanakom. Nex: Real-time view synthesis with neural basis expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8534-8543, 2021.
[74] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20310-20320, 2024.
[75] Wenqi Xian, Jia-Bin Huang, Johannes Kopf. and Changil Kim. Space- time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 9421 111-9431, 2021.
14

---

[76] Haofei Xu. Songyou Peng. Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. Depthsplat: Connecting gaussian splatting and depth. In Proceedings the Computer Vision and Pattern Recognition Conference, pages (6453-16463, 2025.
[77] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438- 5438, 2022.
[78] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In European Conference on Computer Vision, pages 1-20. Springer, 2024.
[79] Zhen Xu, Yinghao Xu, Zhiyuan Yu, Sida Peng, Jiaming Sun, Hujun Bao, and Xiaowei Zhou. Representing long volumetric video with temporal gaussian hicrarchy, ACM Transactions on Graphics, 43(6), November 2024.
[80] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-10381, 2024.
[81] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for conference on computer vision and pattern recognition, pages 20331-20341, 2024.
monocular dynamic scene reconstruction. Proceedings of the IEEE/CVF
[82] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images, 2021.
[83] Chuanrui Zhang. Yingshuang Zou, Zhuoling Li, Minmin Yi, and Haoqian Wang. Transplat: Generalizable 3d gaussian splatting from sparse multi-view images with transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9869-9877, 2025.
[84] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision, pages 1-19. Springer, 2025.
[85] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[86] Shengjun Zhang, Xin Fei, Fangfu Liu, Haixu Song, and Yueqi Duan. Gaussian graph network: Learn- ing efficient and generalizable gaussian representations from multi-view images. Advances in Neural Information Processing Systems, 37:50361-50380 2024.
[87] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018
[88] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490-4499, 2018.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly present the core contributions of the paper: novel feed-forward Fusion and Refinement module that enhances 3D Gaussian Splatting for both static and streaming scene reconstruction. The paper explicitly states that the method improves reconstruction quality, enables zero-shot generalization to dynamic scenes, and achieves interactive runtime.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the method is discussed the Conclusion (Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All theoretical results are presented with the necessary assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed implementation information, including network architecture, training datasets, and hyperparameters, is fully provided in the supplementary materials (Section A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The code is not publicly available at the time of submission. However, the datasets used in the experiments publicly accessible, and sufficient implementation details are provided to support reproducibility. The authors are also willing to assist with any reproduction-related issues.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Pollccyyyyio for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.c//ggieeeeeeee Code//bmeeeeeeee SusSSuumissss iilammsssion sssisssssiccciiiii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the dataset details and implementation details included in the supple- mental material (Section and B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The paper does not include repeated training runs due to the high computational cost. However, extensive comparisons across multiple datasets demonstrate that our model outperforms the baselines consistently.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed training setup, including information on GPU resources, can be found in the supplementary material (Sections A and B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tt.t//iii//////ao/////////////a
Answer: [Yes]
Justification: The research complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses its societal impact in Section
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any models or datasets that pose high risk of misuse, and therefore no specific safeguards are necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in the paper, including datasets and code, are properly cited with their original sources and licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces new learning- based representation, but pretrained models or other assets are publicly released. However, the extensive implementation details and the use of publicly available datasets make the model and results reproducible.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The research does not involve human subjects and therefore does not require IRB approval equivalent review.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

A
Implementation Details
Our framework is implemented in JAX 2 and trained on NVIDIA A100 GPUs. We use the Adam optimizer [24] with an initial learning rate of 4e-4, applying cosine learning rate decay with linear warmup. The warmup period is set to 5000 training steps.
A.1 Network Structure
Our network consists of a Multi-view Transformer backbone and Sparse Voxel Transformer. The multi-view transformer has 24 transformer layers with 1024 hidden dimensions and 16 attention heads. The sparse voxel transformer uses 6 layers with 128 hidden dimensions and 8 attention heads. To demonstrate the effectiveness of our representation, we use a 12-layer multi-view transformer in Table 4, showing that it can outperform GS-LRM by 2dB in PSNR while maintaining a comparable network capacity and similar inference time.
Each transformer layer [67] consists of a self-attention mechanism and feed-forward networks, with all bias terms removed following GS-LRM (84) Additionally, following LVSM (21] incorporate QK-Norm [19 to stabilize training. For efficiency, we utilize the implementation available in Jax.
[IO]
The shallow multi-layer perceptrons MLP coarse and MLP, fine share similar structures with the network in transformer layer. includes two linear layers with LayerNorm [1] and GeLU 18 activation function,and an additional residual connection.
A.2 Hybrid Spxat-Voxel Voxel Representation
Our Voxel model is inspired by Lagrangian- Eulcrian representations in physical simulation where we combine the Gaussian splats (Lagrangian representation) and voxels (Eulerian representation) to assist warping and fusion of history information.
Eulleiinnn
We the official splatting- based rasterizer from 3D Gaussian Splatting 22 render images from predicted Gaussian primitives. To ensure fair comparison with GS-LRM on RealEstate [0K[87], we the spherical harmonics degree to 0 for Table 9 For models trained the DL3DV dataset 35 the degree set to I to enhance view-dependent effects.
Our voxel grid is aligned with the target camera, and is in the normalized device space (NDC), where the z-axis is lincar in disparity. This space is helpful in presenting unbounded, in wild scenes. The full fine volume size is set to [64, H, W]. where the H, W corresponds to the rendered image height and width. The coarse volume size is set to [32, H//8. /8, W//8), meaning each coarse- level volume is subdivided into [2, 8, 8] fine-level volumes. The cost volume also be constructed from any meaningful reference view, such as the average Gaussian Splatting (3DGS) for free-viewpoint novel view synthesis. Moreover, the application of our sparse voxel transformer' is not limited to cost volumes. It naturally generalizes to other voxel structures, such octrees, making it applicable to broader scenarios like SLAMM.
the input cameras, to produce 3D
We initialize the voxels by depositing the Gaussian primitives to the volume with a 3D distance-based kernels used in Material Point Method [59]:
K(x)= -
K
(10)
h
where = (x,, 2) is the offset from the voxel center to the Gaussian primitive center, and the grid size used normalize the offset vector to the range [-1, 1]. The function K K(x) is one-dimensional cubic kernel function given by:
K(x) = - (3/xr1 GZ + 4)
(11)
Rather than directly generating new attributes, we design the voxel transformer to predict residuals that are added to the initial Gaussian rendering attributes on voxels. Specifically, given the input fine-level voxel attributes obtained through the Splat Fusion procedure E, which comprise the fused
23

---

Training Static Scenes
Inference on Dynamic Scenes
Splat Module
Vaxel
Muttivicw Trensformer
Soudsion Splans
Derse Visaals
Viceel Module
Doer Plze
sture weightt
Transformer
3D werp
Sparse Vovals
Spiiet
Moduee
Transformee
3D scene flow only points
3D warp
Goussion Splans
frumcss
tracking
Interpolane 30 scare flom from reartty key poinns
images
stream
points
Figure 5: History-aware Streaming System. Our method can generalize to dynamic scenes during inference while trained only static scenes. The hybrid Aplat-Voxel model first extracts input image features using a multi-view transformer, which outputs pixel-aligned Gaussian splats for each input image with associated features. The splat features are then deposited onto a coarse-to to-fine voxel grid using the decoded position, and a secondary sparse voxel transformer processes the grid features to output final Gaussian parameters. To merge history, we compute the triangulated scene flow from the input views and perform keypoint guided deformations. These deformed splats can be treated identically to the input-aligne splats, and be similarly deposited into voxel grid merge the previous state with the current state.
splat feature F, and rendering parameters G; throught the Splat Fusion procedure, the transformer obtain the final splat rendering parameters G in Eqn. 7 as:
AG; = ML.P fine (O,, E,))
(12)
G/ = G, + AG,.
A.3 History-aware Streaming Reconstruction
Triangulation For solving the close closest point to the N camera rays originating from the 2D projections PI, which can be formulated the following least-squares optimization problem:
X A arg min x.
Σ Ell(: ),d,)) x|
(13)
where o and d, denote the origin and direction of the i-th camera ray, respectively, and X is scalar representing the depth along the ray. This least-squares problems can be converted into solving an linear equation
Ena(s-
d.d!)) -adlsadd))) x En d.d. )o:
(14)
where J3 denotes the three-dimensional identity matrix, and 7/ is visibility mask that indicates whether the projected 2D point is visible throughout the motion occuring in that view. The visibility mask is obtained through combination of 2D tracking occlusion checks and depth-based visi- bility verification, which involves comparing the rendered depth from Gaussian primitives with the projection depth determine whether the point is visible in front of the surface.
We use TAPIR [1] as our 2D tracking backbone, which takes query points and video frames as input and produces 2D tracking positions throughout the video with the corresponding occlusion mask, indicating whether a tracked point is visible in a given frame. obtained from 2D tracking, the visibility mask used in Eqn. 14] for triangulation also depends on depth-based visibility. Depth-basee visibility is determined by comparing the projected depth of given splat with the depth map obtained by rendering all splats to given view. If the projected depth
addition to the occlusion mask
24

---

multi-view stream
render results
time slice
Per-frame model
Time
PSNR: 21.80
GS-LRM
Runtime: 0.04s
History-aware model
Time
PSNR: 27.41
Ours
Runtime 0.07s
Figure 6: (Top) Per-frame reconstruction methods, which produce an independent scene reconstruc- tion at each time step, are prone to flickering artifacts. (Bottom) In constrast, history-aware novel view streaming model merges previous and current frame information, allowing us to better model occluded regions and improve temporal stability. Our method achieve sate-off-hheart visual quality and temporal consistency, and runs in interactive rate (15 fps with a 350ms delay") on two view inputs of resolution 320 x 240.
smaller, the splat considered unoccluded. This process utilizes alpha blending of splat depths to derive rendered depths from 3D Gaussian Splatting, widely adopted technique that has been demonstrated to be effective g
Embedded Deformation Graph Assuming motion continuity between neighboring frames, first estimate optical flow between each keyframe and its subsequent frame using RAFT 65 to identify regions of significant movement. We select regions within the input views where the optical flow norm exceeds threshold of 0.2 pixels as areas significant motion. Each splat is then projected onto the input views determine whether its projected position falls within these significant motion regions and remains unoccluded based on the depth-based visibility verification aforementioned. If splat exhibits significant motion in more than half of the views, it is prioritized for sampling as an anchor splat. During experiments, we found that uniformly sampling 512 points from the prioritized splats using the Farthest Point Sampling method 47 is sufficient for effectively handling moving Gaussians in the keyframes.
To warp splats from the previous frame, first compute deformations the anchor splats using triangulation, and then propagate the motion to all primitives through Linear Blend Skinning (LBS). For each Gaussian, anchor splats. The position displacement óp R is then determined by blending the deformations of the selected anchor points, denoted as E 3g
employ K-nearest neighbors approach with k 4 to identify the closest
óp
wet:
(15)
Wi exp(-Ild: /22
exp (1d
where d, represents the Fuclidean distance between the Gaussian primitive and the i-th anchor point, while a controls the smoothness of the weighting function, set the mean distance between anchor points and the querying splats. To ensure that static splats remain unaffected, we set displacement
25

---

Table Reproduced GS-LRM Performance. Our reproduced GS-LRM achieves improved LPIPS and SSIM scores with slight decrease in PSNR on the RealEstate 10K dataset compared to the officially reported numbers. We also include comparisons with other feed-forward novel view synthesis methods in this table.
Method
PSNR f
SSIM f
LPIPS 4
PiXelNeRF
82
20.43
0.589
0.550
GPNR
60
24.11
0.793
0.255
Du et al.
13
24.78
0.820
0.213
GS-LRM (Official)
84
28.10
0.892
0.114
GS-LRM (Reproduced)
27.78
0.902
0.082
GS-LRM (Reproduced) Ours
28.47
0.907
0.078
Table 10: Inference Time on RealEstate We compare the inference time of our method with other feed-forwed Gaussian Splatting approaches. All runtimes are measured on an NVIDIA A100 GPU. The slight runtime overhead of our method relative to GS-LRM can be mitigated by reducing the number of multi-view transformer layers. As reported in Table 2 our method improves PSNR by over dB while maintaining comparable model size and inference time.
Method
PSNR
SSIM
LPIPS
Inference Time(s)
pixelSplat
26.09
0.863
0.136
0.104
MVSplat
6
26.39
0.869
0.128
0.044
GS-LRM 84
28.10
0.892
0.114
0.041
Ours
28.47
0.907
0.078
0.067
óp to be zero if the distance to the anchors is greater than Ad, where is the mean distance between anchor points and À is hyperparameter controlling the motion graph's 's extent. In practice set between 0.2 to 0,5 depending on the scene.
For efficiency, we only maintain primitives from set of past keyframes, and the contribution of these primitives are smoothly adjusted as older keyframes are replaced by newer ones to ensure seamless temporal transitions. Due to time constraints from querying the 2D tracking model, in practice use two keyframes placed 5 frames apart, each lasting for 10 frames.
Error-aware Streaming Fusion To further mitigate artifacts resulting from warping inaccuracies of historical primitives (e.g. warping error, appearance variation, fusion weighting mechanism during splat fusion based on the per-pixel error between the past and current frame splats when rendered the input views. In more detail, each historical splat is projected onto the input views to assess whether it produces a higher rendering error and appears in front of the rendered depth of the current frame splats. If historical splat results in increased error in more than half of the input views, we set its fusion weight to zero to suppress unreliable contributions. Concurrently, current frame splats predicted from the multi-view transformer that increase rendering error are assigned zero weight. To enhance temporal continuity, we retain 50% of the voxel attributes from the past keyframe in static areas. Static areas are identified during the Splat-to-voxel transfer process, where historical splats undergoing deformation mark their neighboring voxels as regions of motion.
objects), use adaptive
Fig. Fig. showws the system implementation of our history-award streaming reconstruction. The streaming fusion and refinement process enables model to significantly reduce temporal flickering and improve reconstruction quality, as demonstrated in Fig.
B Dataset
B.1 Static Scene Datasets
RealEstate For the RealEstate 10K dataset, we set the training and testing resolution to 256 x 256. Both GS-LRM and our model are trained for 100K iterations. Specifically, we first train
26

---

Table 11: Quantitative Comparison with 4DGS and GS-LRM. Evaluation across all sequences second-best results are
in Long Volumetric Video [79 and Neural3DVideo highlighted.
The
best
and
4DGS
GS GS-LRM
Ours
Datasct
Sequenee
PSNR 1 SSIM f LPIPS Flicker
PSNR
SSIM Ş LPIPS Flicker
PSNR
SSIM 1 LPIPS Flicker
Corgi
17.25
0,530
0.423
8.17
26.24
0.882
0.118
9.95
26.48
0.885
0.109
8.76
Con
Bike
20.26
0.742
0.288
4.89
24.74
0.830
0.158
6.97
24.64
0.844
0.140
6.80
coffee_munt aoftee_aatini
19.29
0.676
0.268
1.33
20.86
0.801
0.149
7.21
2.33
masted beef
25.41
0.851
0.167
3.17
21.95
0.855
0.134
5.51
29.15
0.909
0.096
3.26
Neural3DVV
fame__teak _teak
26.54
0.854
0.159
3.24
22.04
0.866
5.56
3.31
24.41
0.794
0.198
3.49
21.93
0.872
0.119
5.78
29.26
0.906
0.103
3.42
flame_saiee saelsaaaoon__
19.92
0,709
0.250
2.18
21.70
0.827
0.126
5.65
24.70
0.860
0.102
2.52
sear_steok
23.42
0.804
0.206
2.52
22.23
0.869
0.122
4.92
29.14
0.907
0.098
2.64
Table 12: Method Comparison. Unlike existing baselines, our framework uniquely supports the three key requirements of the novel view streaming task targeted in this paper: fast reconstruction with interactive rate, sparse view inputs, and history awareness. In particular, our streaming approach only uses past frames, in constrast to DeformableGS [8]] and DeformableNerf [44) that optimize over all" frames. The history-awarene
in our model mitigates temporal flickering and occlusion artifacts, and is also able to support unlimited sequence lengths.
Method
Interactive Speed Sparse-view Input History Awareness
NeRF 41
InstantNGP
42
X
3DGS 22
X
DeformableGS
81
DeformableNerf 44
x
PixelSplat 4
GS DS-LRRM 84
Quark
15
Ours
our multi-view transformer backbone for 80K iterations, followed by 20K iterations of joint training with the voxel decoder. Each training batch consists of two input views and six target views with baseline of one unit length between the input views, following the training setup of GS-LRM. For evaluation, we use the same input and target indices PixelSplat and GS-LRM.
DL3DV (35) For the DL3DV dataset, we set the training and testing resolution to 384 x 216. For training, we randomly select one image in the scene as the target, and randomly select four of the nearest eight cameras to the target as inputs. We scale the scene such that the cameras fit within unit cube. For evaluation, we use every eighth image as the target set, and for each target we use the nearest four cameras not in the target set inputs. We average metrics per scene, and then average over all scenes. In our experiments on the DL3DV test set (Table 2). both our model and GS-LRRM are trained using 8 GPUs, with a batch size of 2 per GPU, for a total of 200K iterations. Our method supports both joint training of the multi-view Transformer and voxel decoder from scratch, and two-stage training scheme. As shown in Table 8 joint training from scratch outperforms the two-stage approach, where the multi-view Transformer is first trained for 100K iterations, followed by another 100K iterations of joint training with the voxel decoder. Figure Tigure/Jillustee illustrates that after training the multi-view Transformer used in GS-LRM, our proposed sparse voxel Transformer converges to significantly improved results with only small number of additional training steps.
For cross-dataset evaluation on dynamic scene datasets, we train both GS-LRM and our model on DL.DVV using batch size of 128, for total of 300K iterations. The training follows a two-stage process: we first train the view transformer backbone for 200K iterations, then jointly train the voxel decoder for 100K iterations.
27

---

Table 13: Comparison with Gaussian Graph Network (GGN) [86]. Both models are trained on RealEstate (2-view training, 4-eeww test- ing).
Figure 7: Comparison of Multi- and Voxel Trans- formers. Validation curves the DL3DV dataset show that our 3D Sparse Voxel Transformer converges faster and achieves significantly better final performance when initial- ized with pre-trained 2D Multi- View Transformer, com- pared training with the 2D Multi- Transformer alone. Our method also supports joint training of the Multi-Vier and Voxel Transformers, leading to further performance im- provements as shown in Table
Method
PSNRT
SSIM
LPIPS
GGN 86
24.76
0.784
0.172
Ours
28.79
0.914
0.081
Velidation Curves
Table 14: Extension to DepthSplat. Our method can be seamlessly integrated into other feed-forward 3D Gaussian Splatting frameworks to enhance their performance. Both models are trained small subset of DL3DV (2-view training, 4-view testing).
20 Nutt-stnn Thanshomer
30 Sparse Wooel Transfomer
25-
Method
PSNR SSIMf LPIPS.
SppthSplat 76
16.57
0.4108
0.4272
DepthSplat + Ours
17.66
0.4426
0.4019
50 100
250 201 250 320 250 400 430 500 533 660 550 A0 (20 500 Rb0 500 150
TPatning Step (K)
B.2 Dynamic Scene Datasets
For evaluation, we use two input views with a small-baseline setup and across six sequences from Neural3DVideo, and four input views with large-baseline setup and 1,500 frames for each of the two sequences from Long Volumetric Video.
We use six sequences from the dataset for evaluation, each containing 290 frames. The center target camera is selected as the evaluation view, while the two second-closest cameras on either side are used as input cameras. We use resolution 320 x 240 downsampling from their original resolution.
Neural3DVideo
Long Volumetric 799 For evaluation, we use the Corgi and Bike sequences and select 4-input-camenn camprt setup that covers one target view. We failed to find enough baseline coverage for other sequences in Long VolCap using only 4 input cameras. For both the Corgi and Bike sequences, we use 1500 frames for evaluation. We use resolution 384 x 216 for Corgi and 256 x 256 for the Bike sequences, downsampling from their original resolution.
C Baselines
3DGS We use the original 3D Gaussian Splatting implementation 22). and reduce the number of training iterations from original 30,000 to 1,000 for Nearal3DVideo and 1,500 for LongVolu- Volu- metricVided This modification not only accelerates training but also significantly enhances novel view synthesis results (e.g. around PSNR improvement). Given the sparsity of the input data, this adjustment is crucial for preventing overfitting. All other parameters are kept consistent with the original implementation.
3DGStream We use the open-source 3DGStream (63 implementation to process the multi-view videos. We first use the original 3DGS to optimize on the first frame for 1,500 iterations, warm up the NTC cache, and then optimize on the next frame using the previous 3DGS for 150 iterations, following the original implementation.
4DGS We use the open-source 4DGS 74 implementation to process the multi-view videos. Following their implementation, we first optimize the first frame for 3,000 iterations, and then optimize on the full multi-view video sequence for 14,000 iterations.
28

---

TALK
TARY
TARK
TO
GT
GS-LRM
Ours
Figure 8: Reconstruction Close-ups Dynamic Scenes. We show zoomed-in novel view recon- structions of GS-LRM and our model on dynamic scene datasets. Our model better handles occlusion boundaries with sharper detail.
We enhance GS-LRM's speed by replacing the CNN deconvolution with MLP unpatchify layer as proposed [21]. and implementing it in JAX. We maintain the same network parameters and training loss weights in the original paper. Additionally, we use the LPIPS loss employed in our model, achieving improved LPIPS and SSIM scores, with a slight decrease in PSNR, detailed in Table I0 We adopt the same color modeling approach as the official paper for RealEstate 10K, setting the spherical harmonics degree to O. For the GS-LRM baseline trained on DL3DV, set the degree to 1 to ensure fair comparison with method. Additionally, found that this modification enhances GS-LRM's performance on DL3DV.
GS-LRM
D Additional Results
Novel View Streaming Table II presents detailed metrics for our results alongside the two best- performing baselines, 4DGS and GS-LRM, evaluated across all eight sequences. We show close-ups of the reconstruction in Fig. g comparing ours with GS-LRM.
Feed-forward Novel View Synthesis We also provide additional examples comparing GS-LRM with our method on both DL3DD and RealEstatel0K: as shown in Fig. 9 aland Fig. 10
We also compare our method with Gaussian Graph Network (GGN) 86 on the RealEstate dataset, following its experimental configuration. Both models are trained with 2 input views and evaluated with imput views to assess generalization across varying input configurations. As shown in Table 13 our method significantly outperforms this state-of-the-art Gaussian fusion approach.
We further validate the generalization capability of the proposed Fuse-and and-Refine module by inte- grating it into DepthSplat (76). We retrain the publicly available DepthSplat implementation on the DL33V dataset a reduced resolution of 128 192. The training subset includes 1,000 scenes with 2 input views, and evaluation is performed on 10 test scenes with 4 input views. Table shows that incorporating our module leads to a performance gain over the original DepthSplat, demonstrating the adaptability of our approach.
29

---

GT
GS-LRM
GS-IRM-
Ours
Ours GT
Figure 9: DL3DV Static Scene Reconstruction. Additional qualitative examples of GS LS-L and our method trained on DL3DV.
GT
GS-LRM
Ours
GT
GS-LRM
Ours
Figure 10: Re10K Static scene reconstruction. Additional qualitative examples of GS-LRM trained on RealEsttate0kk [87].
30

---

E
Societal Impact
In this section, reflect on the broader socictal implications of our work. Our method facilitates fast and high-quality 3D reconstruction from sparse views, significantly lowering the barrier for creating digital assets from real- world scenes or objects. This has potential benefits across domains such as digital content creation, robotics, autonomous navigation, cultural heritage preservation, and immersive applications in AR/VR. Additionally, supports real-time streaming use cases, toward fully immersive experiences.
However, like many 3D/4D reconstruction methods, our approach could be misused to create detailed models of people, places, or objects without consent, raising concerns about privacy, identity theft, and misleading content. While our work intended for positive and legitimate use, we acknowledge the ethical responsibilities involved and urge users to apply this technique responsibly.
31


Reviews:
[{"review_id": "NKvxcZRUNU", "reviewer": "Reviewer_8DB3", "summary": "The paper lies in the space of feed-forward 3D Gaussian Splatting (3DGS) and is built on top of a feed-forward 3DGS model (GS-LRM) which predicts 3D Gaussian primitives from sparse-view images. The first contribution is a Splat-to-Voxel module, which aims to reduce redundancy and improve density control by assigning the 3D Gaussian primitives to fine and coarse voxel grids and fusing them in a coarse-to-fine manner. The second contribution is a Sparse Voxel Transformer, which enables learning refined Gaussian parameters from coarse voxel features. The Sparse Voxel Transformer is trained on static scenes, but it is shown that it can also generalise to dynamic scenes, where the proposed fuse-and-refine strategy is used to improve temporal consistency in streaming scene reconstruction.", "questions": "1. Does voxel sparsification effectively reduce the number of Gaussian primitives at the output of the feed-forward model? If so, is removing entire (coarse) voxels the most effective approach to culling? Wouldn’t culling at the level of primitives or even fine voxels provide better density control, without creating “dead” regions? Why or why not?\n2. How is the 20% threshold in voxel sparsification decided?  An ablation here would also be helpful. \n3. Can the Voxel Transformer be applied at the fine voxel level, and how would that affect performance? \n4. It is mentioned in the abstract that the model can process **20k** primitives in 15ms, and in L189 that the voxel transformer processes fewer than **10k** tokens in 15ms. Can the authors explain this further? \n5. How are the keyframes chosen in the streaming setting?\n6. In Table 4, why do the keyframe and non-keyframe variants give the same result? \n7. Can the streaming fusion be applied to monocular videos?", "limitations": "Limitations are sufficiently discussed in the paper, but some are primarily attributed to the underlying methods that the paper builds upon (such as limitations of the feed-forward model GS-LRM). I would have also liked to see a discussion of potential limitations of the approach itself.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The paper has a clear structure and is well-written, making it easy to understand.\n* The paper improves reconstruction performance over the baseline model (GS-LRM) with an efficient fuse-and-refine strategy, which takes only 15ms. This strategy reduces redundancy (through voxelization) and refines the Gaussian primitives by leveraging global context (through coarse-voxel features).\n* While the approach is trained on static scenes only, the paper also evaluates on a dynamic dataset, demonstrating the generalisation ability of the model to streaming scene reconstruction. I find this result particularly interesting, even if it may lag behind tailored dynamic scene reconstruction solutions.  \n\n## Weaknesses\n\nA major weakness of the paper is the lack of a detailed discussion and comparisons to related work [1,2,3,4,5]. In particular, quantitative comparisons could include more recent methods such as TranSplat [1] and EVolSplat [2] in the static setting, where the latter also focuses on efficiency, or HiCoM [5] in the dynamic setting. More importantly, Gaussian Graph Network [3] and HiSplat [4] are closely related to this paper, aiming to aggregate Gaussian primitives with graph operations or fusion modules. To fully judge the contribution and effectiveness of the paper, a thorough discussion and, where possible, comparisons to these works should be provided. \n\nDue to the focus on efficiency, the paper should provide a more detailed breakdown of the computation times (e.g., feedforward model, splat-to-voxel transfer, voxel transformer, rendering)  and a time comparison to prior work on the static scene setting. It would be interesting to see in more detail how the voxel sparsification affects efficiency. \n\nSome more model considerations / ablations could be discussed or included in the paper (see Questions).\n\n### References\n[1] Zhang, Chuanrui, et al. \"Transplat: Generalizable 3d gaussian splatting from sparse multi-view images with transformers.\" AAAI 2025.\n\n[2] Miao, Sheng, et al. \"Evolsplat: Efficient volume-based gaussian splatting for urban view synthesis.\" CVPR 2025.\n\n[3] Zhang, Shengjun, et al. \"Gaussian graph network: Learning efficient and generalizable gaussian representations from multi-view images.\" NeurIPS 2024.\n\n[4] Tang, Shengji, et al. \"Hisplat: Hierarchical 3d gaussian splatting for generalizable sparse-view reconstruction.\" ICLR 2025.\n\n[5] Gao, Qiankun, et al. \"Hicom: Hierarchical coherent motion for dynamic streamable scenes with 3d gaussian splatting.\" NeurIPS 2024.", "was_revised": true, "final_justification": "The rebuttal has addressed my open questions well, and I am happy to increase my rating. Reading the other reviewers and responses, I did not find any strong concerns that would prevent acceptance.", "rebuttal": "We appreciate the reviewer’s valuable feedback and their recognition of our method, especially its ability to generalize to streaming scene reconstruction. In the following, we address the identified weaknesses in detail.\n\n## **W1: Discussion and Comparison with Related Work**\nWe thank the reviewer for highlighting relevant works for comparison and discussion, which we will thoroughly incorporate into the revised version. Here, we outline the key advantages of our method over **HiSplat** and **Gaussian Graph Network**, both of which also explore aggregation strategies for Gaussian primitives.\n\n**HiSplat** predicts multi-scale per-pixel Gaussian primitives and fuses these hierarchical predictions at each pixel location. However, the absence of a 3D canonical space for fusion limits its ability to generalize to temporal fusion of Gaussian primitives across frames, which our method is explicitly designed to support. Furthermore, our sparse voxel transformer learns an effective global strategy using full attention. This results in substantially improved performance on the RealEstate10K dataset (shown in the table below) compared to the local per-pixel fusion approach in HiSplat.\n**Gaussian Graph Network** fuses per-pixel primitives using graph convolutions over local neighborhoods, but its scalability is limited since the number of edges grows quadratically with the number of input primitives. By transferring an arbitrary number of primitives into a fixed voxel representation, our voxel transformer maintains constant processing time regardless of input size. Moreover, graph convolutions are inherently limited by their local receptive fields, making it difficult to capture the global organization of primitives. Our approach overcomes these limitations by leveraging full attention within the voxel transformer to aggregate information across the entire scene, enabling long-range context modeling and global structural fusion and refinement.\n\nWe also include a quantitative comparison with more recent feed-forward 3D Gaussian Splatting methods including **TranSplat**, **HiSplat**, **Omni-Scene** on the RealEstate10K dataset. As shown below, our method achieves the best performance among all baselines, demonstrating its effectiveness. Since **EvoSplat** does not report results on RealEstate10K, we include **Omni-Scene** instead, a related method that similarly adopts a voxel-splat representation for enhancing driving scenarios.\n\n\n**Table: Additional Baseline Comparisons on the RealEstate10K Dataset**\n\n|Method|PSNR↑|SSIM↑|LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|Omni-Scene|26.19|0.865|0.131|\n|TranSplat|26.69|0.875|0.125|\n|HiSplat|27.21|0.881|0.117|\n|**Ours**|**28.47**|**0.907**|**0.078**|\n\n\n\n\nWe also compare our method with **Gaussian Graph Network** on the RealEstate10K dataset, following its experimental configuration. Both models are trained with 2 input views and evaluated with 4 input views to assess generalization across varying input configurations. As shown in the table below, our method significantly outperforms this state-of-the-art Gaussian fusion approach.\n\n**Table: Comparison with Gaussian Graph Network on the RealEstate10K Dataset (2-view training, 4-view evaluation)**  \n\n|Method|PSNR ↑|SSIM ↑|LPIPS ↓|\n|:-:|:-:|:-:|:-:|\n|Gaussian Graph Network|24.76|0.784|0.172|\n|**Ours**|**28.79**|**0.914**|**0.081**|\n\n\n\n\nDue to time constraints, we could not include recent streaming methods like HiCoM in the rebuttal, but we will add them in the revision. *HiCoM*, like 3DGStream, is designed for dense-view input and performs poorly under sparse views. While faster than 3DGStream, it still takes over 1 second per frame, making it unsuitable for online reconstruction. In contrast, our method runs at 0.07 seconds per frame.\n\n\n\n## **W2: Efficiency Analysis**\n\nWe present a detailed breakdown of the inference runtime of our method, evaluated on an NVIDIA H100 GPU. We first report the runtime breakdown for static scene reconstruction on RealEstate10K using two input views at 256×256 resolution. \n\n**Table: Runtime Breakdown of Static Scene Reconstruction on the RealEstate10K Dataset**\n\n|Components|Multi-view Transformer|Splat-to-Voxel Transfer|Voxel Transformer|Rendering|\n|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.026|0.005|0.016|0.003|\n\n\n\nWe further analyze the runtime of each component in a streaming reconstruction setting on the Neural3DVideo dataset, using 2 input views at a resolution of 320×240. Note that 2D tracking is applied only to keyframes, which are sampled every 5 frames.\n\n\n\n**Table: Runtime Breakdown of Streaming Reconstruction on the Neural3DVideo Dataset**\n\n|Components|Multi-view Transformer|2D Tracking|3D Warping|Temporal Fuse-and-Refine|Rendering|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.030|0.28|0.011|0.026|0.003|\n\n\n\nWe also present an inference time comparison between our method and other feed-forward Gaussian Splatting approaches, including PixelSplat, MVSplat, and GS-LRM, on the RealEstate10K dataset. All runtimes were measured on an NVIDIA A100 GPU. \n\n**Table: Inference Time Comparison on the RealEstate10K Dataset**\n\n|Method|PSNR↑|SSIM↑|LPIPS↓|Inference Time(s)|\n|:-:|:-:|:-:|:-:|:-:|\n|pixelSplat|26.09|0.863|0.136|0.104|\n|MVSplat|26.39|0.869|0.128|0.044|\n|GS-LRM|28.10|0.892|0.114|**0.041**|\n|**Ours**|**28.47**|**0.907**|**0.078**|0.067|\n\nThe slight runtime overhead of our method compared to GS-LRM can be mitigated by reducing the number of multi-view transformer layers. As shown in Table 1 of our paper, our method improves PSNR by over 2 dB while maintaining comparable model size and inference time.\n\n\n## **Q1&Q3: Effect of Voxel Sparsification and Applying Transformer on Fine-level voxels**\nThe primary goal of voxel sparsification is to reduce the number of input tokens for the voxel transformer. Without sparsification, the token count would scale cubically with the voxel dimensions, with excessive GPU memory usage and slow compute speed. Relatedly, applying the transformer on coarse-level voxels also helps to reduce token count, whereas directly applying the transformer on fine-level voxels becomes infeasible unless we reduce voxel resolution which harms reconstruction quality. While the reviewer is correct that culling the fine voxels can improve rendering speed by reducing the final number of primitives, our coarse-level sparsification achieves two goals: reduceing the final primitive count and making the voxel transformer tractable.\n\nAs shown in Table 1, we include ablation studies for the setting without voxel sparsification (Ours (w/o Coarse-to-Fine)). Due to memory constraints, we reduce the voxel resolution by 75% in this setting, which still results in slower inference and degraded reconstruction quality. Similarly, in the variant without the coarse-to-fine hierarchy (Ours (w/o Coarse-to-Fine)), the voxel transformer is applied only at a single voxel level. Applying the transformer directly on dense-level voxels would require an even greater reduction in resolution due to memory limitations, leading to significantly worse results.\n\n\n## **Q2&Q4: Sparsification Ratio and Token/Primitive Numbers**\nWe apologize for the typo in the abstract regarding the number of primitives. The correct number is 200K (as stated in line 57). \nThe number of primitives and tokens is calculated as follows: given 2 to 4 input views with a resolution of 384 × 216, GS-LRM generates multi-view per-pixel primitives (2~4 × 384 × 216), resulting in approximately 165K to 330K primitives. For these primitives, Our splat-to-voxel transfer converts them into a coarse voxel grid with a resolution of (384 / 8) × (216 / 8) × 32. We then sparsify the grid based on voxel opacity and retain only the top 20%, yielding roughly 8K tokens for the transformer input. This voxel transformer takes approximately 15 milliseconds for the forward pass, as shown in the breakdown above. \n\nThe sparsification ratio (20% threshold) is chosen to keep the number of tokens below 10K, ensuring that the transformer runs within 15 milliseconds. Although using a higher sparsification ratio can further improve speed, it risks discarding useful information. Conversely, a lower ratio increases token count and slows down inference. In practice, we find that a sparsification ratio between 15% and 25% offers a good balance between performance and quality. We also observed that sparsity is common in large-scale unbounded scene modeling, with typically fewer than 30% of voxels having a primitive opacity greater than 0.005.\n\n\n## **Q5&Q6: Keyframing**\nWe select a keyframe every 5 frames to run the 2D tracking model, which takes approximately 0.28 seconds and provides tracking results for the following 5 frames. Only two keyframes are maintained during streaming reconstruction to ensure efficiency.\n\nAs expected, non-keyframe frames may exhibit slightly reduced performance compared to keyframes due to accumulated 3D warping errors. However, we report the average performance over the entire sequence in paper Table 4, which reflects the practical effectiveness of our method under the streaming setting.\n\n## **Q7: Monocular Input**\nOur method currently does not support monocular streaming, as the GS-LRM backbone requires at least two input views to produce reliable Gaussian primitives. However, with a reliable single-view Gaussian Splatting predictor, our framework could naturally be extended to support monocular reconstruction.\n\n## **Limitation**\nWe acknowledge that coarse-level pruning may introduce “dead regions” in the reconstruction. However, we observe that typical 3D scenes do contain large regions of empty space, and our pruning step exploits this fact to improve processing efficiency. We currently use a simple heuristic, assuming a fixed sparsity ratio of 20%, but a potential direction for future work is a more adaptive and scene-aware pruning strategy."}, {"review_id": "YAO2hFoA6J", "reviewer": "Reviewer_LksN", "summary": "The paper presents a novel method for efficient 3D scene reconstruction using a Fuse-and-Refine module that enhances feed-forward 3D Gaussian Splatting. It shows several applications both in static and dynamic reconstructions.\n\nThe Fuse-and-Refine module is motivated by the fact that existing GS LRM models typically predict a gsplat at every incoming pixel, which is very redundant and won't scale well to long sequence or dynamic scenes. \nIt uses a multi-scale (two scale) voxel structure to \"splat\" the splats, prune the undersampled, then globally refine the voxels on the coarse-to-fine level through a transformer, then finally decode the voxels back to gsplats. \nThis module can be trained *after* a pretrained GS-LRM and will quickly converge achieving higher visual quality. The module is also designed with performance in mind, achieving higher PSNR compared to pixel-aligned GS-LRM with similar runtime.", "questions": "The online splat-voxel construction should ideally enable large scene online reconstruction. \nI'm curious how this paper would compare with other splat-voxel representations such as\n- SCube: Instant Large-Scale Scene Reconstruction using VoxSplats (code available)\n- Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction\n\nIn table 4, does the reported time includes also tracking and 3d warping? Is it possible to report each step individually? \nIn the same table, I'm not sure why 3dgs rime is fast? is this the training time or rendering time? in appendix you mentioned shortening the training steps significantly, while the psnr on new views grow 4db, it is not a good reason to use not converged 3dgs.", "limitations": "I think it has some really good ideas and should work well on long sequence static scene reconstruction. \nI reservation is rooted in the paper's strong claim in dynamic scenes. While it is impressive that the method can handle dynamic scenes\nwithout training on dynamics, its dynamic expression mainly comes from continuous tracking (via existing 2D tracking) and robust fusion. From the presented video, such robust fusion often translates to visual \"averaging\",  in terms of reconstruction quality, it will never reach the level of a model that actually trains with dynamic scenes.\n\nThe comparison on this topic focuses on rather simple sequences. Furthermore, the comparison is done with methods that don't support dynamic reconstruction natively. I'm not convinced of the application. \n\nThough I do like the methodology, and think it could be a great paper if the authors focused only on static scenes and built a compelling case in static scene reconstruction by comparing with other strong baselines, such as SCube.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n- High impact: breaking the old format of per-pixel gsplat is a key to scalability, this paper address this challenge in a refreshing way, which is worth sharing to the community.\n- Practical application: high throughput for streaming application, while significantly reduce the gaussian counts, hence further improve rendering speed. \n- Well written, well structured experiments and ablation studies. \n\n# Weakness\n- The position of the paper is not very convincing to me. While spending significantly amount of time on building a use case for dynamic scene reconstruction, the method is ultimately still for static-scene reconstruction. In the video, the model clearly struggles with highly dynamic components, such as the flame. I think being able to robust handle inconsistent input views is a benefit of voxel and global refinement, but it shouldn't lead to claims of 4D scene reconstruction. \n- Related, I think the comparison for novel-view streaming is not really fair. 3DGS and GS-LRM doesn't have any dynamic component there, equivalent 3D warping should be added in order for the comparison to make sense.", "was_revised": true, "final_justification": "My concern over dynamic scene reconstruction wasn’t raised by other reviewers, I concede that the emphasis on that is reasonable for the online streaming reconstruction as the author pointed out.", "rebuttal": "We thank the reviewer for recognizing our fuse-and-refine strategy, as well as for the constructive feedback regarding dynamic scene reconstruction and the suggestion to place greater emphasis on static scenes. Below, we address the discussion points in detail.\n\n\n\n## **W1&W2&Limitation: Dynamic Scene Reconstruction**\nWe would like to clarify that our application to dynamic scene reconstruction specifically focuses on sparse-view online streaming reconstruction, a particularly challenging and underexplored setting within the field. By achieving interactive reconstruction rates of 15 FPS from sparse-view, long streaming input, our method becomes highly applicable to online augmented and virtual reality scenarios. Existing methods fall short in this scenario: optimization-based methods such as 3DGS and 4DGS struggle with sparse-view inputs, face scalability issues with long video sequences, and fail to meet the reconstruction speed requirements of online applications; feed-forward approaches like GS-LRM do not leverage historical information to ensure temporal consistency, often leading to noticeable flickering artifacts.\n\nThanks to the reviewer's suggestion, we additionally apply 2D tracking and 3D warping to GS-LRM and present the results below.  We use the same keyframing strategy as described in the paper, refreshing the warped primitives every 5 frames to mitigate error accumulation.\n\n\n**Table: Comparison for Adding 3D Warping to GS-LRM**\n\n|Method|PSNR↑|SSIM↑|LPIPS↓|Flicker↓|\n|---|:-:|:-:|:-:|:-:|\n|GS-LRM|21.80|0.8488|0.1278|5.714|\n|GS-LRM +  3D Warping|21.40|0.8250|0.1419|**2.163**|\n|GS-LRM + 3D Warping + Fusion (Concat+Dropout)|19.52|0.7429|0.2502|3.983|\n|GS-LRM + 3D Warping + Ours (Fuse-and-Refine)|**27.41**|**0.8863**|**0.1040**|2.916|\n\n\nThe results highlight that leveraging historical information by **directly applying 3D warping to per-frame reconstruction backbones is prone to error**. Specifically, warping past reconstructions to later frames (GS-LRM + 3D Warping) leads to larger reconstruction errors, as it fails to handle newly appearing content and suffers from accumulated warping errors.  Addressing these limitations is non-trivial. A simple fusion baseline (Concat + Dropout) incrementally adds new primitives while discarding 50% per frame to prevent unbounded growth in the number of primitives, but lacks refinement and often removes accurate primitives.\n\n\nTo overcome these challenges, we propose a novel fuse-and-refine mechanism. The module efficiently leverages historical information, functions at interactive rates, and does not require training on the currently limited multi-view temporal datasets. We believe this robust fusion strategy provides a compelling and practical solution to the challenges of online streaming reconstruction. \n\nWe also acknowledge the current limitations in performance, for example in the dynamic components as noted by the reviewer. We believe these challenges are due in part to our simple 3D warping strategy, which may have limited accuracy under the sparse-view setting. Future work could explore more advanced 3D warping backbones to address this issue, which is beyond the scope of this work. Due to the scarcity of available multi-view video data, we currently train our method only on static scenes. Nonetheless, the fuse-and-refine framework holds strong potential for training on dynamic scenes, which could further mitigate temporal artifacts in future work.\n\n\n\n\n## **Limitation: Extended Evaluation for Static Scene Reconstruction**\n\nWe appreciate the reviewer’s recognition of our method’s potential for static scene reconstruction. As shown below, our method outperforms recent feed-forward 3D Gaussian Splatting methods on RealEstate10K:\n\n**Table: Quantitative Comparison with Additional Baselines on the RealEstate10K Dataset**\n|Method|PSNR↑|SSIM↑|LPIPS↓|\n|:----:|:---:|:---:|:-----:|\n|pixelSplat [CVPR2024]|26.09|0.863|0.136|\n|MVSplat [ECCV2024]|26.39|0.869|0.128|\n|GS-LRM [ECCV2024]|28.10|0.892|0.114|\n|TranSplat [AAAI2025]|26.69|0.875|0.125|\n|HiSplat [ICLR2025]|27.21|0.881|0.117|\n|Omni-Scene [CVPR2025]|26.19|0.865|0.131|\n|DepthSplat [CVPR2025]|27.47|0.889|0.114|\n|Ours|**28.47**|**0.907**|**0.078**|\n\n\n\nOur method also demonstrates strong generalization capability across varying input views. We compare it with Gaussian Graph Network, a state-of-the-art method for handling diverse input views through local primitive fusion using graph convolution. We follow its experimental setup to train the model with 2 input views and evaluate with 4 input views on the RealEstate10K dataset. As shown in the table below, our method not only significantly outperforms the Gaussian Graph Network but also maintains consistent reconstruction quality when generalizing from 2 to 4 input views.\n\n\n**Table: Comparison with Gaussian Graph Network on the RealEstate10K Dataset (2-view training, 4-view evaluation)**  \n\n|Method|PSNR↑|SSIM↑|LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|Gaussian Graph Network|24.76|0.784|0.172|\n|**Ours**|**28.79**|**0.914**|**0.081**|\n\n\n\nOur method also performs well on large unbounded scene datasets such as DL3DV. As shown in paper Table 1 and Appendix Figure 9, our fuse-and-refine approach significantly improves the reconstruction quality of GS-LRM, yielding a PSNR improvement of approximately 2dB while maintaining a comparable model size.\n\nTrained with 4 input views on the DL3DV dataset, our method consistently outperforms GS-LRM across a wide range of input configurations as shown in the table below. Since input views are chosen based on proximity to the target, increasing from 4 to 16 introduces a distribution shift while contributing limited new information for target view rendering, which explains the performance drop in both methods. Nevertheless,  our method maintains significantly better reconstruction quality than GS-LRM across all input settings. We believe these results further demonstrate the potential of our method to handle long sequence large scene reconstruction, as noted by the reviewer. \n\n**Table: Quantitative Comparison under Varying Numbers of Input Views on the DL3DV Dataset (4-view Training)**\n\n\n|Method||2 views|||4 views|||8 views|||16 views||\n|:----:|:-----:|:--:|:------:|:-----:|:-----:|:------:|:-----:|:-----:|:------:|:------:|:------:|:--------:|\n|      |PSNR↑|SSIM↑|LPIPS↓|PSNR↑|SSIM↑|LPIPS↓|PSNR↑|SSIM↑|LPIPS↓|PSNR↑|SSIM↑|LPIPS↓|\n|GS-LRM|18.99|0.810|0.135|28.59|0.925|0.063|22.51|0.879|0.106|20.12|0.792|0.177|\n|Ours|**26.32**|**0.876**|**0.093**|**30.34**|**0.934**|**0.054**|**28.98**|**0.930**|**0.060**|**26.06**|**0.889**|**0.093**|\n\n\nWe would also be happy to include more evaluations on static scene reconstruction, and would appreciate it if the reviewer could suggest specific experimental setups.\n\n\n\n\n## **Q1: Comparison with Splat-voxel Representations**\nWe thank the reviewer for highlighting relevant voxel-splat representations and will include the following discussion in the revised related works section.\n**SCube** pretrains a voxel diffusion model using LiDAR and COLMAP point clouds from a self-driving dataset to infer voxel representations at inference. However, it is extremely slow (several seconds per frame) and not applicable to scenes like RealEstate10K that lack LiDAR input. In contrast, our method is trained using only RGB supervision. It also supports large-scale scene reconstruction such as DL3DV, and achieves significantly faster inference times of approximately 70 milliseconds per frame. **Omni-Scene** combines pixel-aligned Gaussians with Gaussians constructed from feature volumes, aiming to exploit the strengths of both pixel-based and volumetric representations. Unlike our method, this design is not easily extendable to dynamic scenes. Additionally, both SCube and Omni-Scene rely on 3D CNNs to process voxel features in a tractable manner, but this design leads to limited perceptual fields. In contrast, our approach introduces an efficient coarse-to-fine voxel hierarchy, allowing us to directly perform full self-attention over the input tokens. As shown in the comparison on the RealEstate10K dataset above, our method significantly outperforms Omni-Scene and achieves state-of-the-art performance.\n\n\n\n## **Q2: Runtime Breakdown**\n\nThe runtime reported in paper Table 4 includes both 3D warping and 2D tracking times. 2D tracking is applied only to keyframes, sampled every 5 frames. Below, we provide a detailed runtime breakdown for both streaming and static scene reconstruction (tested on H100).\n\n\n**Table: Runtime Breakdown of Streaming Reconstruction on the Neural3DVideo Dataset**\n\n|Components|Multi-view Transformer|2D Tracking|3D Warping|Temporal Fuse-and-Refine|Rendering|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.030|0.28|0.011|0.026|0.003|\n\n**Table: Runtime Breakdown of Static Scene Reconstruction on the RealEstate10K Dataset**\n|Components|Multi-view Transformer|Splat-to-Voxel Transfer|Voxel Transformer|Rendering|\n|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.026|0.005|0.016|0.003|\n\n\n## **Q3: Performance of Converged 3DGS**\nThe runtimes of 3DGS and 3DGStream reported in paper Table 4 include both training and rendering times. We apply early termination during training due to a severe overfitting problem observed in 3DGS under the sparse-view setup. To illustrate this, we show the training and validation PSNR curves for 3DGS on a single frame from the 2-view Neural 3D Video dataset. \n\n|Training Iterations|500|1000|2000|3000|10000|15000|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Training Time(s)|4|9|15|22|72|108|\n|Training PSNR|16.84|17.91|37.25|45.58|48.34|**50.11**|\n|Validation PSNR|16.88|**20.88**|17.72|17.52|13.64|13.39|\n\n\nAs shown in the table, validation PSNR peaks near 1000 iterations while training PSNR continues to rise, clearly indicating overfitting. This observation motivates our decision to limit training iterations. However, we are also happy to include the fully converged 3DGS and 3DGStream results using their original configurations in the revised version."}, {"review_id": "jZVWiOCyfP", "reviewer": "Reviewer_skmt", "summary": "This paper proposes a Fuse-and-Refine module to enhance existing feedforward 3DGS models. The core idea is to introduce a hybrid Splat-Voxel representation as a reference for refinement, enabling efficient learning via a coarse-to-fine voxel hierarchy. The proposed method further demonstrates generalization by supporting zero-shot extension to 4DGS tasks.", "questions": "See the weaknesses.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents an efficient method to refine Gaussian primitives generated by existing feedforward 3DGS pipelines, improving reconstruction quality without retraining the entire system.\n2. The proposed coarse-to-fine voxel hierarchy converts 3D Gaussian splats into a more structured voxel representation, which helps reduce the fitting burden on the Transformer and achieves better refinement results with improved computational efficiency.\n3. The fuse idea is conceptually elegant and naturally extends to dynamic scene reconstruction, as it enables fusing primitives across time, offering promising potential for 4DGS applications.\n\nWeaknesses:\n1. While the proposed fusion strategy is conceptually helpful for mitigating redundancies caused by overlapping viewpoints, its benefit is not clearly reflected in visual quality. For example, in Figure 3 (Static scene reconstruction, bottom row), the chair’s backrest becomes somewhat transparent compared to GS-LRM. This raises concerns of hollow or missing geometry, making it unclear whether this represents a genuine improvement or a regression.\n2. The method heavily relies on the quality and sparsity of the initial Gaussian primitives generated by existing methods. It seems that the method also intervenes when the pixel-aligned-based method is almost converging. Can the scheme be trained end-to-end directly?\n3. The overall architecture may be similar to other voxel-based point-cloud networks. The author needs to introduce and compare these works, which are in different fields but have structural similarities.\n\nMinors:\nFigure 6: “Histoy-aware” ->“History-aware”", "was_revised": true, "final_justification": "The author provided detailed and somewhat persuasive responses and supplementary experiments. I agree with the idea of using 2D-3D hybrid networks for Feed-forward reconstruction in the paper. However, due to the limited improvement in performance, I maintain my score as Borderline accept.", "rebuttal": "We thank the reviewer for their valuable comments and appreciation of our fuse-and-refine module. Below, we address the noted weaknesses.\n\n\n## **W1: Visual Improvement**\n\nWe observe that the key visual advantage of our method over GS-LRM lies in its ability to better preserve fine details in thin structures, whereas GS-LRM often produces blurrier edges (e.g., the chair backrest case). Additional qualitative results are presented on the DL3DV dataset in Figure 9 and the RealEstate10K dataset in Figure 10, where our method consistently outperforms GS-LRM in terms of both geometric accuracy and appearance fidelity, producing sharper boundaries and more fine-grained structural details. This advantage also extends to streaming reconstruction scenarios, as illustrated in Figure 4 and the close-up examples in Figure 8.\nQuantitative metrics and visual comparisons alike further support the effectiveness of our fuse-and-refine module in enhancing reconstruction quality. \n\n\n\n## **W2: End-to-end Training**\nWe thank the reviewer for this valuable comment. Our approach supports end-to-end training together with the per-pixel primitive prediction backbone, such as the multi-view transformer used in GS-LRM. Below, we show results on the DL3DV dataset using end-to-end training from scratch for 200K iterations. We compare this to the staged training strategy in Table 1 of the paper, where the multi-view transformer is first trained alone for 100K iterations, then jointly trained with our fuse-and-refine module for another 100K iterations. To ensure a fair comparison in model capacity, both variants of our method use a 12-layer multi-view transformer, whereas GS-LRM employs a 24-layer transformer and is trained for 200K iterations.\n\n|Method|PSNR↑|SSIM↑|LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|GS-LRM |28.59|0.925|0.063|\n|Ours (Staged Training)|30.34|0.934|0.054|\n|Ours (End-to-end training)|**30.61**|**0.935**|**0.052**|\n\nThese results demonstrate that end-to-end training yields further improvements in performance.\n\n## **W3: Comparison with voxel-based point-cloud networks**\n\nWe thank the reviewer for the insightful suggestion regarding the connection to voxel-based point cloud networks. In the revised version, we will include a thorough discussion and citations of relevant voxel-based methods, and we would also like to highlight two key advantages of our approach over existing techniques.\n\n**Fine-grained Reconstruction in Unbounded Scene**: Voxel-based point cloud methods typically convert irregular point sets into uniform grids to enable structured operations such as 3D convolutions [1,2]. While effective for object-level 3D perception tasks, these approaches struggle to scale to large, unbounded environments. Bird’s Eye View (BEV) representations have been adopted in autonomous driving to handle large-scale point clouds more efficiently [3]. However, by projecting the scene onto a top-down plane, BEV representations inherently sacrifice vertical resolution and full 3D geometry, limiting their applicability to general-purpose 3D reconstruction. In contrast, our method constructs a cost volume aligned with the target camera's normalized device coordinates (NDC), where the z-axis is parameterized linearly in disparity to enable finer modeling of near-field geometry. This voxel representation supports high-fidelity reconstruction across large, unconstrained scenes.\n\n**​​Global Perception Field:** While attention mechanisms have been introduced to voxel-based point cloud networks to overcome the locality constraints of traditional 3D convolutions, most approaches [4,5] are still limited to local attention due to memory constraints. Full global attention over all voxels or points remains challenging especially for large-scale unbounded scenes. We address this challenge by introducing a coarse-to-fine voxel hierarchy that enables global attention at the coarsest level. This design facilitates efficient long-range context aggregation while capturing fine-grained details essential for large-scale scene modeling.\n\n[1]: Zhou Y, Tuzel O. Voxelnet: End-to-end learning for point cloud based 3d object detection\n\n[2]: Liu Z, Tang H, Lin Y, et al. Point-voxel cnn for efficient 3d deep learning\n\n[3]: Li Z, Wang W, Li H, et al. Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers\n\n[4]: Mao J, Xue Y, Niu M, et al. Voxel transformer for 3d object detection\n\n[5]: He C, Li R, Li S, et al. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds"}, {"review_id": "7u5gPdHirD", "reviewer": "Reviewer_Wi3w", "summary": "This paper proposes a Fuse-and-Refine module to improve feed-forward 3D Gaussian Splatting for both static and dynamic scene reconstruction. The method introduces a hybrid Splat-Voxel representation that aggregates and refines Gaussian primitives using a sparse voxel transformer. It enables efficient fusion of overlapping or temporally adjacent primitives, significantly improving reconstruction quality and temporal consistency. Notably, the model generalizes to dynamic scenes in a zero-shot manner, achieving interactive performance without retraining.", "questions": "1.  Could the authors elaborate on how their method fundamentally differs from SplatFormer? Specifically, what architectural or functional advantages does the proposed Fuse-and-Refine pipeline offer over that prior work?\n\n2. Could you provide further analysis or experimental evidence to explain why the performance improvement is more pronounced in dynamic scenes than in static ones? For instance, does the temporal fusion mechanism contribute more significantly than the spatial refinement in achieving this gain?\n\n3. Could you provide a detailed breakdown of the computational bottlenecks introduced by your method? Have you explored any optimization strategies to reduce runtime while preserving the benefits, especially in dynamic scene reconstruction?\n\n4. Could you include experiments or analysis showing how your proposed modules perform when integrated with other state-of-the-art 3D Gaussian Splatting frameworks beyond GS-LRM? This would help demonstrate the generalizability and flexibility of your approach.", "limitations": "Yes. The authors have clearly acknowledged both technical limitations, such as reliance on high-quality initial primitives and residual temporal artifacts, and potential societal risks, including privacy concerns from unauthorized 3D reconstruction. They also propose future directions and ethical considerations, demonstrating responsible awareness. Further improvement could include outlining safeguards to mitigate misuse.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is clearly written and well structured.\n2. The hybrid Splat-Voxel representation, coupled with the sparse voxel transformer, efficiently refines \\~20 k Gaussian primitives in \\~15 ms and sustains interactive rendering (\\~15 FPS) while preserving detail, striking a strong balance between fidelity and computational cost.\n3. The zero-shot history-aware streaming fusion generalizes to dynamic scenes without retraining and yields state-of-the-art PSNR, SSIM, and LPIPS on both static and dynamic benchmarks, highlighting the method’s practical impact.\n\n**Weaknesses**\n\n1. The paper claims novelty in introducing a Splat-to-Voxel transfer followed by Transformer-based refinement, but the proposed design closely resembles SplatFormer \\[1], which is not cited or discussed. This omission raises concerns about the originality of the approach and the authors’ positioning of their contributions within the current literature.\n\n2. The method demonstrates more significant performance gains on dynamic scenes than on static ones, yet the explanation for this difference is shallow. The paper lacks in-depth analysis or ablation to clarify how design components like temporal fusion or the hybrid representation contribute to dynamic reconstruction advantages, and why static scenes benefit less.\n\n3. While the integration with GS-LRM shows promising improvements, it incurs substantial computational overhead, which undermines claims of real-time performance. The paper does not provide a breakdown of the time-consuming components or suggest optimizations to reduce latency and memory usage, especially important for practical deployment.\n\n4. The method’s evaluation is limited to GS-LRM and GS-LAM, making it difficult to assess generalizability. There is no validation on other recent 3D Gaussian Splatting frameworks, which weakens the claim that the proposed modules are broadly applicable across different architectures.\n\n5. The visual results do not adequately demonstrate the model’s handling of challenging cases such as occlusion or temporal inconsistency. More targeted visualizations and metrics are needed to evaluate robustness in synthesizing coherent views under occlusion or motion blur.\n\n6. Several technical terms and components lack sufficient clarity. For example, “history-aware streaming reconstruction” is not properly defined; the semantic role of the feature vector $F_k$ is unspecified; and the voxel pruning strategy (retaining top 20% by weight) is a fixed heuristic without adaptive guidance or performance feedback.\n\n7. The learning-based fusion strategy lacks theoretical grounding or interpretability analysis. Similarly, the error-aware fusion applies fixed thresholds to discard noisy primitives, but this may also remove useful information, potentially limiting reconstruction quality in complex or noisy scenes.\n\n8. The paper does not report inference-time GPU memory usage or system footprint, which is critical for assessing feasibility on mobile or edge devices. Moreover, minor presentation issues—such as incorrect references in Table 1 and missing discussion of related works like Mini-Splatting [2], Scaffold-GS [3], and Octree-GS [4], should be corrected to improve rigor and completeness.\n\n[1] Chen Y, Mihajlovic M, Chen X, et al. SplatFormer: Point Transformer for Robust 3D Gaussian Splatting[J]. arXiv preprint arXiv:2411.06390, 2024.\n\n[2] Fang G, Wang B. Mini-splatting: Representing scenes with a constrained number of gaussians[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 165-181.\n\n[3] Lu T, Yu M, Xu L, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20654-20664.\n\n[4] Ren K, Jiang L, Lu T, et al. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians[J]. arXiv preprint arXiv:2403.17898, 2024.", "was_revised": true, "final_justification": "My primary concerns regarding the “Comparison with SplatFormer” and “Integration with other Frameworks” sections remain unaddressed.\n\n1. Naming Convention of Point Transformer: The term “Point Transformer” conventionally refers to the seminal work by Zhao et al. [1], which introduced the foundational framework for point cloud processing via transformer architectures. In contrast, SplatFormer [2] represents a distinct, recently proposed extension tailored for 3D Gaussian splatting. This distinction should be explicitly clarified to avoid terminological ambiguity and ensure proper attribution.\n\n2. Hierarchical Aggregation in Point-based Designs: Point Transformer [1] have demonstrated hierarchical feature learning through iterative k-nearest neighbor (k-NN) aggregation and multi-scale sampling strategies. A more rigorous discussion contrasting the proposed hierarchical mechanism with the proposed approach would strengthen the technical contribution.\n\n3. Limited Validation of Framework Integration: While the integration with GS-LRM [3] is presented, the evaluation lacks generality to substantiate the method’s broader compatibility. To better demonstrate extensibility, experiments integrating with established frameworks should be included. This would more convincingly validate the architectural flexibility and scalability of the proposed approach.\n\n[1] Zhao H, Jiang L, Jia J, et al. Point transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 16259-16268.\n\n[2] Chen Y, Mihajlovic M, Chen X, et al. Splatformer: Point transformer for robust 3d gaussian splatting[J]. arXiv preprint arXiv:2411.0\n\n[3] Zhang K, Bi S, Tan H, et al. Gs-lrm: Large reconstruction model for 3d gaussian splatting[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 1-19.\n\nOverall, I tend to keep my rating (Borderline reject).", "rebuttal": "We appreciate the reviewer’s valuable feedback and thoughtful questions. We will correct the formatting issues and incorporate the relevant related work in the revised version. Below, we address the discussion points in detail.\n\n\n## **W1&Q1: Comparison with SplatFormer**\nWe respectfully note that SplatFormer is discussed and cited in line 201. However, we are happy to further elaborate here and will expand the discussion in the related work section of the revised version. Notably, our design yields two significant benefits:\n\n**Fusion capability and applicability to dynamic scenes:** Our fuse-and-refine sparse voxel transformer is capable of fusing and refining an arbitrary number of input primitives into a fixed number of output tokens. In contrast, SplatFormer uses a Point Transformer that outputs the same number of Gaussian primitives as its input, without fusing nearby points spatially. As a result, SplatFormer is not suitable for online streaming reconstruction tasks, as it cannot fuse redundant primitives over time. Our hybrid splat-voxel representation supports efficient and flexible fusion in both space and time, a capability that SplatFormer's purely point-based design cannot support. \n\n**Effectively Handling of unbounded scene:** SplatFormer’s Point Transformer is sensitive to the spatial distribution of input points and is primarily designed for object-centric settings. According to its original paper, SplatFormer demonstrates limited scalability when applied to large, unbounded scenes. Its uniform downsampling strategy in the point Transformer disproportionately reduces the resolution of foreground objects relative to expansive backgrounds, making it challenging to preserve fine-grained details in complex environments. In contrast, our method employs a coarse-to-fine voxel hierarchy, with the depth dimension of the voxels representing equal increments in disparity, and  prunes empty space where appropriate to reallocate compute to occupied regions. Our design enables effective and scalable reconstruction of unbounded scenes, while achieving significantly higher efficiency. When processing a similar number of primitives, our method completes a forward pass in under 15 ms, compared to over 100 ms for SplatFormer. \n\n\n\n\n\n\n\n## **W2&Q2:  Better Improvement on Dynamic Scenes**\nWhen applied to dynamic scenes, our fusion-and-refine module operates in both spatial and temporal domains. In addition to the multi-view spatial fusion-and-refinement also used in static scenes, it leverages temporal information by incorporating historical reconstructions.\nThat said, the more pronounced improvements observed in dynamic scenes can be attributed to the combined effects of both spatial and temporal fusion, as demonstrated in the ablation study presented in Table 3. Specifically, adding the hybrid representation for spatial fusion and refinement without temporal information (Ours without History-aware Fusion) achieves a reconstruction PSNR of 27.17, compared to GS-LRM’s 21.80. This indicates that spatial fusion is the primary contributor to the overall improvement in reconstruction quality. However, this variant suffers from severe flickering artifacts due to the lack of temporal consistency, with a flicker score of 5.91 worse than GS-LRM’s 5.71. By incorporating temporal fusion, our full method further improves the reconstruction quality from 27.17 to 27.41 PSNR and significantly reduces flicker from 5.91 to 2.92.  These results underscore the essential role of temporal fusion in maintaining temporal consistency, which is critical for high-quality dynamic scene reconstruction. Overall, we emphasize that the effectiveness of our fusion-and-refinement module lies in its ability to operate jointly across spatial and temporal domains, which is a key strength of our approach.\n\n\n## **W3&Q3: Efficiency Analysis**\n\n\nHere, we present a detailed breakdown of the inference runtime of our method for both static scene and streaming reconstruction, evaluated on an NVIDIA H100 GPU. We begin by reporting the runtime of individual components for static scene reconstruction on the RealEstate10K dataset, using two input views at a resolution of 256×256.\n\n**Table: Runtime Breakdown of Static Scene Reconstruction on the RealEstate10K Dataset**\n\n|Components|Multi-view Transformer|Splat-to-Voxel Transfer|Voxel Transformer|Rendering|\n|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.026|0.005|0.016|0.003|\n\n\nWe further analyze the runtime of each component in a streaming reconstruction setting on the Neural3DVideo dataset, using 2 input views at a resolution of 320×240. Note that 2D tracking is applied only to keyframes, which are sampled every 5 frames.\n\n**Table: Runtime Breakdown of Streaming Reconstruction on the Neural3DVideo Dataset**\n\n|Components|Multi-view Transformer|2D Tracking|3D Warping|Temporal Fuse-and-Refine|Rendering|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Time(s)|0.030|0.28|0.011|0.026|0.003|\n\n\nThe slight runtime overhead of our method compared to GS-LRM for static scene reconstruction can be mitigated by reducing the number of multi-view transformer layers in the GS-LRM backbone. As shown in Table 1 of our paper, our method **improves PSNR by over 2 dB while maintaining comparable model size and inference time**.\n\nFor streaming reconstruction, the primary bottlenecks are the 2D tracking and 3D warping modules. We believe that adopting a faster 2D tracking backbone and exploring more advanced 3D warping strategies beyond our current simple warping algorithm represent promising directions for future work.\n\n\n## **W4&Q4: Integration with Other frameworks**\nOur method can be naturally integrated with other feed-forward 3D Gaussian Splatting frameworks. Since the fuse-and-refine module directly operates on 3D primitives, it can be seamlessly plugged into the training pipelines of various feed-forward methods. In this paper, we choose GS-LRM as the backbone due to its state-of-the-art performance among existing feed-forward methods (see Table 2 of our paper).\nDue to time constraints, we were unable to include experiments with alternative backbones in the rebuttal. However, we plan to explore such integrations to further demonstrate the generalizability of our approach.\n\n## **W5: Occlusion Handling and Temporal Consistency**\nWe evaluate temporal consistency using the flicker metric, as described in Line 270 of the paper. As shown in Table 4, our method achieves the highest reconstruction quality while maintaining strong temporal consistency. A key challenge in evaluating occluded regions is obtaining accurate occlusion masks. We would be happy to include this metric and would appreciate any suggestions for generating reliable masks in sparse, in-the-wild videos.\n\nAs illustrated in Figure 4, GS-LRM struggles to reconstruct the window due to occlusion by the person. In contrast, our method achieves consistent reconstruction by leveraging historical information to effectively handle occlusions, as also shown at 2:00 in the supplementary video.\n\n\n## **W6: Clarification of Technical Terms**\nWe thank the reviewer for pointing out these unclear terms and will revise them accordingly. The term history-aware streaming reconstruction refers to leveraging information from past frames to enhance the reconstruction of new frames in arbitrarily sparse-view, long streaming inputs, thereby improving both quality and temporal stability.  \nThe feature vector $F_k$ is introduced in Line 135 as additional splat features derived from the image to enhance the fuse-and-refine module. Its effect is analyzed in paper Table 1 under the ablation setting \"Ours (w/o Splat Feature).\" \n\n## **W7: Interpretability Analysis of the Fuse-and-refine Strategy**\n\nThe motivation behind our learning-based fusion strategy is to merge redundant primitives across both spatial views and temporal frames. As discussed in Line 140, deriving an analytical solution for fusing Gaussian primitives is highly challenging. Our fuse-and-refine approach addresses this by learning a global fusion mechanism from large-scale data using a voxel Transformer backbone. Given a set of initial Gaussian primitives, we aggregate local features from Gaussian primitives into a voxel representation, and then apply the Transformer to generate refined Gaussian primitives from voxel features. Through end-to-end training with photometric rendering supervision, the module is optimized to refine imperfect primitives into a set of high-quality Gaussians. We empirically validate the effectiveness of our fusion strategy through comparisons with GS-LRM in both static and streaming scene reconstruction, and present a comprehensive ablation study of its design components in paper Table 1. \n\nOur voxel sparsification strategy is motivated by the observation that large-scale unbounded scenes are typically sparse. The sparsification ratio (20% threshold) is chosen to keep the number of tokens below 10K, ensuring that the transformer runs within 15 milliseconds. In practice, we find that a sparsification ratio between 15% and 25% offers a good balance between performance and quality.  A promising direction for future work is to develop a more adaptive, scene-aware pruning strategy.\n\nThe error-aware fusion module is designed to filter out primitives with large warping errors by comparing the rendered results of warped primitives against ground truth. We demonstrate the efficacy of this module in paper Table 3. Since useful primitives generally produce low errors, they are more likely to be preserved. Additionally, trained on large-scale datasets with diverse types of Gaussian primitive inputs, our learned fuse-and-refine strategy demonstrates robustness to variations in input primitives.\n\n## **W8: Runtime GPU Memory**\n\nOur method consumes approximately 10 GB of GPU memory during inference, primarily due to the naive implementation of voxel grid storage. This can potentially be optimized by adopting a sparse data structure."}]

Author rebuttal:

Meta-review: This paper considers the task of feed-forward 3D Gaussian Splatting (3DGS) from images, where pixelwise Gaussians are processed in 3D space to reduce redundancy, control density and refine the primitives with global context. The identified strengths include that the approach processes Gaussians in 3D-space rather than image-space, facilitating merging and refinement; the reconstruction performance; the good balance between fidelity and throughput; the unexpected and interesting generalization to streaming scene reconstruction; and the structure and clarity of the writing. The identified weaknesses include a significant amount of closely related work that was not discussed in the submission [8DB3, LksN, skmt, Wi3w]; closely related work that was not quantitatively compared to the proposed approach [8DB3, LksN]; missing efficiency analyses [Wi3w]; concerns about the dynamic scene performance and analysis [skmt, Wi3w]; and compatibility with other GS frameworks [Wi3w]. Several of these concerns were addressed in the author responses to the satisfaction of the reviewers, including the literature discussion, comparisons, efficiency analyses, and the dynamic streaming setting.

The initial ratings were relatively consistent but borderline (2xBR/2xBA). After reading all reviews, author responses, and engaging in discussion with the authors and each other, the reviewers are on-the-whole positive about this work, with the modal response recommending acceptance. The dissenting reviewer has ongoing concerns with the technical differences w.r.t. SplatFormer and the compatibility of the method with other frameworks. The latter, in the AC's view, was satisfactorily addressed in the authors final response, and the former can be handled by extending the discussion about this work in the revision. Overall, the AC sees no reason to override the view of the reviewer majority and is satisfied by the technical merit of the paper and its potential to make an impact on the field.
Decision: Accept (poster)