FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes
Authors: Yunsong Wang
Tianxin Huang
Hanlin Chen
Gim Hee Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian Splatting
Generalization
3D from multi-view sensors
Novel View Synthesis
3D Computer Vision
Primary area: machine_vision
TL;DR: We propose a generalizable 3D Gaussian Splatting framework to progressively fuse multi-view pixel-aligned 3D Gaussians for large-scale scene photorealistic reconstruction.

Abstract: Empowering 3D Gaussian Splatting with generalization ability is appealing. However, existing generalizable 3D Gaussian Splatting methods are largely confined to narrow-range interpolation between stereo images due to their heavy backbones, thus lacking the ability to accurately localize 3D Gaussian and support free-view synthesis across wide view range. In this paper, we present a novel framework FreeSplat that is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis.Specifically, we firstly introduce Low-cost Cross-View Aggregation achieved by constructing adaptive cost volumes among nearby views and aggregating features using a multi-scale structure. Subsequently, we present the Pixel-wise Triplet Fusion to eliminate redundancy of 3D Gaussians in overlapping view regions and to aggregate features observed across multiple views. Additionally, we propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views. Our empirical results demonstrate state-of-the-art novel view synthesis peformances in both novel view rendered color maps quality and depth maps accuracy across different numbers of input views. We also show that FreeSplat performs inference more efficiently and can effectively reduce redundant Gaussians, offering the possibility of feed-forward large scene reconstruction without depth priors. Our code will be made open-source upon paper acceptance.

FreeSplat: Generalizable 3D Gaussian Splatting Towards Pree-View Synthesis of Indoor Scenes
Yunsong Wang Tianxin Huang
Hanlin Chen
Gim Hee Lee
School of Computing, National University of Singapore
yunsong@comp nus edu. sg gimhee lee@nus.eu... edu. sg
https: htap::://githhne comm/vaagyss1f/rreSplaat
Rdf
Our Dopth Pred.
3D Gaussians
Interpolated Novel View Rendering
MVSplat
FreeSplat
Figure 1: Comparison between FreeSplat and previous methods. pixelSplat ① and MVSplat fail to reconstruct geometrically consistent global 3D Gaussians, while our FreeSplat is proposed to accurately localize 3D Gaussians from long sequence input and support free view synthesis.
Abstract
Empowering 3D Gaussian Splatting with generalization ability is appealing. How- ever, existing generalizable 3D Gaussian Splatting methods are largely confined narrow-range interpolation between stereo images due to their heavy backbones, thus lacking the ability to accurately localize 3D Gaussian and support free-view synthesis across wide view range. In this paper, we present novel framework FreeSplat that is capable reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. Specifically, we firstly introduce Low-cost Cross-View Aggregation achieved by constructing adaptive cost volumes among nearby views and aggregating features using multi-scale structure. Subsequently, we present the Pixel-wise Triplet Fusion to eliminate redundancy of 3D Gaussians in overlapping view regions and to aggregate features observed across multiple views. Additionally, we propose simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views. Our empirical results demonstrate state- of-the-art novel view synthesis peformances in both novel view rendered color maps quality and depth maps accuracy across different numbers of input views. We also show that FreeSplat performs inference more efficiently and can effectively reduce redundant Gaussians, offering the possibility of feed-forward large scene reconstruction without depth priors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recent advancements has emerged 3 45 6 in reconstructing 3D scenes from multiple viewpoints.
is capable of learning the implicit 3D geometry and radiance fields without depth information. Nonetheless, computational cost remains to be the inherent bottleneck in ray- may-marching- baaseinng--eedddd volume rendering, preventing it from real-time rendering. 3D Gaussian Splatting [0] TT 12 [3] has recently been proposed as efficient representation for photorealistic reconstruction of 3D scenes from multi-views. The explicit representation of 3D Gaussians are optimized to be densified in the textured regions, and the rasterization-base based volume rendering avoids the costly ray marching scheme. Consequently, 3D Gaussian Splatting has achieved real-time rendering of high-quality images from novel views. Nonetheless, vanilla 3D Gaussian Splatting lacks generalizability and requires per-scene optimization.
Based on may-marching-base volume rendering, Neural Radiance Fields 3
7
9
Several attempts 2 15 16 have been made to give 3D Gaussian Splatting generalization ability.. Despite showing promising performance, these methods are limited to narrow-range scene- level view interpolation [ limitation is that these existing methods depend on dense view matching across multi-view images with transformers to predict Gaussian primitives, which consequently becomes computationally intractable with longer sequences and thus restricting the supervision of these methods narrow-range interpolated views. As we show in Figure 4 supervision by narrow-range interpolated views often result in poorly localized 3D Gaussians that can become floaters when rendered from extrapolated views. Additionally, the problem is further aggrevated by existing methods typically merging multi- view 3D Gaussians through simple concatenation and thus inevitably lead noticeable redundancy in overlapping areas (cf. Table 2). In view of the above-mentioned problems. it is therefore imperative to design a method that is capable of long sequence reconstruction of global 3D Gaussians, which has the significant potential of supporting real-time rendering from arbitrary poses.
2 [S] and object-centric synthesis [14] [6]. The primary reason for the
In this paper, propose FreeSplat tailored for indoor long sequence free view synthesis. Unlike existing methods limited to view interpolation in narrow ranges, our method can effectively reconstruct explicit global 3DGS for novel view synthesis across wide view ranges. Our pipline consists of Low-cost Cross- Aggregation and Pixel-wise Triplet Fusion (PTF). In Low-cost Cross-Vies Aggregation, we introduce efficient CNN-based backbones and adaptive cost volumes formulation among nearby views for low-cost feature extraction and matching, then we leverage Multi-Scale Feature Aggregation structure broaden the receptive field of cost volume and predict Depths and Gaussian Triplets. Subsequently, we present Pixel-wise Alignment with progressive Gaussian fusion PTF to adaptively fuse local Gaussian Triplets from multi-views and avoid Gaussian redundancy in the overlapping regions. Moreover, due to our efficient feature extraction and matching, we propose a Free-View Training (FVT) strategy to disentangle generalizable 3DGS with specific number of views and train the model on long sequences.
The contributions of our paper are summarized follows:
1. We present Low-cost Cross-View Aggregation to predict initial Gaussian triplets, where the low computational cost makes it possible for feature matching between more nearby views and training on long sequence reconstruction;
2. We propose Pixel-wise Triplet Fusion to fuse Gaussian triplets, which can effectively reduce the Gaussian redundancy in the overlapping regions and aggregate multi-view 3D Gaussian latent features;
3. To the best of our knowledge, we are the first to explore generalizable 3DGS for long sequence reconstruction. Extensive experiments on indoor dataset ScanNet (18] and Replica [19] demonstrate our superiority on both image rendering quality and novel view depth rendering accuracy when given different lengths of input views.
2 Related Work
Novel View Synthesis. Traditional attempts in novel view synthesis mainly employed voxel grids
have drawn growing interest using ray-marching-banr based volume rendering to backpropagate image color error to the implicit geometry and radiance fields, such that the 3D geometry can be implicitly learned
20 21 or multiplane images (22). Recently, Neural Radiance Fields (NeRF) 3
5
23
24
25

---

to satisfy the multi-view color consisteney. Nonetheless, one inherent bottleneck of NeRFs-based method is the computation intensity of ray marching, which requires the costly volume sampling in the implicit fields for each pixel during rendering. To this end, recently 3DGS [10
have attracted increasing attention due to its high efficiency and photorealistic rendering. Instead of relying on MLPs to represent the goordincate-baser implicit fields, 3DGS learns an explicit field using set of 3D Gaussians. They optimize the 3D Gaussians parameters and perform adaptive densify control to fit to the given set of images, such that the 3D Gaussians are encouraged to perform densification only in the textured regions and refrain from over-densification. During rendering, 3DGS performs tile-based rasterization to differentiably accumulate color images from the explicit 3D Gaussian primitives, which is significantly faster than the ray- marching- baaseinn--bsddd volume rendering and achieves real-time rendering speed.
26 27
Generalizable Novel View Synthesis. Another drawback of the traditional NeRF-based and 3DGS- based methods is the requirement of per-scene optimization instead of direct feeding-forward. forward. To this end, there have been line of work 28 8 29 focusing on learning effective priors to predict 3D geometry from given images in a feed forward fashion, where the common practice is to project ray-marching sampled points onto given source views to aggregate multi-view features, conditioning the prediction of the implicit fields source views instead of point coordinates. Recently, there have also been attempts towards generalizable 3DGS [ 2 15 14]. pixelSplat M and EeS-Gaussian [17 propose to predict pixel-aligned 3D Gaussian parameters in feed forward fashion. MVSplat [2] replaces the epipolar line transformer of pixelSplat with a lightweight cost volume to perform more efficient image encoding. GGRt [15] concatenates pixelSplat predicted 3D Gaussians in a sequence of images and simultaneously perform pose optimization. latentSplat 14] encodes 31) Variational Gaussians and leverages discriminator to help produce more indistinguable images. Nonetheless, existing methods do not reconstruct the global 3D Gaussians from arbitrary length of inputs, and are limited view interpolation [ I7 1 or object/human-centric scenes (17 [4]. In contrary, in this paper we focus on reconstructing large scenes from arbitrary length of inputs without depth priors, unleashing the potential of generalizable 3DGS for large scene explicit representation.
Indoor Scene Reconstruction. One line of efforts in sced-forward indoor scene reconstruction focuses extracting 3D mesh using voxel volumes [30] 31 32] and TSDF-fusion [33). while do not perform photorealistic novel view synthesis. On the other hand, the SLAM- methods
34
35 36) require dense sequence of RGB-D input and per-scene tracking and mapping. Another
paradigm of 3D reconstruction [37 38, 39
learns implicit Signed Distance Fields from RGB input,
while demanding intensive per-scene optimization. Another recent work SurfeINeRF (40 learns feed-forward framework to map sequence of images to 3D surfels which support photorealistic image rendering, while they rely on external depth estimator or ground truth depth maps. In contrary, we propose an end-to-end model without ground truth depth map input or supervision, enabling accurate 3D Gaussian localization using only photometric losses.
3 Preliminary
Vanilla 3DGS. 3D-GS 10 explicitly represents a 3D scene with a set of Gaussian primitives which are parameterized via 3D covariance matrix E and mean jæ:
1 (p 1)) (D' 14),
G(p) exp(
2
R using scaling matrix S and a rotation matrix R to maintain positive semi-definiteness. During rendering, the 3D Gaussian is transformed into the image coordinates with world-to-camera transform matrix W and projected onto image plane with projection matrix J. and the 2D covariance matrix Σ' is computed as E' JWEW JJ" We then obtain a 2D Gaussian G2D with the covariance Σ' in 2D, and the color rendering is computed using point-based alpha-blending on each ray:
where Σ is decomposed into Σ = RSS T
JJ"
i-1
C(x) = EGa. (x) II II c,)o j=1
(2)
IEN
where is the number of Gaussian primitives, as is a learnable opacity, and is view-dependent color defined by spherical harmonics (SH) coefficients s. The Gaussian parameters are optimized by photometric loss to minimize the difference between renderings and image observations.
3

---

Global Cusssien Triplei
2
FF
D1-2
Progressive Goussions Fusion
Cost Volume
Multi--Scale Feature Aggregation
Efficient Feature Extraction
[F]
Wagined San
Gaussian Latent Decoder
Fix
Neareat N views
F?-
Cost Volume
Factedee
PTF
Algreatt
FF !)
Mabi-iclee
Loca Goundan Triplen
Weighesenn San
Nearest N views
Cost Volume
PTF
Algreutt
(FS)
Weightd
Rendering
Input Sparse Image Sequence
Umproject
Matching Flow
Global Flow
God Wiluine
Appond
Local Flow
Figure 2: Framework of FreeSplat. Given input sparse sequence of images, we construct cost volumes between nearby views and predict depth maps and corresponding feature maps, followed by unprojection to Gaussian triplets with 3D positions. We then propose Pixel-aligned Triplet Fusion (PTF) module, where we progressively aggregate and update local/global Gaussian triplets based on pixel-wise alignment. The global Gaussian triplets be later decoded into Gaussian parameters.
Generalizable 3DGS. Unlike vanilla 3DGS that optimizes per-scene Gaussian primitives, recent generalizable 3DGS 17 predict pixel-aligned Gaussian primitives {E, a, a) and depths d, such that the pixel-aligned Gaussian primitives can be unprojected to 3D coordinates μ.. The Gaussian parameters are predicted by 2D encoders, which are optimized by the photometric loss through rendering from novel views. However, existing methods are still limited to view interpolation within narrow view range, which leads inaccurately localized 3D Gaussians that fail support large scene reconstruction and view extrapolation (cf. Figure
I 4. To this end, we propose FreeSplat towards global 3D Gaussians reconstruction with accurate localization that supports free-view synthesis.
4
Our Methodology
4.1
Overview
The overview of our method illustrated in Figure 2 Given a sparse sequence of RGB images, we build cost volumes adaptively between nearby views, and predict depth maps to unproject the 2D feature maps into 3D Gaussian triplets. We then propose the Pixel-aligned Triplet Fusion (PTF) module to progressively align the global with the local Gaussian triplets, such that we can fuse the redundant 3D Gaussians in the latent feature space and aggregate cross cross-view Gaussian features before decoding. Our method is capable of efficiently exchanging cross-view features through cost volumes, and progressively aggregating pie-view 3D Gaussians with cross-view alignment and adaptive fusion.
4.2 Low-cost Cross-Vice Aggregation
Efficient 2D Feature Extraction. Given a sparse sequence of posed images [['))= we first feed them into a shared 2D backbone to extract multi-scale embeddings F. and matching feature
Ft. Unlike [ 2 which rely on patch-wise transformer- based backbones 41 42 that can lead
to quadratically expensive computations, we leverage pure CNN-based backbones feature extraction for efficient performance on higher resolution inputs.
43 for 2D
Adaptive Cost Volume Formulation. To explicitly integrate camera pose information given arbitrary length of input images, we propose adaptively build cost volumes between nearby views. For current view I' with pose p' and matching feature Ft. C.C. x 4 44, we adaptively select its N nearby views [I'm] |== with poses [Ptm]_)- based on pose proximity, and construct cost volume via plane sweep stereo [55 46]. Specifically, we define set K virtual depth planes (dl ) that uniformly spaced within (drear, dforr, and warp the nearby view features to each depth plane da

---

Pixel-wise Alignment
Gaussian Latent Fusion
Gaussian Triplet
Local Gasssian Triplet
Pg
1-1
dg
Geometrical Weighttdd Sum
GRU Fusion
e-th View
Figure 3: Visual illustration of PTE. The PTF incrementally projects current global Gaussians to input views and computes their pixel-wise distance with local Gaussians. Nearby local Gaussians then fused using a lightweight Gate Recurrent Unit (GRU) network 44
of current view:
Fl..k Trans F'
(3)
where Trans Pt) the transformation matrix from view by to t. The cost volume ev E RR x 4x 4 is then defined as:
Ft. (k)) fo (1 Σ cos Cos(FIt F))"))
1 ΣE
where FL, [k] is the k-th dimension of F cos(.) is the cosine similarity, @ is feature-wise concate- nation, and fo(-) is
CNN mapping to dimension of 1.
Multi-Scale Feature Aggregation. The embedding of the cost volume plays significant part to accurately localize the 3D Gaussians (ef. Table 5). To this end, inspired by previous depth estimation methods 47 33 we design an multi-scale encoder-decoder structure, such that to fuse multi-scale image features with the cost volume and propagate the cost volume information broader receptive fields. Specifically, the multi-scale encoder takes in Ft. and the output is concatenated with {F.) before sending into UNet++ -like decoder to upsample to full resolution and predict depth candidates map Dr E x x and Gaussian triplet map F/ E RR CCCHX We then predict the depth map through soft-argmax to bound the depth prediction between near and far:
D' softmax(D!)) d+.
(5)
Finally, the pixel-aligned Gaussian triplet map F/ unprojected 3D Gaussian triplet [ut, w/, ff where ne R3xHW are the Gaussian centers, w/ E IR weights between (0, 1), and fi E I(C-1)xHW are Gaussian triplet features.
4.3 Pixel-wise Triplet Fusion
One limitation previous generalizable 3DGS methods is the redundancy Gaussians. Since we need multi-view observations to predict accurately localized 3DD Gaussians in indoor scenes, the pixel-aligned Gaussians become redundant in frequently observed regions. Furthermore, previous methods integrate multi-view Gaussians of the same region simply through their opacities, leading to suboptimal performance due to lack of post aggregation (cf. Table 5). Consequently, inspired by previous methods [3] (40), we propose the Pixel-wise Triplet Fusion (PTF) module which can significantly remove redundant Gaussians in the overlapping regions and explicitly aggregate multi- view observation features in the latent space. We align the pir-view local Gaussians with global ones using Pixel-wise Alignment to select the redundant 3D Gaussian Triplets, and progressively fuse the local Gaussians into the global ones.
Pixel-wise Alignment. Given the Gaussian triplets (uf, f[] we start from t = where the global Gaussians latent is empty. In the t-th step, first project the global Gaussian triplet centers 44 E x onto the 1-th view:
P = {x vy d P P'
(6)
5

---

where (x y d']l R33M are the projected 2D coordinates and corresponding depths. We then correspond the local Gaussian triplets with the pixel-wise nearest projections within a threshold. Specifically, for the i-th local Gaussian with 2D coordinate (x) yf() and depth (i),, we first find its intra-pixel global projection set S;:
S = (i [+0))
where [-] is the rounding operator. Subsequently, search for valid correspondence with minimum depth difference under a threshold:
jES; arg min d () if || - min d/0) k di())
jES:
(8)
otherwise
where d is a ratio threshold. We define the valid correspondence set as:
F ((i. 1..... HW; m, 7 Ø).
(9)
Gaussian Triplet Fusion. After the pixel-wise alignment, we remove the redundant 3D Gaussians through merging the validly aligned triplet pairs. Given a pair (i. m2) E Ft, we compute the weighted sum of their center coordinates and sum their weights to restrict the 3D Gaussian centers to lie between the triplet pair:
(m,,) wi (1) + +
(m.))ug (m)) where (m.)) (i) + ( (m;).
(10)
(i) + (m,)
We then aggregate the aligned local and global Gaussian latent features through a lightweight GRU network:
(m.) = GRU(F) (f
(m:)),
(11)
and then append with the other unaligned local Gaussian triplets.
Gaussian primitives decoding. After the Pixel-wise Triplet Fusion, we can decode the global Gaussian triplets into Gaussian primitives:
E, E,a.s=
(12)
and Gaussian centers μ4 u. Our proposed fusion method can incrementally integrate the Gaussians with geometrical constraints and learnable GRU network for feature update. Consequently, our fusion method capable of significantly removing redundant Gaussians and perform post feature aggregation across multiple views, and can be trained with the other framework components end-to-end with eligible computation overhead.
4.4 Training
Loss Functions. After predicting the 3D Gaussian primitives, we render from novel views following
the rendering equations in Eq. (2) Similar to pixelSplat [1] and MVSplat 22, train our framework
using only photometric losses, Lee. a combination MSE loss and LPIPS 50 loss, with weights of 1
and 0.05 following 212]
Free-Viee Training. We propose a Free- View Training (FVT) strategy to add more geometrical constraints the localization of 3D Gaussians, and to disentangle the performance of generalizable 3DGS with specific number of input views. To this end, we randomly sample number of context views (in experiments view interpolations. The long sequence training is made feasible due to our efficient feature extraction and aggregation. We empirically find that FVT significantly contributes to depth estimation from novel views (cf. Table
set 7 between and 8), and supervise the image renderings in the broader
Experiments
5.1 Experimental Settings
Datasets. We leverage the real-world indoor dataset ScanNet [8 for training. ScanNet large RGB-D dataset containing 1, 513 indoor scenes with camera poses, and we follow [5] 40] to use

---

Table I: Generalizable Novel View Interpolation results on ScanNet [18]. FreeSplat-fr is trained with our FVT strategy, and the other methods all trained specific numher of views form complete comparison. Time(s) indicates the total time encoding input images and rendering one image.
2 views
3 views
Method
PSNR
SSIM
LPIPS| Time(s)) #GS(k)
PSNRT
SSIMT
LPIPS! Time(s)] #GS(k)
NeuRay 9
25.65
0.840
0.264
3.103
25.47
0.843
0.264
4.278
PixelSplat [
26.03
0.784
0.265
0.289
1180
25.76
0.782
0.270
0.272
1769
MVSplat 2
27.27
0.822
0.221
0.117
393
26.68
0.814
0.235
0.192
590
FreeSplat-spec
28.08
0.837
0.211
0.103
278
27.45
0.829
0.222
0.121
382
FreeSplat-fv 27.67
0.830
0.215
0.104
279
27.34
0.826
0.226
0.122
390
Table 2: Long Sequence (10 views) Explicit Reconstruction results on ScanNet. The results of pixelSplat, MVSplat and FreeSplat-spec given using their 3-views version.
View Interpolation
View Extrapolation
Method
Time(s)
#GS(k)
PSNR
SSIM
LPIPS..
PSNR1
SSIM
LPIPS].
pixelSplat 1
0.948
5898
21.26
0.714
0.396
20.70
0.687
0.429
MVSplat
1.178
1966
22.78
0.754
0.335
21.60
0.729
0.365
PreeSplat-3views 0.599
882
25.15
0.800
0.278
23.78
0.774
0.309
FreeSplat-fv 0.596
899
25.90
0,808
0.252
24.64
0.786
0.277
100 scenes for training and 8 scenes for testing. To evaluate the generalization ability of our model, further perform zero-shot evaluation the synthetic indoor dataset Replica [19] for which
follow 122 to select 8 scenes for testing.
Implementation Details. Our FreeSplat is trained end-to-end using Adam [53] optimizer with initial learning rate of Le - 4 and cosine decay following [2]. Due the large GPU requirements of 2 given high-resolution images, all input images resized to 384 x 512 and batch size is set to 1, to form a fair comparison between different methods. We mainly compare with previous generalizable 3DGS methods in 2, 3, 10 reference view settings, where the distance between input views is fixed, thus evaluating the models' performance under different view ranges. For 10 views setting, also choose target views that beyond the given sequence of reference views to evaluate the view extrapolation results.
5.2 Results on ScanNet
View Interpolation Results. On ScanNet, we evaluate the generalizable novel view interpolation results given 2 and reference views as shown in Table Comparing to pixelSplat and MVSplat, our FreeSplat-spec consistently improves rendering quality and efficiency on 2-views setting and 3-views setting. Although slightly underperforming on SSIM comparing to NeuRay [, we show significant improvements PSNR and LPIPS over NeuRay and 300x faster inference speed. Moreover, our FreeSplat-fv consistently offers competitive results given arbitrary number of views, and performs more similarly as FreeSplat-spe when number of input views increases.
Table 3: Novel View Depth Rendering results on ScanNet. 1 10-views results of pixelSplat, MVSplat and FreeSplat spec are given using their 3-views version.
views Abs Diff] Reljō < 1.25 Abs Diff Abs Relfō < 1.25 1.bs Abs Relj8 < 1.25 1
views
views
Method
NeuRay
9
0.358
0.200
0.755
0.231
0.117
0.873
0.202
0.108
0.875
pixelSplat
1.205
0.745
0.472
0.698
0.479
0.836
0,970
0.621
0.647
MVSplat
2
0.192
0.106
0,912
0.164
0.079
0.929
0.142
0.080
0.914
FreeSplat-spec FreeSplat-fv
0.157
0.086
0,919
0.161
0.077
0.930
0.120
0.070
0.945
0.153
0.085
0,923
0.162
0.077
0.928
0.097
0.059
0.961

---

Ref.
Target View
Ours fv
Ours spec
MVSplat
paceSpplt
Figure 4: Qualitative Results of Long Sequence Explicit Reconstruction. For each sequence, the first two rows are view interpolation results, and the last two rows are view extrapolation results.
Long Sequence Results. As shown in Table 2 we further evaluate the long sequence results where sample reference views with length of 10, and compare both view interpolation and extrapolation results. The results reveal that generalizable 3DGS methods underperform when given long sequence input images, which is due the complicated camera trajectories ScanNet. and the inaccuracy of 3D Gaussian localization that leads to errors when observed from wide view ranges, Our FreeSplat-3view 3views significantly outperforms pixelSplat and MVSplat on view interpolation and view extrapolation results. Through our proposed FVT that can be easily plugged into our model due our low requirement on GPU, our FreeSplat-fv consistently outperforms our 3-views version. Our PTF module can also reduce the number of Gaussians by around 55. 55.0%, which becomes indispensable long sequence reconstruction due to the pixel-wise unprojection nature generalizable 3DGS. The qualitative results are shown in Figure 4 which clearly reveal that FreeSplat-spec outperforms MVSpla and pixelSplat in localizing 3D Gaussian and preserving fine-grained details, and FreeSplat- fy further improves localizing and fusing multi-view Gaussians.

---

Table 4: Zero-Shot Transfer Results Replica 19
Views LPIPS|
Views LPIPS|
Method
PSNRT
SSIMT
< 1.25 f #GS(k)
PSNR*
SSIM
< 1.25 f #GS(k)
pixelSplat M
26.24
0.829
0.229
0.576
1769
19.23
0.719
0.414
0.375
5898
MVSplat 2
26.16
0.840
0.173
0.670
590
18.66
0.717
0,360
0.565
1966
FreeSplat-spec FreeSplat-fv
26.98
0.848
0.171
0.682
423
21.11
0.762
0.312
0.720
1342
26.64
0.843
0,184
0.682
421
21.95
0.777
0.290
0.742
1346
Table 5:
Ablation ScanNet. CV: Cost Volume, PTF: Pixel-wise Triplet Fusion, FVT: Free-View Training.
3 views
10 views
CV PTF FVT
PSNRÍ
SSIM
LPIPS 8 < 1.25 f d < 1.10 1 PSNR SSIM
LPIPS| 6 < 1.25 5 < 1.10
27.12
0.825
0.224
0.925
0.762
24.23
0.792
0.277
0.942
0.804
22.10
0.696
0.359
0.639
0.311
17.94
0.607
0.487
0.543
0.216
27.45
0.829
0.222
0.930
0.773
25.15
0.800
0.278
0.945
0.823
26.41
0.806
0.232
0.919
0.746
25.40
0.799
0.252
0.950
0.831
V
27.34
0.826
0.226
0.928
0.764
25.90
0.808
0,252
0.961
0.858
Views
Views
Target View
Full model
w/o FVT
w/o PTF
w/o CV
Figure Qualtitative Ablation Study. The first and second input view lengths of 3 and 10.
Novel View Depth Estimation Results. We also investigate the correctness of 3D Gaussian lo- calization of different methods through comparing their depth rendering results. We report the Absolute Difference (Abs. Diff), Relative Difference (Rel. Diff), and threshold tolerance < 1.25 results from novel views in Table 3 We find that FreeSplat consistently outperforms pixelSplat and MVSplat in predicting accurately localized 3D Gaussians, where FreeSplat-fv reaches 94.9% of 8 1.25, enabling accurate unsupervised depth estimation novel views. The improved depth estimation accuracy of FreeSplat-fv highlights the importance of depth estimation in supporting free-view synthesis across broader view range.
5.3 Zero-Shot Transfer Results Replica
We further evaluate the zero-shot transfer results through testing Replica dataset, with results in Table 4 Our view interpolation and novel view depth estimation results still outperforms existing methods. The long sequence results degrade due to inaccurate depth estimation and domain gap, indicating potential future work in further improving the depth estimation in zero-shot tranferring.
5.4 Ablation Study
We conduct a detailed ablation study as shown Table 5 Figure The results indicate that: 1) cost volume is essential in accurately localizing 3D Gaussians; 2) our proposed PTF module can consistently contribute rendering quality and depth estimation results. The PTF module learns to incrementally fuse multi-view 3D Gaussians and contributes significantly when varying number of input views, and serves as a multi-view localization regularization that helps unsupervised depth estimation; 3) Our FVT module excels in long sequence reconstruction quality well novel view

---

depth rendering results, which provides stricter constrains on 3D Gaussian localization and can be seamlessly combined with the PTF module to fit to varying length of input views.
Conclusion
In this study, we introduced FreeSplat, a generalizable 3DGS model that is tailored to accommodate an arbitrary number of input views and perform free-view synthesis using the global 3D Gaussians. We developed a Low-cost Cross-Viee Aggregation pipeline that enhances the model's ability to efficiently process long input sequences, thus incorporating stricter geometry constraints. Additionally, have devised a Pixel-wise Triplet Fusion module that effectively reduces redundant pixel-aligned 3D Gaussians in overlapping regions and merges multi-view Gaussian latent features. FreeSplat consistently improves the fidelity of novel view renderings in terms of both color image quality and depth map accuracy, facilitating fcrd-forward global Gaussians reconstruction without depth priors.
7
Acknowledgement
This work supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L760021).
References
[1] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312.12337, 2023.
[2] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403. /4627, 2024.
[3] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[4] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106 1nn::2206..0880 10689, 2021.
[5] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1- 15, 2022.
[6] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[7] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Martin-Brualla, Brualla, Noah Snavely, and Thomas Funkhouser. Ibmet: Learning
Barron, Ricardo Martin multi-view image-based rendering. In CVPR, 2021.
[8] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-
14133, 2021.
[9] Yuan Liu, Sida Peng. Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7824-7833, 2022.
[10] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42((4: 2023.
10

---

[11] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. arXiv preprint arXiv:2311.16493, 2023.
[12] Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octree- gs: Towards consistent real-time rendering with lod-structured 3d gaussians. arXiv preprint arXiv:2403.17896 2024.
[13] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308. 0971 13, 2023.
[14] Christopher Wewer, Kevin Raj, Eddy 1lg. Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint vrXiv:2403.16291 2024.
[15] Hao Li, Yuanyuan Gao, Dingwen Zhang, Chenming Wu, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Junwei Han. Ggrt: Towards generalizable 3d gaussians without pose priors in real-time. arXiv preprint arXiv:2403. 10147, 2024.
[16] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:23 22.00//33//.9944.. 2023.
[17] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wine wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02 155, 2023.
[18] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[19] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: digital replica of indoor spaces. arXiv preprint arXiv: (et.. 05797, 2019.
[20] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: 1906.0:906.07770 07751, 2019.
[21] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nicßner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels:
Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446, 2019.
[22] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint 2018.
[23] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bautin-Bruaala, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[24] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/VV Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[25] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[26] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308, 09713, 2023.
[27] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08558, 2023.
[28] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587 2021.

---

[29] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 5480-5490, 2022.
[30] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Esion-ECCV/ 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[31]
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real- time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15598-15600 2021.
[32] Noah Stier, Alexander Rich, Pradeep Sen, and Tobias Höllerer. Vortx: Volumetric 3d recon- struction with transformers for voxelwise view selection and fusion. In 2021 International Conference on 3D Vision (3DV), pages 320-330. IEEE, 2021.
[33] Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. Simplerecon: 3d reconstruction without 3d convolutions. In European Conference on Computer Vision. pages 1-19. Springer, 2022.
[34] Zihan Zhu, Songyou Peng. Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12796, 2022.
[35] Chi Yan, Delin Qu, Dong Wang, Dan Xu, Zhigang Wang, Bin Zhao, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. arXiv preprint arXiv:231 17700, 2023.
[36] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgh-d slam, arXiv preprint arXiv:2312.022 126, 2023.
[37] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-48155 2021.
[38] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems. 35:25018- 10000088222222..1 2022.
[39] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8456-8465. 2023.
[40] Yiming Gao, Yan-Pei Cao, and Ying Shan. Surfelnerf: Neural surfel radiance fields for online photorealistic reconstruction of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 108-118, 2023.
[41] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint arXiv:2010. 1929, 1929, 2010.
[42] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[43] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105 61 14. PMLR, 2019.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[45] Robert T Collins. A space-sweep approach to true mulli-imaae matching, In Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition, pages 358-363. Icee, 1996.
[46] Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and So Kweon. Dpsnet: End-to-end deep plane sweep stereo. arXiv preprint arXiv: 2019.
12

---

[47] Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale. Mihai Dusmanu, and Mare Pollefeys. Deepvideomvs: Multi-view stereo on video with recurrent Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15324-15333, 2021.
fusion. In
[48] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, pages 3-11. Springer, 2018.
[49] Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pages 1597-1600. IEEE, 2017.
[50] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[51] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing
radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449-5458, 2022.
[52] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
13

---

A Appendix supplemental material
A.1 Experimental Environment
We conduct all the experiments on single NVIDIA RTX A6000 GPU. The experimental environment
is PyTorch 2.1.2 and CUDA 12.2.
A.2 Additional Implementation Details
We the number of virtual depth planes K 128, matching feature dimension C. 64, and dnear = 0.5, far = 15.0 for cost volume formulation, and set = 0.05 in Eq.(8) for pixel-wise alignment. To train the 3-views version of pixelSplat [T] on a single NVIDIA RTX A6000 GPU, we change their ViT patch size from 8 x 8 to 16 x 16. During inference on the 10 views setting, the epipolar line sampling in pixelSplat and the cross-view attention in MVSplat [2] are performed between nearby views similarly as ours to save GPU requirements and form a fair comparison. For the free-view version of FreeSplat we set the number of nearby views as N for training. For the testing of long sequence explicit reconstruction, cost volumes are formed between nearby 8 views.
A.3 Additional Experiments
Table Comparison on computational cost and whole scene reconstruction (30 input views). We report the required GPU for Train Test, the Encoding Time, the rendering FPS, and PSNR of novel views. denotes that we are not able to run pixelSplat inference using 30 input views due to its increasing GPU requirement.
Method
GPU (GB) Time (s) FPSt PSNR
pixelSplat-3views
44.9
MVSSlat-3views
34.8 /4.0
3.004
39
17.57
PreeSplat-3views
16.9/21.0
1.191
57
21.33
FreeSplat-fr w/o PTF
/212
1.006
39
21.82
FreeSplat-fi
42.2/21.0
1.205
22.32
Computational Cost. As shown in Table 6 we compare the required GPU memory for training and testing, the encoding time, rendering FPS, and PSNR for whole scene reconstruction. pixelSplat- 3views and MVSplat-3views already consume 30 50 GB GPU memory for training due to their quadratically increasing GPU memory requirement with respect the image resolution / sequence length. Therefore, it becomes infeasible to extend their methods to higher resolution inputs or longer sequence training. In comparison, our low-cost framework design enable to effectively train on long sequence inputs while requiring lesser GPU memory compared to the 3 views version of existing methods. Furthermore, our proposed PTF module can effectively reduce redundant 3D Gaussians, improving rendering speed from 39 to 72 FPS. This becomes increasingly important when reconstructing larger scenes since generalizable 3DGS normally perform pixel-win wise unprojection, which can easily result in redundancy in the overlapping regions.
Table 7: Results on REIOK and ACID with input views. We train our model on RE10K, and report its results on RE10K and ACID. denotes that our model is trained on our previously downloaded 9,266 scenes instead of 11,075 scenes used by baselines.
REIOK 2 Views
ACID -2 Views
Method
PSNR1
SSIM1 LPIPS
PSNR
SSIM
LPIPS
pixelSplat
25.89
0.858
0.142
27.64
0.830
0.160
MVSplat 26.39
0.869
0.128
28.15
0.841
0.147
Ours' 26.41
0.871
0.132
27.94
0.838
0.157
Experiments on RE10K and ACID. To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using View setting and 5-View setting respectively. The results are shown in Table 7 g and Figure 6 Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2- View setting, we perform better than pixelSplat and on par MVSplat on both datasets. In the 5- View setting, we outperform both baselines by clear margin. We analyze the main causes of the above results as follows:
14

---

Table 8: Results RE10K and ACID with input views. The models trained REIOK with imput
views.
REIOK
Views
ACID -DD-5 Views
Method
PSNR1
SSIM1
LPIPS
PSNR
SSIM
LPIPS
pixelSplat
24.78
0.850
0.150
26.84
0.833
0.173
SVSplat
25.38
0.866
0.132
27.81
0.863
0.134
Ours
25.95
0.873
0.128
28.35
0.870
0.130
Inputs
Target View
Ours
MVSplat
pixelSplat
Figure 6: Qualitative Results on RE10K and ACID. We visualize the 2-Views results on REI0K and 5- V-eesss results on ACID.
In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set be large. On average, the interval between image stereo is 66 in REIOK and in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result minimum view overlap between the image stereo, which means that our cost volume can be much sparser and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses heavy 2D backbone that can potentially become stronger monocular depth estimator. our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians.
Therefore, our model not specifically designed for highly sparse view inputs, but it designed as low-cost model that can easily take in much longer sequences of higher- inputs, that is suitable for indoor scene reconstruction. Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend such setting as evaluated in main paper.
Comparison with SurfelNeRF. We further compare with SurfeINeRF as shown in Table 9 and Figure 7 We evaluate on the same novel views as theirs, sampling input views along their input sequences with an interval 20 between nearby views. Note that the number of input views changes when the input length changes, while our FreeSplat-fv can seamlessly conduct inference with arbitrary numbers of inputs. Our method performs significantly better than SurfelNeRF in both rendering quality and efficiency. Our end-to-end framework jointly learns depths and 3DGS using an MVS-based backbone. while SurfelNeRF relies on depths and does not aggregate multi-view features to assist their surfel feature prediction.
15

---

Table Comparison with SurfeINeRE. We compare with SurfeINeRF the same sequences their test set. denotes the rendering speed reported from SurfeINeRF.
Method
Time (s)
FPS1
PSNRT
SSIMI
LPIPS!
SurfeINeRF
3.242
5"
24.20
0.694
0.477
FreeSplat-fv
0.302
224
27.06
0.818
0.223
Misy
Moey FreeSplat-fv
Target View
SurfeINeRF
Figure 7: Qualitative Comparison with SurfeINeRF.
A.4 Additional Qualitative Results
and 3-View Interpolation Results. The qualitative results are shown in Figure g where FreeSplat more precisely localizes 3D Gaussians and captures more fine-grained details comparing to previous methods. FreeSplat can also localize 3D Gaussians more accurately and renders precise depth maps, supporting high-quality rendering from broader view range (ef. FreeSplat- results in Figure 4
Results on Replica. We show the qualitative results on Replica in Figure IO. where our superiority over MVSplat and pixelSplat remains. The results indicate the generalization ability of FreeSplat across indoor datasets for the view interpolation task.
Results of Whole Scene Reconstruction. We also show qualitative results of our whole scene reconstruction in Figure Despite the long input sequence (~~ 40 images) covering the whole scene, FreeSplat can still perform efficient feed-forward ~ 1s on single NVIDIA RTX A6000, and can render high-quality images and accurate depth maps from novel views. On the other hand, it is still difficult to accurately predict depth of textureless (e.g. wall) and specular (e.g. light reflection on the floor) regions. However, we hope our work provides an initial step towards accurate geometry reconstruction without ground truth depth priors.
A.5 Limitations
Although approach excels in novel view rendering depth estimation and support arbitrary number of input views, the GPU requirement becomes expensive (> 40GB) when inputting extremely long image sequence (> 50). On the other hand, due to unsupervised scheme of depth estimation, there is still a gap between our 3D reconstruction accuracy and the state-of-the-arr methods with 3D supervision 33 32 or RGB-D inputs (36] 35) (e.g. shown in Figure the textureless and specular regions). Our main focus is to explore the feed-forward indoor scene photorealistic reconstruction purely based on 2D supervision.
16

---

Ref.
Target View
Ours-spec
MVSplat
pixelSplat
21 Sineon
3 Views
Figure 8: Qualtitative Results given 2 and 3 reference views. We show the rendered color images (first row) and depth maps (second row) for each batch of reference views.
17

---

Target View
Ours-spec
MVSplat
pike Splat
Figure 9: Qualitative Results Replica.
3D Gausslans
Renderings
Target
Figure 10: Qualitative Results of whole scene reconstruction.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We mainly focus on generalizable 3DGS for indoor scene reconstruction, where we support arbitrary number of inputs through adaptive cost volume (Sec. and gaussian fusion (Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed Sec. A.5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
We do not propose new theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include detailed description of our framework in Sec. 4 and implementation
details Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Our code will be released upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /otee///uudddeeeeSSuuubbmiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The detailed deseriptions of the experiment settings are illustrated in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow our related works in the setting for error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the experimental environment in Sec.
A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///www.pppp.cccppc////aaaaa
Answer: [Yes]
Justification: To the best of our knowledge, our work is conducted with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To the best of our knowledge, we do not foresee societal impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the used existing datasets and models.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed descriptions about method (Sec. well as its limitations (Sec. A.5 Our code will be released upon paper acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "2GCPNB79su", "reviewer": "Reviewer_gTxT", "summary": "This paper proposes FreeSplat to reconstruct geometrically consistent 3D scenes from long sequence inputs. To this end, the paper presents Low-cost Cross-View Aggregation for feature matching and Pixel-wise Triplet Fusion for Gaussian triplets fusion. The outstanding results of long sequence 3DGS generalization shows the superiority beyond baseline methods.", "questions": "1. What are the differences between Freeplat-spec and Freesplat-fv?\n2. The author provided 3-view version results in the long sequence reconstruction experiment in Table 2. I’m doubting why do you provide the 10-view version of the baselines? Just because their original paper did not provide the 10-view version, or because they will fail to reconstruct a scene under 10-view inputs, or other reasons? Further clarification will help to understand this experiment.\n3. The author claims that the proposed pixel-wise triplet fusion module helps to remove redundant Gaussians in the overlapping regions and the experimental results of final Gaussian number demonstrates this. However, it is somewhat not intuitive. It would be better to provide a visualization to show how the Gaussian ellipsoids are reduced in the overlapping regions.", "limitations": "The author discussed the limitations in the appendix about the memory consumption and the limited performance of generalizable Gaussians, which I think is valuable. It remains a significant problem for further study.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and the technical contributions are clearly clarified and easy to understand.\n2. The experimental results over baselines are attractive and convincing, which demonstrates the effectiveness of the proposed method.\n3. The proposed pixel-wise triplet fusion is reasonable and novel.", "weaknesses": "1. The low-cost cross-view aggregation module is not novel enough. It seems that the three submodules (2D feature extraction, cost volume formulation, multi-scale feature aggregation) are all borrowed from existing methods and the low-cost cross-view aggregation is a simple stack of these submodules. I suggest the author to further clarify the differences between the proposed module and the existing MVS methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer gTxT (#R1):\n\n1. **Differences from existing MVS methods:** Compared to traditional MVS methods [1,2], the main difference of our backbone lies in the unsupervised scheme of depth estimation supervised purely by color images, while reaching comparable depth estimation accuracy. Due to the unsupervised nature of our method, the pixel-aligned depth values are predicted through weighted summing along the candidate depth planes instead of predicting absolute depth values, in order to bound the depth values within a reasonable range. Compared to existing MVS-based NeRF methods [3, 4], our backbone design avoids expensive 3D CNN / ray transformer, and our explicit 3DGS reconstruction can greatly benefit from real-time renderings. Compared to MVS-based 3DGS methods [5, 6], our main contributions lie on designing a transformer-free 2D backbone to encode extended input sequences of high-resolution images, and adaptively formulate cost volumes within nearby views based on pose affinities. More importantly, we are the first to explore explicit long sequence reconstruction based on generalizable 3DGS. Both pixelSplat and MVSplat struggle to extend to long sequences of high-resolution inputs due to their patch-wise 2D Transformers / cross-view Transformers, which consume quadratically increasing GPU memory with image sequence length and resolution. On the other hand, our low-cost 2D backbone is designed to easily encode such indoor scene sequences. Therefore, although our 2D backbone bears high-level design similarity to existing MVS methods, the differences in our design are specifically targeting long sequence reconstruction with precise unsupervised depth estimation which we demonstrated to be significant for our goal.\n2. **Differences between FreeSplat-spec and FreeSplat-fv:** FreeSplat-spec is trained with fixed number of input views (eg. 2, 3), and FreeSplat-fv is trained using our free-view training strategy using *2~8* input views, in order to extend our method to arbitrary length of inputs. We will include the above explanation in the final version.\n3. **Why providing 3-view version of baselines in Table 2:** The main reason is the enormous GPU consumption of pixelSplat and MVSplat when inputting more views. As shown in our ***rebuttal pdf Table 1***, pixelSplat and MVSplat already require **30~50 GB** GPU when training with 3 input views and batch size of 1 due to their heavy patch-based 2D Transformers and cross-view attention. Their GPU requirements quadratically increase with respect to the inputs resolution and length, thus making training their 10-views version nearly infeasible. In contrast, our model can train on 8 input views while requiring a smaller GPU memory compared to 3 views version of pixelsplat and MVSplat, supporting training and inference on long sequences of high-resolution inputs. Therefore, in our main paper Table 2, we report pixelSplat and MVSplat's 10-views inference results using their 3-views trained models due to the infeasibility of training their models using more views. To form a fair comparison, we also compare our 3-view version on 10-view inference, where we consistently outperform the 3-views baselines. On the other hand, the further improvements brought by our FreeSplat-fv over our 3-view version are creditable to our low-cost backbone design.\n4. **Visualization of PTF for removing redundant 3D Gaussians:** To illustrate our proposed PTF more intuitively, we draw an illustration figure in our ***rebuttal pdf Figure 1***, and visualize the fusion process as shown in our ***rebuttal pdf Figure 2***. Our proposed PTF module can greatly remove the redundant 3D Gaussians that lie very closely to the existing ones, such that we avoid redundant unprojection of 3D Guassians to the regions that are observed multiple times. PTF can also fuse the redundant 3D Gaussians latent features to aggregate multi-view observations at point-level, alleviating artifacts caused by lightning conditions, etc. Furthermore, as shown in our ***rebuttal pdf Table 1***, applying PTF module can increase the rendering speed from **39 to 72 FPS** during whole scene reconstruction, which becomes more essential when the 3D scene is larger with more input views. Since generalizable 3D Gaussians methods normally unproject pixel-wise 3D Gaussians for each input view that can easily result in redundancy, removing the redundant ones become more important to achieve real-time rendering.\n\n[1] Yao, Yao, et al. \"Mvsnet: Depth inference for unstructured multi-view stereo.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[2] Sayed, Mohamed, et al. \"Simplerecon: 3d reconstruction without 3d convolutions.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Chen, Anpei, et al. \"Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[4] Wang, Qianqian, et al. \"Ibrnet: Learning multi-view image-based rendering.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[5] Charatan, David, et al. \"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[6] Chen, Yuedong, et al. \"Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.\" arXiv preprint arXiv:2403.14627 (2024).\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aIMIEnkJOk", "reviewer": "Reviewer_emYN", "summary": "1. Low-cost Cross-View Aggregation: This efficient methodology constructs adaptive cost volumes between proximate views and aggregates features utilizing a multi-scale structure. This approach enables the processing of extended input sequences and the incorporation of more stringent geometric constraints.\n2. Pixel-wise Triplet Fusion (PTF): This module aligns and merges local Gaussian triplets into global representations, thereby mitigating redundancy in overlapping regions and consolidating features observed across multiple viewpoints.\n3. Free-View Training (FVT): This novel training strategy decouples the model's performance from a predetermined number of input views, thus enabling robust view synthesis across an expanded range of viewpoints.\n4. The authors demonstrate that FreeSplat surpasses existing methodologies in both novel view synthesis quality and depth map accuracy on the ScanNet and Replica datasets. The proposed method exhibits enhanced performance in view interpolation and extrapolation tasks, particularly for extended input sequences. Moreover, FreeSplat reduces the quantity of redundant Gaussians and offers more computationally efficient inference compared to preceding approaches.", "questions": "Please see the weaknesses.", "limitations": "The authors discuss some limitations of their work:\n1. High GPU memory requirements (>40GB) for extremely long input sequences (>50 images).\n\n2. A gap in 3D reconstruction accuracy compared to state-of-the-art methods that use 3D supervision or RGB-D inputs.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured, offering lucid explanations of the technical approach. It includes detailed architectural diagrams and pseudocode to enhance reproducibility. Comprehensive ablation studies are presented to isolate the impact of different components. The overall writing is coherent and accessible.\n2. This research enables free-viewpoint rendering and 3D reconstruction from arbitrary numbers of input views, broadening the applicability of 3D Gaussian splatting techniques. It enhances efficiency and minimizes redundancy in 3D Gaussian representations for extended sequences.", "weaknesses": "1. Experimental Scope and Comparability: PixelSplat presents experimental results on the Re10k and ACID datasets. For a comprehensive and equitable comparison, it would be beneficial for the proposed method to also report results on these two datasets. This would facilitate a direct comparison with pixelSplat across multiple benchmarks, thereby strengthening the validity of the authors' claims. The absence of experimental results on Re10k and ACID raises questions about the proposed method's performance and generalizability across diverse datasets. I am wondering why the experimental results on Re10k  and ACID are absent.\n2. Analysis of Failure Cases and Limitations: The paper would be significantly enhanced by a more thorough examination of failure cases and limitations of the proposed approach. While the appendix briefly touches on this aspect, a more in-depth analysis would provide valuable insights into the method's robustness and potential areas for improvement. Such an analysis could include specific examples of scenarios where the method underperforms, a discussion of the underlying causes for these failures, and potential strategies for addressing these limitations in future work. This level of critical self-evaluation would not only increase the paper's scientific rigor but also provide a more balanced perspective on the method's capabilities and constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer emYN (#R2):\n\n1. **Experiments on Re10k and ACID:** To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in our ***rebuttal pdf Table 2, 3 and Figure 3***. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat [1] and on par as MVSplat [2] on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:\n\n   In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in **minimum view overlap** between the image stereo (e.g. as shown in our ***rebuttal pdf Figure 4(b)***), which means that our cost volume can be **much sparser** and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians. \n\n   Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction (we also offer a quantitative comparison on computation cost in our ***rebuttal Table 1*** to emphasize our strengths). Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.\n\n   We really appreciate your question, which also helped us delve into the comparison on a larger experimental scope and analyze the reasons behind the results. We hope that our analysis can help demonstrate our contributions more clearly. We will add the corresponding results and the above illustrations in our final version paper to improve the completeness of our paper and highlight the contributions of our paper comparing to existing works.\n\n2. **Failure cases and potential future works:** As shown in our ***rebuttal pdf Figure 3***, we visualize the whole scene reconstruction results highlighting (a) the errorenously estimated depth for the specular/texture-less regions, and (b) Difficulty of accurate depth estimation when the given input stereo has extremely large interval. Such errors are mainly due to the following aspects: \n\n   **(1) Lack of depth regularizations.** The nature of our color-supervised depth estimation methods makes it difficult to accurately estimate depth for such regions. One potential solution is to leverage depth supervision / priors, e.g. regularize the depth estimation results using GT depth / sparse depth from COLMAP / monocular depth estimation methods. Future works can also explore the addition of geometric constraints on Gaussians localisation (eg. depth smoothness regularization), or multi-view consistency regularizations to enhance the 3D Gaussian localization.  \n\n   **(2) Straightforward 3D Gaussian fusion method.** Although we have proposed the PTF module to reduce redundant 3D Gaussians and improve the depth estimation performance, it is still not enough to reach satisfactory multi-view fusion results. Future works can learn from TSDF fusion methods [3] and enhance the unprojection of 3D Gaussians, e.g. given the initial 3D Gaussians from the first frame, search for its projections on all the remaining frames, and unproject them together.\n\n   **(3) Ineffectivity of MVS encoder when faced minimum view overlap between inputs:** When the given inputs only have minimum view overlap, our MVS can hardly find correspondences on the reference view, which results in highly sparse cost volume and ineffectivity of multi-view feature aggregation. It would be beneficial to leverage a cross-view attention (e.g. as in MVSplat) which can work despite minimum view overlap. Although our transformer-free backbone may underperform in such extreme cases, it was specifically designed for relatively dense input of long sequences. In the main paper, we have evaluated that our low-cost backbone is essential for accurate indoor scene reconstruction.\n\n   We will add the above illustrations in our final version to delve deeper into the analysis of failure cases and discuss potential future works that can be explored upon our method.\n\n[1] Charatan, David, et al. \"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.\" CVPR. 2024.\n\n[2] Chen, Yuedong, et al. \"Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.\" arXiv preprint arXiv:2403.14627 (2024).\"\n\n[3] Choe, Jaesung, et al. \"Volumefusion: Deep depth fusion for 3d scene reconstruction.\" *CVPR*. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XsDiI1nNq5", "reviewer": "Reviewer_Hd37", "summary": "This paper proposes a FreeSplat, aiming at generalizable 3D gaussian splitting for long sequence inputs. Specifically, it uses an efficient CNN-based cost volume and eliminates redundant 3D gaussians observed across multiple views. Extensive experiments show that FreeSplat effectively reduces inference costs and improves the novel view quality for long sequence inputs.", "questions": "The proposed method is interesting, but the lack of evaluation details, lack of comparative experiments, and inconsistencies in the results make it difficult to verify the contribution. The authors should address the \"weaknesses\" and strengthen the manuscript.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is good, making it easy to understand and follow.\n\n* The proposed Pixel-Aligned Triplet Fusion (PTF) is interesting as it effectively handles the redundant 3D gaussians for long sequence inputs.", "weaknesses": "* I think the evaluation details are not clear. For example, during the evaluation, I am curious whether the maximum gap between input views is fixed regardless of the number of input views. Does \"view range\" increase as the number of input views increases? If so, please describe the evaluation details.\n\n* It is weird that the novel view image quality drops as the number of input views increases, but the novel view depth quality increases. I'd like more clarification on this.\n\n* The existing pixelSplat and MVSplat have been experimented with on RealEstate10k [1] and ACID [2], and both datasets also contain long sequence inputs. The authors should have included comparisons on at least one of the RealEstate10k and ACID datasets.\n\n[1] Zhou et al., Stereo magnification: Learning view synthesis using multiplane images, SIGGRAPH 2018.\n\n[2] Liu et al., Infinite nature: Perpetual view generation of natural scenes from a single image, ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer Hd37 (#R3):\n\n1. **Evaluation Details:** For view range, as mentioned in Line \\#431-432 in the appendix, the distance between nearby input views is fixed to 20 in ScanNet and 10 in Replica, thus the maximum gap increases linearly with the number of input views. Therefore, the long sequence reconstruction covers larger regions of the scene and supports whole scene reconstruction. During evaluation, we randomly select a sequence of images with specific length (2, 3, 10) and fixed interval between nearby views, e.g. we choose the 30, 50, 70-th images of a ScanNet scene as input views for the 3-view setting. When evaluating interpolation results, we randomly choose novel views between each interval of nearby views, and for extrapolation we choose novel views that are beyond the input image sequence as illustrated in Line \\#226-227 in our main paper. We will include the above illustrations in the final version to describe the evaluation details more clearly.\n\n2. **Decreased image quality when the number of input views increases:** It is because of the increased difficulty when explicitly reconstructing longer sequences. Similarly as existing works, when evaluating interpolation results we select novel views that are within the view interval between nearby views. When using 2 or 3 input views, and the reconstructed 3D Gaussians only needs to give high-quality renderings from a relatively narrow view range, which means that they may still give good renderings even if the 3D Gaussians are erroneously located. However, when increasing the number of input views to 10, the reconstructed region needs to give reasonable renderings from a broader view range, where the distant novel views can serve as extrapolation views, thus evaluating the precision of Gaussian localisation. For example, given 10 input views of the 10, 30, 50, 70, 90, 110, 130, 150, 170, 190-th images of the scene, we render novel view from the pose of the 185-th image, then it may view at the same region as in the 10-th and 30-th images but from a significantly differently view direction. Such a case requires the reconstructed 3D Gaussians to be precisely localized, otherwise they may become floaters when viewing from extrapolated views. On the other hand, the depth map quality only evaluates the localization accuracy of 3D Gaussians, which becomes more precise when given more reference views and training on long sequences. Overall, training the model on long sequences is important for precise 3D Gaussian localization since the distant novel views can serve as extrapolated views to regularize the depth estimation, which is one of the key focuses of our paper.\n\n3. **Experiments on Re10k and ACID:**  To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in our ***rebuttal pdf Table 2, 3 and Figure 3***. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat [1] and on par as MVSplat [2] on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:\n\n   In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in **minimum view overlap** between the image stereo (e.g. as shown in our ***rebuttal pdf Figure 4(b)***), which means that our cost volume can be **much sparser** and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians. \n\n   Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction (we also offer a quantitative comparison on computation cost in our ***rebuttal Table 1*** to emphasize our strengths). Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.\n\n   We really appreciate your question, which helped us dive deeper into broader experimental comparisons and explore the underlying reasons, reaching a better clarification of our research focus and contributions. We will add the corresponding results and illustrations in final version to improve the completeness and highlight the contributions of our paper.\n\n[1] Charatan, David, et al. \"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.\" CVPR. 2024.\n\n[2] Chen, Yuedong, et al. \"Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.\" arXiv preprint arXiv:2403.14627 (2024).\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c5mHYKubeR", "reviewer": "Reviewer_ZuRj", "summary": "This paper introduces a generalizable 3DGS model which is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. The key idea of this paper is introducing Low-cost Cross-View Aggregation, which makes it is possible to use more nearby views for feature matching. Experiments show the advance in long sequence novel view synthesis.", "questions": "1. I want to make it clear that FreeSplat-fv indicates using Free-View Training (FVT) strategy and FreeSplat-spec indicates using specific number of reference views, such as two or three.\n2. Gaussian Splatting SLAM is one of the 3DGS-based SLAM system, which also can be run without depth as input information.It would be better if the authors can explain the strength of their work compared with this kind of 3DGS-based RGB SLAM, since the SLAM method can run in real time and process a longer input sequence.", "limitations": "Yes. As the authors discussed in this paper, the GPU requirement becomes expensive (> 40GB) when inputting extremely long image sequence (> 50). Besides, there is still a gap between the method of this paper and the other methods that rely on depth information as input.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Low-cost Cross-View Aggregation is introduced to predict initial Gaussian triplets, with which the computational cost would be lower, making it possible for feature matching between more nearby views and training on long sequence reconstruction.\n2. The Gaussian triplets are fused by Pixel-wise Triplet Fusion. This module can effectively reduce the pixel-aligned Gaussian redundancy in the overlapping regions and aggregate multi-view 3D Gaussian latent features.\n3. The experiments show the advance in long sequence novel view synthesis.", "weaknesses": "1. It would be better if the authors could report the training time of their model, since the authors addressed that both the Low-cost Cross-View Aggregation module and Pixel-wise Triplet Fusion module can lower the computation cost.\n2. It would be better to include the comparison with SurfelNeRF, since using a depth estimator differs from the GT depth. \n3. The definition of FreeSplat-fv and  FreeSplat-spec is not given clearly in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## To Reviewer ZuRj (#R4):\n\n1. **Computational Cost:** The full training time is around **2 days** for our 2-view and 3-view  versions of our method and baselines, and **3 days** for our free-view version. Although our training time is similar to pixelSplat [1] and MVSplat [2], we consume **much fewer GPU hours** due to our lower GPU requirements. As shown in our ***rebuttal pdf Table 1***, the 3-view version of pixelSplat and MVSplat already consume **30~50 GB** for training. It is due to their heavy patch-based 2D transformer and cross-view attention that results in quadratically increasing GPU requirements with increased image resolution and input sequence length. Therefore, their models are not suitable for high-resolution, long sequence inputs. In contrast, our proposed low-cost backbone can greatly reduce GPU consumption given a long sequence of high-resolution inputs, making whole scene reconstruction much more feasible. Furthermore, our proposed PTF module can effectively reduce redundant 3D Gaussians, improving the rendering speed from 39 to 72 FPS in whole scene reconstruction, which becomes even more important when reconstructing larger scenes to achieve real-time rendering. Overall, the efficiency of our model mainly includes: (1) lowering required GPU to make it feasible to train on long sequences of high-resolution inputs, and also reducing training cost (required GPU hours); (2) removing redundant 3D Gaussians to increase rendering speed and handle large 3D scenes.\n\n2. **Comparison with SurfelNeRF:** As shown in our ***rebuttal pdf Table 3 and Figure 4***, we conduct a comparison experiment with SurfelNeRF. We evaluate our FreeSplat-*fv* on the same novel views as theirs. Note that their input views are much denser than ours: e.g. they input 40 views covering a length of 120 images, with an approximate image interval of 3; while we still use relatively sparse inputs that have a fixed image interval of 20. We pick input views along their image sequences and evaluate at the same novel views. Note that the number of input views changes with the length of image sequence, while our FreeSplat-*fv* can perfectly fit to such varying numbers of input views. We outperform them largely in both rendering quality and efficiency. We analyze that such improvements are creditable to two main reasons: \n\n   (1) Their surfel-based representation is less representative comparing to 3DGS; \n\n   (2) Their model is not trained end-to-end with MVS encoder. SurfelNeRF uses external GT depth maps or MVS depth maps and do not jointly train an MVS-based encoder with the surfel-predictor, so their surfel-predictor does not have multi-view information, resulting in sub-optimal rendering quality. \n\n3. **Definition of FreeSplat-spec and FreeSplat-fv:** You are correct that FreeSplat-spec is trained using fixed number of input views for a fair comparison with the baselines FreeSplat-fv uses Free-View Training to reconstruct larger regions for stricter supervision from broader view range, to enforce precise 3D Gaussian localization. We will add the above illustration in the final version to improve the clarity.\n\n4. **Comparison with 3DGS-based SLAM methods:** Comparing to 3DGS-based SLAM methods, our proposed generalizable 3DGS-based method has the following two main strengths:\n\n   **(1) Efficiency:** Although the existing 3DGS-based SLAM can reach *1~3* FPS for per-scene optimization, they still require *10~20* minutes to finish reconstructing the whole 3D scene (*cf.* MonoGS [3] paper Table 9, 10; Splatam [4] paper Table 6). In contrast, our method only requires *1~2* seconds to parallelly mapping all the input views to 3D Gaussians. Furthermore, our proposed PTF can largely improve the rendering FPS (*cf.* our ***rebuttal pdf Table 1***), making the scene-level generalizable 3DGS more feasible. After fast feeding-forward, we can optionally conduct fast per-scene optimization using our predicted 3D Gaussians as the initalization to further improve the rendering quality.\n\n   **(2) Less constraints on Input Data:** Majority of the existing SLAM methods require dense input RGB-D sequence to estimate camera trajectories on-the-fly. Although several works [3, 5] can handle monocular inputs, there is still a gap between their camera tracking accuracy and RGB-D methods (*cf.* MonoGS paper Table 1). Such on-the-fly methods may also suffer from the drift problem due to accumulated errors. In contrast, our method is designed to explore learning strong priors from training data and perform fast offline prediction instead of on-the-fly application, making it more compatible with COLMAP and recent feed-forward pose estimation methods like DUSt3R [6]. Furthermore, our method does not require sensor depths during training or inference, and only requires relatively sparse inputs, which can be more easily generalized to various domains. \n\n\n\n[1] Charatan, David, et al. \"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Chen, Yuedong, et al. \"Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.\" arXiv preprint arXiv:2403.14627 (2024).\"\n\n[3] Matsuki, Hidenobu, et al. \"Gaussian splatting slam.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[4] Keetha, Nikhil, et al. \"SplaTAM: Splat Track & Map 3D Gaussians for Dense RGB-D SLAM.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[5] Teed, Zachary, and Jia Deng. \"Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.\" *Advances in neural information processing systems* 34 (2021): 16558-16569.\n\n[6] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## To all Reviewers:

We first thank all reviewers for your valuable time and inspiring comments. As summarized by our reviewers, our proposed method is "reasonable and novel" (\#R1) and "interesting" (\#R3), and our experimental results are "attractive and convincing" (\#R1), "exhibits enhanced performance" (\#R2), "advance in long sequence novel view synthesis" (\#R4). 

Regarding the raised questions, we have conducted extensive experiments as shown in our rebuttal pdf, and we provide corresponding introductions as follows:

1. **Comparison of computational cost and whole scene reconstruction:** As shown in our ***rebuttal pdf Table 1***, we compare the required GPU memory for training and testing, the encoding time, rendering FPS, and PSNR for whole scene reconstruction. pixelSplat-3views and MVSplat-3views already consume *30~50 GB* GPU memory for training due to their quadratically increasing GPU memory requirement w.r.t image resolution / sequence length. Therefore, it becomes infeasible to extend their methods to higher resolution inputs or longer sequence training. In comparison, our low-cost framework design enable us to effectively train on long sequence inputs while requiring lesser GPU memory compared to the 3 views version of existing methods. Furthermore, our proposed PTF module can effectively reduce redundant 3D Gaussians, improving rendering speed from 39 to 72 FPS . This becomes increasingly important when reconstructing larger scenes since generalizable 3DGS normally perform pixel-wise unprojection, which can easily result in redundancy in the overlapping regions (as shown in our ***rebuttal pdf Figure 2***).

2. **Experiments on RE10K and ACID:** To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in our ***rebuttal pdf Table 2, 3 and Figure 3***. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat [1] and on par as MVSplat [2] on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:

   In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in **minimum view overlap** between the image stereo (e.g. as shown in our ***rebuttal pdf Figure 4(b)***), which means that our cost volume can be **much sparser** and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians. 

   Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction (we also offer a quantitative comparison on computation cost in our ***rebuttal Table 1*** to emphasize our strengths). Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.

3. **Comparison with SurfelNeRF:** We further compare with SurfelNeRF as shown in our ***rebuttal pdf Table 3 and Figure 4***. We evaluate on the same novel views as theirs, sampling input views along their input sequences with an interval of 20 between nearby views. Note that the number of input views changes when the input length changes, while our FreeSplat-*fv* can seamlessly conduct inference with arbitrary numbers of inputs. Our method performs significantly better than SurfelNeRF in both rendering quality and efficiency. Our end-to-end framework jointly learns depths and 3DGS using an MVS-based backbone, while SurfelNeRF relies on depths and does not aggregate multi-view features to assist their surfel feature prediction.

[1] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024)."

Meta-review: The submission got four positive recommendations. The reviewers were concerned about the experiments, evaluations, limitations, missing details, and more analysis on the methods. The authors did a good job in the rebuttal and had good discussions with reviewers during the reviewer and author discussion period, both of which are helpful to address most reviewers’ concerns. The reviewers reached a consensus of acceptance before the AC and reviewers discussion period. The AC read through the paper, review, rebuttal, and the discussion. The AC supports the reviewers’ decision, and makes a decision to accept this submission. This decision was approved by the SAC.
Decision: Accept (poster)