Title: FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Gaussian Splatting

Generalization

3D from multi-view sensors

Novel View Synthesis

3D Computer Vision
Abstract: Empowering 3D Gaussian Splatting with generalization ability is appealing. However, existing generalizable 3D Gaussian Splatting methods are largely confined to narrow-range interpolation between stereo images due to their heavy backbones, thus lacking the ability to accurately localize 3D Gaussian and support free-view synthesis across wide view range. In this paper, we present a novel framework FreeSplat that is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis.Specifically, we firstly introduce Low-cost Cross-View Aggregation achieved by constructing adaptive cost volumes among nearby views and aggregating features using a multi-scale structure. Subsequently, we present the Pixel-wise Triplet Fusion to eliminate redundancy of 3D Gaussians in overlapping view regions and to aggregate features observed across multiple views. Additionally, we propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views. Our empirical results demonstrate state-of-the-art novel view synthesis peformances in both novel view rendered color maps quality and depth maps accuracy across different numbers of input views. We also show that FreeSplat performs inference more efficiently and can effectively reduce redundant Gaussians, offering the possibility of feed-forward large scene reconstruction without depth priors. Our code will be made open-source upon paper acceptance.

FreeSplat: Generalizable 3D Gaussian Splatting Towards Free-View Synthesis of Indoor Scenes

Abstract

Empowering 3D Gaussian Splatting with generalization ability is appealing. However, existing generalizable 3D Gaussian Splatting methods are largely confined to narrow-range interpolation between stereo images due to their heavy backbones, thus lacking the ability to accurately localize 3D Gaussian and support free-view synthesis across wide view range. In this paper, we present a novel framework FreeSplat that is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. Specifically, we firstly introduce Low-cost Cross-View Aggregation achieved by constructing adaptive cost volumes among nearby views and aggregating features using a multi-scale structure. Subsequently, we present the Pixel-wise Triplet Fusion to eliminate redundancy of 3D Gaussians in overlapping view regions and to aggregate features observed across multiple views. Additionally, we propose a simple but effective free-view training strategy that ensures robust view synthesis across broader view range regardless of the number of views. Our empirical results demonstrate stateof-the-art novel view synthesis peformances in both novel view rendered color maps quality and depth maps accuracy across different numbers of input views. We also show that FreeSplat performs inference more efficiently and can effectively reduce redundant Gaussians, offering the possibility of feed-forward large scene reconstruction without depth priors.

1 Introduction

Recent advancements has emerged [3, 4, 5, 6] in reconstructing 3D scenes from multiple viewpoints. Based on ray-marching-based volume rendering, Neural Radiance Fields [3, 7, 8, 9] is capable of learning the implicit 3D geometry and radiance fields without depth information. Nonetheless, computational cost remains to be the inherent bottleneck in ray-marching-based volume rendering, preventing it from real-time rendering. 3D Gaussian Splatting [10, 11, 12, 13] has recently been proposed as an efficient representation for photorealistic reconstruction of 3D scenes from multi-views. The explicit representation of 3D Gaussians are optimized to be densified in the textured regions, and the rasterization-based volume rendering avoids the costly ray marching scheme. Consequently, 3D Gaussian Splatting has achieved real-time rendering of high-quality images from novel views. Nonetheless, vanilla 3D Gaussian Splatting lacks generalizability and requires per-scene optimization.

Several attempts [1, 2, 14, 15, 16, 17] have been made to give 3D Gaussian Splatting generalization ability. Despite showing promising performance, these methods are limited to narrow-range scenelevel view interpolation [1, 2, 15] and object-centric synthesis [14, 16]. The primary reason for the limitation is that these existing methods depend on dense view matching across multi-view images with transformers to predict Gaussian primitives, which consequently becomes computationally intractable with longer sequences and thus restricting the supervision of these methods to narrow-range interpolated views. As we show in Figure 4, supervision by narrow-range interpolated views often result in poorly localized 3D Gaussians that can become floaters when rendered from extrapolated views. Additionally, the problem is further aggrevated by existing methods typically merging multiview 3D Gaussians through simple concatenation and thus inevitably lead to noticeable redundancy in overlapping areas ( cf. Table 2). In view of the above-mentioned problems, it is therefore imperative to design a method that is capable of long sequence reconstruction of global 3D Gaussians, which has the significant potential of supporting real-time rendering from arbitrary poses.

In this paper, we propose FreeSplat tailored for indoor long sequence free view synthesis. Unlike existing methods limited to view interpolation in narrow ranges, our method can effectively reconstruct explicit global 3DGS for novel view synthesis across wide view ranges. Our pipline consists of Low-cost Cross-View Aggregation and Pixel-wise Triplet Fusion (PTF). In Low-cost Cross-View Aggregation, we introduce efficient CNN-based backbones and adaptive cost volumes formulation among nearby views for low-cost feature extraction and matching, then we leverage a Multi-Scale Feature Aggregation structure to broaden the receptive field of cost volume and predict Depths and Gaussian Triplets. Subsequently, we present Pixel-wise Alignment with progressive Gaussian fusion in PTF to adaptively fuse local Gaussian Triplets from multi-views and avoid Gaussian redundancy in the overlapping regions. Moreover, due to our efficient feature extraction and matching, we propose a Free-View Training (FVT) strategy to disentangle generalizable 3DGS with specific number of views and train the model on long sequences.

The contributions of our paper are summarized as follows:

We present Low-cost Cross-View Aggregation to predict initial Gaussian triplets, where the low computational cost makes it possible for feature matching between more nearby views and training on long sequence reconstruction;

We propose Pixel-wise Triplet Fusion to fuse Gaussian triplets, which can effectively reduce the Gaussian redundancy in the overlapping regions and aggregate multi-view 3D Gaussian latent features;

To the best of our knowledge, we are the first to explore generalizable 3DGS for long sequence reconstruction. Extensive experiments on indoor dataset ScanNet [18] and Replica [19] demonstrate our superiority on both image rendering quality and novel view depth rendering accuracy when given different lengths of input views.

2 Related Work

Novel View Synthesis. Traditional attempts in novel view synthesis mainly employed voxel grids [20, 21] or multiplane images [22]. Recently, Neural Radiance Fields (NeRF) [3, 5, 23, 24, 25] have drawn growing interest using ray-marching-based volume rendering to backpropagate image color error to the implicit geometry and radiance fields, such that the 3D geometry can be implicitly learned

to satisfy the multi-view color consistency. Nonetheless, one inherent bottleneck of NeRFs-based method is the computation intensity of ray marching, which requires the costly volume sampling in the implicit fields for each pixel during rendering. To this end, recently 3DGS [10, 26, 27, 11] have attracted increasing attention due to its high efficiency and photorealistic rendering. Instead of relying on MLPs to represent the coordincate-based implicit fields, 3DGS learns an explicit field using a set of 3D Gaussians. They optimize the 3D Gaussians parameters and perform adaptive densify control to fit to the given set of images, such that the 3D Gaussians are encouraged to perform densification only in the textured regions and refrain from over-densification. During rendering, 3DGS performs tile-based rasterization to differentiably accumulate color images from the explicit 3D Gaussian primitives, which is significantly faster than the ray-marching-based volume rendering and achieves real-time rendering speed.

Generalizable Novel View Synthesis. Another drawback of the traditional NeRF-based and 3DGSbased methods is the requirement of per-scene optimization instead of direct feeding-forward. To this end, there have been a line of work [28, 8, 7, 29] focusing on learning effective priors to predict 3D geometry from given images in a feed forward fashion, where the common practice is to project ray-marching sampled points onto given source views to aggregate multi-view features, conditioning the prediction of the implicit fields on source views instead of point coordinates. Recently, there have also been attempts towards generalizable 3DGS [1, 17, 2, 15, 14]. pixelSplat [1] and GPS-Gaussian [17] propose to predict pixel-aligned 3D Gaussian parameters in feed forward fashion. MVSplat [2] replaces the epipolar line transformer of pixelSplat with a lightweight cost volume to perform more efficient image encoding. GGRt [15] concatenates pixelSplat predicted 3D Gaussians in a sequence of images and simultaneously perform pose optimization. latentSplat [14] encodes 3D Variational Gaussians and leverages a discriminator to help produce more indistinguable images. Nonetheless, existing methods do not reconstruct the global 3D Gaussians from arbitrary length of inputs, and are limited to view interpolation [1, 2, 17, 15] or object/human-centric scenes [17, 14]. In contrary, in this paper we focus on reconstructing large scenes from arbitrary length of inputs without depth priors, unleashing the potential of generalizable 3DGS for large scene explicit representation.

Indoor Scene Reconstruction. One line of efforts in feed-forward indoor scene reconstruction focuses on extracting 3D mesh using voxel volumes [30, 31, 32] and TSDF-fusion [33], while do not perform photorealistic novel view synthesis. On the other hand, the SLAM-based methods [34, 35, 36] require dense sequence of RGB-D input and per-scene tracking and mapping. Another paradigm of 3D reconstruction [37, 38, 39] learns implicit Signed Distance Fields from RGB input, while demanding intensive per-scene optimization. Another recent work SurfelNeRF [40] learns a feed-forward framework to map a sequence of images to 3D surfels which support photorealistic image rendering, while they rely on external depth estimator or ground truth depth maps. In contrary, we propose an end-to-end model without ground truth depth map input or supervision, enabling accurate 3D Gaussian localization using only photometric losses.

3 Preliminary

Vanilla 3DGS. 3D-GS [10] explicitly represents a 3D scene with a set of Gaussian primitives which are parameterized via a 3D covariance matrix Σ and mean µ :

where Σ is decomposed into Σ = RSS ⊤ R ⊤ using a scaling matrix S and a rotation matrix R to maintain positive semi-definiteness. During rendering, the 3D Gaussian is transformed into the image coordinates with world-to-camera transform matrix W and projected onto image plane with projection matrix J , and the 2D covariance matrix Σ ′ is computed as Σ ′ = JWΣW ⊤ J ⊤ . We then obtain a 2D Gaussian G 2 D with the covariance Σ ′ in 2D, and the color rendering is computed using point-based alpha-blending on each ray:

where N is the number of Gaussian primitives, α i is a learnable opacity, and c i is view-dependent color defined by spherical harmonics (SH) coefficients s . The Gaussian parameters are optimized by a photometric loss to minimize the difference between renderings and image observations.

…

…

Efficient 2D Feature Extraction

𝑰

𝒕"𝟐

𝑰

𝒕"𝟏

𝑰

𝒕

𝑭

𝒎

𝒕"𝟐

𝑭

𝒔

𝒕"𝟐

Nearest N views

Multi-Scale Feature Aggregation

𝑫

𝒕"𝟐

𝑭

𝒍

𝒕"𝟐

𝑭

𝒎

𝒕"𝟏

Nearest N views

𝑭

𝒕 𝒎

Nearest N views

𝑫

𝒕"𝟏

𝑭

𝒍

𝒕"𝟏

𝑫

𝒕

𝑭

𝒍

𝒕

𝝁

𝒈

𝒕#𝟐

, 𝝎

𝒈

𝒕#𝟐

, 𝒇

𝒈

𝒕#𝟐

PTF

PTF

PTF

Pixel-wise

Alignment

Pixel-wise

Alignment

Pixel-wise

Alignment

Progressive Gaussians Fusion

𝑮

𝒈

𝒕"𝟐

𝑮

𝒈

𝒕"𝟏

𝑮

𝒈

𝒕

U

U

U

U

Unproject

Local Flow

Matching Flow

Global Flow

A

Append

Image Sequence

Cost Volume

Cost Volume

Cost Volume

Weighted Sum

& GRU

Weighted Sum

& GRU

Weighted Sum

& GRU

Gaussian Latent Decoder

A

A

A

Local Gaussian Triplet

Global Gaussian Triplet

𝑭

𝒔

𝒕"𝟏

𝑭

𝒔

𝒕

Multi-Scale

Image Feature

Image Feature

Cost Volume

Image Color

𝝁

𝒍

𝒕#𝟏

𝝎

𝒍

𝒕#𝟏

𝒇

𝒍

𝒕#𝟏

𝝁

𝒍

𝒕#𝟐

𝝎

𝒍

𝒕#𝟐

𝒇

𝒍

𝒕#𝟐

𝝁

𝒍

𝒕

𝝎

𝒍

𝒕

𝒇

𝒍

𝒕

Matching Feature

Rendering

→

→

Generalizable 3DGS. Unlike vanilla 3DGS that optimizes per-scene Gaussian primitives, recent generalizable 3DGS [1, 17] predict pixel-aligned Gaussian primitives { Σ , α , s } and depths d , such that the pixel-aligned Gaussian primitives can be unprojected to 3D coordinates µ . The Gaussian parameters are predicted by 2D encoders, which are optimized by the photometric loss through rendering from novel views. However, existing methods are still limited to view interpolation within narrow view range, which leads to inaccurately localized 3D Gaussians that fail to support large scene reconstruction and view extrapolation ( cf. Figure 1, 4). To this end, we propose FreeSplat towards global 3D Gaussians reconstruction with accurate localization that supports free-view synthesis.

4 Our Methodology

4.1 Overview

The overview of our method is illustrated in Figure 2. Given a sparse sequence of RGB images, we build cost volumes adaptively between nearby views, and predict depth maps to unproject the 2D feature maps into 3D Gaussian triplets. We then propose the Pixel-aligned Triplet Fusion (PTF) module to progressively align the global with the local Gaussian triplets, such that we can fuse the redundant 3D Gaussians in the latent feature space and aggregate cross-view Gaussian features before decoding. Our method is capable of efficiently exchanging cross-view features through cost volumes, and progressively aggregating per-view 3D Gaussians with cross-view alignment and adaptive fusion.

4.2 Low-cost Cross-View Aggregation

Efficient 2D Feature Extraction. Given a sparse sequence of posed images { I t } T t =1 , we first feed them into a shared 2D backbone to extract multi-scale embeddings F t e and matching feature F t m . Unlike [1, 2] which rely on patch-wise transformer-based backbones [41, 42] that can lead to quadratically expensive computations, we leverage pure CNN-based backbones [43, 44] for 2D feature extraction for efficient performance on higher resolution inputs.

Adaptive Cost Volume Formulation. To explicitly integrate camera pose information given arbitrary length of input images, we propose to adaptively build cost volumes between nearby views. For current view I t with pose P t and matching feature F t m ∈ R C m × H 4 × W 4 , we adaptively select its N nearby views { I t n } N n =1 with poses { P t n } N n =1 based on pose proximity, and construct cost volume via plane sweep stereo [45, 46]. Specifically, we define a set of K virtual depth planes { d k } K k =1 that are uniformly spaced within [ d near , d far ] , and warp the nearby view features to each depth plane d k

Local Gaussian Triplet

Global Gaussian Triplet

𝑡

-th View

𝑑

!

"

-𝑑

#

"

< 𝛿 & 𝑑

!

"

𝜇

!

"#$

, 	𝜔

!

"#$

, 𝑓

!

"#$

𝜇

%

"

, 						𝜔

%

"

,							𝑓

%!

"

Geometrical

Weighted Sum

GRU

Fusion

Gaussian Latent Fusion

Pixel-wise Alignment

of current view:

where Trans( P t n , P t ) is the transformation matrix from view t n to t . The cost volume F t cv ∈ R K × H 4 × W 4 is then defined as:

where F t cv [ k ] is the k -th dimension of F t cv , cos( · ) is the cosine similarity, ⊕ is feature-wise concatenation, and f θ ( · ) is a 1 × 1 CNN mapping to dimension of 1 .

Multi-Scale Feature Aggregation. The embedding of the cost volume plays a significant part to accurately localize the 3D Gaussians ( cf. Table 5). To this end, inspired by previous depth estimation methods [47, 33], we design an multi-scale encoder-decoder structure, such that to fuse multi-scale image features with the cost volume and propagate the cost volume information to broader receptive fields. Specifically, the multi-scale encoder takes in F t cv and the output is concatenated with { F t s } before sending into a UNet++ [48]-like decoder to upsample to full resolution and predict a depth candidates map D t c ∈ R K × H × W , and Gaussian triplet map F t l ∈ R C × H × W . We then predict the depth map through soft-argmax to bound the depth prediction between near and far:

Finally, the pixel-aligned Gaussian triplet map F t l is unprojected to 3D Gaussian triplet { µ t l , ω t l , f t l } , where µ t l ∈ R 3 × HW are the Gaussian centers, ω t l ∈ R 1 × HW are weights between (0 , 1) , and f t l ∈ R ( C -1) × HW are Gaussian triplet features.

4.3 Pixel-wise Triplet Fusion

One limitation of previous generalizable 3DGS methods is the redundancy of Gaussians. Since we need multi-view observations to predict accurately localized 3D Gaussians in indoor scenes, the pixel-aligned Gaussians become redundant in frequently observed regions. Furthermore, previous methods integrate multi-view Gaussians of the same region simply through their opacities, leading to suboptimal performance due to lack of post aggregation ( cf. Table 5). Consequently, inspired by previous methods [31, 40], we propose the Pixel-wise Triplet Fusion (PTF) module which can significantly remove redundant Gaussians in the overlapping regions and explicitly aggregate multiview observation features in the latent space. We align the per-view local Gaussians with global ones using Pixel-wise Alignment to select the redundant 3D Gaussian Triplets, and progressively fuse the local Gaussians into the global ones.

Pixel-wise Alignment. Given the Gaussian triplets { µ t l , f t l } T t =1 , we start from t = 1 where the global Gaussians latent is empty. In the t -th step, we first project the global Gaussian triplet centers µ t -1 g ∈ R 3 × M onto the t -th view:

where [ x t g , y t g , d t g ] ∈ R 3 × M are the projected 2D coordinates and corresponding depths. We then correspond the local Gaussian triplets with the pixel-wise nearest projections within a threshold. Specifically, for the i -th local Gaussian with 2D coordinate [ x t l ( i ) , y t l ( i )] and depth d t l (j) , we first find its intra-pixel global projection set S i :

where [ · ] is the rounding operator. Subsequently, we search for valid correspondence with minimum depth difference under a threshold:

where δ is a ratio threshold. We define the valid correspondence set as:

̸

Gaussian Triplet Fusion. After the pixel-wise alignment, we remove the redundant 3D Gaussians through merging the validly aligned triplet pairs. Given a pair ( i, m i ) ∈ F t , we compute the weighted sum of their center coordinates and sum their weights to restrict the 3D Gaussian centers to lie between the triplet pair:

We then aggregate the aligned local and global Gaussian latent features through a lightweight GRU network:

and then append with the other unaligned local Gaussian triplets.

Gaussian primitives decoding. After the Pixel-wise Triplet Fusion, we can decode the global Gaussian triplets into Gaussian primitives:

and Gaussian centers µ = µ ⊤ g . Our proposed fusion method can incrementally integrate the Gaussians with geometrical constraints and learnable GRU network for feature update. Consequently, our fusion method is capable of significantly removing redundant Gaussians and perform post feature aggregation across multiple views, and can be trained with the other framework components end-to-end with eligible computation overhead.

4.4 Training

Loss Functions. After predicting the 3D Gaussian primitives, we render from novel views following the rendering equations in Eq. (2). Similar to pixelSplat [1] and MVSplat [2], we train our framework using only photometric losses, i.e. a combination of MSE loss and LPIPS [50] loss, with weights of 1 and 0.05 following [1, 2].

Free-View Training. We propose a Free-View Training (FVT) strategy to add more geometrical constraints on the localization of 3D Gaussians, and to disentangle the performance of generalizable 3DGS with specific number of input views. To this end, we randomly sample T number of context views (in experiments we set T between 2 and 8 ), and supervise the image renderings in the broader view interpolations. The long sequence training is made feasible due to our efficient feature extraction and aggregation. We empirically find that FVT significantly contributes to depth estimation from novel views ( cf. Table 3, 4).

5 Experiments

5.1 Experimental Settings

Datasets. We leverage the real-world indoor dataset ScanNet [18] for training. ScanNet is a large RGB-D dataset containing 1 , 513 indoor scenes with camera poses, and we follow [51, 40] to use

100 scenes for training and 8 scenes for testing. To evaluate the generalization ability of our model, we further perform zero-shot evaluation on the synthetic indoor dataset Replica [19], for which we follow [52] to select 8 scenes for testing.

Implementation Details. Our FreeSplat is trained end-to-end using Adam [53] optimizer with an initial learning rate of 1 e -4 and cosine decay following [2]. Due to the large GPU requirements of [1, 2] given high-resolution images, all input images are resized to 384 × 512 and batch size is set to 1, to form a fair comparison between different methods. We mainly compare with previous generalizable 3DGS methods in 2, 3, 10 reference view settings, where the distance between input views is fixed, thus evaluating the models' performance under different view ranges. For 10 views setting, we also choose target views that are beyond the given sequence of reference views to evaluate the view extrapolation results.

5.2 Results on ScanNet

View Interpolation Results. On ScanNet, we evaluate the generalizable novel view interpolation results given 2 and 3 reference views as shown in Table 1. Comparing to pixelSplat and MVSplat, our FreeSplatspec consistently improves rendering quality and efficiency on 2-views setting and 3-views setting. Although slightly underperforming on SSIM comparing to NeuRay [9], we show significant improvements on PSNR and LPIPS over NeuRay and 300 × faster inference speed. Moreover, our FreeSplatfv consistently offers competitive results given arbitrary number of views, and performs more similarly as FreeSplatspec when number of input views increases.

Ref.

Target View

Ours-

spec

MVSplat

pixelSplat

Ours-

fv

…

…

Long Sequence Results. As shown in Table 2, we further evaluate the long sequence results where we sample reference views with length of 10, and compare both view interpolation and extrapolation results. The results reveal that generalizable 3DGS methods underperform when given long sequence input images, which is due to the complicated camera trajectories in ScanNet, and the inaccuracy of 3D Gaussian localization that leads to errors when observed from wide view ranges. Our FreeSplat3views significantly outperforms pixelSplat and MVSplat on view interpolation and view extrapolation results. Through our proposed FVT that can be easily plugged into our model due to our low requirement on GPU, our FreeSplatfv consistently outperforms our 3-views version. Our PTF module can also reduce the number of Gaussians by around 55 . 0% , which becomes indispensable in long sequence reconstruction due to the pixel-wise unprojection nature of generalizable 3DGS. The qualitative results are shown in Figure 4, which clearly reveal that FreeSplatspec outperforms MVSplat and pixelSplat in localizing 3D Gaussian and preserving fine-grained details, and FreeSplatfv further improves on localizing and fusing multi-view Gaussians.

Method

3 Views

10 Views

PSNR

↑

SSIM

↑

LPIPS

↓

δ <

1

.

25

↑

#GS(k) PSNR

↑

SSIM

↑

LPIPS

↓

δ <

1

.

25

↑

#GS(k)

pixelSplat [1]

26.24

0.829

0.229

0.576

1769

19.23

0.719

0.414

0.375

5898

MVSplat [2]

26.16

0.840

0.173

0.670

590

18.66

0.717

0.360

0.565

1966

FreeSplat-

spec

26.98

0.848

0.171

0.682

423

21.11

0.762

0.312

0.720

1342

FreeSplat-

fv

26.64

0.843

0.184

0.682

421

21.95

0.777

0.290

0.742

1346

CV PTF FVT

3 views

10 views

PSNR

↑

SSIM

↑

LPIPS

↓

δ <

1

.

25

↑

δ <

1

.

10

↑

PSNR

↑

SSIM

↑

LPIPS

↓

δ <

1

.

25

↑

δ <

1

.

10

✓

27.12

0.825

0.224

0.925

0.762

24.23

0.792

0.277

0.942

0.804

✓

22.10

0.696

0.359

0.639

0.311

17.94

0.607

0.487

0.543

0.216

✓

✓

27.45

0.829

0.222

0.930

0.773

25.15

0.800

0.278

0.945

0.823

✓

✓

26.41

0.806

0.232

0.919

0.746

25.40

0.799

0.252

0.950

0.831

✓

✓

✓

27.34

0.826

0.226

0.928

0.764

25.90

0.808

0.252

0.961

0.858

Target View

Full model

w/o FVT

w/o PTF

3 Views

10 Views

Novel View Depth Estimation Results. We also investigate the correctness of 3D Gaussian localization of different methods through comparing their depth rendering results. We report the Absolute Difference (Abs. Diff), Relative Difference (Rel. Diff), and threshold tolerance δ < 1 . 25 results from novel views in Table 3. We find that FreeSplat consistently outperforms pixelSplat and MVSplat in predicting accurately localized 3D Gaussians, where FreeSplatfv reaches 94 . 9% of δ < 1 . 25 , enabling accurate unsupervised depth estimation on novel views. The improved depth estimation accuracy of FreeSplatfv highlights the importance of depth estimation in supporting free-view synthesis across broader view range.

5.3 Zero-Shot Transfer Results on Replica

We further evaluate the zero-shot transfer results through testing on Replica dataset, with results in Table 4. Our view interpolation and novel view depth estimation results still outperforms existing methods. The long sequence results degrade due to inaccurate depth estimation and domain gap, indicating potential future work in further improving the depth estimation in zero-shot tranferring.

5.4 Ablation Study

We conduct a detailed ablation study as shown in Table 5 and Figure 5. The results indicate that: 1) cost volume is essential in accurately localizing 3D Gaussians; 2) our proposed PTF module can consistently contribute to rendering quality and depth estimation results. The PTF module learns to incrementally fuse multi-view 3D Gaussians and contributes significantly when varying number of input views, and serves as a multi-view localization regularization that helps unsupervised depth estimation; 3) Our FVT module excels in long sequence reconstruction quality as well as novel view

depth rendering results, which provides stricter constrains on 3D Gaussian localization and can be seamlessly combined with the PTF module to fit to varying length of input views.

6 Conclusion

In this study, we introduced FreeSplat, a generalizable 3DGS model that is tailored to accommodate an arbitrary number of input views and perform free-view synthesis using the global 3D Gaussians. We developed a Low-cost Cross-View Aggregation pipeline that enhances the model's ability to efficiently process long input sequences, thus incorporating stricter geometry constraints. Additionally, we have devised a Pixel-wise Triplet Fusion module that effectively reduces redundant pixel-aligned 3D Gaussians in overlapping regions and merges multi-view Gaussian latent features. FreeSplat consistently improves the fidelity of novel view renderings in terms of both color image quality and depth map accuracy, facilitating feed-forward global Gaussians reconstruction without depth priors.

References

David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312.12337 , 2023.

Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.14627 , 2024.

Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , pages 405-421. Springer, 2020.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACMTransactions on Graphics (ToG) , 41(4):115, 2022.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR , 2021.

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1412414133, 2021.

Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7824-7833, 2022.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. arXiv preprint arXiv:2311.16493 , 2023.

Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octreegs: Towards consistent real-time rendering with lod-structured 3d gaussians. arXiv preprint arXiv:2403.17898 , 2024.

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 , 2023.

Christopher Wewer, Kevin Raj, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoencoding variational gaussians for fast generalizable 3d reconstruction. arXiv preprint arXiv:2403.16292 , 2024.

Hao Li, Yuanyuan Gao, Dingwen Zhang, Chenming Wu, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, and Junwei Han. Ggrt: Towards generalizable 3d gaussians without pose priors in real-time. arXiv preprint arXiv:2403.10147 , 2024.

Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 , 2023.

Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02155 , 2023.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 , 2019.

Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 , 2019.

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2437-2446, 2019.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 , 2018.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mipnerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19697-19705, 2023.

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 , 2023.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 , 2023.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5480-5490, 2022.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 523-540. Springer, 2020.

Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Realtime coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15598-15607, 2021.

Noah Stier, Alexander Rich, Pradeep Sen, and Tobias Höllerer. Vortx: Volumetric 3d reconstruction with transformers for voxelwise view selection and fusion. In 2021 International Conference on 3D Vision (3DV) , pages 320-330. IEEE, 2021.

Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. Simplerecon: 3d reconstruction without 3d convolutions. In European Conference on Computer Vision , pages 1-19. Springer, 2022.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12786-12796, 2022.

Chi Yan, Delin Qu, Dong Wang, Dan Xu, Zhigang Wang, Bin Zhao, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. arXiv preprint arXiv:2311.11700 , 2023.

Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 , 2023.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems , 35:25018-25032, 2022.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8456-8465, 2023.

Yiming Gao, Yan-Pei Cao, and Ying Shan. Surfelnerf: Neural surfel radiance fields for online photorealistic reconstruction of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 108-118, 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint arXiv:2010.11929 , 2010.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Robert T Collins. A space-sweep approach to true multi-image matching. In Proceedings CVPR IEEE computer society conference on computer vision and pattern recognition , pages 358-363. Ieee, 1996.

Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. Dpsnet: End-to-end deep plane sweep stereo. arXiv preprint arXiv:1905.00538 , 2019.

Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, and Marc Pollefeys. Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15324-15333, 2021.

Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4 , pages 3-11. Springer, 2018.

Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS) , pages 1597-1600. IEEE, 2017.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5449-5458, 2022.

Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

A Appendix / supplemental material

A.1 Experimental Environment

We conduct all the experiments on single NVIDIA RTX A6000 GPU. The experimental environment is PyTorch 2.1.2 and CUDA 12.2.

A.2 Additional Implementation Details

We set the number of virtual depth planes K = 128 , matching feature dimension C m = 64 , and d near = 0 . 5 , d far = 15 . 0 for cost volume formulation, and set δ = 0 . 05 in Eq.(8) for pixel-wise alignment. To train the 3-views version of pixelSplat [1] on a single NVIDIA RTX A6000 GPU, we change their ViT patch size from 8 × 8 to 16 × 16 . During inference on the 10 views setting, the epipolar line sampling in pixelSplat and the cross-view attention in MVSplat [2] are performed between nearby views similarly as ours to save GPU requirements and form a fair comparison. For the free-view version of FreeSplat we set the number of nearby views as N = 4 for training. For the testing of long sequence explicit reconstruction, cost volumes are formed between nearby 8 views.

A.3 Additional Experiments

Computational Cost. As shown in Table 6, we compare the required GPU memory for training and testing, the encoding time, rendering FPS, and PSNR for whole scene reconstruction. pixelSplat3views and MVSplat-3views already consume 30 50 GB GPU memory for training due to their quadratically increasing GPU memory requirement with respect to the image resolution / sequence length. Therefore, it becomes infeasible to extend their methods to higher resolution inputs or longer sequence training. In comparison, our low-cost framework design enable us to effectively train on long sequence inputs while requiring lesser GPU memory compared to the 3 views version of existing methods. Furthermore, our proposed PTF module can effectively reduce redundant 3D Gaussians, improving rendering speed from 39 to 72 FPS . This becomes increasingly important when reconstructing larger scenes since generalizable 3DGS normally perform pixel-wise unprojection, which can easily result in redundancy in the overlapping regions.

Experiments on RE10K and ACID. To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in Table 7, 8 and Figure 6. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat and on par as MVSplat on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:

Target View

Ours

MVSplat

pixelSplat

Inputs

In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in minimum view overlap between the image stereo, which means that our cost volume can be much sparser and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians.

Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction. Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.

Comparison with SurfelNeRF. We further compare with SurfelNeRF as shown in Table 9 and Figure 7. We evaluate on the same novel views as theirs, sampling input views along their input sequences with an interval of 20 between nearby views. Note that the number of input views changes when the input length changes, while our FreeSplat-fv can seamlessly conduct inference with arbitrary numbers of inputs. Our method performs significantly better than SurfelNeRF in both rendering quality and efficiency. Our end-to-end framework jointly learns depths and 3DGS using an MVS-based backbone, while SurfelNeRF relies on depths and does not aggregate multi-view features to assist their surfel feature prediction.

FreeSplatfv

SurfelNeRF

A.4 Additional Qualitative Results

2 and 3-View Interpolation Results. The qualitative results are shown in Figure 8, where FreeSplat more precisely localizes 3D Gaussians and captures more fine-grained details comparing to previous methods. FreeSplat can also localize 3D Gaussians more accurately and renders precise depth maps, supporting high-quality rendering from broader view range ( cf. FreeSplatspec results in Figure 4).

Results on Replica. We show the qualitative results on Replica in Figure 10, where our superiority over MVSplat and pixelSplat remains. The results indicate the generalization ability of FreeSplat across indoor datasets for the view interpolation task.

Results of Whole Scene Reconstruction. We also show qualitative results of our whole scene reconstruction in Figure 9. Despite the long input sequence ( ∼ 40 images) covering the whole scene, FreeSplat can still perform efficient feed-forward in ∼ 1s on single NVIDIA RTX A6000, and can render high-quality images and accurate depth maps from novel views. On the other hand, it is still difficult to accurately predict depth of textureless ( e.g. wall) and specular ( e.g. light reflection on the floor) regions. However, we hope our work provides an initial step towards accurate geometry reconstruction without ground truth depth priors.

A.5 Limitations

Although our approach excels in novel view rendering depth estimation and support arbitrary number of input views, the GPU requirement becomes expensive ( > 40 GB) when inputting extremely long image sequence ( > 50 ). On the other hand, due to our unsupervised scheme of depth estimation, there is still a gap between our 3D reconstruction accuracy and the state-of-the-art methods with 3D supervision [33, 32] or RGB-D inputs [36, 35] ( e.g. as shown in Figure 9, the textureless and specular regions). Our main focus is to explore the feed-forward indoor scene photorealistic reconstruction purely based on 2D supervision.

Ref.

Target View

Ours-

spec

MVSplat

pixelSplat

2 Views

3 Views

Ref.

Target View

Ours-

spec

MVSplat

pixelSplat

Target View

Our Renderings

3D Gaussians

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We mainly focus on generalizable 3DGS for indoor scene reconstruction, where we support arbitrary number of inputs through adaptive cost volume (Sec. 4.2) and gaussian fusion (Sec. 4.3).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in Sec. A.5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not propose new theory in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include detailed description of our framework in Sec. 4 and implementation details in Sec. 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our code will be released upon paper acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The detailed descriptions of the experiment settings are illustrated in Sec. 5.1. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow our related works in the setting for error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the experimental environment in Sec. A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our knowledge, our work is conducted with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: To the best of our knowledge, we do not foresee societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite the used existing datasets and models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide detailed descriptions about our method (Sec. 4) as well as its limitations (Sec. A.5). Our code will be released upon paper acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a generalizable 3DGS model which is capable of reconstructing geometrically consistent 3D scenes from long sequence input towards free-view synthesis. The key idea of this paper is introducing Low-cost Cross-View Aggregation, which makes it is possible to use more nearby views for feature matching. Experiments show the advance in long sequence novel view synthesis.\n\nSTRENGTHS:\n1. Low-cost Cross-View Aggregation is introduced to predict initial Gaussian triplets, with which the computational cost would be lower, making it possible for feature matching between more nearby views and training on long sequence reconstruction.\n2. The Gaussian triplets are fused by Pixel-wise Triplet Fusion. This module can effectively reduce the pixel-aligned Gaussian redundancy in the overlapping regions and aggregate multi-view 3D Gaussian latent features.\n3. The experiments show the advance in long sequence novel view synthesis.\n\nWEAKNESSES:\n1. It would be better if the authors could report the training time of their model, since the authors addressed that both the Low-cost Cross-View Aggregation module and Pixel-wise Triplet Fusion module can lower the computation cost.\n2. It would be better to include the comparison with SurfelNeRF, since using a depth estimator differs from the GT depth. \n3. The definition of FreeSplat-fv and  FreeSplat-spec is not given clearly in this paper.\n\nQUESTIONS:\n1. I want to make it clear that FreeSplat-fv indicates using Free-View Training (FVT) strategy and FreeSplat-spec indicates using specific number of reference views, such as two or three.\n2. Gaussian Splatting SLAM is one of the 3DGS-based SLAM system, which also can be run without depth as input information.It would be better if the authors can explain the strength of their work compared with this kind of 3DGS-based RGB SLAM, since the SLAM method can run in real time and process a longer input sequence.\n\nLIMITATIONS:\nYes. As the authors discussed in this paper, the GPU requirement becomes expensive (> 40GB) when inputting extremely long image sequence (> 50). Besides, there is still a gap between the method of this paper and the other methods that rely on depth information as input.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a FreeSplat, aiming at generalizable 3D gaussian splitting for long sequence inputs. Specifically, it uses an efficient CNN-based cost volume and eliminates redundant 3D gaussians observed across multiple views. Extensive experiments show that FreeSplat effectively reduces inference costs and improves the novel view quality for long sequence inputs.\n\nSTRENGTHS:\n* The writing is good, making it easy to understand and follow.\n\n* The proposed Pixel-Aligned Triplet Fusion (PTF) is interesting as it effectively handles the redundant 3D gaussians for long sequence inputs.\n\nWEAKNESSES:\n* I think the evaluation details are not clear. For example, during the evaluation, I am curious whether the maximum gap between input views is fixed regardless of the number of input views. Does \"view range\" increase as the number of input views increases? If so, please describe the evaluation details.\n\n* It is weird that the novel view image quality drops as the number of input views increases, but the novel view depth quality increases. I'd like more clarification on this.\n\n* The existing pixelSplat and MVSplat have been experimented with on RealEstate10k [1] and ACID [2], and both datasets also contain long sequence inputs. The authors should have included comparisons on at least one of the RealEstate10k and ACID datasets.\n\n[1] Zhou et al., Stereo magnification: Learning view synthesis using multiplane images, SIGGRAPH 2018.\n\n[2] Liu et al., Infinite nature: Perpetual view generation of natural scenes from a single image, ICCV 2021.\n\nQUESTIONS:\nThe proposed method is interesting, but the lack of evaluation details, lack of comparative experiments, and inconsistencies in the results make it difficult to verify the contribution. The authors should address the \"weaknesses\" and strengthen the manuscript.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. Low-cost Cross-View Aggregation: This efficient methodology constructs adaptive cost volumes between proximate views and aggregates features utilizing a multi-scale structure. This approach enables the processing of extended input sequences and the incorporation of more stringent geometric constraints.\n2. Pixel-wise Triplet Fusion (PTF): This module aligns and merges local Gaussian triplets into global representations, thereby mitigating redundancy in overlapping regions and consolidating features observed across multiple viewpoints.\n3. Free-View Training (FVT): This novel training strategy decouples the model's performance from a predetermined number of input views, thus enabling robust view synthesis across an expanded range of viewpoints.\n4. The authors demonstrate that FreeSplat surpasses existing methodologies in both novel view synthesis quality and depth map accuracy on the ScanNet and Replica datasets. The proposed method exhibits enhanced performance in view interpolation and extrapolation tasks, particularly for extended input sequences. Moreover, FreeSplat reduces the quantity of redundant Gaussians and offers more computationally efficient inference compared to preceding approaches.\n\nSTRENGTHS:\n1. The paper is well-structured, offering lucid explanations of the technical approach. It includes detailed architectural diagrams and pseudocode to enhance reproducibility. Comprehensive ablation studies are presented to isolate the impact of different components. The overall writing is coherent and accessible.\n2. This research enables free-viewpoint rendering and 3D reconstruction from arbitrary numbers of input views, broadening the applicability of 3D Gaussian splatting techniques. It enhances efficiency and minimizes redundancy in 3D Gaussian representations for extended sequences.\n\nWEAKNESSES:\n1. Experimental Scope and Comparability: PixelSplat presents experimental results on the Re10k and ACID datasets. For a comprehensive and equitable comparison, it would be beneficial for the proposed method to also report results on these two datasets. This would facilitate a direct comparison with pixelSplat across multiple benchmarks, thereby strengthening the validity of the authors' claims. The absence of experimental results on Re10k and ACID raises questions about the proposed method's performance and generalizability across diverse datasets. I am wondering why the experimental results on Re10k  and ACID are absent.\n2. Analysis of Failure Cases and Limitations: The paper would be significantly enhanced by a more thorough examination of failure cases and limitations of the proposed approach. While the appendix briefly touches on this aspect, a more in-depth analysis would provide valuable insights into the method's robustness and potential areas for improvement. Such an analysis could include specific examples of scenarios where the method underperforms, a discussion of the underlying causes for these failures, and potential strategies for addressing these limitations in future work. This level of critical self-evaluation would not only increase the paper's scientific rigor but also provide a more balanced perspective on the method's capabilities and constraints.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThe authors discuss some limitations of their work:\n1. High GPU memory requirements (>40GB) for extremely long input sequences (>50 images).\n\n2. A gap in 3D reconstruction accuracy compared to state-of-the-art methods that use 3D supervision or RGB-D inputs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FreeSplat to reconstruct geometrically consistent 3D scenes from long sequence inputs. To this end, the paper presents Low-cost Cross-View Aggregation for feature matching and Pixel-wise Triplet Fusion for Gaussian triplets fusion. The outstanding results of long sequence 3DGS generalization shows the superiority beyond baseline methods.\n\nSTRENGTHS:\n1. The paper is well-written and the technical contributions are clearly clarified and easy to understand.\n2. The experimental results over baselines are attractive and convincing, which demonstrates the effectiveness of the proposed method.\n3. The proposed pixel-wise triplet fusion is reasonable and novel.\n\nWEAKNESSES:\n1. The low-cost cross-view aggregation module is not novel enough. It seems that the three submodules (2D feature extraction, cost volume formulation, multi-scale feature aggregation) are all borrowed from existing methods and the low-cost cross-view aggregation is a simple stack of these submodules. I suggest the author to further clarify the differences between the proposed module and the existing MVS methods.\n\nQUESTIONS:\n1. What are the differences between Freeplat-spec and Freesplat-fv?\n2. The author provided 3-view version results in the long sequence reconstruction experiment in Table 2. I’m doubting why do you provide the 10-view version of the baselines? Just because their original paper did not provide the 10-view version, or because they will fail to reconstruct a scene under 10-view inputs, or other reasons? Further clarification will help to understand this experiment.\n3. The author claims that the proposed pixel-wise triplet fusion module helps to remove redundant Gaussians in the overlapping regions and the experimental results of final Gaussian number demonstrates this. However, it is somewhat not intuitive. It would be better to provide a visualization to show how the Gaussian ellipsoids are reduced in the overlapping regions.\n\nLIMITATIONS:\nThe author discussed the limitations in the appendix about the memory consumption and the limited performance of generalizable Gaussians, which I think is valuable. It remains a significant problem for further study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}