Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis
Authors: Zhiyuan Min
Yawei Luo
Jianwen Sun
Yi Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: epipolar-free
3DGS
generalizable novel view synthesis
Primary area: machine_vision

Abstract: Generalizable 3D Gaussian splitting (3DGS) can reconstruct new scenes from sparse-view observations in a feed-forward inference manner, eliminating the need for scene-specific retraining required in conventional 3DGS. However, existing methods rely heavily on epipolar priors, which can be unreliable in complex real-world scenes, particularly in non-overlapping and occluded regions. In this paper, we propose eFreeSplat, an efficient feed-forward 3DGS-based model for generalizable novel view synthesis that operates independently of epipolar line constraints. To enhance multiview feature extraction with 3D perception, we employ a self-supervised Vision Transformer (ViT) with cross-view completion pre-training on large-scale datasets. Additionally, we introduce an Iterative Cross-view Gaussians Alignment method to ensure consistent depth scales across different views. Our eFreeSplat represents a new paradigm for generalizable novel view synthesis. We evaluate eFreeSplat on wide-baseline novel view synthesis tasks using the RealEstate10K and ACID datasets. Extensive experiments demonstrate that eFreeSplat surpasses state-of-the-art baselines that rely on epipolar priors, achieving superior geometry reconstruction and novel view synthesis quality.

Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis
Zhiyuan Min
Yawei Luo
Jianwen Sun?
Yi Yang'
'Zeejiang University
Central China Normal University
Reference Views
Novel Views
pixelSplat [6]
MVSplat [9]
elrreSplat (Ours)
Epipolar
Valid Epipolar Priors
Invalid Epipolar Priors
Lines
Fig. 1. Epipolar priors be unreliable across extremely sparse views, especially in non-overlapping or occluded areas. Our model, eFreeSplat, generalizes to novel scenes without relying on epipolar priors, offering superior appearance and geometric perception.
Abstract
Generalizable 3D Gaussian splitting (3DGS) can reconstruct new scenes from sparse-view observations in feed-forward inference manner, eliminating the need for scene-specific retraining required in conventional 3DGS. However, existing methods rely heavily on epipolar priors, which can be unreliable in complex real- world scenes, particularly in non-overlapping and occluded regions. In this paper, we propose SprrtSppat, an efficient weed-forward 3DGS-based model for generaliz- able novel view synthesis that operates independently of epipolar line constraints. To enhance multiview feature extraction with 3D perception, we employ a self- supervised Vision Transformer (ViT) with cross-view completion pre-training large-scale datasets. Additionally, we introduce an Iterative Cross-view Gaussians Alignment method ensure consistent depth scales across different views. Our eFreeSplat represents an innovative approach for generalizable novel view synthe- sis. Different from the existing pure geometry-free methods, eFFreeSplat focuses more on achieving epipolar-free free feature matching and encoding by providing 3D priors through crews-view pretraining. We evaluate eFreeSplat on wiselbaseline novel view synthesis tasks using the RealEstate 10K and ACID datasets. Extensive experiments demonstrate that eFreeSplat surpasses ofte-oo-theear baselines that rely on epipolar priors, achieving superior geometry reconstruction and novel view synthesis quality. Project page: https://tatakai I. giituub.iolefressplal.
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Rendering novel views from sparse observations has long been a challenging research task in the 3D vision community. Recently, generalizable novel view synthesis (GNVS) techniques have emerged as a promising solution. These models, trained on large-scale multiview datasets, can directly synthesize novel views of new scenes from a few observations, eliminating the need for scene-specific retraining. Notable works in this vein include NeRF-based GNVS 38, 55, 64] and Light Field Network- based GNVS [12, 48, 49]. An enabling factor in their generalizability is the use of epipolar priors, which help determine the precise location of pixel in one image on the corresponding epipolar line in another viewpoint [17, 70]. More recently, generalizable 3D Gaussian splatting methods, such as pixeISplat [6] and MVSplat [9], have been proposed. These methods leverage the benefits of a primitive-based 3D representation, offering fast and memory-efficiet rendering along with an interpretable 3D structure for generalizable view synthesis. Like previous approaches, most DDGSS-baeed GNVS methods [6, 9, 61] depend on epipolar priors to achieve high-quality and fast cross-scene novel view rendering.
Despite significant advancements utilizing epipolar priors, a new and underexplored issue has emerged in GNVS: epipolar priors prove unreliable in non-ovellapping and occluded regions of complex real- world scenes, where corresponding points on epipolar lines are absent. As depicted in Fig. I, epipolar lines (marked in green) effectively identify geometric correspondences in multiview overlapping areas. Conversely, epipolar lines (marked in red) become invalid in those non non-overlapping regions, leading to unreliable geometric reconstructions. Moreover, sampling on invalid epipolar lines and employing attention mechanism will produce a lot of redundant calculations [6, 38, 49].
newly proposed geometry-free 3D reconstruction method [56], which captures multiview consistent knowledge from a versatile model pre-trained on cross-view data, has inspired our development of novel GNVS method that circumvents the dependence on epipolar priors through data-driven 3D priors. Leveraging this insight, we propose eFrreSpplat an efficient feed-forward 3D Gaussian Splatting model for GNVS that operates independently of epipolar line priors. eFreeSplat built upon 3DGS [23] originally designed for single-scene NVS and extends its advantages to GNVS. The overview of our method illustrated in Fig. To capture 3D structural information across sparse views without unreliable epipolar priors, we utilize self-supervised pre-training model for 3D vrew completion [59, 60). This model uses Vision Transformer (ViT) [11] encoder and cross-attention decoder to predict parts of the masked images from reference views. In eFreeSplat, the pre-training model retains all patches, effectively capturing spatial relationships and acting as "cross-view mutual perceiver", This approach provides robust geometric biases for global 3D representation via cross-view view completion pre-training on large-scale datasets [25, 35, 40, 44, 45].
Experimentally, we found that without an explicit 3D constraint, the scale of predicted depth maps of per-pixel 3D points from different views tends to be inconsistent [4, 53], leading to artifacts or pixel displacement in images from novel views. To address the issue of inconsistent depth scales across different views, we introduce an Iterative Cross-view Gaussians Alignment (ICGA) technique to epratS ICGA is based on the fact that the features of most surface points projected onto the camera planes of different views remain consistent. Specifically, we obtain the warped features for each view based on the predicted depths via U-Net. We then calculate the fine depths for the next iteration via the correlation between the warped features and the features from other views. Unlike the plane-sweep stereo approach [9, 62, 63]. our updating and alignment strategy does not require numerous depth candidates, thereby reducing computational and storage costs.
The main contributions of this paper are summarized as follows:
We introduce eFreeSplat, method with novel insights into GNVS that operates without relying on epipolar priors in the process of multi-view geometric perception. eFreeSplat demonstrates robustness in generalizing new scenarios with sparse and non-overlapping observations.
To ensure depth scale consistency across different viewpoints without explicit epipolar con- straints, we propose an Iterative Cross-view Gaussians Alignment method, which alleviates artifacts and pixel displacement issues in renderings.
eFreeSplat achieves competitive cross-scene rendering performance on the RealEsttae [72] and ACID [26] datasets, surpassing otate-of-the-art approaches such as pixelSplat [6] and MVSplat [9].
2

---

2 Related Work
Single-Scene 3DGS, 3D Gaussian Splatting (3DGS) [23] marks a significant shift in 3D scene representation. It employs millions of learnable 3D Gaussians to explicitly map spatial coordinates to pixel values, enhancing rendering efficiency and quality via a rasterization- based splatting approach, and boosting various downstream tasks [34, 36.. Unlike early 3D neural representation methods [37, 39, 46] that require intensive computations and large memory usage (e.g., neural fields [2, 3, 67] and volume rendering [27, 65, 66)) 3DGS enables real-time rendering and editability with minimized computational demands [8]. Existing single-scene 3DGS-liked methods [10, 18, 23] demand dense views for each scene via the expensive per-scene gradient back-pru prockk-pppagationn process. our work, we employ single feedforward network to deduce the parameters of Gaussian primitives using merely two images.
Cross-Scene Generalizable 3DGS. Cross-scene generalizable 3DGS learns robust priors from large-scale scenarios to predict Gaussian primitive parameters and render novel view images using sparse inputs. pixelSplat [6] and LatentSplat [61] leverage the epipolar transformer [17] to find cross-view correspondences and learn per-pixel Gaussian depth distributions. However, this can fail in non-overlapping and occluded areas, leading to inaccurate geometry and surface reconstructions. Splatter Image [50] merges Gaussian primitives from single-view regressions but lacks cross-view information, limiting its multiview applications. GPS- EPS-Gaussian [71] and MVSplat [9] improve feature matching with cost volumes for better geometries; however, GPS-Gaussian is limited to human body reconstruction with depth ground truth, and MVSplat, using plane-sweep stereo [28, 29, 62, 63], still relies on the epipolar priors [13, 15, 633. Triplane-Gaussinn [73] encodes single- view images into latent 3D point clouds and triplane features, outputting 3D Gaussian properties via MLP decoders. However, it focuses on single-view reconstruction, with rendering quality dependent on initial geometry. Our method bypasses 3D priors through sampling along epipolar lines or cost volumes, instead using cross- -visw competition pre-training [59, 60] on large-scale datasets [25, 35, 40, 44, 45].
Solving 3D Tasks using Geometry-free Methods. Priors are crucial for visual tasks to provide generalized features [14, 30, 31, 32, 33). Capitalizing on the geometric priors, methods based on re-projection features [21, 51, 64], cost volume [7, 19, 22, 63], and image warping [5] have performed well in downstream 3D activities. However, these methods rely on task-specific designs and struggle with complex scenarios, such as occlusions non-overlapping views. Recently, some geometry- alternatives have been proposed to this challenge. SRT [43] and GS-LRM [68] are epipolar-fres GNVS methods that boldly eschew any explicit geometric inductive biases. SRT encodes patches from all reference views using a Transformer encoder and decodes the RGB color for target rays through Transformer decoder. GS-LRM's network, composed of a large number of Transformer blocks, implicitly learns 3D representations. However, due to the lack of targeted scene encoding, these methods either limited to specific datasets suffer from unacceptable computational efficiency and carbon footprint. Some pose-free GNVS methods [20, 43, 54] are also epipolar- perform epipolar line
free. These methods, lacking known camera poses, find it challenging sampling. They often reduce task complexity through specially designed feature representations (e.g.. Learned 3D Neural Volume in LEAP [20] and Triplane in PF-LRM [54]), but this reduction comes at the cost of decreased model generalization. Different from the above methods, our method focuses on data-driven 3D priors and does not require any consumnnu and complex structured feature representations, such as cost volumes. CroCo [59]. a self-supervised pre- trainininn method for 3D vision tasks, uses cross-view completion to recover occluded parts of an image from different viewpoints without any 3D inductive biases, significantly enhancing downstream 3D vision tasks. DUSI3 [56] introduces a novel paradigm for dense and unconstrained stereo 3D reconstruction from arbitrary image collections, operating without prior information about camera calibration. These geometry-free pioneers pave the way for more adaptable and efficient 3D vision systems capable of performing accurately across diverse and challenging environments.
3 Methodology
3.1
Overview
Our objective is to predict per-pixel 3D Gaussian [23] primitives {ms E., a,, SH using N reference views images { camera intrinsics matrices {Km} and poses matrices {
3

---

Crussxjew Completion Pre- Tee-Triinig
Iterative Cross-view Gaussions Alignment
Novel View Rendering
similaity
Decoder
depch depch Addephh
Epipular-Frox 3D Prioes
3D Gaussion Splatting
1o,
Caursinos Alignnent
Sinndwwcigh VIT
Decoder
ungreject
attention
To, I. SHH
Fpipolar--ere Cress-vion Mutuzl Purcoption
Neretionn
Fig. 2. Overview of eFreeSplat. (a) Epipolar-free Cross-view Mutual Perception leverages self- supervised cross-view completion pre-training [60] to extract robust 3D priors. The ViT [1i] with shared weights processes the reference images, followed by cross-attention decoder to generate multiview feature maps, forming 3D perception without epipolar priors. (b) Iterative Cross-view Gaussians Alignment module iteratively refines Gaussian attributes through 2D U-Net. The process involves warped features to align corresponding features and depths, ensuring consistent depth scales across different views. (c) The final step involves employing rasterization-based volume rendering [23] to generate high-quality geometry and realistic novel view images.
in single feed-forward inference. The 3D Gaussian primitives include position μ., covariance E. opacity a, and spherical harmonics SH for colors. Given a W sized reference image, the number of 3D Gaussian primitives can be calculated as M = N x H x W. The position of the 3D Gaussians μt determines the geometric shape of the scene, which corresponds to pixel u calculated using the camera origin o, the ray direction d,,, and the predicted depth d:
μ4 0 + d du.
where du is calculated by the camera intrinsic and pose matrix: du PKP 11" However, when the number of reference views is extremely sparse, predicting accurate depths d and reconstructing high-quality geometric structures and appearances become particularly challenging. Particularly in non-overlapping and occluded areas, prevalent methods [6, 12] based epipolar line sampling fail to introduce valid geometric priors.
In this paper, we propose eFreeSplat, generalizable 3D Gaussian Splatting model from sparse reference views? that operates independently of epipolar line priors. illustrated in Fig. 2, the pre- trained ViT model based on cross view completion via self-supervised training [59, 60] in large-scale datasets provides robust geometric priors, serving as our Epipolar-free Cross-view Mutual Perception (Sec. 3.2). Unlike recent works [6, 9, 61], which directly combine per-view 3D Gaussians, we propose Iterative Cross-view Gaussians Alignment (ICGA) Sec. 3.3. This module iteratively updates the position and features of Gaussians by calculating the similarity between warped features and corresponding features, alleviating the issues of local geometric inaccuracies caused by inconsistent depth scales. In Sec. 3.4, predict the centers of the 3D Gaussians by unprojecting the aligned depth maps while calculating other 3D Gaussian parameters based on the aligned features.
Epipolar-free Cross-view Mutual Perception
To realize the cross-view mutual preception without relying on the epipolar prior, we extract cross- view image features using a -hight ViT &g, and cross-attention decoder Dø both pre- trained on large-scale cross-view completion tasks in self-supervised manner [60]. Following
In our experiments, the number of reference views N 2, which is consistent with previous methods [6, 9]. For convenience, all subsequent discussions will assume 2-views input scenario.

---

the methodologies of CroCo v2 [60] and ViT [11], both images I1 and I2 are divided into 2n non- overlapping patches via linear projection, with each patch measuring 16 16 pixels. Additionally. relative positional embeddings [47] are added to the RGB patches before inputted into series of stacked Transformer modules for encoding tokens E;
{ = {Ee, (1,)
(2)
After encoding and I2 via VIT independently, the cross-attention decoder Dø, takes E and e2 conditioned on each other for cross-view features F; E R
F (Do, (21, F2 f (Do, (82,
The structure of the cross-attention decoder consists of alternating multi-head self attention blocks and multi-head cross-attention blocks. The mapping function f refers to unflattening the tokens back to the original image size. The multi-head self-attention blocks learn token representations from the first viewpoint, while the multi-head cross-attention blocks facilitate cross-view information exchange conditioned on the token representations from the second view.
The CroCo model [59, 60], as a variant of masked image modeling [1, 16, 58] that leverages cross- view information from the same scene capture the spatial relationship between two images, can significantly enhance performance on 3D downstream tasks. Based on cross-view completion self- supervised pre-training on large-scale datasets, our epipolar-Free cross-view mutual perception method provides robust 3D priors information by understanding the spatial relationship between the two images [59]. Due the randomness of the masking process during pre-training, the pre-traines trained model is capable of reasoning about non-overlapping and occluded areas, which is hard for traditional geometric methods to achieve. Therefore, our epipolar-Free mutual perception possesses a more global and robust feature-matchin inductive bias compared to methods [6, 9, 12, 61] that rely on epipolar line sampling [17] or the plane-sweep stereo approach [63].
3.3 Iterative Cross-view Gaussians Alignment
To address the issue of inconsistent depth scales across different views, we utilize cross-view feature matching information align and update per-pixel Gaussians' centers and features iteratively.
Firstly, we predict per-pixel Gaussians' depths d and features via a 2D U-Net [42] mapping U with cross-view attention, similar to [9]:
91 = U(F
Next, to establish vrews-view correspondences, we endeavor to make the features of each 3D Gaussian point projected onto the known camera planes to be as similar as possible. Taking the first view an example, we calculate the warped features G1,2 of the first view on the second view's 's features map via the predicted coarse depth dj;
(R,'1, RT nf
W1.2 = K2R₂
R
KT
(5)
di
G12((u) G2(M,,uu,,))
(6)
where W denotes the homographic warping matrix. represents pixel location in the first view. R. and t; are the rotation and translation parameters of the camera pose P.. n, refers to the normal
vector of the target plane. We compute the similarity S S" between the warped feature map 91.2 and the corresponding feature map 91 based on Ad S² is obtained by the dot product of 9 and G1,2, where C denotes the feature dimension of the 3D Gaussian primitives.
91,)2, S 91.912
s' = -91 - -
(7)
VC
Finally, we update the coarse per-pixel 3D Gaussian features and predicted depths.
AG = <([ 9 St )) S², Ad, = dd -d.5 S² 9 = 9 Ag d = d, - Ad,
(8)
(9)
5

---

Table 1. Quantitative comparisons. We evaluate our method by rendering three novel view images
from two reference viewpoints for each scene. The performance is determined by averaging across all scenes. The dataset's training and testing split follows the protocol established by pixelSplat [6]. The
inference time includes both scene encoding and rendering time, tested on single RTXX-4990 GPU.
Rstaee 10K [72]
ACID [26]
Inference Time
Methods
PSNRT
SSIM
LPIPS
PSNR
SSIM
LPIPS|
(s)
Du al. [12]
24.78
0.820
0.213
26.88
0.799
0.218
1,578
GPNR [49]
24.11
0.793
0.255
25.28
0.764
0.332
13.180
pixelSplat [6]
25.89
0,858
0.142
28.14
0.839
0.150
0.100
MVSplat [9]
26.39
0.869
0.128
28.25
0.843
0.144
0.046
eFFeeSplat
26.45
0,865
0.126
28.30
0.851
0.140
0.061
where [-]-]] refers to the concatenation operation of tensors. We employ the mapping function şp 220x 1 XW + HX through lightweight convolutional blocks.
The updated features and depths serve as inputs for Eq. (4) (5) and (6), bootstrapping the next iteration of Gaussian updates. Our cross-vics view Gaussians alignment method, during each iteration, involves establishing match for target pixel in the first view with matching pixel in the second view. This process is akin to considering all neighboring pixels of the projected pixel u'y based on the current coarse depth due to the locality inductive bias inherent in convolutions. During each querying process, the discrepancy between 11 and the true matching u2 progressively decreases, thereby harmonizing the consistency of depth scales across multiple views.
3.4 Gaussian Parameters Prediction
We calculate the per-view Gaussians' centers based on the refined depths and camera parameters using Eq. (I). We predict additional Gaussian primitives: E, a. SH via an additional U-Net. Following other HDGS-baseed methods [6, 9, 23], the covariance matrix Σ is composed of scaling matrix and rotation matrix. The spherical harmonic coefficients SH are used compute RGB values given a direction. Since we have harmonized the depth scale across different viewpoints, we Nx directly merge all views' Gaussian primitives {u,, E,, a₁ SH X xW
4 Experiments
4.1 Experimental Settings
Datasets. eFreeSplat is trained on RealEstate10K [72] and ACID [26]. The RealEstate 10K dataset consists of home tour videos, providing a wealth of scenes and a variety viewpoint changes. The ACID dataset contains aerial landscape videos, featuring expansive views and complex terrains. Both datasets provide estimated camera parameters. Following pixelSplat [6], we use the provided training and testing splits and evaluate three novel view images on each test scene.
Evaluation Metrics and Training Losses. We employ standard image quality metrics to validate and compare our results quantitatively: pixel-level PSNR, patch-level SSIM [57], and feature-level LPIPS [69]. During the training phase, the loss is composed of linear combination of MSE and LPIPS loss, with loss weights and 0.05, respectively. Since existing methods conduct experiments 256 256, we also set the resolution of our training and testing images for fair comparison.
Comparison Methods. We compared four feed-forward methods for sparse view novel view synthesis. Du et al. [12] and GPNR [49] are the methods based light field rendering that combines features on epipolar lines aggregated by the epipolar transformer. pixelSplat [6] and MVSplat [9] are the latest ADGS-based models based epipolar sampling and multi-plane sweeping, respectively. Our method compared the qualitative and quantitative results with these four methods.
Implementation details. The VIT-B vision transformer [11] and cross-attention decoder [59] have been pretrained by CroCo v2 [60], which underwent self-supervised cross-view completion training on large-scale datasets [25, 35, 40, 44, 45]. The Iterative Alignment and Updating strategy

---

Ref.
et al. [12]
pixelSplat [6]
MVSplat [9]
eFreeSplat
Ground Truth
Fig. 3. We provide qualitative comparisons on the RealEstate 10K (first four rows) and the ACID (last two rows). Compared to baselines, method produces fewer artifacts in rendering results (red boxes). Moreover, our approach can perform better in non-overlapping areas (Ist, 2nd, 5th and 6th rows) and occluded areas 3th and 4th rows) without relying on unreliable epipolar priors.
is implemented through 2 iterations. All models are trained on 4 RTX-4090 GPUs for 300, 000 iterations using the Adam optimizer [24]. More details are provided in Appendix C.
4.2 Comparative Studies
Image quality comparison. We report quantitative results against baselines [6, 9, 12, 49] on the RealEstate 10K and ACID datasets in Tab. 1. Our method, eFreeSplat, outperforms the SOTA method, MVSplat [9] by 0.06dB in PSNR on the RealEstate 10K dataset and by 0.05dB on the ACID dataset. The evaluation metrics for all baselines are derived from experimental results published in the papers on pixelSplat [6] and MVSplat [9].
7

---

Ref.
pixelSplat [6]
MV [9]
eFreeSplat
Fig. 4. Comparison results about 3D Gaussians (top) and predicted depth maps of the reference viewpoints (bottom). Compared to SOTA 3DGS-based methods [6, 9}, our method achieves higher quality in 3D Gaussian Splatting and produces smoother depth maps.
Table In more challenging scenarios, we classify the RealEstate 10K dataset [72] into three subsets based on the overlap size of the reference images: scenes with an overlap below 0.7, 0.6, and 0.5.
Overlap 0.7
Overlap 0.6 SSIM
Overlap 0.5
Methods
PSNR
SSIM
LPIPS.
PSNR
LPIPS
PSNRT
SSIM
LPIPS.
pixelSplat [6]
25.05
0.852
0.145
24.79
0.849
0.149
24.96
0.846
0.149
MVSplat [9]
25.11
0.854
0.139
24.70
0.841
0.146
24.64
0.840
0.150
eFreeSplat
25.72
0.861
0.132
25.48
0.859
0.135
25.46
0.853
0.139
Our method's qualitative comparison with baselines is illustrated in Fig. Our rendering results show
fewer artifacts or object deformations, especially in non-overiie non-overlapping or occluded areas. Competitive methods like pixeISplat [64] and MVSplat [9], based on sampling along the epipolar lines, produce unreliable reconstructions in these challenging areas. It demonstrates that eFreeSplat provides more robust 3D priors than epipolar priors, offering global 3D perception even in challenging areas.
Geometry quality comparison. As illus- trated in Fig. 4, our method produces higher- quality 3DGS reconstructions and smoother priors without the epipolar priors. pixel- Splat [6]. despite additional finetuning via depth regularization during training, exhibits noticeable artifacts in its reconstructed 3DGS and depth maps. MVSplat [9] generates com- petitive depth maps by building a cost volume representation [63], which directly merges vie-view Gaussians, resulting in significant point cloud shifts. Our method, which does not rely on sampling along epipolar lines or additional depth regularization finetuning, surpasses current SOTA methods in 3DGS reconstruction quality. Please refer to Ap- pendix A for additional comparison and anal- ysis.
cFreeSplat excels
MVSplat excels
Frequenyy
100
0.5
0.6
0.7
0.8
0.9
10
Overlap
Fig. Our method reconstructs more reliable results than MVSplat when the reference views overlap is low. the histogram, the blue bars represent the frequency which method exceeds MVSplat in rendering quality under the current overlap conditions, while the orange bars indicate the opposite.
Performance with Low-overlapped obser- vations. In this section, we analyze the differences between our method and 3DGS- based methods when the reference view- points have lower overlap. First, we counted the number scenes where our method and MVSplat [9] outperform each other in PSNR on the RealEstate 1OK dataset, selecting the top 400 scenes with the largest PSNR differences for

---

Ground Truth
w/o mutual perception
weo Gaussians alignment
w/o pre-training
Ours
Ref.
Fig. 6. Ablations. The first row displays the novel viewpoint images, while the last row shows the reference viewpoints and the depth maps the novel views. Our full model renders higher-quality RGB images and smoother depth maps.
Table
Ablations. All ablation experiments were conducted by training and evaluating on the RealEEstate 10K dataset [72]. Each ablation model was derived from our full model by removing the corresponding modules.
Model
PSNR1
SSIM
LPIPS
eFreeSplat (Full)
26.45
0.865
0.126
w/o mutual perception
22.04
0.723
0,212
w/o Gaussians alignment
23.03
0.758
0,187
w/o pre-training weights
24.81
0.829
0.153
each. As shown in Fig. 5, our method performs better in scenes with more minor viewpoint overlaps, method outperforms other 3DGS
while MVSplat excels when the overlap is close 1. In Tab. baselines [6, 9] in settings with more minor overlaps by 3.1% f in PSNR and 8.6% J. in LPIPS. It confirms the robustness of our method in non-overlapping areas. However, methods based on epipolar priors have advantages in scenes where reference viewpoints are closer, and reconstruction quality declines as the overlap decreases.
4.3 Ablation Studies
As shown in Tab. 3 and Fig. 6, we conducted ablation studies on the eFreeSplat model on the RealEstate dataset. We will detail the analysis in the following three subsections.
Importance of epipolar-free cross-view mutual perception. Epipolar-free cross-view mutual weight ViT [11] and cross-attentin decoder. According to Tab. 3, this module's absence results in a 4.41 decrease in PSNR. In Fig. 6, the absence of cross-view mutual perception results in significant offsets in the depth map and noticeable artifacts.
perception extracts cross-view image features using shared-weight
Importance of iterative cross-view Gaussians alignment. Iterative cross-view Gaussian alignment updates per-pixel Gaussian features and depths through warped U-Net features, thereby aligning the view 3D Gaussian point clouds. The lack of Gaussian alignment can lead to pixel displacement or unreliable local geometric details (e.g., the lamp's 's position in Fig 6). Additionally, we conducted extra experiments with to 3 iterations. As shown in Fig. 7, using iterations significantly reduces artifacts and inconsistent depth in novel view rendering. This validates that the iterative mode helps align the depth scale across multiple views. When the iteration count increases 3, there is no notable improvement reconstruction and rendering quality. For further analysis and results, please refer Appendix B.
Importance self-supervised cross-view completion pre-training. In Fig. 6, the absence of cross-view completion pre-training weights results in unaccuracy depth maps. Self-supervised pre-

---

Ground Truth
wro Gaussians alignment
Iteration
Iteration
Iteration
Ref.
Fig. 7. Ablation of gaussians alignment module. Additional iterations can significantly aid in aligning the depth scale and reducing artifacts that occur during novel view rendering.
training by cross-view completion [60] on large-scale datasets allows our model to perceive spatial correspondences, thereby enabling it to predict more reliable and smoother depth maps.
Conclusion
Our work introduces eFreeSplat, novel generalizable 3D Gaussian Splatting model tailored for novel view synthesis across new scenes, designed to function independently of epipolar constraints that might be unreliable when large viewpoint changes occur. By leveraging a Vision Transformer archi- tecture self-supervised pre-trained by cross-view completion [60] on large-scale datasets, eFreeSplat excels in handling sparse and challenging viewing conditions that traditional methods [17, 63) strug- gle with. This model's s ability to unify the consistency of depth scales across different views marks significant improvement over existing techniques, effectively addressing issues like artifacts and misalignment rendered images. Our experiments have demonstrated that our method provides high-quality geometric reconstructions and novel viewpoint images. In settings with large bascline from 2-view inputs, it outperforms the latest state-of tfth--of-the-rt methods [6, 9] that rely on epipolar priors.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (62293554, 62206249, U2336212). "Pioneer" and "Leading Goose" R&D Program Zhejiang (2024C01073), Ningbo Innovation "Yongjiang 2035" Key Research and Development Programme (2024Z292).
10

---

References
[1] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin. Michael G. Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-efficient Learning. CoRR, abs/2204.07 141, 2022, 5
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasam. Mip-NeRF; A Multiscale Representation for Anti-aliasing Neural Radiance Fields. CoRR, 0os/2103.13415, 2021. 3
[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-aliased Neural Radiance Fields. CoRR, abs/21 1.777 2021.
[4] Jia- Wang Bian. Zhichao Li. Naiyan Wang. Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian D. Reid. Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video. CoRR, abs/1908. 10553, 2019. 2
[5] Thomas Brox. Andrés Bruhn, Nils Papenberg. and Joachim Weickert High Accuracy Optical Flow Estimation Based a Theory for Warping. In Computer Vision- ECCV 2004, 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004, Proceedings, Part IV, pages 25-36. Springer, 2004.
[6] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CORR, abs/2312.12337,, 2023. 1, 2, 4,
5, 6, 7, 8, 9, 10, 15, 16, 18
[7] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. MVS- NeRF; Fast Generalizable Radiance Field Reconstruction from Multi-view Stereo. CoRR, abs/2103.
2021. 3
[8] Guikun Chen and Wenguan Wang. A Survey on 3D Gaussian Splatting. CoRR, abs/2401 .03890, 2024. 3
[9] Yuedong Chen, Haofei Xu, Chuanxia Zheng. Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images. CoRR, abs/2403. 14627, 2024. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 18
[10] Kai Cheng, Xiaoxiao Long, Kaizhi Yang. Yao Yao, Wei Yin, Yuexin Ma, Weaping Wang. and Xuejin Chen. GaussianPro: 3D Gaussian Splatting with Progressive Propagation. CoRR, abs/2402. 14650, 2024. 3 [11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. CoRR.
abs/2010.11922 2020. 2, 4, 5, 6, 9, 18
[12] Yilun Du Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to Render Novel Views from
Wide-baseline Stereo Pairs CoRR, abs/2304.08446 2023. 2, 4. S, 6, 7, 15
[13] Ziyue Feng. Leon Yang. Pengsheng Guo, and Bing Li. CVRecon: Rethinking 3D Geometric Feature Learning For Neural Reconstruction. CoRR, abs/2304.1466 2023. 3
[14] Yuan Gan, Ruijie Quan, and Yawei Luo. Expavatar: High-fidelity avatar generation of unseen expressions with 3d face priors. ACM Transactions on Multimedia Computing, Communications and Applications. 2024. 3
[15] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade Cost Volume for High-resolution Multi-view Stereo and Stereo Matching. CoRR, Abss/1912.06311 2019. 3
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked Autoencoders Are Scalable Vision Learners. CoRR, 0abs/2111.0333 20217,2021.
2021. 5
[17] Yihui Hc, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar Transformers. CoRR, abs/2005.04551
2020. 2, 3,5, 5, 10
[18] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2D Gaussian Splatting for
Geometrically Accurate Radiance Fields. CoRR, abs/2403. 17888, 2024. 3
[19] Zhaoyang Huang. Xiaoyu Shi, Chao Zhang, Qiang Wang. Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. FlowFormer: A Transformer Architecture for Optical Flow. CoRR, ahs/2203.16 16194,1114444 2022.
[20] Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXtv:2310.014111 2023.
[21] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. GeoNeRF: Generalizing NeRF with Geometry Priors. CoRR, ahs/21 1.15539, 2021.
[22] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end Learning of Geometry and Context for Deep Stereo Regression. CoRR.
abs/1703.04409, 24309,201..3 3
[23] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-time Radiance Field Rendering. CoRR, abs/2308. 004099 2023. 2,
[24] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 7

---

[25] Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang. Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation.
CoRR, abs/2203. 11433, 2022. 2. 3, 6, 16
[26] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa.
Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image. CoRR, 20000/00,,0955,
2020. 2, 6, 16
[27] Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh. Neural Volumes: Learning Dynamic Renderable Volumes from Images. CoRR, abss/190.00751..
2019. 3
[28] Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo. P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10452-10461, 2019. 3
[29] Keyang Luo. Tao Guan, Lili Ju. Yuesong Wang. Zhuo Chen, and Yawei Luo. Attention multi view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1590-1599, 2020. 3
[30] Yawei Luo and Yi Yang. Large language model and domain-specific model collaboration for smart education. Frontiers of Information Technology & Flectronic Figgineringg, 25(3):333-341, 2024. 3
[31] Yawei Luo, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, 2019. 3
[32] Yawei Luo, Ping Liu, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Category- adversarial adaptation for semantic segmentation using purified features. IEEF Transactions on Pattern Analysis and Machine Intelligence, 2021. 3
[33] Yawei Luo, Ping Liu, and Yi Yang. Kill two birds with one stone: Domain generalization for semantic segmentation via network pruning. International Journal of Computer Vision, 2024. 3
[34] Shaojie Ma, Yawei Luo, and Yi Yang. Reconstructing and simulating dynamic 3d objects with mesh- adsorbed gaussian splatting, arXiv preprint arXiv:2406. 01593, 2024. 3
[35] Nikolaus Mayer, Eddy 1lg. Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene
Flow Estimation. CoRR, abs/1512... 0ttp/1512.0214 2015. 2, 3, 6, 16
[36] Qiaowei Miao, Yawei Luo, and Yi Yang. Pla4d: Pixel-level alignments for text-to-4d gaussian splatting.
arXiv preprint arXiv:2405. /9957, 2024. 3
[37] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NcRF: Representing Scenes as Neural Radiance Fields for View Synthesis. CoRR, abs/2003.088 009934, 2020. 2,
[38] Zhiyuan Min, Yawei Luo, Wei Yang. Yuesong Wang, and Yi Yang. Entangled view-epipolar information aggregation for generalizable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4906-4916, 2024, 2
[39] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with Multiresolution Hash Encoding. CoRR, abs/2201. .05989, 2022. 3
[40] Pierluigi Zama Ramirez, Fabio Tosi, Matteo Poggi. Samuele Salti, Stefano Mattoccia, and Luigi Di Stefano.
Open Challenges in Deep Stereo: the Booster Dataset. CoRR, abs/2206.00 00677,, 2022. 2, 3. 6, 16
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution
Image Synthesis with I Latent Diffusion Models. CoRR, abb/2112..1752. 2021. 18
[42] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted InterventionMICCAAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[43] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario I Læčić Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022. 3
[44] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nesic, Xi Wang, and Porter Westling. High-resolutton Stereo Datasets with aubpixel-accurate accurate Ground Truth. In Pattern Recognition 36th German Conference, GCPR 2014, Münster. Germany, September 2-5, 2014, Proceedings, pages
31-42. Springer, 2014. 2, 3, 6, 16
[45] Thomas Schöps, Johannes I. Schönherger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Mare Pollefeys, and Andreas Geiger. A Multi-view Stereo Benchmark with High-resolution Images and Multi- camera Videos. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 2538-2544 2547. IEEE Computer Society. 2017. 3.6, 16 [46] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießnnrr Gordon Wetzstein, and Michael Zollhöfer. DeepVoxels: Learning Persistent 3D Feature Embeddings. CoRR, 1b2/1812.01 2018. 3
12

---

[47]
Jianlin Su, Yu Lu. Shengfeng Pan. Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. CoRR, abs/2104,09864, 2021. 5
[48] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light Field Neural Rendering.
CoRR,
2021. 2
[49] Mohammed Suhail. Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable Patch-based Neural
Rendering. CoRR, abs/2207. 10662. 2022. 2, 6,
[50] Stanislaw Srymanowiez, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view
3d reconstruction. arXiv preprint arXiv:2312.13155 20..3. 3
2023.
[51] Mukund Varma T. Peihao Wang. Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is Attention All That NeRF Needs? The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.. 3
[52] Jaxiing Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. LGM: Large Multi-view Gaussian Model for High-resolution 3D Content Creation. CoRR, abs/2402. 0550544 2024. 18 [53] Lijun Wang. Yifan Wang, Linzhao Wang. Yunlong Zhan, Ying Wang, and Huchuan Lu. Can Scale- consistent Monocular Depth Be Learned in Self-supervised Scale-invariamt Manner? In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021,
pages 12707-12716. IEEE, 2021. 2
[54] Peng Wang, Hao Tan, Sai Bí, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv
preprint 1. I. 12024, 2023. 3
[55] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas A. Funkhouser. IBRNet: Learning Multi-view
Image-based Rendering. CoRR, abs/2102. 13090, 2021. 2
[56] Shuzhe Wang. Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. DUS(3R: Geometric
3D Vision Made Easy. CoRR, abs/2312. 141322 2023. 3.
[57] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from
ertor visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004. 6
[58] Chen Wei, Haoqi Fan, Saining Xic, Chao-Yuse Wu, Alan L. Yuille, and Christoph Feichtenhofer. Masked Feature Prediction for Self- supervieee Visual Pre-training. CoRR, abs/2112.09113. 2021. 5
[59] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. CroCo: Self-supervised Pre- training for 3D Vision Tasks by Cross-view Completion. CoRR, abs/2210.1071e 2022. 2, 3, 4,5, 5, 6, 16,
18
[60] Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre- -rettraining for Stereo Matching and Optical Flow. In JEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 17923 17934. IEEE, 2023. 2, 4, 6, 10, 16
[61] Christopher Wewer, Kevin Raj. Eddy Ilg. Bernt Schiele, and Jan Eric Lenssen. latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction. CoRR, abs/2403. 16292, 2024. 2, 3, 4, 5
[62] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying Flow, Stereo and Depth Estimation. CoRR, abs/2211 01/222110550 005783, 2022. 2,3
[63] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo. CoRR, abs/ 804.02505. 2018, 2, 3, 8, 10
[64] Alex Yu. Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields from One or Few Images. CoRR, abs/2012.02199 2020. 2, 8
[65] Alex Yu, Sara Fridovich-Keil Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. CORR, abs/2112.0513/1 2021. 3
[66] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer
Vision. pages 5752-5761, 2021. 3
[67] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and Improving Neural Radiance Fields. CoRR, abs/2010.07492. 2020.
[68] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatring. arXiv preprint arXiv:2404. 19702, 2024. 3
[69] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features Perceptual Metric. CoRR, abs/1801 .392449244... 2018. 6
[70] Zhengyou Zhang. Determining the Epipolar Geometry and its Uncertainty: A Review. Int. J. Comput. Vis., 1792)1611115, 1998. 2
[71] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gas-Saassi Generalizable Pixel- wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis. CoRR, ht.b///31.02155.... 2023. 3
13

---

[72] Tinghui Zhou, Richard Tucker, John Flynn. Graham Fyffe. and Noah Snavely. Stereo Magnification:
Learning View Synthesis using Multiplane Images. CoRR, abs/1805.0983 2018. 2, 6, 8, 9, 15, 16
[73] Zi-Xin Zou, Zhipeng Yu. Yuan-Chen Guo, Yangguang Li. Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane Meets Gaussian Splatting: Fast and Generalizable Single-view 3D Reconstruction with Transformers. CoRR, abs/2312.09147,, 2023.
14

---

A Additional Experimental results
We provide additional qualitative comparisons against baselines. The visualization results on the RealEstate are shown in Fig. 8. Additionally, we provide more geometry reconstruction compari- son results, as shown in Fig. 9. Our method reconstructs high-quality 3DGS without using epipolar priors or depth regularization finetuning.
Ref.
al. [12]
pixelSplat [6]
MVSplat [9]
eFreeSplat
Ground Truth
Fig. Additional visualization results on the RealEstate 10K [72]. Our method, eFreeSplat, outper- forms baselines in rendering results, producing fewer artifacts and scene distortions.
15

---

Ref.
pixelSplat [6]
MVSplat [9]
eFreeSplat
Fig. 9. Additional geometry reconstruction quality comparison results. Our method achieves higher quality 3D Gaussian Splatting and produces smoother depth maps than pixelSplat [6] and MVSplat [9].
Table 4. Quantitative results of gaussians alignment module under the settings to "Memory" refers to GPU memory usage, and "TTme" indicates the inference time.
Iterations
PSNRT
SSIM
LPIPS.
Memory(M)
Times(s)
23.36
0.768
0.182
2410
0.058
2
26.45
0.865
0.126
2452
0.061
26.40
0.861
0.126
2488
0.086
Ground Truth
wro Gaussians aligument
Iteration
Iteration
Iteration
Ref.
Fig. 10. Visualization of gaussians alignment module under the settings of
Additional Experimental Analysis
More ablations. this section, we provide both quantitative and qualitative results of the Gaussians Alignment module under the settings of to iterations. As shown in Tab. 4 and Fig. 10, setting the iteration count to effectively reduces artifacts caused by inconsistent depth scales. When the iteration count is set to 3, we had to reduce the model's parameter size and batch size to avoid OOM errors, which might be one of the reasons for the lack of significant improvement in image reconstruction metrics.
Failure cases. Our method relies on the prior knowledge provided by CroCo [59] pre-trained weights. However, the input viewpoint overlap in the pre-trained dataset does not exceed 0.75 [60], while the input viewpoint overlap the RealEstate 10K and ACID datasets mainly ranges from 0.9 to 1.0. As shown in Fig. 11, our method renders unreliable results when the input viewpoints are very close, which can be attributed to the distribution bias between the GNVS dataset [26, 72] and the pre-trained dataset [25, 35, 40, 44, 45].
16

---

Ground Truth
eFreeSplat
Exp.A
w/o pre-training
Exp.B
Ref.
Fig. 11. Failure cases. Our method may produce unstable results in scenarios where the input viewpoints are very close. Exp.A and Exp.B indicate that fine-tuning the CroCo model on RelOk helps mitigate this issue.
Table 5. Exp. A involves fine-tuning the CroCo pretrained weights using the REIOK training set, while Exp. trains CroCo directly using the REIOK training set without loading the pretrained weights. w/o pre-training refers to neither using CroCo pre-trained weights nor performing fine- tuning.
Model
PSNR1
SSIM1
LPIPS
raw eFreeSplat
26.45
0.865
0.126
Exp.A
26.32
0.862
0.129
w/o pre-training
24.81
0.829
0.153
Exp.B
25.12
0.839
0.144
Limitations. Our method lacks geometric inductive biases, our model is data data-hungry and sensitive to the training data distribution. Joint training with richer multiview datasets across different scenes could be viable direction. Additionally, the per-pixel 3D Gaussian mapping struggles to reconstruct parts of the scene that are occluded or missing from input viewpoints, such as an obscured chair. Therefore, introducing high-level features for scene completion might be future research direction for generalizable 3D Gaussian Splatting work.
Fine-tuning of the CroCo model. aforementioned limitation. We conducted relevant Experiments and B regarding fine-tuning the CroCo model using the RE10K dataset. Experiment A involved fine-tuning the CroCo pretrained weights with the RE10K training set, while Experiment involved training CroCo directly with the RE1OK training set without loading the pretrained weights. Finally, retrained eFreeSplat using new pretrained weights. As shown in Fig. and Tab. 5, the results indicate that pretraining the backbone model on the RE10K training set effectively addresses the model's 's poor performance in low-overlap scenarios. However, in the RE10K test set, Experiment A's reconstruction metrics were slightly lower than those of the original model, which may be due to insufficient training iterations. We will further investigate the positive impact fine-tuning the CroCo pretrained model on novel view synthesis and 3D reconstruction in future work.
We have conducted preliminary explorations to address the
Potential negative societal impacts. Our model could be misused for unethical purposes, such as creating false evidence manipulating media, which threatens information integrity and personal privacy. Additionally, the model introduces security risks contexts like autonomous driving, as may produce incorrect reconstructions in real and complex scenarios. These concerns underscore the importance of implementing stringent ethical guidelines and security measures when deploying such technology, to prevent misuse and ensure that it used responsibly.
17

---

C Additional Implementation Details
The cross-attention decoder. Following the CrossBlock decoder architecture in CroCo [59], the cross-attention decoder comprises self-attention module and cross- cross-attention module. Let
e1.82 E IR NxC be the tokens of the two viewpoints outputted by Vision Transformer [11]. The computation process of the decoder is as follows:
ē; = LayerNorm (e;),
1,2
E = E; + Attention (6,,
= 1,2 (10)
E e + Attention (LayerNorm (2,
22 e/ + Attention (LayerNorm ,1,
output, e" MLP (LayerNorm
1,2
In Equations 10, Attention is derived from the classic attention computation. The inputs Q, K, V
undergo projection transformations using Wg, Wk, Wo:
Q' ZW=WQQ K' = WoK, = WWV,
Q'K'T
(1I)
Attention (Q, K, V) Linear
softmax
VC V
The cross-view U-Net. For the Gaussian Alignment Strategy and the prediction of Gaussian primitives, we utilize a 2D Cross-Vies U-Net inspired by [41, 52], 2024. We concatenate and flatten multiview feature maps for cross-view information exchange, similar to the structure of the U-Net used for cost volume refinement in MVSplat [9]. Specifically, for the Gaussian Alignment Strategy, we apply four times of 2 x down-sampling and add attention the 16 x down-sampled level, with the channel dimensions being [32. 32, 64, 128, 256]. For the prediction of Gaussian primitives, we keep the channel dimension fixed at 32, while the rest of the architecture remains the as that of the U-Net used in the Gaussian Alignment Strategy.
More training details. Our model is trained and tested on 4 RTX-4090 GPUs using the Adam optimizer with a learning rate 2c-4. The per-GPU batch size during training is 4. Similar to pixelSplat [6], the distance between the two input viewpoints gradually increases throughout training. However, to learn more robust 3D prior information, our setup allows for a maximum viewpoint distance of 60 frames, compared to the 45 frames used by pixelSplat [6] and MVSplat [9].
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: Our contributions and scope are claimed clearly in the abstract and introduction (last paragraph).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the Limitation section in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We introduce our method and experimental details in the paper. We will open source the code and provide model weights in the future.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification:
Our datasets is open source, and we will open source our code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct / wwwinnoolll. onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail all the training and test details in the Experiments and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include statistical experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerp//.///ee cc
Answer:
[Yes]
Justification: Our research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appenidx B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow the license CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "v3ijwnSiGI", "reviewer": "Reviewer_w4rj", "summary": "This work presents eFreeSplat, a model to address generalizable novel view synthesis without relying on epipolar prior. Specifically, it extracts the cross-view mutual perception by leveraging a pre-trained CroCo model. It then improves the alignment of multi-view Gaussians by using an iterative updating strategy. Finally, novel views are rendered by an off-the-shelf 3DGS renderer. Experiments on two benchmarks, RE10K and ACID, demonstrate the effectiveness of the introduced eFreeSplat.", "questions": "Kindly refer to the [Weaknesses]", "limitations": "Kindly refer to the [Weaknesses]", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation of using the pre-trained model to address the non-overlapping and occluded regions is interesting.\n\n* Extensive experiments showcase that eFreeSplat achieves significantly better results on scenes where input views contain less overlap, which is well-aligned with the motivation.\n\n* The paper is well-written and easy to follow.", "weaknesses": "* Performances regarding cross-dataset generation. Similar to the cross-dataset experiments shown in MVSplat, it would be interesting to see how eFreeSplat performs when trained on RE10K but tested on ACID and DTU. This experiment will help us better understand how such a data-driven approach generalizes across different data distributions.\n\n* It would be better to provide a more detailed analysis of the Iterative Cross-view Gaussians Alignment module. For example, showcase the depth maps of both input views when setting the iteration to 1, 2, and 3. This would be helpful for verifying whether the iterative solution can really help align the depth scale across multiple views.\n\n* Will it be beneficial to perform a fine-tuning of the CroCo model? As mentioned in the failure case, eFreeSplat performs worse on cases with significant overlaps, potentially because CroCo is trained on image pairs with slight overlaps. Can these failure cases be addressed by fine-tuning CroCo on the RE10K training set? Similarly, as reported in Tab. 3, the performance of eFreeSplat w/o pre-training weights is much worse compared to other state-of-the-art models, while MVSplat achieves reasonably good results even when training from scratch. Can this gap be reduced by pre-training the backbone model using the RE10K training set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weaknesses 1. Performances regarding cross-dataset generation**\nThank you for your valuable suggestions on our work. We fully agree with your view on the importance of cross-dataset testing and, following your advice, have added cross-dataset test results on the DTU dataset. For specific experimental results, please see **Table 1 and Figure 1 in the Rebuttal PDF**.\n\nThese tests further validate eFreeSplat's generalization ability across different data distributions. The results demonstrate that even though training was conducted on the RE10K dataset, eFreeSplat performs excellently on the DTU dataset, which further confirms the robustness and generalization performance of our method.\n### **Weaknesses 2. Supplementary analysis of the Iterative Cross-view Gaussians Alignment module**\nThank you for your valuable suggestions. We have added ablation experiments for iteration counts of 1-3 and presented the corresponding depth maps based on your feedback. For specific experimental results, please see **the Table 2 and Figure 2 in the Rebuttal PDF**.\n\nThe results indicate that setting the iteration count to 2 achieves the best balance between reconstruction accuracy and rendering efficiency. When the iteration count is set to 3, there is no significant improvement in image reconstruction metrics, which may be due to training overfitting and the subsequent iterations losing the 3D perception provided by the original CroCo features.\n\n### **Weaknesses 3. Fine-tuning of the CroCo model**\nThank you for your valuable suggestions. We conducted relevant Experiments A and B regarding fine-tuning the CroCo model using the RE10K dataset. Experiment A involved fine-tuning the CroCo pretrained weights with the RE10K training set, while Experiment B involved training CroCo directly with the RE10K training set without loading the pretrained weights. Finally, we retrained eFreeSplat using the new pretrained weights. The CroCo pretraining for Experiments A and B was performed on 2 RTX 4090 GPUs, with total iterations of 4000 and 6000, respectively (due to time constraints), learning rates of 2e-5 and 2e-4, and a batch size of 12. The viewpoint overlap was set the same as in the main model training.\nFor the quantitative and qualitative results of Experiments A and B, please see **Table 3 and Figure 3 in Rebuttal PDF**.\n\nThe results indicate that pretraining the backbone model on the RE10K training set effectively addresses the model's poor performance in low-overlap scenarios. However, in the RE10K test set, Experiment A's reconstruction metrics were slightly lower than those of the original model, which may be due to insufficient training iterations. We will further investigate the positive impact of fine-tuning the CroCo pretrained model on novel view synthesis and 3D reconstruction in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmYPnhnZUt", "reviewer": "Reviewer_MtQr", "summary": "This paper introduces a robust pipeline for generalizable 3D Gaussian novel view synthesis, utilizing a cross-attention model trained on large-scale datasets. This approach enables the model to generalize effectively to new scenes without depending on epipolar geometry, which is commonly used in traditional methods. To address inconsistencies in depth scales, the pipeline features an iterative refinement process that adjusts the attributes of 3D Gaussians based on cross-view feature correspondences.", "questions": "- In formula (7) , the C is used without definition. I assume it is the dimensionality of the feature of 3D Gaussian primitive? It should be made clearer. \n- Since you are using a Croco pre-trained model could you give some metrics on 512 x 512 resolution? \n- Given the significant memory demands of the global alignment process in DUSt3R, could you detail the memory requirements for your method? It would be helpful to understand if similar memory constraints apply to your approach.", "limitations": "The author have made a commendable effort in acknowledging both the technical limitations and potential negative societal impacts of their method.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The motivation is well-grounded. The paper identifies and tackles one of the core problems of the sparse 3D Novel view synthesis. \n\n- The experiments show solid results on image pair settings, especially in sparse and non-overlapping scenarios. The evaluation convinces me of the usefulness of each component.", "weaknesses": "- Overall, the paper has good ideas, but the experiments are not enough to support their claim. The paper claimed to be a multiview generalizable novel view synthesis, but most of the experiments are done in an image pair setting, it should add more experiments in a sparse-view setting and compare with other sparse-view Novel view synthesis methods on datasets like mipnerf360. \n- 59 \"obtain the warped features for each view based on the predicted depths via U-Net\" and formula (5) seems similar to the the coarse-to-fine cost volume formulation utilized in the MVSNet. Could you elaborate on how these two are different?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weaknesses 1. Missing experiments in a sparse-view setting and comparison with other sparse-view methods**\nWe thank you for identifying the novelty of our idea. Nevertheless, maybe we did not illustrate the difference between GNVS and sparse-view NVS clear enough and make you miss the focus of our experiments. Please allow us to reiterate the concept of the generalizable novel view synthesis (GNVS) task:\nThe GNVS task aims to render new viewpoint images by leveraging the generalization ability of cross-scene 3D representations. When encountering new scenes not present in the training set, this process requires no additional single-scene optimization, achieving rendering through a single network feed-forward pass. Therefore, sparse-view novel view synthesis and GNVS are distinct tasks. The methods and datasets we compare in our experiments are commonly used settings in the GNVS task.\nMoreover, eFreeSplat is theoretically straightforward to extend to multi-view inputs. Our choice of datasets and dual-view input settings is solely for fairer comparisons with current GNVS methods. For instance, recent works such as pixelSplat and MVSplat also use dual-view input in their experimental settings, with their datasets being RE10K and ACID. Through fair experiments, we have validated our method's generalization capability and its advantages in reconstructing challenging regions.\n\n### **Weaknesses 2. Differences between Eq. (5) and cost volume construction method in MVSNet**\nHere, we provide a detailed explanation of the differences between Equation (5) and the coarse-to-fine cost volume construction method in MVSNet.\n\nEquation (5) represents a commonly used warped transformation formula in multi-view geometry. Unlike MVSNet, which requires sampling along the canonical view ray N times to obtain N depth values $\\[d_{i}\\]_{i=1}^{N}$  when constructing the cost volume, our method does not require multiple depth direction samples. Specifically, in each iteration, our method directly predicts the coarse depth value $d$ using a U-Net network (as detailed in Equation (4)), rather than sampling each depth plane to calculate the cost volume as MVSNet does.\n\nThe intuition behind this approach stems from the 3D prior knowledge provided by the CroCo pretrained model, which replaces the multiple depth sampling process required in the MVSNet cost volume. By leveraging CroCo's 3D priors, we can effectively obtain multi-view feature point matching relationships in each iteration, simplifying computation and improving efficiency.\n\n### **Q1. Missing definition**\nThank you very much for your thorough review and valuable suggestions. We sincerely apologize for the omission of the definition of $C$ in Equation (7). Your assumption is indeed correct: $C$ represents the feature dimension of the 3D Gaussian primitives. We will include a clear definition of $C$ in the revised manuscript.\n\n### **Q2. More experiments on 512 x 512 resolution**\nThank you for your valuable feedback. The baseline methods use the RE10K and ACID datasets, which have a resolution of 360 x 640, lower than 512 x 512. For a fair comparison, we did not conduct experiments at other resolutions that were different from the baselines and other popular methods.\n\nAccording to our research, current 3DGS-based GNVS methods all use fixed, lower-resolution datasets for testing. Although high resolution and variable resolution are not the focus of this study, they could indeed make models more applicable in real-world scenarios, so we have included them in our future work plans. Thank you again for your suggestion.\n\n### **Q3. The memory requirements**\nThank you for your valuable suggestion. We have added ablation experiment results for iteration counts of 1-3, including comparisons of memory usage, rendering speed, and reconstruction metrics. For details, please see **Table 2 and Figure 2 in the Rebuttal PDF**.\n\nThe experiments show that while our method's memory usage varies with different iteration counts, the overall memory demand remains relatively low. Compared to DUSt3R, our method requires less memory and achieves better rendering efficiency and reconstruction accuracy with two iterations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PEKnVdpvzv", "reviewer": "Reviewer_AnyG", "summary": "This paper addresses the task of 2-view generalizable novel view synthesis. It introduces a cross-view completion model as prior assistance and incorporates cross-view Gaussian alignment after predicting Gaussian attributes to enhance cross-view consistency.", "questions": "1. Table 3 seems to indicate that all three modules are important. In the author's opinion, which module has the greatest impact on the performance improvement?\n2. Does the iterative nature of the Cross-view Gaussians Alignment module slow down the training speed? Is choosing to iterate twice a trade-off between effectiveness and efficiency?\n3. Is Cross-view Gaussians Alignment required during inference?", "limitations": "The author provides a detailed discussion of limitations and social impacts in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of pretraining the cross-view completion prior for assisting the generalizable novel view synthesis task is an interesting and novel approach. \n2. Extensive experiments, including comparisons with baseline methods such as pixelSplat/MVSplat, demonstrate that the proposed approach can generate higher-quality novel view images, particularly in scenes with smaller overlaps (Tab. 2, Fig. 5).", "weaknesses": "The ablation analysis in Tab. 3 does not clearly identify which module contributes the most to the performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1. The most influential module in Table 3**\nThe results in Table 3 show that the absence of the epipolar-free cross-view mutual perception results in the most significant performance decline. This highlights the crucial role of the pretraining and network structure of Croco in providing 3D priors and cross-view features. Additionally, as illustrated in Fig. 6, the exclusion of this module leads to noticeable artifacts in both novel view images and depth maps, underscoring its importance. We will add this analysis to the ablation study  section in the paper.\n\n### **Q2. Supplementary analysis of the Iterative Cross-view Gaussians Alignment module**\nWhile the iterative nature of the Cross-view Gaussians Alignment module does slightly reduce the rendering speed during training or inference, it substantially enhances the reconstruction metrics. The supplementary experiments with iterations set to 1-3, as presented in the table below, demonstrate that iterating twice strikes a balance between effectiveness and efficiency, providing significant improvements without excessively increasing computational demands. For details, please see **Table 2 and Figure 2 in the Rebuttal PDF**.\n\n### **Q3. Whether Cross-view Gaussians Alignment required during inference**\nYes, cross-view Gaussians Alignment is required and plays a crucial role during inference, and it does not require much time (less than 0.061s, as reported in Tab 1 of the manuscript). The scene representation is obtained from input perspective images through Epipolar-free Cross-view Mutual Perception and Cross-view Gaussians Alignment, which matches the scene representation trained by the single-scene optimized 3DGS method, allowing for direct inference of new viewpoints. As shown in Fig. 2 of the manuscript, the pipeline of eFreeSplat remains the same for both the training and testing sets: it first uses Croco-pretrained ViT to provide feature maps with 3D priors, then employs the Cross-view Gaussians Alignment module to acquire pixel-wise 3D Gaussian primitives for the 3D scene representation, and finally, the inference is performed using the original 3DGS tile-based rendering method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HucZ2qKh4I", "reviewer": "Reviewer_1Kz3", "summary": "This paper tackles the generalizable NVS problem from sparse-view inputs. The main claim is that relying on epipolar geometry cues harms the performance, and, thus, it removes these geometric priors. It adopts the pre-training and supervised reconstruction/NVS training pipeline as in Dust3r. The authors verify the performance on RealEstate and ACID.", "questions": "Please see the weaknesses.", "limitations": "The biggest concern is the novelty. This paper is a combination of Duster (cross-view pre-training + 2-view geometry without epipolar prior) and generalizable NVS using Gaussian Splatting (e.g. MVSplat).\n\nI believe the authors need to discuss the prior works more properly and better word their contributions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method demonstrates comparable performance to the current SoTA method MVSplat. \n- I like the idea of not using epipolar geometry priors, but I do have some problems regarding the claims and technique. Please see details in weaknesses.\n- I like the idea of doing ablation with different input view overlap ratio, which is trying to convince me on their claim in the introduction and Figure 1.", "weaknesses": "- Overclaim. The authors claim that the proposed method is the first epipolar-free work (Line 64), but actually there are a lot prior works that are epipolar-free, espeicially for those works doing reconstruction from unposed images where the epipolar lines cannot be computed without the access of ground-truth poses. These works including SRT (the unposed version UpSRT), LEAP [1], PF-LRM [2] and Dust3r+Croco, where they do cross-view attention between different input views. LEAP also shows visualization results of the cross-view attention for implicit cross-view feature matching and correspondence. Besides, there are also some of lightfield-based methods are epipolar-free. I don't know why the authors have already discussed SRT and Dust3r+Croco in the related work section, but they still think they are the first epipolar-free work. All previously mentioned works should be discussed properly.\n\n[1] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024.\n[2] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.\n\n- Related work. More related work on generalizable NVS using gaussian splatting should be discussed, for example, GS-LRM [3].\n\n[3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ECCV 2024.\n\n- Evaluation. The evaluation doesn't verify the claims of the authors. Even though the idea of Table 2 is good, I don't see a clear different of the gain over the epipolar-aware baseline pixelSplat when the overlap is smaller.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weaknesses 1. Overclaim Issue** \nWe would like to clarify any confusion regarding our use of the term \"epipolar-free.\" This term specifically indicates our method's avoidance of epipolar sampling and cost volume techniques, which are commonly used in generalizable novel view synthesis (GNVS) tasks. While approaches like UpSRT, LEAP, and PF-LRM also do not utilize epipolar geometry, our method targets multiview GNVS tasks with precise camera poses. Thus, our approach is epipolar-free within the GNVS context but not entirely devoid of geometric information. As mentioned in the related work section, \"Solving 3D Tasks using Geometry-free Methods,\" our focus is distinct from those methods that are entirely geometry-free, such as LEAP and PF-LRM, which do not directly apply to the GNVS tasks addressed in this paper.\nWe will reiterate our method's contributions and intuition. The novelty of our approach lies in the fact that most current multi-view GNVS methods heavily rely on epipolar line sampling or cost volume, which struggle to provide effective priors in non-overlapping and occluded areas. Therefore, we innovatively utilize a 3D cross-view pretraining model to obtain epipolar-free 3D priors and the cross-view Gaussians Alignment module to acquire accurate Gaussian attribute-matching features. The GNVS experiments demonstrate the advantages of our epipolar-free approach over existing methods dependent on epipolar line sampling or cost volume techniques.\n\n### **Weaknesses 2. Related Work** \nWe appreciate your suggestion to include more related works on GNVS using Gaussian splatting. To the best of our ability, we have discussed 6 recent works on GNVS using Gaussian splatting, except for some studies very close to the submission date, such as GS-LRM. We will add discussions on GS-LRM and other relevant works to provide a more comprehensive context for our contributions.\n\n### **Weaknesses 3. Evaluation** \nWe appreciate your valuable feedback on our evaluation section. We would like to emphasize that our method demonstrates clear advantages over both pixelSplat and MVSplat when the overlap is smaller.\nSpecifically, our method achieves an improvement of 0.5dB PSNR compared to pixelSplat, along with faster rendering speeds. This improvement is comparable to the PSNR gain over pixelSplat reported in the MVSplat paper on the RE10K dataset. According to the evaluation criteria in the MVSplat paper, a PSNR increase of 0.5dB is considered a relatively significant improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback! We are encouraged that they found our motivation for using a pre-trained model to address non-overlapping and occluded regions interesting and well-grounded (w4rj, MtQr, 1Kz3). They also praised our paper's well-written presentation (w4rj) and the novelty of introducing pretraining the cross-view completion prior for assisting in the generalizable novel view synthesis task (AnyG). Additionally, they appreciated the beneficial improvements demonstrated through extensive experiments, particularly in scenarios with sparse and non-overlapping input views (w4rj, MtQr, AnyG). Here, we address the concerned questions below and will incorporate all responses in the revision.

Meta-review: This paper receives borderline reviews. It gets 1x accept, 2x borderline accepts and 1x borderline reject. The reviewers agree that the motivation of using the pre-trained model to address the non-overlapping and occluded regions, and the idea of not using epipolar geometry priors are interesting and well-grounded. Extensive experiments, including comparisons with baseline methods such as pixelSplat/MVSplat, demonstrate that the proposed approach can generate higher-quality novel view images, particularly in scenes with smaller overlaps. The major concern of Reviewer MtQr who gave a borderline reject is that the experiments do not seem enough to support the claims of the paper. This concern seems to be addressed by the authors during the rebuttal. It was pointed out in the rebuttal that "maybe we did not illustrate the difference between GNVS and sparse-view NVS clear enough and make you miss the focus of our experiments". The weaknesses mentioned by other reviewers are mostly clarifications on the experiments, which seem to have been well-addressed during the rebuttal and discussion phase. On the consideration that the strengths outweigh the weaknesses of the paper, the meta-reviewers decide to accept the paper.
Decision: Accept (poster)