Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis

Authors: zhiyuan  min
yawei  luo
jianwen  sun
yi  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:36:26

Abstract: Generalizable 3D Gaussian splitting (3DGS) can reconstruct new scenes from sparse-view observations in a feed-forward inference manner, eliminating the need for scene-specific retraining required in conventional 3DGS. However, existing methods rely heavily on epipolar priors, which can be unreliable in complex real-world scenes, particularly in non-overlapping and occluded regions. In this paper, we propose eFreeSplat, an efficient feed-forward 3DGS-based model for generalizable novel view synthesis that operates independently of epipolar line constraints. To enhance multiview feature extraction with 3D perception, we employ a self-supervised Vision Transformer (ViT) with cross-view completion pre-training on large-scale datasets. Additionally, we introduce an Iterative Cross-view Gaussians Alignment method to ensure consistent depth scales across different views. Our eFreeSplat represents a new paradigm for generalizable novel view synthesis. We evaluate eFreeSplat on wide-baseline novel view synthesis tasks using the RealEstate10K and ACID datasets. Extensive experiments demonstrate that eFreeSplat surpasses state-of-the-art baselines that rely on epipolar priors, achieving superior geometry reconstruction and novel view synthesis quality.

Main paper: 
 introduction: 
 Rendering novel views from sparse observations has long been a challenging research task in the 3D vision community. Recently, generalizable novel view synthesis (GNVS) techniques have emerged as a promising solution. These models, trained on large-scale multiview datasets, can directly synthesize novel views of new scenes from a few observations, eliminating the need for scene-specific retraining. Notable works in this vein include NeRF-based GNVS [37, 38, 55, 64] and Light Field Networkbased GNVS [12, 48, 49 ]. An enabling factor in their generalizability is the use of epipolar priors, which help determine the precise location of a pixel in one image on the corresponding epipolar line in another viewpoint [17, 70] . More recently, generalizable 3D Gaussian splatting methods, such as pixelSplat [6] and MVSplat [9] , have been proposed. These methods leverage the benefits of a primitive-based 3D representation, offering fast and memory-efficient rendering along with an interpretable 3D structure for generalizable view synthesis. Like previous approaches, most 3DGS-based GNVS methods [6, 9, 61] depend on epipolar priors to achieve high-quality and fast cross-scene novel view rendering. Despite significant advancements utilizing epipolar priors, a new and underexplored issue has emerged in GNVS: epipolar priors prove unreliable in non-overlapping and occluded regions of complex realworld scenes, where corresponding points on epipolar lines are absent. As depicted in Fig. 1 , epipolar lines (marked in green) effectively identify geometric correspondences in multiview overlapping areas. Conversely, epipolar lines (marked in red) become invalid in those non-overlapping regions, leading to unreliable geometric reconstructions. Moreover, sampling on invalid epipolar lines and employing attention mechanism will produce a lot of redundant calculations [6, 38, 49] .A newly proposed geometry-free 3D reconstruction method [56] , which captures multiview consistent knowledge from a versatile model pre-trained on cross-view data, has inspired our development of a novel GNVS method that circumvents the dependence on epipolar priors through data-driven 3D priors. Leveraging this insight, we propose eFreeSplat, an efficient feed-forward 3D Gaussian Splatting model for GNVS that operates independently of epipolar line priors. eFreeSplat is built upon 3DGS [23] originally designed for single-scene NVS and extends its advantages to GNVS. The overview of our method is illustrated in Fig. 2 . To capture 3D structural information across sparse views without unreliable epipolar priors, we utilize a self-supervised pre-training model for 3D cross-view completion [59, 60] . This model uses a Vision Transformer (ViT) [11] encoder and cross-attention decoder to predict parts of the masked images from reference views. In eFreeSplat, the pre-training model retains all patches, effectively capturing spatial relationships and acting as a "cross-view mutual perceiver". This approach provides robust geometric biases for global 3D representation via cross-view completion pre-training on large-scale datasets [25, 35, 40, 44, 45] .Experimentally, we found that without an explicit 3D constraint, the scale of predicted depth maps of per-pixel 3D points from different views tends to be inconsistent [4, 53] , leading to artifacts or pixel displacement in images from novel views. To address the issue of inconsistent depth scales across different views, we introduce an Iterative Cross-view Gaussians Alignment (ICGA) technique to eFreeSplat. ICGA is based on the fact that the features of most surface points projected onto the camera planes of different views remain consistent. Specifically, we obtain the warped features for each view based on the predicted depths via U-Net. We then calculate the fine depths for the next iteration via the correlation between the warped features and the features from other views. Unlike the plane-sweep stereo approach [9, 62, 63] , our updating and alignment strategy does not require numerous depth candidates, thereby reducing computational and storage costs.The main contributions of this paper are summarized as follows:• We introduce eFreeSplat, a method with novel insights into GNVS that operates without relying on epipolar priors in the process of multi-view geometric perception. eFreeSplat demonstrates robustness in generalizing to new scenarios with sparse and non-overlapping observations. • To ensure depth scale consistency across different viewpoints without explicit epipolar constraints, we propose an Iterative Cross-view Gaussians Alignment method, which alleviates artifacts and pixel displacement issues in renderings. • eFreeSplat achieves competitive cross-scene rendering performance on the RealEstate10K [72] and ACID [26] datasets, surpassing state-of-the-art approaches such as pixelSplat [6] and MVSplat [9] . 
related work: 
 Single-Scene 3DGS. 3D Gaussian Splatting (3DGS) [23] marks a significant shift in 3D scene representation. It employs millions of learnable 3D Gaussians to explicitly map spatial coordinates to pixel values, enhancing rendering efficiency and quality via a rasterization-based splatting approach, and boosting various downstream tasks [34, 36] . Unlike early 3D neural representation methods [37, 39, 46] that require intensive computations and large memory usage (e.g., neural fields [2, 3, 67] and volume rendering [27, 65, 66] ) , 3DGS enables real-time rendering and editability with minimized computational demands [8] . Existing single-scene 3DGS-liked methods [10, 18, 23] demand dense views for each scene via the expensive per-scene gradient back-propagation process. In our work, we employ a single feedforward network to deduce the parameters of Gaussian primitives using merely two images.Cross-Scene Generalizable 3DGS. Cross-scene generalizable 3DGS learns robust priors from large-scale scenarios to predict Gaussian primitive parameters and render novel view images using sparse inputs. pixelSplat [6] and LatentSplat [61] leverage the epipolar transformer [17] to find cross-view correspondences and learn per-pixel Gaussian depth distributions. However, this can fail in non-overlapping and occluded areas, leading to inaccurate geometry and surface reconstructions.Splatter Image [50] merges Gaussian primitives from single-view regressions but lacks cross-view information, limiting its multiview applications. GPS-Gaussian [71] and MVSplat [9] improve feature matching with cost volumes for better geometries; however, GPS-Gaussian is limited to human body reconstruction with depth ground truth, and MVSplat, using plane-sweep stereo [28, 29, 62, 63] , still relies on the epipolar priors [13, 15, 63] . Triplane-Gaussian [73] encodes single-view images into latent 3D point clouds and triplane features, outputting 3D Gaussian properties via MLP decoders. However, it focuses on single-view reconstruction, with rendering quality dependent on initial geometry. Our method bypasses 3D priors through sampling along epipolar lines or cost volumes, instead using cross-view competition pre-training [59, 60] on large-scale datasets [25, 35, 40, 44, 45] .Solving 3D Tasks using Geometry-free Methods. Priors are crucial for visual tasks to provide generalized features [14, 30, 31, 32, 33] . Capitalizing on the geometric priors, methods based on re-projection features [21, 51, 64] , cost volume [7, 19, 22, 63] , and image warping [5] have performed well in downstream 3D activities. However, these methods rely on task-specific designs and struggle with complex scenarios, such as occlusions or non-overlapping views. Recently, some geometry-free alternatives have been proposed to this challenge. SRT [43] and GS-LRM [68] are epipolar-free GNVS methods that boldly eschew any explicit geometric inductive biases. SRT encodes patches from all reference views using a Transformer encoder and decodes the RGB color for target rays through a Transformer decoder. GS-LRM's network, composed of a large number of Transformer blocks, implicitly learns 3D representations. However, due to the lack of targeted scene encoding, these methods are either limited to specific datasets or suffer from unacceptable computational efficiency and carbon footprint. Some pose-free GNVS methods [20, 43, 54] are also epipolarfree. These methods, lacking known camera poses, find it challenging to perform epipolar line sampling. They often reduce task complexity through specially designed feature representations (e.g., Learned 3D Neural Volume in LEAP [20] and Triplane in PF-LRM [54] ), but this reduction comes at the cost of decreased model generalization. Different from the above methods, our method focuses on data-driven 3D priors and does not require any time-consuming and complex structured feature representations, such as cost volumes. CroCo [59] , a self-supervised pre-training method for 3D vision tasks, uses cross-view completion to recover occluded parts of an image from different viewpoints without any 3D inductive biases, significantly enhancing downstream 3D vision tasks. DUSt3R [56] introduces a novel paradigm for dense and unconstrained stereo 3D reconstruction from arbitrary image collections, operating without prior information about camera calibration. These geometry-free pioneers pave the way for more adaptable and efficient 3D vision systems capable of performing accurately across diverse and challenging environments.3 Methodology 
overview: 
 Our objective is to predict per-pixel 3D Gaussian [23] [60] to extract robust 3D priors. The ViT [11] with shared weights processes the reference images, followed by a cross-attention decoder to generate multiview feature maps, forming 3D perception without epipolar priors. (b) Iterative Cross-view Gaussians Alignment module iteratively refines Gaussian attributes through a 2D U-Net. The process involves warped features to align corresponding features and depths, ensuring consistent depth scales across different views. (c) The final step involves employing rasterization-based volume rendering [23] to generate high-quality geometry and realistic novel view images. in a single feed-forward inference. The 3D Gaussian primitives include position µ, covariance Σ, opacity α, and spherical harmonics SH for colors. Given a H × W sized reference image, the number of 3D Gaussian primitives can be calculated as M = N × H × W . The position of the 3D Gaussians µ i determines the geometric shape of the scene, which corresponds to pixel u is calculated using the camera origin o, the ray direction d u , and the predicted depth d:EQUATIONwhere d u is calculated by the camera intrinsic and pose matrix: d u = P K -1 [u, 1] T . However, when the number of reference views is extremely sparse, predicting accurate depths d and reconstructing high-quality geometric structures and appearances become particularly challenging. Particularly in non-overlapping and occluded areas, prevalent methods [6, 9, 12] based on epipolar line sampling fail to introduce valid geometric priors.In this paper, we propose eFreeSplat, a generalizable 3D Gaussian Splatting model from sparse reference views2 that operates independently of epipolar line priors. As illustrated in Fig. 2 , the pretrained ViT model based on cross-view completion via self-supervised training [59, 60] in large-scale datasets provides robust geometric priors, serving as our Epipolar-free Cross-view Mutual Perception (Sec. 3.2). Unlike recent works [6, 9, 61] , which directly combine per-view 3D Gaussians, we propose Iterative Cross-view Gaussians Alignment (ICGA) in Sec. 3.3. This module iteratively updates the position and features of Gaussians by calculating the similarity between warped features and corresponding features, alleviating the issues of local geometric inaccuracies caused by inconsistent depth scales. In Sec. 3.4, we predict the centers of the 3D Gaussians by unprojecting the aligned depth maps while calculating other 3D Gaussian parameters based on the aligned features.To realize the cross-view mutual preception without relying on the epipolar prior, we extract crossview image features using a shared-weight ViT E θ1 and a cross-attention decoder D θ2 , both pretrained on large-scale cross-view completion tasks in a self-supervised manner [60] . Following the methodologies of CroCo v2 [60] and ViT [11] , both images I 1 and I 2 are divided into 2n nonoverlapping patches via a linear projection, with each patch measuring 16 × 16 pixels. Additionally, relative positional embeddings [47] are added to the RGB patches before inputted into a series of stacked Transformer modules for encoding tokens ε j :EQUATIONAfter encoding I 1 and I 2 via ViT independently, the cross-attention decoder D θ2 takes ε 1 and ε 2 conditioned on each other for cross-view features F j ∈ R C×H×W :EQUATIONThe structure of the cross-attention decoder consists of alternating multi-head self-attention blocks and multi-head cross-attention blocks. The mapping function f refers to unflattening the tokens back to the original image size. The multi-head self-attention blocks learn token representations from the first viewpoint, while the multi-head cross-attention blocks facilitate cross-view information exchange conditioned on the token representations from the second view.The CroCo model [59, 60] , as a variant of masked image modeling [1, 16, 58] that leverages crossview information from the same scene to capture the spatial relationship between two images, can significantly enhance performance on 3D downstream tasks. Based on cross-view completion selfsupervised pre-training on large-scale datasets, our epipolar-Free cross-view mutual perception method provides robust 3D priors information by understanding the spatial relationship between the two images [59] . Due to the randomness of the masking process during pre-training, the pre-trained model is capable of reasoning about non-overlapping and occluded areas, which is hard for traditional geometric methods to achieve. Therefore, our epipolar-Free mutual perception possesses a more global and robust feature-matching inductive bias compared to methods [6, 9, 12, 61] that rely on epipolar line sampling [17] or the plane-sweep stereo approach [63] .To address the issue of inconsistent depth scales across different views, we utilize cross-view feature matching information to align and update per-pixel Gaussians' centers and features iteratively.Firstly, we predict per-pixel Gaussians' depths d and features G via a 2D U-Net [42] mapping U with cross-view attention, similar to [9] :EQUATIONNext, to establish cross-view correspondences, we endeavor to make the features of each 3D Gaussian point projected onto the known camera planes to be as similar as possible. Taking the first view as an example, we calculate the warped features G 1,2 of the first view on the second view's features map via the predicted coarse depth d 1 :EQUATIONEQUATIONwhere W denotes the homographic warping matrix. u represents a pixel location in the first view. R i and t i are the rotation and translation parameters of the camera pose P i . n i refers to the normal vector of the target plane. We compute the similarity S 1 , S 2 between the warped feature map G 1,2 and the corresponding feature map G 1 based on ∆d cos . S 2 is obtained by the dot product of G 1 and G 1,2 , where C denotes the feature dimension of the 3D Gaussian primitives.EQUATIONFinally, we update the coarse per-pixel 3D Gaussian features and predicted depths. The updated features and depths serve as inputs for Eq. ( 4) ( 5) and ( 6), bootstrapping the next iteration of Gaussian updates. Our cross-view Gaussians alignment method, during each iteration, involves establishing a match for target pixel u 1 in the first view with matching pixel u 2 in the second view. This process is akin to considering all neighboring pixels of the projected pixel u ′ 2 based on the current coarse depth due to the locality inductive bias inherent in convolutions. During each querying process, the discrepancy between u ′ 2 and the true matching u 2 progressively decreases, thereby harmonizing the consistency of depth scales across multiple views.EQUATIONEQUATIONWe calculate the per-view Gaussians' centers µ based on the refined depths and camera parameters using Eq. ( 1). We predict additional Gaussian primitives: Σ, α, SH , via an additional U-Net. Following other 3DGS-based methods [6, 9, 23] , the covariance matrix Σ is composed of a scaling matrix and a rotation matrix. The spherical harmonic coefficients SH are used to compute RGB values given a direction. Since we have harmonized the depth scale across different viewpoints, we directly merge all views' Gaussian primitives {µ i , Σ i , α i , SH i } N ×H×W i=1 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Datasets. eFreeSplat is trained on RealEstate10K [72] and ACID [26] . The RealEstate10K dataset consists of home tour videos, providing a wealth of scenes and a variety of viewpoint changes. The ACID dataset contains aerial landscape videos, featuring expansive views and complex terrains. Both datasets provide estimated camera parameters. Following pixelSplat [6] , we use the provided training and testing splits and evaluate three novel view images on each test scene.Losses. We employ standard image quality metrics to validate and compare our results quantitatively: pixel-level PSNR, patch-level SSIM [57] , and feature-level LPIPS [69] . During the training phase, the loss is composed of a linear combination of MSE and LPIPS loss, with loss weights of 1 and 0.05, respectively. Since existing methods conduct experiments at 256 × 256, we also set the resolution of our training and testing images for fair comparison.Comparison Methods. We compared four feed-forward methods for sparse view novel view synthesis. Du et al. [12] and GPNR [49] are the methods based on light field rendering that combines features on epipolar lines aggregated by the epipolar transformer. pixelSplat [6] and MVSplat [9] are the latest 3DGS-based models based on epipolar sampling and multi-plane sweeping, respectively. Our method compared the qualitative and quantitative results with these four methods.Implementation details. The ViT-B vision transformer [11] and cross-attention decoder [59] have been pretrained by CroCo v2 [60] , which underwent self-supervised cross-view completion training on large-scale datasets [25, 35, 40, 44, 45] . The Iterative Alignment and Updating strategy Ref.Du et al. [12] pixelSplat [6] MVSplat [9] eFreeSplat Ground Truth Fig. 3 . We provide qualitative comparisons on the RealEstate10K (first four rows) and the ACID (last two rows). Compared to baselines, our method produces fewer artifacts in rendering results (red boxes). Moreover, our approach can perform better in non-overlapping areas (1st, 2nd, 5th and 6th rows) and occluded areas ( 3th and 4th rows) without relying on unreliable epipolar priors.is implemented through 2 iterations. All models are trained on 4 RTX-4090 GPUs for 300, 000 iterations using the Adam optimizer [24] . More details are provided in Appendix C.Image quality comparison. We report quantitative results against baselines [6, 9, 12, 49] on the RealEstate10K and ACID datasets in Tab. 1. Our method, eFreeSplat, outperforms the SOTA method, MVSplat [9] by 0.06dB in PSNR on the RealEstate10K dataset and by 0.05dB on the ACID dataset.The evaluation metrics for all baselines are derived from experimental results published in the papers on pixelSplat [6] and MVSplat [9] .Ref.pixelSplat [6] MVSplat [9] eFreeSplat Fig. 4 . Comparison results about 3D Gaussians (top) and predicted depth maps of the reference viewpoints (bottom). Compared to SOTA 3DGS-based methods [6, 9] , our method achieves higher quality in 3D Gaussian Splatting and produces smoother depth maps.Table 2 . In more challenging scenarios, we classify the RealEstate10K dataset [72] into three subsets based on the overlap size of the reference images: scenes with an overlap below 0.7, 0.6, and 0.5. [64] and MVSplat [9] , based on sampling along the epipolar lines, produce unreliable reconstructions in these challenging areas. It demonstrates that eFreeSplat provides more robust 3D priors than epipolar priors, offering global 3D perception even in challenging areas.)UHTXHQF\ H)UHH6SODWH[FHOV 096SODWH[FHOV Fig. 5 . Our method reconstructs more reliable results than MVSplat when the reference views overlap is low. In the histogram, the blue bars represent the frequency at which our method exceeds MVSplat in rendering quality under the current overlap conditions, while the orange bars indicate the opposite.As illustrated in Fig. 4 , our method produces higherquality 3DGS reconstructions and smoother priors without the epipolar priors. pixel-Splat [6] , despite additional finetuning via depth regularization during training, exhibits noticeable artifacts in its reconstructed 3DGS and depth maps. MVSplat [9] generates competitive depth maps by building a cost volume representation [63] , which directly merges per-view Gaussians, resulting in significant point cloud shifts. Our method, which does not rely on sampling along epipolar lines or additional depth regularization finetuning, surpasses current SOTA methods in 3DGS reconstruction quality. Please refer to Appendix A for additional comparison and analysis.Performance with Low-overlapped observations. In this section, we analyze the differences between our method and 3DGSbased methods when the reference viewpoints have a lower overlap. First, we counted the number of scenes where our method and MVSplat [9] outperform each other in PSNR on the RealEstate10K dataset, selecting the top 400 scenes with the largest PSNR differences for As shown in Fig. 5 , our method performs better in scenes with more minor viewpoint overlaps, while MVSplat excels when the overlap is close to 1. In Tab. 2, our method outperforms other 3DGS baselines [6, 9] in settings with more minor overlaps by 3.1% ↑ in PSNR and 8.6% ↓ in LPIPS. It confirms the robustness of our method in non-overlapping areas. However, methods based on epipolar priors have advantages in scenes where reference viewpoints are closer, and reconstruction quality declines as the overlap decreases.As shown in Tab. 3 and Fig. 6 , we conducted ablation studies on the eFreeSplat model on the RealEstate10K dataset. We will detail the analysis in the following three subsections.Importance of epipolar-free cross-view mutual perception. Epipolar-free cross-view mutual perception extracts cross-view image features using a shared-weight ViT [11] and a cross-attention decoder. According to Tab. 3, this module's absence results in a 4.41dB decrease in PSNR. In Fig. 6 , the absence of cross-view mutual perception results in significant offsets in the depth map and noticeable artifacts.Importance of iterative cross-view Gaussians alignment. Iterative cross-view Gaussian alignment updates per-pixel Gaussian features and depths through warped U-Net features, thereby aligning the cross-view 3D Gaussian point clouds. The lack of Gaussian alignment can lead to pixel displacement or unreliable local geometric details (e.g., the lamp's position in Fig 6) . Additionally, we conducted extra experiments with 1 to 3 iterations. As shown in Fig. 7 , using 2 iterations significantly reduces artifacts and inconsistent depth in novel view rendering. This validates that the iterative mode helps align the depth scale across multiple views. When the iteration count increases to 3, there is no notable improvement in reconstruction and rendering quality. For further analysis and results, please refer to Appendix B.Importance of self-supervised cross-view completion pre-training. In Fig. 6 , the absence of cross-view completion pre-training weights results in unaccuracy depth maps. Self-supervised pre- training by cross-view completion [60] on large-scale datasets allows our model to perceive spatial correspondences, thereby enabling it to predict more reliable and smoother depth maps. 
conclusion: 
 Our work introduces eFreeSplat, a novel generalizable 3D Gaussian Splatting model tailored for novel view synthesis across new scenes, designed to function independently of epipolar constraints that might be unreliable when large viewpoint changes occur. By leveraging a Vision Transformer architecture self-supervised pre-trained by cross-view completion [60] on large-scale datasets, eFreeSplat excels in handling sparse and challenging viewing conditions that traditional methods [17, 63] struggle with. This model's ability to unify the consistency of depth scales across different views marks a significant improvement over existing techniques, effectively addressing issues like artifacts and misalignment in rendered images. Our experiments have demonstrated that our method provides high-quality geometric reconstructions and novel viewpoint images. In settings with a large baseline from 2-view inputs, it outperforms the latest state-of-the-art methods [6, 9] that rely on epipolar priors. 
acknowledgement: 
 This work was supported by the National Natural Science Foundation of China (62293554, 62206249, U2336212), "Leading Goose" R&D Program of Zhejiang (No. 2024C01161), and Young Elite Scientists Sponsorship Program by CAST (2023QNRC001). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We follow the license CC-BY 4.0. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide additional qualitative comparisons against baselines. The visualization results on the RealEstate10K are shown in Fig. 8 . Additionally, we provide more geometry reconstruction comparison results, as shown in Fig. 9 . Our method reconstructs high-quality 3DGS without using epipolar priors or depth regularization finetuning.Ref.Du et al. [12] pixelSplat [6] MVSplat [9] eFreeSplat Ground Truth Fig. 8 . Additional visualization results on the RealEstate10K [72] . Our method, eFreeSplat, outperforms baselines in rendering results, producing fewer artifacts and scene distortions.Ref.pixelSplat [6] MVSplat [9] eFreeSplat Fig. 9 . Additional geometry reconstruction quality comparison results. Our method achieves higher quality in 3D Gaussian Splatting and produces smoother depth maps than pixelSplat [6] and MVSplat [9] . Ref. More ablations. In this section, we provide both quantitative and qualitative results of the Gaussians Alignment module under the settings of 1 to 3 iterations. As shown in Tab. 4 and Fig. 10 , setting the iteration count to 2 effectively reduces artifacts caused by inconsistent depth scales. When the iteration count is set to 3, we had to reduce the model's parameter size and batch size to avoid OOM errors, which might be one of the reasons for the lack of significant improvement in image reconstruction metrics.Failure cases. Our method relies on the 3D prior knowledge provided by CroCo [59] pre-trained weights. However, the input viewpoint overlap in the pre-trained dataset does not exceed 0.75 [60] , while the input viewpoint overlap in the RealEstate10K and ACID datasets mainly ranges from 0.9 to 1.0. As shown in Fig. 11 , our method renders unreliable results when the input viewpoints are very close, which can be attributed to the distribution bias between the GNVS dataset [26, 72] and the pre-trained dataset [25, 35, 40, 44, 45] . Limitations. Our method lacks geometric inductive biases, so our model is data-hungry and sensitive to the training data distribution. Joint training with richer multiview datasets across different scenes could be a viable direction. Additionally, the per-pixel 3D Gaussian mapping struggles to reconstruct parts of the scene that are occluded or missing from input viewpoints, such as an obscured chair. Therefore, introducing high-level features for scene completion might be a future research direction for generalizable 3D Gaussian Splatting work.Fine-tuning of the CroCo model. We have conducted preliminary explorations to address the aforementioned limitation. We conducted relevant Experiments A and B regarding fine-tuning the CroCo model using the RE10K dataset. Experiment A involved fine-tuning the CroCo pretrained weights with the RE10K training set, while Experiment B involved training CroCo directly with the RE10K training set without loading the pretrained weights. Finally, we retrained eFreeSplat using the new pretrained weights. As shown in Fig. 11 and Tab. 5, the results indicate that pretraining the backbone model on the RE10K training set effectively addresses the model's poor performance in low-overlap scenarios. However, in the RE10K test set, Experiment A's reconstruction metrics were slightly lower than those of the original model, which may be due to insufficient training iterations. We will further investigate the positive impact of fine-tuning the CroCo pretrained model on novel view synthesis and 3D reconstruction in future work.Potential negative societal impacts. Our model could be misused for unethical purposes, such as creating false evidence or manipulating media, which threatens information integrity and personal privacy. Additionally, the model introduces security risks in contexts like autonomous driving, as it may produce incorrect reconstructions in real and complex scenarios. These concerns underscore the importance of implementing stringent ethical guidelines and security measures when deploying such technology, to prevent misuse and ensure that it is used responsibly.The cross-attention decoder. Following the CrossBlock decoder architecture in CroCo [59] , the cross-attention decoder comprises a self-attention module and a cross-attention module. Let ε 1 , ε 2 ∈ R N ×C be the tokens of the two viewpoints outputted by a Vision Transformer [11] . The computation process of the decoder is as follows:EQUATIONIn Equations 10, Attention is derived from the classic attention computation. The inputs Q, K, V undergo projection transformations using W q , W k , W v :EQUATIONThe cross-view U-Net. For the Gaussian Alignment Strategy and the prediction of Gaussian primitives, we utilize a 2D Cross-View U-Net inspired by [41, 52] , 2024. We concatenate and flatten multiview feature maps for cross-view information exchange, similar to the structure of the U-Net used for cost volume refinement in MVSplat [9] . Specifically, for the Gaussian Alignment Strategy, we apply four times of 2 × down-sampling and add attention at the 16 × down-sampled level, with the channel dimensions being [32, 32, 64, 128, 256] . For the prediction of Gaussian primitives, we keep the channel dimension fixed at 32, while the rest of the architecture remains the same as that of the U-Net used in the Gaussian Alignment Strategy.More training details. Our model is trained and tested on 4 RTX-4090 GPUs using the Adam optimizer with a learning rate 2e-4. The per-GPU batch size during training is 4. Similar to pixelSplat [6] , the distance between the two input viewpoints gradually increases throughout training. However, to learn more robust 3D prior information, our setup allows for a maximum viewpoint distance of 60 frames, compared to the 45 frames used by pixelSplat [6] and MVSplat [9] .NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In our experiments, the number of reference views N = 2, which is consistent with previous methods[6,9]. For convenience, all subsequent discussions will assume a 2-views input scenario. 


Decision: This paper receives borderline reviews. It gets 1x accept, 2x borderline accepts and 1x borderline reject. The reviewers agree that the motivation of using the pre-trained model to address the non-overlapping and occluded regions, and the idea of not using epipolar geometry priors are interesting and well-grounded. Extensive experiments, including comparisons with baseline methods such as pixelSplat/MVSplat, demonstrate that the proposed approach can generate higher-quality novel view images, particularly in scenes with smaller overlaps. The major concern of Reviewer MtQr who gave a borderline reject is that the experiments do not seem enough to support the claims of the paper. This concern seems to be addressed by the authors during the rebuttal. It was pointed out in the rebuttal that "maybe we did not illustrate the difference between GNVS and sparse-view NVS clear enough and make you miss the focus of our experiments". The weaknesses mentioned by other reviewers are mostly clarifications on the experiments, which seem to have been well-addressed during the rebuttal and discussion phase. On the consideration that the strengths outweigh the weaknesses of the paper, the meta-reviewers decide to accept the paper.

Reviews: review: paper_summary: paper_summary: This work presents eFreeSplat, a model to address generalizable novel view synthesis without relying on epipolar prior. Specifically, it extracts the cross-view mutual perception by leveraging a pre-trained CroCo model. It then improves the alignment of multi-view Gaussians by using an iterative updating strategy. Finally, novel views are rendered by an off-the-shelf 3DGS renderer. Experiments on two benchmarks, RE10K and ACID, demonstrate the effectiveness of the introduced eFreeSplat.
strength_weakness: strength_weakness: * Performances regarding cross-dataset generation. Similar to the cross-dataset experiments shown in MVSplat, it would be interesting to see how eFreeSplat performs when trained on RE10K but tested on ACID and DTU. This experiment will help us better understand how such a data-driven approach generalizes across different data distributions.

* It would be better to provide a more detailed analysis of the Iterative Cross-view Gaussians Alignment module. For example, showcase the depth maps of both input views when setting the iteration to 1, 2, and 3. This would be helpful for verifying whether the iterative solution can really help align the depth scale across multiple views.

* Will it be beneficial to perform a fine-tuning of the CroCo model? As mentioned in the failure case, eFreeSplat performs worse on cases with significant overlaps, potentially because CroCo is trained on image pairs with slight overlaps. Can these failure cases be addressed by fine-tuning CroCo on the RE10K training set? Similarly, as reported in Tab. 3, the performance of eFreeSplat w/o pre-training weights is much worse compared to other state-of-the-art models, while MVSplat achieves reasonably good results even when training from scratch. Can this gap be reduced by pre-training the backbone model using the RE10K training set?
questions: questions: Kindly refer to the [Weaknesses]
limitations: limitations: Kindly refer to the [Weaknesses]
paper_summary: paper_summary: This paper introduces a robust pipeline for generalizable 3D Gaussian novel view synthesis, utilizing a cross-attention model trained on large-scale datasets. This approach enables the model to generalize effectively to new scenes without depending on epipolar geometry, which is commonly used in traditional methods. To address inconsistencies in depth scales, the pipeline features an iterative refinement process that adjusts the attributes of 3D Gaussians based on cross-view feature correspondences.
strength_weakness: strength_weakness: - Overall, the paper has good ideas, but the experiments are not enough to support their claim. The paper claimed to be a multiview generalizable novel view synthesis, but most of the experiments are done in an image pair setting, it should add more experiments in a sparse-view setting and compare with other sparse-view Novel view synthesis methods on datasets like mipnerf360. 
- 59 "obtain the warped features for each view based on the predicted depths via U-Net" and formula (5) seems similar to the the coarse-to-fine cost volume formulation utilized in the MVSNet. Could you elaborate on how these two are different?
questions: questions: - In formula (7) , the C is used without definition. I assume it is the dimensionality of the feature of 3D Gaussian primitive? It should be made clearer. 
- Since you are using a Croco pre-trained model could you give some metrics on 512 x 512 resolution? 
- Given the significant memory demands of the global alignment process in DUSt3R, could you detail the memory requirements for your method? It would be helpful to understand if similar memory constraints apply to your approach.
limitations: limitations: The author have made a commendable effort in acknowledging both the technical limitations and potential negative societal impacts of their method.
paper_summary: paper_summary: This paper addresses the task of 2-view generalizable novel view synthesis. It introduces a cross-view completion model as prior assistance and incorporates cross-view Gaussian alignment after predicting Gaussian attributes to enhance cross-view consistency.
strength_weakness: strength_weakness: The ablation analysis in Tab. 3 does not clearly identify which module contributes the most to the performance improvement.
questions: questions: 1. Table 3 seems to indicate that all three modules are important. In the author's opinion, which module has the greatest impact on the performance improvement?
2. Does the iterative nature of the Cross-view Gaussians Alignment module slow down the training speed? Is choosing to iterate twice a trade-off between effectiveness and efficiency?
3. Is Cross-view Gaussians Alignment required during inference?
limitations: limitations: The author provides a detailed discussion of limitations and social impacts in the paper.
paper_summary: paper_summary: This paper tackles the generalizable NVS problem from sparse-view inputs. The main claim is that relying on epipolar geometry cues harms the performance, and, thus, it removes these geometric priors. It adopts the pre-training and supervised reconstruction/NVS training pipeline as in Dust3r. The authors verify the performance on RealEstate and ACID.
strength_weakness: strength_weakness: - Overclaim. The authors claim that the proposed method is the first epipolar-free work (Line 64), but actually there are a lot prior works that are epipolar-free, espeicially for those works doing reconstruction from unposed images where the epipolar lines cannot be computed without the access of ground-truth poses. These works including SRT (the unposed version UpSRT), LEAP [1], PF-LRM [2] and Dust3r+Croco, where they do cross-view attention between different input views. LEAP also shows visualization results of the cross-view attention for implicit cross-view feature matching and correspondence. Besides, there are also some of lightfield-based methods are epipolar-free. I don't know why the authors have already discussed SRT and Dust3r+Croco in the related work section, but they still think they are the first epipolar-free work. All previously mentioned works should be discussed properly.

[1] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024.
[2] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.

- Related work. More related work on generalizable NVS using gaussian splatting should be discussed, for example, GS-LRM [3].

[3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ECCV 2024.

- Evaluation. The evaluation doesn't verify the claims of the authors. Even though the idea of Table 2 is good, I don't see a clear different of the gain over the epipolar-aware baseline pixelSplat when the overlap is smaller.
questions: questions: Please see the weaknesses.
limitations: limitations: The biggest concern is the novelty. This paper is a combination of Duster (cross-view pre-training + 2-view geometry without epipolar prior) and generalizable NVS using Gaussian Splatting (e.g. MVSplat).

I believe the authors need to discuss the prior works more properly and better word their contributions.

Comments: comment: Thanks for your effort during the rebuttal period. I will raise the score.
Thank you for your support of our work and for providing valuable suggestions. We have carefully considered the issues you raised and plan to revise them as follows:

### **Revision of Controversial Claims**
Upon careful review, the original claims regarding the novelty and contribution of our method were as follows:
* Our eFreeSplat represents a new **paradigm** for generalizable novel view synthesis.
*	To our best knowledge, **the first multiview GNVS paradigm** that operates without relying on epipolar priors.
*	a **groundbreaking** generalizable 3D Gaussian Splatting model designed for novel view synthesis across scenes, **free from the constraints of epipolar priors**.
*	has inspired our development of a **novel GNVS paradigm** that circumvents the dependence on epipolar priors.

After an in-depth discussion during the rebuttal, we recognize the validity of your suggestions, which significantly aid in more accurately describing our contribution. We will revise the content as follows:

*	Our eFreeSplat represents an innovative approach for generalizable novel view synthesis. Different from the existing pure geometry-free methods, eFreeSplat focuses more on achieving epipolar-free feature matching and encoding by providing 3D priors through cross-view pretraining.
*	The approach with novel insights into GNVS that operates without relying on epipolar priors in the process of multi-view geometric perception.
*	A novel generalizable 3D Gaussian Splatting model tailored for novel view synthesis across new scenes, designed to function independently of epipolar constraints that might unrelieble when large viewpoint changes occur.
*	has inspired our development of a novel GNVS method that circumvents the dependence on epipolar priors through data-driven 3D priors.


### **Revision of Discussion on Related Works**
After thorough review, we acknowledge that the original manuscript's description of closely related work included the following content but lacked discussion on methods like GS-LRM, which do not require epipolar line sampling:

*	SRT[1] is a geometry-free, generalizable NVS method that boldly eschews any explicit geometric inductive biases. SRT encodes patches from all reference views using a Transformer encoder and decodes the RGB color for target rays through a Transformer decoder

After in-depth discussion in the Rebuttal, we agree that your suggestion is reasonable, and we will provide a more detailed description of related work in our revised version:

*	SRT[1] and GS-LRM[2] are epipolar-free GNVS methods that boldly eschew any explicit geometric inductive biases. SRT encodes patches from all reference views using a Transformer encoder and decodes the RGB color for target rays through a Transformer decoder. GS-LRM's network, composed of a large number of Transformer blocks, implicitly learns 3D representations. However, due to the lack of targeted scene encoding, these methods are either limited to specific datasets or suffer from unacceptable computational efficiency and carbon footprint.

*	Some pose-free GNVS methods[1][3][4] are also epipolar-free. These methods, lacking known camera poses, find it challenging to perform epipolar line sampling. They often reduce task complexity through clever structured feature representations (e.g., Learned 3D Neural Volume in LEAP[3] and Triplane in PF-LRM[4]), but this reduction comes at the cost of decreased model generalization. Different from the above methods, our proposed eFreeSplat focuses on data-driven 3D priors and does not require any time-consuming and complex structured feature representations, such as cost volumes.


As for the experiments on a more overlap-balanced dataset, due to time constraints, as you acknowledged, we regret that it is challenging to prepare such balanced data and train our model in time to include the analysis within the eFreeSplat work. Nevertheless, we commit to establishing such an experimental scene in the forthcoming revised draft and to eliminating the influence of data distribution gaps on the experimental results, thereby providing a more thorough analysis of our method's advantages in scenarios with significant viewpoint changes.

**We will do our utmost in the revised version to resolve the controversial claims and address all the issues mentioned above. Once again, thank you for your professional feedback and evaluation of our work.**

[1] Sajjadi M S M, Meyer H, Pot E, et al. “Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations” CVPR 2022.

[2] Zhang K, Bi S, Tan H, et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting” ECCV 2024.

[3] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024. 

[4] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.
Thanks for the detailed and helpful reply!

The motivation of using epipolar-free methods makes sense to me. While this field is super competitive as there are many recent works, releasing unnecessary inductive bias on geometric designs is promising to me, especially when having large data. The competition in this field makes me a bit tired, as I can feel most of the works are a kind of incremental. I appreciate the effort to further improving Dust3r training pipeline on the GNVS task using the proposed epipolar-free method and the iterative cross-view gaussian alignment method.

Regarding the performance analysis, maybe the reason for my previous question is the data distribution (45:8.5:1 as mentioned). The data amount with small overlap ratio, e.g. 0.5, can be too less, so the numbers are also noisy. I sincerely suggest the authors to find some other data that are more balanced to perform the analysis, e.g. DTU or DL3DV, where you can control the overlap ratio as the two datasets provide dense views. I won't request this experiment as the discussion period is coming to the end soon.

At the same time, we still want to let the authors know I expect the prior works can be correctly discussed, especially removing some controversial words like first "the first epipolar-free GNVS method". After the discussion, I don't want to reject this paper because of the overclaiming problem as I saw the effort behind this paper. 

Thus, I would like to raise the score to **a neutral borderline** (with a score of 4.5, due to the uncertainty of how the over-claiming problem will be resolved) and I don't against accepting this paper. As there is no neutral borderline for the reviewer, I will just kept the current rating now, but I will let AC know if I keep the a neutral borderline until the end of discussion.

**At the same time, if you have better wording for resolving the controversial claims, please let me know before the discussion period ends. If the revised version is satisfying to me, I will raise the score to a borderline accept.**
Thank you for your thoughtful response. Please find below our detailed comments on the novelty of our work and an analysis of the performance gain.

## **Novelty of the Work:**

* **Epipolar-Free Approach to Address Overlap Challenges.**
We have identified that mainstream multi-view GNVS methods, which rely on epipolar geometry, may encounter significant challenges in scenarios with large viewpoint overlaps. To address this, we explored the role of cross-view pretraining models that provide 3D prior knowledge for the GNVS task, and we introduced a cross-view mutual perception mechanism to partially mitigate the limitations of current methods.

* **Iterative Cross-view Gaussians Alignment Module.**
Beyond cross-view mutual perceptive pretraining, we have specifically introduced the Iterative Cross-view Gaussians Alignment module. We observed that relying solely on the CroCo pretraining model is insufficient to effectively address challenges such as ambiguity in GNVS, primarily due to the absence of precise local feature matching. Unlike previous generalizable NVS methods utilizing Gaussian Splatting (e.g., MVSplat), our approach circumvents the need for time-consuming and less generalizable structured feature representations, bypasses complex depth sampling processes, and enables efficient per-pixel Gaussian attribute prediction. The ablation experiments presented in Table 3 and Figure 6 demonstrate the critical importance of the Iterative Cross-view Gaussians Alignment module.

**Our approach not only effectively manages scenarios where epipolar geometry may fail but also eliminates the time-consuming process of sampling along epipolar lines. This makes our method particularly efficient and effective for GNVS tasks, especially in scenarios with extensive viewpoint overlaps.**

## **Performance Gain Analysis:**

With regard to Table 2 of the manuscript, the three subsets with overlaps below 0.7, 0.6, and 0.5 each represent challenging experimental conditions where epipolar geometry becomes unreliable, and it would be inaccurate to assert that one is inherently more challenging than the others. **Moreover, the performance gain across these subsets does not follow a linear progression.** The reasons are as follows:

* **Distribution Gap Between Subsets and Training Set.**
These three low-overlap subsets differ from the training set and constitute a small fraction of the test set. The majority of training data involves input viewpoints that overlap well above 0.7. In the test scenes, the ratio of scenes with overlaps greater than 0.7 to those below 0.7, 0.6, and 0.5 is 45:8:5:1. Thus, from a dataset distribution perspective, these three subsets exhibit significant variability, and the overlap size does not directly correlate with difficulty. This distribution gap explains why the performance gain remains similar across these subsets, as the model’s performance is more heavily influenced by distributional differences than by the specific overlap percentage.

* **Non-linear Error Growth with Decreasing Overlap.**
In 3D computer vision, local matching errors can propagate and affect the global output, meaning that overall error does not necessarily increase linearly as overlap decreases. This is because incorrectly matched features might be propagated as outliers into subsequent modules (especially in modules with global awareness, such as transformers), thereby amplifying their impact on overall performance. Once overlap falls below a certain threshold, traditional methods are significantly compromised. For instance, if the feature point matching error rate is too high, SFM algorithms may fail. Consequently, scenes with overlaps below 0.7–0.5 all represent scenarios where epipolar geometry is unreliable, and there is no strictly linear relationship for such "unreliableness". Thus, the similarity in performance gains for overlaps of 0.5 and 0.7 reflects the model’s consistent capability in managing these challenging scenarios where epipolar constraints are inadequate.

Finally, thank you again for your excellent suggestions, which have helped us to think more deeply about the essence of performance improvement in our method. We will incorporate this analysis of the performance gain into the revised manuscript.
Thanks for the detailed explanation.

I can understand the argument of the reviewer. However, I still cannot agree with the authors that categorizing the methods into epipolar-free and geometry-free is appropriate. In my perspective, epipolar-free method is a subset of geometry-free methods. Thus, I am not able to agree that splitting unposed methods like SRT to geometry-free means it is not epipolar-free. The authors replied "*'epipolar-free' means that epipolar line sampling is not required*", and SRT, GS-LRM, indeed don't use epipolar line sampling.

At the same time, I agree that SRT is geometry-free, but I don't agree GS-LRM is geometry-free. Moreover, I agree that unposed GNVS and posed GNVS are two tasks. However, whether the method is epipolar-free is orthogonal to whether the method uses GT poses, and it's very clear that the authors claimed "we are the first epipolar-free GNVS work".

I realized this discussion can be a never-ending battle. So I would switch the context to my other comments that are not addressed. Is there any reply to my comments regarding the novelty of this work? And do you have any comments on why the performance gain is quite similar for data with overlap of 0.5 and 0.7?
Thanks so much again for the time and effort in our work. May I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.
Thanks so much again for the time and effort in our work. May I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.
Thanks so much again for the time and effort in our work. May I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.
Thank you for your professional feedback. We believe our work does not involve any over-claiming. 

Firstly, "epipolar-free" and "geometry-free" are two distinct concepts. In our method, "epipolar-free" means that epipolar line sampling is not required, but it does not mean that we completely avoid using geometric information. In fact, our Iterative Cross-view Gaussians Alignment method relies on the warp transformation formula from the known geometry, as shown in Eq. (5) of the manuscript. In contrast, methods like SRT and GS-LRM are entirely geometry-free, meaning they do not use geometric information at all, making them true "geometry-free" methods. Our "epipolar-free" approach is more akin to a semi-geometry approach, so SRT and GS-LRM do not fall under the category of epipolar-free methods as proposed in this paper. We will further clarify the distinction between epipolar-free and geometry-free methods in the related works section of the revised version.

Secondly, at this stage, unposed GNVS tasks and GNVS tasks with known poses are two different tasks. Here are three reasons to illustrate this:

1. Although pose-free GNVS methods can be applied to GNVS tasks with known GT poses, the current state-of-the-art methods, such as LEAP or PF-LRM, struggle to match the accuracy and efficiency of contemporaneous GNVS methods with known GT poses. These pose-free methods mainly compare themselves with known GT pose methods like pixelNeRF (CVPR 2021).

2. Pose-free GNVS methods are currently limited to simple object-level datasets (e.g., DTU or Omniobject3D). In contrast, our method not only conducts experiments at the complex scene level (RE10K) but also generalizes directly to object-level datasets (see Figure 1 and Table 1 in the rebuttal PDF).

3. The primary challenges of unposed GNVS methods and GNVS methods with known poses are different at this stage. Unposed GNVS methods need to address how to implicitly learn geometric relationships or recover camera poses without any known 3D priors. They often reduce task complexity through clever structured feature representations (e.g., Learned 3D Neural Volume in LEAP and Triplane in PF-LRM), but this comes at the cost of reduced model generalization. On the other hand, GNVS methods with known poses focus more on generalization ability and handling larger baselines. Our epipolar-free method is designed to enhance performance on larger baselines: eFreeSplat significantly outperforms SOTA GNVS methods in challenging areas such as regions with smaller viewpoint overlaps.
Hi reviewers,

Thank you for your hard work in reviewing the paper! 
Please check out the authors' responses and ask any questions you have to help clarify things by Aug 13.

--AC
Thanks for the reply. However, the rebuttal didn't solve my concerns, especially the over-claiming.

The rebuttal says "While approaches like UpSRT, LEAP, and PF-LRM also do not utilize epipolar geometry, our method targets multiview GNVS tasks with precise camera poses." As far as I know, SRT can also do GNVS with perfect poses without using epipolar geometry. Besides, LEAP or PF-LRM can also do GNVS, and actually using GT poses in this work is a stronger assumption. I cannot agree that this paper is the first epipolar-free GNVS work.