FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training

Authors: ruihong  yin
vladimir  yugay
yue  li
sezer  karaoglu
theo  gevers
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:46:21

Abstract: The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability to render novel views accurately. While Gaussian Splatting performs well when a sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting in poor rendering performance. To address this, we present a 3D Gaussian-based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching-based consistency constraints imposed on the novel views without relying on pre-trained depth estimation or diffusion models. This is achieved by using the matches of the available training images to supervise the generation of the novel views sampled between the training frames with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance of our method in few-shot novel view synthesis compared to existing state-of-the-art methods.

Main paper: 
 introduction: 
 Reconstruction of a 3D scene representation from sparse 2D observations that can render novel views unseen during training has been an active field of research with wide applications in VR/AR and navigation. Recently, neural radiance fields (NeRF) [20] utilizing differentiable volume rendering and implicit scene representation achieved great results in novel view synthesis (NVS). The NeRF framework was extended to the sparse-view setting (few-shot NVS) using efficient training schemes, regularization losses, depth consistency, and image priors [11, 22, 8, 34] . While being good at rendering, NeRFs typically take a long time to be optimized, and their rendering speed is far from real-time which greatly limits their practical application. 3D Gaussian Splatting [13] has introduced an unstructured radiance field represented with 3D Gaussians. Gaussians are initialized from a sparse Structure-from-Motion (SfM) [29] point cloud and dynamically added or removed from the scene during training. With images observing a static scene from different viewpoints, Gaussian parameters are optimized using photometric loss. This method became widely adopted due to its training efficiency, unprecedented rendering speed, and quality. However, the rendering performance dramatically drops when fewer images are used for training. While well-observed regions can be accurately rendered, less-supervised scene geometry is under-reconstructed, Gaussian parameters are overfitted to the views observing the scene [44] , geometry artifacts [38] appear. In addition, optimization is highly dependent on the SfM initialization point cloud whose quality is also affected by the sparsity of the training views. There are several concurrent works exploring few-shot Gaussian splatting [15, 38, 24, 6, 46] . However, all of them rely on depth estimation [15, 38, 24, 6, 46] or diffusion priors [38] . In these methods, depth-estimation networks predict depth up to scale on unseen scenes, making them sensitive to domain shifts, while diffusion models increase training time. In this paper, we introduce FewViewGS, a Gaussian splatting-based novel view synthesis method achieving state-of-the-art rendering results in a few-shot setup. We break the training process into the pre-training, intermediate, and tuning stages to better propagate the information to the novel views. During the pre-training stage, we use only training views to get a basic representation of the scene. This enables us to obtain a fundamental point cloud and scaled training view depth maps. In the intermediate stage, the emphasis is on optimizing the new perspectives. We use multi-view geometry and a novel view interpolation sampling to render unseen views coherent with the training images. For this, we match the pairs of training images, robustly warp the matches to the randomly sampled novel views between them, and enforce novel view consistency by applying geometry, color, and semantic losses. In the final stage, the scene representation undergoes refinement through a limited number of iterations, utilizing only the known views. During training, the 3D Gaussians are regularized by our proposed locality preserving regularization to maintain their local properties, eliminating the artifacts on the novel views. In summary, our contributions are as follows:• A few-shot NVS Gaussian splatting-based system not relying on pre-trained depth estimation or diffusion models achieving SoTA rendering results. • A multi-stage training scheme enabling seamless knowledge transfer from known to novel views.• A robust warping-based novel view consistency constraint ensuring the coherence of the synthesized unseen images. • 3D Gaussian locality preserving regularization handling visual artifacts.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We highlight our novel multi-stage training scheme and correspondence-driven losses on sampled novel views which greatly improves the final NVS results. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Novel View Synthesis. Neural Radiance Fields [20] model a 3D scene using a neural network that takes the viewing direction and 3D location as input and predicts color and density. During training, the network is optimized to render the images of a scene from different angles. The network trained in such a fashion implicitly learns how to interpolate and extrapolate [39] to render the scene from the angles unseen during training at a very high resolution. Due to its capabilities, NeRF became widely adopted for 3D scene reconstruction [1, 18, 8, 34, 14] , human body modeling [26, 25, 36, 16] , robotics [41, 28] , and medical imaging [7] . However, classical NeRFs require extensive training time and computational resources, making them less practical for real-time applications. There are several methods [2, 1, 21, 10, 42] focusing on efficiency by using multiscale scene representations [1] or efficient data structures for scene encoding [21, 10, 42, 2] . However, all those approaches come at the cost of lower rendering quality. Recently introduced, 3D Gaussian Splatting [13] uses 3D Gaussians to represent the scene. To render a view of a scene, the 3D Gaussians in the camera frustum view are splatted to the image plane. Having several training images observing a static scene from different angles, the parameters of the 3D Gaussians are optimized by minimizing the photometric loss. Due to explicit geometry representation, this method achieves real-time rendering and fast optimization without compromising rendering quality. However, just as NeRF, Gaussian splatting requires multiple views observing the scene from various angles. Few-shot Novel View Synthesis. Few-shot novel view synthesis aims to train a scene representation capable of generating new views of a scene using only a sparse number of training images. Due to the lack of multi-view constraints on the scene and usage of photometric losses, training such a representation remains challenging. Several works explore how to train better NeRFs for the novel view synthesis in a sparse setting. One group of methods explores the regularization with priors from pre-trained neural networks [8, 34, 37] . For example, DSNeRF [8] and SparseNeRF [34] use depth regularization from pre-trained depth estimators on known views to guide optimization. DiffusioNeRF [37] uses weight regularization and priors from diffusion models.Another line of work [4, 22, 11, 33, 31] focuses on imposing additional supervision on the novel views during training. For example, GeoAug [4] randomly samples novel views around the known frames and then calculates the color loss between the warped novel view and the known view. RegNeRF [22] designs depth smooth regularization on unobserved views. DietNeRF [11] argues that the high-level semantic information should be similar for individual scenes and proposes to apply semantic consistency loss on the novel views. SPARF [33] integrates multi-view correspondence and geometry loss to the optimization. ViP-NeRF [31] regularizes the network with visibility prior, which is generated by plane sweep volumes. Simultaneously with our research, numerous methods have emerged focusing on few-shot Gaussian splatting for novel view synthesis. FSGS [46] proposes a Gaussian Unpooling strategy to generate dense Gaussians. SparseGS [38] adopts a floater removal strategy to remove unnecessary Gaussians close to the camera. DNGaussian [15] , CoherentGS [24] , DRGS [6] focus on regularizing the depth maps. Remarkably, all the few-shot novel view synthesis methods based on 3DGS try to integrate the pre-trained depth estimation networks in the pipelines. Novel view regularization has shown great results in sparse novel view synthesis. However, sampling random novel views in the camera frustum can make the rendering results inconsistent. We therefore propose to use correspondences between the known views to regularize the novel views sampled between them robustly. Moreover, we avoid relying on complex priors. Using simple image matching in the 2D-pixel space, we avoid using depth estimation models predicting depth up to scale and heavy diffusion models which struggle on out-of-distribution datasets, while achieving state-of-the-art rendering results.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We address the limitation of our method at the end of Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The key idea of our approach is to enforce consistency between the sparse known views and the novel views in visually overlapping areas. As presented in Fig. 1 , we introduce a multi-stage training scheme consisting of pre-training, intermediate, and tuning stages to gradually optimize the scene and obtain depth maps for the training views. After the pre-training stage, we start enforcing consistency of the novel views. For this, we first sample random training frames and match them with each other in pixel space. Further, we randomly sample a pose between the pairs of training frames and warp the matched pixels to it. We then filter out unreliable warped pixels and apply our new color, geometry, and semantic losses to the rendered novel view. This ensures the novel view remains consistent only in the regions that overlap with the known view pairs. In addition, we apply localitypreserving regularization to remove typical artifacts that appear in few-shot scenarios. We now outline our pipeline, beginning with an overview of Gaussian Splatting [13] , followed by the novel view consistency mechanism, regularization losses, and our training scheme.Gaussian splatting [13] is a recent radiance field representation that replaces the implicit neural network with explicit optimizable 3D Gaussians. A single 3D Gaussian is parameterized by its mean µ ∈ R 3 , covariance Σ ∈ R 3×3 , opacity o ∈ R, and a set of spherical harmonics coefficients sh ∈ R 3(l+1) 2 with degree l for view-dependent color. The mean of a projected (splatted) 3D Gaussian in the 2D image plane µ I is computed as follows:EQUATIONwhere μ denotes homogeneous mean, T wc ∈ SE(3) is the world-to-camera extrinsics, P ∈ R 4×4 is an OpenGL-style projection matrix, π is the projection to pixel coordinates. The 2D covariance Σ I of a splatted Gaussian is computed as:EQUATIONwhere J ∈ R 2×3 is the Jacobian of the affine approximation of the projective transformation, R wc ∈ SO(3) is the rotation component of T wc . For further details on the splatting process, we refer to [47] . Parameters of the 3D Gaussians are iteratively optimized by minimizing the photometric loss between rendered and training images. During optimization, covariance is decomposed as Σ = RSS T R T , where R ∈ R 3×3 and S = diag(s) ∈ R 3×3 are rotation and scale respectively to preserve covariance Given the projected and virtual renders, color, geometry, and semantic losses are calculated at the projected pixels in the new views.positive semi-definite property. Color c for a pixel influenced by n ordered Gaussians is rendered as:EQUATIONwith c i the color of i th Gaussian and α i computed as:EQUATIONwhere ∆ i ∈ R 2 is the offset between the 2D mean of a splatted Gaussian and the pixel coordinate.Similarly, this blending process could be extended to render pixel depth. We compute the depth d of a pixel that is influenced by n ordered Gaussians as:EQUATIONwhere µ z i is the z component of the mean , α i and T i are the same as in Eq. ( 3). This splatting approach provides significantly faster training and inference time with no compromise on rendering quality compared to NeRF. During training, new Gaussians are added and removed from the scene based on the heuristics introduced in [13] to improve the scene coverage.Our main goal is to ensure the novel views are consistent with the training views in overlapping regions, maintaining visual coherence. However, determining which specific pixels in the training views should match those in the novel views is challenging. Other few-shot methods [4, 22, 11, 33, 31] try to address this by randomly sampling novel views within the viewing frustum of a training frame and then applying regularization techniques to them. In contrast, our approach uses pairs of training views to more robustly identify which pixels need regularization in the novel views. This method enhances accuracy and consistency, aligning novel views more effectively with the training data, resulting in higher quality and more coherent visual output.As the first step, for a pair of training images i and j we find the pixel correspondences C i , C j using a robust image matcher [9] . Since the training views' poses P i and P j are known, a novel view pose P k is randomly sampled between them. Further, we render the depth maps of the matched pixels Z i , Z j of the training views, and warp them to the novel view, defined by:EQUATIONwhere C u,i are the pixels of the novel view, π and π -1 are the projections from 3D to 2D and from 2D to 3D respectively.Having projected the matched pixels to the novel view, they are used to supervise novel view synthesis by applying color, geometric, and semantic losses. However, natural errors stemming from color and depth rendering, or image matching, make supervision over all the projected pixels unstable. To mitigate these effects, an agreement masking is proposed and defined as follows:EQUATIONwhere x i , x j are image-shape matrices, θ ∈ R is an agreement threshold, and I is an indicator function.In addition, the warped values often end up in the wrong locations when warped from the high color gradient regions, contaminating the losses. This happens mostly because of inaccurate matches in such areas. To address this, we introduce a color gradient weighting W which alleviates the effect of errors in texture-rich regions. For a color gradient G at a pixel (u, v), it is computed as:EQUATIONwhere |G u,v | is gradient magnitude, θ grad is a scalar threshold. Denoting matches between known images i and j as P i,j , we enforce the geometric consistency of the novel view using the loss:EQUATIONwhere Z p k is the rendered depth of a novel view k at a pixel p, Zp ik , Zp jk are the projected depth values from the known views i and j respectively, θ g is an agreement threshold.t is set to i if |Z p k -Zp ik | 1 is less than |Z p k -Zp jk |1 and to j otherwise. We take the minimum over the pair of depth discrepancy to further reduce the influence of inaccurate projections. To regularize the appearance of the novel view, the loss in the color space is applied as follows:EQUATIONwhere c p k is the rendered color of a novel view k at pixel p. t is defined following the same logic as in Eq. ( 9). Finally, the novel view should have similar semantics to the training views in the overlapping regions. Since pre-trained deep neural networks can encode semantic information of the images [30, 27, 23] , the loss is added on the feature space of the novel views:EQUATIONwhere f i , f j , f k are image-sized features extracted with a pre-trained network from the views i, j, k respectively. t is defined following the same logic as in Eq. ( 9). The final equation for the novel view consistency loss is expressed as:EQUATIONwhere α, β, γ are the respective scalar weights.After examining optimized 3D scenes in a few-shot setting, it was found that accurate rendering results from Gaussian splatting occur when color values are smooth within local neighborhoods. However, in a standard setting, the photometric loss does not explicitly enforce this. Although this is not an issue when numerous frames are available, it leads to artifacts in a few-shot setup. Therefore, a locality-preserving regularization is proposed to address this issue. For every Gaussian i, its neighborhood N i is defined by finding the K nearest neighbors in 3D space. Furthermore, the color parameters of Gaussian i are enforced to be proportionally closer to those of its neighborhood:EQUATION)where δ is a scaling factor, c i and c k are the colors of the Gaussian i and its neighbor k respectively. The weight function exp(-δ• |µ k -µ i | 2) measures the influence of the neighbor colors based on their distance to the Gaussian center. The effect of locality regularization is demonstrated quantitatively in Tab. 4 and qualitatively in Tab. 2.Our training scheme serves two goals. First, to avoid overfitting to the training views and to provide a coarse initialization for synthesizing novel views. Second, to provide scaled depth estimates for the training views to allow warping.During the pre-training stage, only the training views are used to optimize the 3D Gaussians for a small number of iterations using the loss:EQUATIONwhere L photometric is the photometric loss from [13] , L opacity is the L2 regularization term for the Gaussians' opacity, and λ, χ, ζ are the scalar weights. Opacity regularization [32] is added to remove the unobserved floating Gaussians that cause floating artifacts.In the intermediate stage, novel views are synthesized and used as supervision for the Gaussians to avoid overfitting. We found that sparse supervision from feature matching and noise in pseudo labels for novel views leads to model collapse in certain scenes. To address this, training views are also used to regularize the scene representation, with a downscaled L pre-training . Since the depth maps become available after the pre-training stage, we start using novel view consistency losses as described in Sec. 3.2:EQUATIONwhere κ, η are the scalar weights for the respective losses. The tuning stage is designed to refine the network using ground truth, provide supervision for unmatched pixels during the intermediate phase, and reduce the impact of noise in novel views. At the same time, it is crucial not to overfit the training views. Only training views are used for optimizing the L pre-training loss for a small number of iterations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Our method focuses on the research topic few-show novel view synthesis, which poses no noticeable societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Here we describe our experimental setup and then evaluate our approach against state-of-the-art few-shot NVS methods on commonly used datasets [12, 19, 20] . In addition, we compare our method with concurrent works. The tables highlight best results as first , second , third . Concurrent works in the tables are marked with an asterisk * .Figure 2 : Qualitative comparison on DTU [12] and LLFF [19] datasets. The results show that RegNeRF tends to produce blurred outcomes. 3DGS and DNGaussian introduce artifacts in the novel view. In contrast, our method generates better qualitative results.Datasets. Our approach is evaluated on commonly used real-word [12, 19] and synthetic [20] datasets. Following previous methods, 15 scenes are selected on the DTU [12] dataset in our experiments, in which the background is removed during evaluation. LLFF [19] dataset is composed of 8 real-world scenes. Following FreeNeRF [40] , we choose the 8th image for inference and then uniformly sample sparse views from the remaining images as the training set. For both datasets, the training set consists of 3 views. We use 1 4 resolution images on DTU and 1 8 resolution on LLFF. Synthetic Blender dataset [20] contains 8 scenes. We use the same setup as in [11] . Evaluation Metrics. We use PSNR [35] , SSIM, and LPIPS [45] to measure rendering performance. For LPIPS, a lower value denotes better performance, while a higher value is preferable for PSNR and SSIM. Our method is compared to NeRF-based and concurrent 3DGS-based methods, including NeRF [20] , SRF [5] , PixelNeRF [43] , MVSNeRF [3] , Mip-NeRF [1] , DietNeRF [11] , Reg-NeRF [22] , FreeNeRF [40] , SparseNeRF [34] , 3DGS [13] , DNGaussian [15] , FSGS [46] , DRGS [6] , and SparseGS [38] . Implementation details. Our method is trained for 10000 iterations in total on all the datasets. We use the same schedule by spending 2000 iterations on the pre-training stage, 7500 on the intermediate stage, and 500 on the tuning stage. In the pre-training stage and tuning stages, loss weights {λ, χ, ζ} are set to {1.0, 0.001, 0.001} respectively. In the intermediate stage, geometry, color, and semantic loss weights α, β, γ are set to 0.5, 0.05, and 0.001. Consistency and pre-training loss weights κ, η are set to 1.0 and 0.05. δ in the locality preserving regularization is set to 2.0. The gradient threshold θ grad is 0.1. θ in the agreement masking is 10.0. The scaling factor for the locality preserving loss δ is set to 0.2. We use a pre-trained VGG16 to extract semantic features. The experiments are run on one NVIDIA A6000 GPU.We report our rendering performance on commonly used real-world datasets in Tab. 1. We compare our method against NeRF-based methods and concurrent Gaussian-based approaches (marked with an asterisk * ). Our method performs on par or better than most of the presented approaches significantly improving over the baseline. On the DTU dataset, FreeNeRF [40] performs slightly better since it Table 1 : Quantitative evaluation on DTU [12] and LLFF [19] . We use 3 training views across all the datasets. Concurrent works are marked with an asterisk * .Setting DTU LLFF PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ SRF[5]Trained on DTU hallucinates over the missing regions, while Gaussian Splatting can't due to its explicit geometry encoding. Our method shows SoTA results on LLFF datasets. Interestingly, our method shows compelling results even with random Gaussian initialization. Finally, our method shows state-of-theart results on the synthetic Blender dataset in Tab. 3.We ablate over our key design choices presented in the paper on the DTU dataset with 3 training views in Tab. 4, in which random initialization is used for 3D Gaussians. In the ablations, the effects of each contribution are examined in isolation.Locality Preserving Regularization. The locality preservation loss improves rendering when added standalone to the vanilla Gaussian Splatting (row ii) and when added to the consistency losses in a multi-stage training setup (row xiv). Novel View Consistency. We ablate over novel view consistency losses and show our results in rows iii to ix. First, it is shown in rows iii to v that geometry, appearance, and semantic losses contribute to the higher quality rendering when added separately. In particular, geometry and color consistency remarkably improve the performance, since they directly influence the attributes of 3D Gaussians and are highly related to the view-synthesis task. In rows iii, iv, v it is shown that all three consistency losses lead to even better results. Experiments are conducted on different large-scale feature extraction backbones for the semantic loss, such as DINOv2 [23] , CLIP [27] , and VGG [30] in vii, viii, ix. The results demonstrate that DINOv2 and CLIP perform worse than VGG. This occurs because DINO and CLIP significantly down-sample features early in their encoders, leading to missing details and blurring, especially in the boundary regions, as illustrated in Fig. 6 . In contrast, our method utilizes the low-level features of VGG16 for the semantics constraint, balancing the positive effect from local semantic features and missing details. Minimum Loss in Consistency Losses. We demonstrate in the row x the effect of the min operation in Eq. ( 9) -Eq. (11) . min softens the constraints imposed on the 3D Gaussians by reducing the influence of errors in feature matching and rendered depth. Feature Matching. Experiments are also conducted using only a single known view for the consistency losses in row xii. We utilize a single training view, sample a random novel view in its frustum view, and reproject the points to it for additional supervision. There's a drastic difference in rendering performance compared to the variant where feature matching is not used in xii. This happens because the rendered depth in a training view may contain many errors that mislead the novel view synthesis. Feature matching algorithm plays an important role by choosing points appearing in several views for the warping, providing the ability to compute the agreement mask for filtering out unreliable color, depth, and features. RoMa [9] is particularly effective since it provides many correspondences between the images that are quite far from each other. Experiments are conducted with a classical feature matching algorithm [17] in xiii, but it fails due to a limited number of matches (see Fig. 7 ). 
conclusion: 
 We introduced FewViewGS, a few-shot novel view synthesis system based on 3D Gaussian Splatting as the scene representation that enables accurate renderings using only a handful of training images.We proposed an effective multi-stage training scheme, novel view consistency constraints, and regularization losses. Compared to previous state-of-the-art sparse novel view synthesis systems, we achieve high-quality rendering without relying on complex priors like depth estimation or diffusion models. We demonstrated that FewViewGS yields compelling results in rendering on both real-world and synthetic datasets. Limitations. Our method may struggle to render texture-rich regions accurately, particularly when novel views diverge significantly from the input views. Additionally, as indicated by our ablation study, utilizing a more precise feature-matching network to create dense matched pairs would be beneficial, as this can offer more robust supervision for the novel views. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our few-shot novel view synthesis pipeline generates results that are scenespecific and does not pose such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have included and cited all the datasets and methods we have used in our pipeline, also we have checked that their licenses are suitable for research purpose.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Comparison with SOTA methods under 6-view and 9-view settings. The results for 6-view and 9-view settings are given in Tab. 5. Our method shows improved performance over the baseline 3DGS with both SFM and random initialization. Notably, with SFM initialization, our method surpasses NeRF-based methods on both datasets.Comparison with SOTA methods on training and inference time. Tab. 11 displays the training time on a single NVIDIA A6000 GPU under 3-view setting on DTU. Our method demonstrates comparable training time and is faster than SparseGS [38] (which uses a pre-trained diffusion model) and FSGS [46] (which relies on a pre-trained depth network). inference time, since all methods utilize the same rendering process as 3DGS, their inference time is similar across these methods, at around 300 FPS.Ablation study for hyperparameters. We provide evaluations for hyperparameters and the multistage settings in Tab. 6 -Tab. 10 on DTU with 3 training views. ( 1) α in Eq. ( 12): we set it to a low weight due to the lack of ground truth for the depth. It can be seen that, in Tab. 7, larger (e.g. 0.2) gets poorer results, primarily due to noise in predicted depth values. ( 2) β in Eq. ( 12): due to the importance of color supervision for the novel view synthesis, a large weight is utilized in the experiment. In Tab. 6, a value of 0.5 achieves the best performance. (3) γ in Eq. ( 12): The results in Tab. 9 demonstrate that our method is robust to different γ values, with γ = 0.001 being optimal.(4) η in Eq. ( 15): The pre-training loss in the intermediate stage is employed to ensure sufficient supervision for novel views. In Tab. 8, a lower η (0.01) does not solve this very well, while higher η (0.1 and 0.5) fails to prevent overfitting to the training views. ( 5) Multi-stage settings: we evaluate the influence of iterations for each stage in our 3-stage training in Tab. 10. The last two rows show that a fewer/more iteration in the first stage leads to worse results, due to underfitting/overfitting to training views. The 1st-5th rows demonstrate that the fine-tuning stage enhances performance and achieves optimal results with 500 iterations. Visualization of unseen regions. In Fig. 3 , we compare the results of unseen regions between our method and FSGS [46] . The visualization shows that our method produces fewer artifacts and outperforms FSGS (which relies on a pre-trained depth network). This improvement is due to our multi-view consistency constraints, which offer accurate supervision for seen regions, while the proposed locality-preserving regularization maintains local smoothness and reduces artifacts.Visualization of predicted depth. Fig. 4 presents the generated depth by 3DGS [13] and our method. Our method produces clearer and more accurate depth estimations than the baseline 3DGS, demonstrating its effectiveness.Ablation study on multi-view constraints. In Fig. 5 and SIFT [17] implement feature matching between the Image1 and Image2.Ablation study on feature matching. In Fig. 8 , the results with and without feature matching are presented. It is shown that the model without feature matching generates false surfaces due to noise in the multi-view projection. In Fig. 7 , we show the difference between matching results generated by RoMa and SIFT. RoMa produces denser and more accurate matches than SIFT, which significantly enhances the model's performance when using RoMa. NeurIPS Paper Checklist 


Decision: This paper proposes a method for 3D Gaussian reconstruction from sparse view images. The framework includes three stages, where the first stage initializes 3D Gaussians using sparse views, the second stage synthesizes novel views with multi-view consistency constraint, the last stage further refines the scene for texture improvement. 

During rebuttal, reviewers raised concerns about the motivation of 3-stage design (Eqzk), performance on 6/9-views (8CAh), hyper-parameters (8CAh, oLUf), novelty (8CAh, ZPEG, oLUf) and initialization. The authors have provided additional experiments for 6/9-view setting, ablation studies on hyper-parameters, as well as differentiating with existing methods that utilize depth prior. For the discussion on initialization, the method follows standard dataset pre-processing (i.e., predicting camera poses using all views, and point cloud from training views only). This initialization facilitates a fair comparison with existing baseline methods.

Based on these consideration, this paper provides useful insights as well as effective designs for sparse-view reconstruction and is recommended for acceptance. The authors are encouraged to include additional experiments and ablation studies in the revision.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of few view (or sparse view) 3DGS multistage training with correspondence-driven losses that enforce projected colors, depths, and semantic features (extracted by a pre-trained VGG) are consistent. Contributions are straightforward and geometrically inspired.

In addition, the authors propose a locality preservation loss, which enforces color smoothness in neighbouring gaussians.
strength_weakness: strength_weakness: 1. Intuitions behind a 3-stage training strategy are not well established. If intermediate training already includes L_pre_training, why is further fine-tuning with L_pre_training only needed?
questions: questions: See weaknesses
limitations: limitations: Ok
paper_summary: paper_summary: This paper proposes a new method for sparse-view novel view synthesis. It proposes a multi-stage training scheme including pre-training, intermediate, and tuning stages. It introduces pre-trained dense matching models to find pixel correspondences between different-view images and encourage consistency. A Locality Preserving Regularization is proposed to encourage local smoothness.
strength_weakness: strength_weakness: 1. The comparisons are not enough. It lacks comparisons on more input views, such as the 6-view and 9-view settings used in FreeNeRF, which is also important to evaluate the proposed method in sparse-view settings.
2. The ablation studies are not enough. There are lots of hyperparameters listed in the implementation details, however, it is unclear how these hyperparameters are selected and how they impact the performance, which is important to evaluate the robustness of the proposed method.
3. The proposed method introduces ore-trained dense matching networks, thus I think it is still similar to those introduced pre-trained depth estimation networks.
questions: questions: 1. I wonder is the proposed method still work with more input views such as 6 views and 9 views.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper introduces a novel few-shot Gaussian Splatting method for synthesizing novel views. Unlike conventional approaches that rely on pre-trained monocular depth estimation or diffusion methods, the proposed method leverages the matches of available training views to generate novel sample views between the training frames. It employs color, depth, and image feature losses. Additionally, a novel regularization loss is introduced to preserve the local structure of the object. Experimental results demonstrate that the proposed method achieves significant performance improvements in both real and synthetic datasets.
strength_weakness: strength_weakness: The weaknesses of the paper are as follows:

1. The authors claim that relying on depth estimation or diffusion priors requires longer training times, but there is no comparison or ablation study provided to justify this claim. It would be beneficial to perform a comparison between state-of-the-art methods and the proposed method in terms of training and inference time.
2. The paper does not provide a justification for why the proposed method, which relies on matches between training views, performs better than methods that rely on pre-trained depth estimation or diffusion methods. A more in-depth analysis is needed to demonstrate how the proposed method outperforms these approaches.
questions: questions: Based on the weaknesses, the following questions arise:

1. How do the training and inference times of the proposed method compare to those of state-of-the-art methods?
2. How well does the performance of the proposed method generalize to unseen data? Given that state-of-the-art methods typically employ large pre-trained models for data reconstruction, the proposed method, which relies solely on matches between known views, may encounter difficulties when dealing with unseen regions.
limitations: limitations: The proposed method may face challenges when dealing with texture-rich regions that are not visible from the input views.
paper_summary: paper_summary: This paper presents FewViewGS to regularize sparse view 3DGS from unseen viewpoints without relying on pre-trained depth estimators. The main contribution is to reproject the pixels to an unseen view and calculate the losses at corresponding pixels found by image matching. Experiments on LLFF, DTU, and Blender datasets demonstrate the proposed method can achieve competitive or superior performance compared to existing SOTA methods.
strength_weakness: strength_weakness: - The main contribution of this paper shares the same insight with SPARF [1], which also uses image matching to add supervision on unseen views. The main difference is that SPARF uses depth, while this work uses depth, color and semantic features, for which the novelty in this is limited. And also the multi-stage training is just a common composition of " initialization + regular training + refinement".

- The proposed equations have too many manual factors and artificial designs that lack mathematical theory (e.g. Eq (7, 8,13)). These all seem to be engineering attempts but not technical contributions. 

- Experiment is insufficient and unclear. 1) Although a  (Rand. Init.) is mentioned at one item in Table 1, I can not find any description of the used initialization for other settings, which may have a very huge influence on the performance. 2) And serving as a geometry reconstruction method, some geometry visualizations like depth are wished to report. 3) As the reprojection operation and semantic feature extraction may cost a lot of time,  I'm wondering if the training time would significantly increase.

- Performance improvements are somewhat limited compared to current methods. And although this work can be free from some pre-trained models, it has introduced other dependencies like VGG features and RoMa matching.

- The paper has some errors, for example: 1) Eq (6) is unclear on the part (C, Z), which is a 2D->3D back projection I guess.  2) The citation information of SPARF [1] (numbered as [34] in the paper) is wrong.

[1] Truong, Prune, et al. "Sparf: Neural radiance fields from sparse and noisy poses." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
questions: questions: Please clarify the weaknesses, and make additional explanations of the unclear parts.
limitations: limitations: The authors have not discussed the limitations. This work seems to heavily rely on some manual hyperparameters, and some pre-trained models like VGG features and RoMa matching, which may limit the applicability to more real-world scenes.

Comments: title: Reply to Reviewer oLUf
Reply to Reviewer oLUf
Reply to Reviewer Eqzk
Reply to Reviewer oLUf
Thanks for your reply
comment: We sincerely appreciate your time and efforts in reviewing our responses. We hope that our responses below adequately address your concerns.

We believe that the focus on specific terminology, rather than the actual process and results, is not conducive to the evaluation of the work's quality. We have been transparent in our methodology and provided a thorough explanation in our previous responses (we initialize the 3D Gaussians with dense reconstruction in COLMAP.). This step is performed within the COLMAP framework, specifically using the convert.py script in our provided FSGS code to extract the camera poses.

Our use of SfM followed by MVS is technically accurate and appropriately implemented in the context of our research.
The authors alway passby the most important problems but only quibble by mentioning some irrelevant and trivial things, attempting to divert the attention. They can't face the exposed problems but only deny them. Such a response is meaningless. As above, this paper has many serious problems. The quality can not match NeurIPS. I'll lower my rating.

BTW, for the  initialization part, the author  said that "We follow the SfM process provided in FSGS ('tools/colmap_llff.py' in FSGS code) to generate camera poses and 3D points". Thus, do the authors still claim that "This process involves two parts, SfM followed by MVS, and does estimate camera extrinsics during SfM, therefore, the statement that "this process does not estimate any camera poses" is simply incorrect"? See what 'tools/colmap_llff.py' exactly includes. Where is SfM in this file?
We sincerely appreciate your time and efforts in reviewing our responses. We hope that our responses below adequately address your concerns.

***Q1: Clarification of our novelty***

***A1:*** We strongly disagree with the reviewer's statement about novelty. This assertion is not only inaccurate but also a misrepresentation of our previous responses. Our method aims to address the overfitting and maintain the multi-view consistency, as detailed in Section 3.2. While SPARF also aims to mitigate overfitting, a challenge common to sparse view synthesis, the similarity with our method ends there.

Our method utilizes the correspondence priors to guide multi-view projection, filtering out outliers and generating reliable pseudo-labels, which have been verified to be crucial. Subsequently, we compute the appearance/geometry/semantic losses with gradient weighting for the novel view. This not only broadens the available view information but also maintains multi-view consistency. Notably, during loss computation, we again utilize correspondence priors to select the minimal loss among each matched pair, which can reduce the effect of large noises. Table 4ix and 4x prove the effectiveness of the minimal operation. 

Our method diverges significantly from SPARF in both approach and outcomes, delivering superior performance on 3DGS. The depth consistency loss (DCons) employed in SPARF is fundamentally limited by its reliance on single-view projections, where depth inaccuracies lead to flawed projections. The DCons in SPARF's original paper improves PSNR by 0.20 (from 20.81 to 21.01) on NeRF—a result that aligns with our own reproduction on 3DGS. This proves that the single-view projection on SPARF is insufficient to address the overfitting issue on both NeRF and 3DGS. In contrast, our depth supervision for novel views in Table 4iii improves the PNSR by 3.13.

***Q2: Analysis for semantic loss***

***A2:*** As explained in Lines 271-277 and Lines 441-443, DINOv2 and CLIP down-sample features early in the encoder with a large stride, e.g. 16. It is widely recognized that features extracted with a large stride often result in a loss of detail, which is the reason why the semantic loss with DINOv2 and CLIP obtains poorer performance. Additionally, Figure 3 further illustrates that using DINOv2 and CLIP as feature extractors generates worse results in some boundary and detailed regions. In contrast, our semantic loss using VGG can reduce artifacts and maintain details.

***Q3: Details of SFM initialization***

***A3:*** We have already explained in our previous response that we initialize the 3D Gaussians with dense reconstruction in COLMAP. This process involves two parts, SfM followed by MVS, and does estimate camera extrinsics during SfM, therefore, the statement that "this process does not estimate any camera poses" is simply incorrect. Training and test views are both served as input for SfM to obtain their poses, then only the training views are used during MVS to get the fused point clouds. The technically sound initialization method name for the shorthand "SfM initialization" would be "Initialization based on the poses estimated by SfM for all views, and the fused point clouds from MVS on training views only", which will need to be shortened when referring to. Thus the initialization method we used is actually SfM+MVS, rather than just MVS as the reviewer suggested.

The integrity of our initialization is by no means affected by the name, we ensure the dense point clouds are only from the training views, making it fair and comparable with the baselines.

***Q4: Comparison of training time and results***

***A4:*** Our method, which incorporates RoMa and VGG, achieves significantly reduced training time while delivering superior performance in sparse view synthesis compared to networks relying on pre-trained depth/diffusion models, such as FSGS and SparseGS. As noted in our previous responses, only the VGG model introduces additional training time. However, as detailed in Table S7 of the uploaded file, our method requires just 5.82 minutes of training—substantially less than the 51.78 minutes needed by SparseGS (which uses a pre-trained diffusion model) and the 10.90 minutes required by FSGS  (which relies on a pre-trained depth network).

Furthermore, our approach not only enhances efficiency but also surpasses SparseGS and FSGS in performance, as demonstrated in Table 1. While incorporating VGG does involve some additional training time compared to raw 3DGS, the efficiency and speed trade-off is outweighed by the significant improvements in performance. Our method offers considerable gains over both raw 3DGS and methods reliant on pre-trained depth or diffusion models. For example, it improves PSNR by 4.09 dB on DTU and 3.32 dB on Blender compared to 3DGS.

In summary, the integration of RoMa and VGG enhances the overall speed and effectiveness of our method, making it a notable advancement over previous approaches.
I'll respond to some important problems, but not mean that other parts are good:

1. For **novel view consistency constraints**: According to the response, the authors agree that their work lacks novelty, since the novelty claimed by the authors, eq (7), is just a simple filter mask supported by a manual hyperparameter, while the other parts in "novel view consistency constraints“ is strongly similar to SPARF. 

    on the part of the effect, the provided reproduction experiment of SPARF on 3DGS is not convincing for me, as there is not any explanation about why the performance gap happens. BTW, according to its paper, SPARF can achieve the performance of PSNR 21.01, SSIM 0.87, LPIPS 0.10 on NeRF backbone in the 3-view DTU setting, which can already fully validate the effect of their strategies. Considering the working principle of this work is extremely close to SPARF without significant innovation, it can be considered as an incremental work to transfer SPARF to 3DGS backbone.

 2. For **semantic loss**: According to Table 4 vi and ix, the improvement is very marginal. What's worse, when applying an unsatisfactory type of feature, the performance will drop significantly. On the other hand, Despite that the authors try to prove the effect through "The comparisons between Table 4i and 4v show a 3.2% increase in PSNR and a 4.5% reduction in LPIPS", the analysis is partial. It only compares the situations with raw 3DGS. According to the comparisons between Table 4 vi, vii, viii and ix, its effect significantly shrinks to none after other constraints are applied, showing its redundancy.


3. For the **so-called "SFM initialization"**, it shows that the authors lack the basic knowledge in multi-view geometry and the ability of discernment. Following the description and code process of FSGS, the author declared they use "the SfM process ...  to generate camera poses and 3D points, using only training views without any post-processing". However, first, there is a significant mistake in FSGS that it calls the process they used as SfM but actually is an MVS method. Second, this process does not estimate any camera poses. So, the initialization method the authors used is actually MVS, rather than SfM they announced in the paper and rebuttal. This problem shows that the authors do not even know what they are actually doing. Considering this problem is to be revealed after the reviewer's twice asks, I'm worrying about this work's quality.

4. For **introducing pre-trained models**: the authors did not directly reply to my questions. First, are RoMa and VGG pre-trained models? Then, can these pre-trained models escape from the problems that "make training time longer and highly dependent on the quality of the pre-trained models"?  In other words, is there not any extra time cost in order to make the proposed method as fast as raw 3DGS, especially when needing to use VGG to generate the semantic embedding for novel view k in eq (11) online? Or can it also work well when using models with poor pre-training quality? If not, how can the authors claim that their method is different from previous works in this part?
We sincerely appreciate your precious time and efforts in reviewing our paper and responses. We hope that the response provided below adequately addresses your concerns. 

***Q1: Is it possible to show those "unmatched pixels during the intermediate stage" to support your claims?***

***A1:*** We appreciate the review's suggestion on visualizations to better support our claim. We provided visualizations for the matched results in Figure 7. The white regions in Figure 7c and the areas without red lines in Figure 7d represent unmatched pixels, which lack effective supervision during the intermediate stage. We will include detailed descriptions in our revised version. 

***Q2: I am also wondering why you did not include an experiment with 1st: 2000 2nd: 7500 Third: 0, for a more direct comparison.***

***A2:*** We thank the reviewer for the suggestion on the experiment. The setting with '1st: 2000 2nd: 7500 Third: 0' achieves a performance of PSNR 18.37, SSIM 0.782, and LPIPS 0.198. We will integrate these results into our revised version.
We sincerely appreciate your time and efforts in reviewing our responses. We hope that our responses below adequately address your concerns.

***Q1: Clarification of our novelty***

***A1:*** In this response, we would like to clarify our novelty and design details. 
**(1) Novel view consistency constraints:** Our method offers a new solution to address overfitting and ensure multi-view consistency in sparse view synthesis by using correspondence priors to generate labels for novel views. Since accurate depth prediction is crucial for reliable view projection, our method projects matched pairs from training views to the novel view and filters out pairs with significant depth discrepancies in the novel view (see Eq 7). This design enhances label reliability over SPARF's single-view projection. The table below shows the results of SPARF's multi-view correspondence loss (MV-Corr) and depth consistency loss (DCons) on 3DGS, which is much worse than our method (PSNR 19.13, SSIM 0.792, LPIPS 0.186). Furthermore, DCons in SPARF shows limited improvement, see Row i and Row ii. In contrast, our depth supervision for novel views in Table 4iii (PSNR 18.17, SSIM 0.736, and LPIPS 0.198) greatly improves the performance of 3DGS.
|| Method | PSNR | SSIM |LPIPS |
| --- | --- | --- | ---| --- |
| i| 3DGS | 15.04  | 0.676 |   0.246 |
| ii | + DCons | 15.17 | 0.699  |   0.232 |
| iii| + MV-Corr | 16.25 | 0.722 |  0.219 |
| iv| + MV-Corr + DCons | 16.37 |  0.729| 0.214 |

**(2) Multi-stage training:** Common multi-stage training typically involves an additional teacher network or large dataset. In contrast, our approach tailors the multi-stage training to sparse view synthesis, eliminating the need for extra networks or datasets. It alternates between training and novel views, expanding the view information and solving overfitting. As shown in Table 4ix and 4xi, this multi-stage approach improves the PSNR by 4.4\%. **(3) Locality preserving regularization:** Our locality regularization explicitly constrains the color attributes of 3D Gaussians to preserve local smoothness in 3D space. Table 2 demonstrates reduced artifacts and improved accuracy, and Table 4i and 4ii highlight gains in all metrics. **(4) Loss design:** Our proposed loss functions integrate additional constraints directly, which is intuitive and effective to address issues in sparse view synthesis, without increasing inference time. **(5) $L_{sem}$:** Our work introduces the semantic constraint to integrate local constraints and reduce artifacts. The comparisons between Table 4i and 4v show a 3.2\% increase in PSNR and a 4.5\% reduction in LPIPS. Indeed, as discussed in Lines 272-277, the local constraint needs to balance the benefits of local semantics with the risk of missing details. **(6) Other details:** We apply consistent values for $\theta$ and $\theta_{grad}$ in Eqs (9, 10, 11), to avoid too many human factors. Fine-tuning these parameters and exploring better mapping functions could improve results. 

***Q2: Details of SFM initialization***

***A2:*** The code for SFM process will be released, providing all the details. **(1) SfM initialization**: We follow the SfM process provided in FSGS ('tools/colmap\_llff.py' in FSGS code) to generate camera poses and 3D points, using only training views without any post-processing. The choice of training/test views is consistent with FSGS on LLFF and DNGaussian on DTU. Novel views are randomly interpolated between training views. **(2) Visualizations**: Figure S2 shows 3DGS results generated with SfM initialization, a 3-view setting, and size_threshold=None (as used in  FSGS and DNGaussian). We will release the model for further clarification. **(3) COLMAP in DTU**: Following FSGS, we use the dense reconstruction in COLMAP to generate 3D points, with only training views, which successfully yields point clouds on all scenes except 'scan110'. They are used to initialize 3D Gaussians. 'Scan110' is initialized randomly.

***Q3: Comparison with other works based on pre-trained depth/diffusion models.***

***A3:*** We try to clarify the confusion below. **(1) Training time:** We pre-compute and store VGG features and RoMa-matched pairs during data pre-processing, avoiding extra training time and redundant computations. However, pre-trained depth/diffusion models generate results online, making them more computationally expensive than pre-trained Roma and VGG models. **(2) Quality:** Networks using pre-trained depth/diffusion models generate labels for the novel view often ignore the priors in training views and lack multi-view consistency. In contrast, our method projects labels from training views to novel views, ensuring higher accuracy and multi-view consistency. Using a pre-trained depth network for novel views on 3DGS yielded PSNR 16.74, SSIM 0.734, LPIPS 0.214. Our method with depth supervision alone in Table 4iii (PSNR 18.17, SSIM 0.736, and LPIPS 0.198) outperforms this.
Thanks for supplementing your results. It is now clear the 3rd stage provides additional performance improvements. Is is possible to show those "unmatched pixels during  the intermediate stage" to support your claims?

I am also wondering why you did not include an experiment with 1st: 2000 2nd: 7500 Third: 0, for a more direct comparison.
Thanks for the reply from the authors. After reading the rebuttal, this work still has the following problems.

1. Lack of Novelty. Summarized by the authors in the paper, this work has three main contributions: an effective multi-stage training scheme, novel view consistency constraints, and a color smooth regularization loss. First, as in the review, the multi-stage training is just a common composition of " initialization + regular training + refinement", with no independent novelty on this point. And the color smooth regularization loss is just a trick to add local smoothness to reduce some artifacts, which only improved PSNR but not SSIM and LPIPS, demonstrating its limited effect on the overall quality. If these are declared to be the most important novelties in the paper, I can only believe there are no other valuable things to show. 

    And for the novel view consistency constraints, it is still the incremental technique of SPARF even according to the rebuttal. The contribution lies only in some loss designs with no interesting insight, moreover, has too many manual hyperparameters in eq(7,9,10,11) and replaceable mapping functions like $exp()$ in eq(8, 13).  Besides, L_sem is less effect according to Table 4 (vi, ix, xiv) and Figure 4. Especially, some details became worse after adding L_sem in Figure 4. It is also sensitive to the type of feature. I do not think these novelties can match the bar at NeurIPS.

2) Details of "SFM initialization" is still not clear enough. Please explain how many views are involved in the SfM process to get the camera poses and point cloud, how to align the excluded test views into the camera coordinate system, and whether is there any post-processing of the point cloud.  As far as I know, original 3DGS can not recover such precise scenes shown in Figure S2 when just use 3-view SfM point cloud as initialization, and COLMAP will fail in some DTU scenes when using only three training views. I have doubts about these.

3) The authors claimed introducing depth or diffusion models will "make training time longer and highly dependent on the quality of the pre-trained models" (lines 40-41), which is the stem of this work. However, why introducing pre-trained RoMa and VGG can escape from these weaknesses? The logic is weird.

I'll temporarily keep my rating.