Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

Authors: liang  han
junsheng  zhou
yu-shen  liu
zhizhong  han
shen  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 16:38:31

Abstract: Novel view synthesis from sparse inputs is a vital yet challenging task in 3D computer vision. Previous methods explore 3D Gaussian Splatting with neural priors (e.g. depth priors) as an additional supervision, demonstrating promising quality and efficiency compared to the NeRF based methods. However, the neural priors from 2D pretrained models are often noisy and blurry, which struggle to precisely guide the learning of radiance fields. In this paper, We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior as supervision. Our key idea lies in exploring the self-supervisions inherent in the binocular stereo consistency between each pair of binocular images constructed with disparity-guided image warping. To this end, we additionally introduce a Gaussian opacity constraint which regularizes the Gaussian locations and avoids Gaussian redundancy for
improving the robustness and efficiency of inferring 3D Gaussians from sparse views. Extensive experiments on the LLFF, DTU, and Blender datasets demonstrate that our method significantly outperforms the state-of-the-art methods.

Main paper: 
 introduction: 
 3D reconstruction technologies [28, 19] have demonstrated significant advances in synthesizing realistic novel views given a set of dense input views. To explore the challenging task in harsh real-world situations where only sparse inputs are available, some studies learn NeRF [28] with specially designed constraints [17, 44, 43, 49] and regularizations [30, 54, 50] on the view scarcity. However, NeRF-based methods often suffer from slow training and inference speeds, leading to high computational costs that restrict their practical applications. 3D Gaussian Splatting (3DGS) [19] has achieved notable advantages in rendering quality and efficiency. However, 3DGS is still facing severe challenges with inputting sparse views, where the unstructured 3D Gaussians with limited constraints tend to overfit the given few views, resulting in geometric inaccuracies for scene learning. Some recent studies [31, 71, 22, 51] on sparse view synthesis based on 3DGS employ the commonly-used depth priors from pre-trained models as additional constraints on the Gaussians geometries. However, the neural priors are often noisy and blurry, which struggle to precisely guide the learning of radiance fields.In this paper, we aim to design a method that does not require external prior as supervision, which directly explores the self-supervisions from the few input views for improving the quality and efficiency of sparse 3DGS. We justify that the key factors in achieving this goal include 1) learning more accurate scene geometry of Gaussians which leads to consistent views synthesis, and 2) avoiding redundant Gaussians near the surface for better efficiency and filtering noises.For learning more accurate scene geometry of Gaussians, we explore the self-supervisions inherent in the binocular stereo consistency to constrain on the rendered depth of 3DGS solely utilized from given input views and synthesized novel views. Our key insight lies in the observation that binocular image pairs implicitly involve the property of view consistency, as demonstrated in the binocular stereo vision methods [12, 11, 58] . Specifically, we first translate the camera of one input view slightly to the left or right to obtain a translational view, from which we render the image and depth from 3DGS. The rendered image and the input one thus form a left-right view pair as in binocular stereo vision. We then leverage the rendered depth and the known camera intrinsic to compute the disparity of the view pair. We conduct the supervisions by warping the rendered image of the translational view to the viewpoint of the input image using the disparity, and constrains on the consistency between the warped and input images.To further reduce redundant Gaussians near the scene surface and enhance the quality and efficiency of novel view synthesis, we propose a decay schema for the opacity of Gaussians. Specifically, we simply apply a decay coefficient to the opacity property of the Gaussians, penalizing the opacity during training. To this end, Gaussians with lower opacity gradients (i.e., where the increase in opacity is smaller than the decay) are pruned, while Gaussians with higher opacity gradients (i.e., where the increase in opacity is greater than the decay) are retained. As the optimization process continues, redundant Gaussians are filtered out, and those newly generated (copied or split) Gaussians that are closer to the scene surface are retained, resulting in cleaner and more robust Gaussians. This opacity decay strategy significantly reduces artifacts in novel views and decreases the number of Gaussians, improving both the rendering quality and optimization efficiency of 3DGS.Additionally, to achieve better geometry initialization for improving 3DGS quality when conducting optimization on sparse views, we use pre-trained keypoints matching network [42] to generate a dense initialization point cloud. The dense point cloud describes the geometry of the scene more accurately, preventing Gaussians from appearing far from the scene surface, especially in low-texture areas where the distribution of Gaussians is subject to limited constraints.In summary, our main contributions are as follows.• We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior as supervision. We explore the self-supervisions inherent in the binocular stereo consistency to constrain the rendered depth, solely obtained from existing input views and synthesized views.• We propose an opacity decay strategy which significantly regularizes the learning of Gaussians and reduces redundancy among Gaussians, leading to better rendering quality and optimization efficiency for novel view synthesis from sparse view with Gaussian Splatting.• Extensive experiments on widely-used forward-facing and 360-degree scene datasets demonstrate that our method achieves state-of-the-art results compared to existing sparse novel view synthesis methods.2 Related WorksQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction clearly state the contributions and scope of the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
neural radiance field: 
 Neural implicit functions have made great progress in surface reconstruction [45, 65, 16, 64, 56, 62, 63, 60] , 3D representation [67, 53, 68, 23, 26, 66, 24] and generation [7, 70, 69, 47, 61, 25] . Detailed and realistic 3D scene representation has always been the research goal in the field of computer vision, and neural radiance fields (NeRFs) [28] has brought fundamental innovation to this domain. NeRF can reconstruct high-quality 3D scenes from sparse 2D images and generate realistic images from arbitrary viewpoints by representing scenes as continuous volume radiance functions.However, NeRF requires a large number of views as input during the training stage and exhibits limitations in both training and inference speed. Consequently, the following researches have primarily focused on addressing these bottlenecks by improving computational efficiency [10, 29, 39 , 19, 15, 4, 5, 9] or reducing the number of input views [30, 17, 44, 43, 6, 49, 37, 21, 8, 38, 36, 40, 50] , while continuously striving for enhanced rendering quality [1, 2, 3] .Notably, recent approach 3D Gaussian Splatting [19] has shown promising results in achieving real-time rendering capabilities without compromising rendering quality. 3D Gaussian Splatting [19] is an emerging method for novel view synthesis, which reconstructs high quality scenes rapidly by utilizing a set of 3D Gaussians to represent the radiance field in the scene. This method performs excellently when dealing with real-world scenes, particularly excelling in handling high-frequency details. Moreover, 3D Gaussian Splatting demonstrates significant advantages in inference speed and offers more intuitive editing and interpretability capabilities.Recent researches [30, 17, 44, 43, 6, 49, 37, 21, 8, 38, 36, 40, 50] have explored various approaches to generate novel views from sparse input images, focusing on enhancing both rendering quality and efficiency. Methods such as NeRF [28] and 3DGS [19] have been refined through various techniques, resulting in continuous improvement in the quality of novel view synthesis.Using depth priors obtained from pre-trained networks [55, 32, 33] to supervise neural radiance fields is a widely used technique [38, 44] . Some methods [30, 54, 50, 8] introduce regularization terms in NeRF to address the problem, such as frequency regularization [54] and ray entropy regularization [20] . Additionally, there are also some methods [50, 49, 17] that leverage pre-trained models such as Diffusion model to enhance the rendering quality of novel views. However, most methods incur high costs during training and inference. While some methods have improved inference efficiency through generalizable models [5, 57, 52] or by using voxel grids [40] , they often sacrifice rendering quality.Currently, with the advent of 3D Gaussian splatting, some methods use 3DGS for sparse view synthesis, such as FSGS [71] , SparseGS [51] , DNGaussian [22] and CoherentGS [31] . These methods utilize depth priors obtained from pre-trained models [32] as supervision. Depth constraints enable the unstructured Gaussians to approximately distribute along the scene surface, thereby enhancing the quality of novel view images. However, the depth priors from pre-trained models often contain significant errors and cannot make the Gaussians distribute to optimal positions. In contrast, our method employs view consistency constraints based on binocular vision, resulting in more accurate depth information and thereby achieving a more optimal distribution of Gaussians. Some concurrent studies [48, 34] also employed the concept of binocular stereo in 3DGS [19] , but they feed binocular images into a pre-trained network to obtain depth priors, instead of self-supervision.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Our paper includes a dedicated "Limitation" paragraph where potential limitations are discussed.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 The pipeline of our method is depicted in Figure 1 . In this section, we first review the 3D representation method based on 3D Gaussians. Then, we explain how to construct stereo view pair and utilize it to enforce view consistency in a self-supervised manner. Next, we introduce the opacity decay strategy and the dense initialization method for 3D Gaussians. Finally, we present the overall loss function used for optimization.Gaussian splatting [19] represents scene with a set of 3D Gaussians. Each 3D Gaussian is defined by a central location µ ∈ R 3 , and a covariance matrix Σ ∈ R 3×3 . Formally, it is defined asEQUATIONwhere the covariance matrix Σ has physical meaning only when it is positive semi-definite. Therefore, it can be decomposed into Σ = RSS T R T , S ∈ R 3 is a diagonal scaling matrix with 3 parameters, R ∈ R 4 is a rotation matrix analytically expressed with quaternions. In addition, for rendering the image, each Gaussian also stores an opacity value α ∈ R and a color feature f ∈ R k .The 3D Gaussian is projected into the 2D image space when rendering an image, the projected 2D Gaussian is sorted by its depth value, and then alpha blending is used to calculate the color of each pixel,EQUATIONwhere c i is the color computed from feature f and α ′ is the opacity of the 2D Gaussian, which is obtained by multiplying the covariance Σ ′ of the 2D Gaussian by the opacity α of the corresponding 3D Gaussian. The 2D covariance matrix Σ ′ is calculated by Σ ′ = JW ΣW T J T , where J is the Jacobian of the affine approximation of the projective transformation. W is the view transformation matrix.The optimization process of Gaussian splatting is initialized with a set of 3D Gaussians from a sparse point cloud from SfM. Subsequently, the density of the Gaussian set is optimized and adaptively controlled. During optimization, a fast tile-based renderer is employed, allowing competitive training and inference times compared to the fast NeRF based methods.The key factor in improving the rendering quality of 3DGS is to guide the Gaussians to be distributed close to the exact scene surfaces. To this end, we aim to design a constraint on the rendered depth of 3DGS which directly represents the Gaussian geometry. Previous studies commonly adopt the depth priors from pretrained models to guide the depth of 3DGS. However, the depth priors are often noisy and blurry, especially for complex scenes, which fails in providing accurate guidance.In this paper, we propose a novel prior-free method which leverages the binocular stereo consistency in 3D Gaussian splatting. We constrain on the rendered depth of 3DGS solely obtained by the existing input views and the novel views rendered by 3DGS. Specifically, we first render a novel view from a view point which is translated from one of the input views, leading to a pair of binocular vision images. We then shift the novel view to the perspective of the input image using disparity. Finally, we constrain the rendered image using the consistency between the input image and the warped image.Specifically, given an image view I l as input, we translate its corresponding camera position C EQUATIONIn practice, to make the reconstructed left image I shif ted differentiable, we use a bilinear sampler to interpolate on the right image I r to obtain each pixel of I shif ted . The sampling coordinates for each pixel are directly obtained from the disparity. Formally, we define the loss function byEQUATIONTo achieve better optimization with the loss function, we do not use a fixed camera position shift in practice. Instead, we randomly translate the input camera to the left or right by a distance within a range [-d max , d max ] in each iteration.Although some methods [38, 6, 21] use unseen views or neighboring training views as source views and warp them to the reference view for consistency constraints, they do not account for the impact of the distance and angle between the source and reference views on the effectiveness of the supervision.When the source view is rotated or moved away from the reference view, the warped image suffers severe distortion due to depth errors and occlusions. Compared to the ground truth (GT) image, this leads to larger errors, hindering the convergence of the image-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth. For the results using different source views, please refer to the Appendix.We further justify that relying solely on the depth constraints does not always lead to correct Gaussian geometries that are closely aligned with the exact scene surfaces. The reason is that the rendered depth varies with changes in the scale and opacity of the Gaussians, rather than being solely determined by their positions. While 3DGS flexibly optimizes the scale and opacity during training, which leads to deviations in the novel views. To address this issue, we design a simple strategy by applying a decay coefficient λ to the opacity α of the Gaussians, penalizing the opacity during training.EQUATION)Opacity increase due to gradient backward Opacity decay Surface We illustrate the opacity decay strategy in Figure 2 . Assuming all Gaussians are in the initialized state, due to the accumulation of constraints from multiple views, Gaussians near the scene surface typically have larger opacity gradients, allowing opacity to rise rapidly and construct the scene surface. However, some Gaussians far from the surface fail to get their opacity decreased and be pruned due to insufficient multiview consistent constraints, eventually affecting the rendering quality of novel views.We aim to improve and stabilize the optimization of 3DGS by filtering out the far away Gaussians which indicates incorrect geometry while remain the ones close to the exact scene surfaces. This is achieved by applying the opacity decay strategy. We justify that the strategy does not lead all Gaussians' opacity going down, such as the ones on the surface, since these Gaussians' opacity progressively increase. As illustrated in Figure 2 , the Gaussians far from the scene surface have lower opacity gradients due to fewer constraints, meaning their opacity increases less than those of the Gaussians on the scene surface. Under the opacity decay strategy, the opacity of Gaussians with lower opacity gradients gradually decreases until they are pruned. Conversely, the increase in opacity for Gaussians near the scene surface exceeds the decay magnitude, ultimately achieving a balance between the opacity increase and the decay, thereby preserving Gaussians close to the surface. Previous 3DGS methods [19, 71, 51] usually utilize a sparse point cloud generated by Structure from Motion (SfM) [35] to initialize 3D Gaussians. However, the point cloud produced by sparse views is too sparse to adequately describe the scene to be reconstructed. Although the splitting strategy in 3DGS can replicate new Gaussians to cover the under-reconstructed area, they are subject to limited geometric constraints and cannot adhere well to the scene surfaces, especially for low-texture areas where the distribution of Gaussians may be arbitrary. We therefore seek a robust approach to achieve better geometry initialization for improving 3DGS quality when optimizing from sparse views. Note that this prior is just used for initialization, and we do not use any this kind of priors during learning 3D Gaussians.To achieve this, we use a pre-trained keypoints matching network [42] to generate a dense initialization point cloud. Specifically, we arbitrarily select two images from the input images, input them into the matching network, and obtain matching points. We then leverage the triangulation method [14] , along with the camera parameters corresponding to these images, to project the matching points into 3D space. This forms a dense point cloud, providing a more robust initialization for the Gaussians.Compared with the sparse point cloud, the dense point cloud describes the geometry of the scene more accurately, preventing Gaussians from appearing far from the scene surface and ultimately leading to improved quality in novel view synthesis. The final loss function consists of two parts: the proposed binocular stereo consistency loss L consis as introduced in Eq. ( 4) and the commonly-used color reconstruction loss L color of 3DGS [19] . We define the overall loss function byEQUATIONwhere L color is composed of an L 1 loss and a structural similarity loss L D-SSIM , as denoted byEQUATION4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use the publicly datasets and provided sufficient instructions on how to use them for our method in the supplementary materials. The source code will be publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
datasets: 
 We conduct experiments on three public datasets, including the LLFF dataset [27] , the DTU dataset [18] and the NeRF Blender Synthetic dataset (Blender) [28] . Following prior works [30, 54, 17] , we used 3, 6, and 9 views as training sets for the LLFF and DTU datasets, and 8 images for training on the Blender dataset. The selection of test images remained consistent with previous works [17, 30, 54 ]. The downsampling rates for the LLFF, DTU, and Blender datasets are 8, 4, and 2, respectively.Since the LLFF and DTU are datasets of forward-facing scenes, they cannot be constrained by views from other directions during the optimization process. On the other hand, Blender is a dataset of 360-degree scenes, 8 images from different viewpoints are used as input during training, providing stronger constraints. Therefore, for the LLFF and DTU datasets, we utilize a pre-trained matching network PDC-Net+ [42] to obtain keypoints from input images, which are then used as the dense initialization point clouds. For the Blender dataset, we adopt random initialization as in the original 3DGS [19] . We choose some state-of-the-art NeRF-based and 3DGS-based sparse view synthesis methods for comparison. NeRF-based methods include DietNeRF [17] , RegNeRF [30] , FreeNeRF [54] , SparseNeRF [44] , ReconFusion [49] and MuRF [52] . 3DGS-based methods include DNGaussian [22] and FSGS [71] . Additionally, we compare against the original 3DGS [19].LLFF. Table 1 shows the quantitative results of the LLFF dataset with 3, 6, and 9 input views, respectively. Our method surpasses all baseline methods in terms of PSNR, SSIM [46] , and LPIPS [59] scores under different number of input views. When the number of input views increases to 9, it is almost adequate to provide sufficient color constraints. However, the evaluation scores of DNGaussian [22] do not show a significant improvement, indicating that it does not perform well with an increased number of input views. This is because errors in the depth prior negatively affect the optimization process. From the rendered results of novel views, several state-of-the-art methods are all perceptually acceptable. However, regarding the depth maps, RegNerf [30] , FreeNerf [54] , and FSGS [71] exhibit poor quality, indicating significant errors in their geometric estimation. SparseNeRF [44] and DNGaussian [22] indirectly utilize relative depth information from depth priors to supervise geometry estimation, hence achieving adequate rendering depth. Although FSGS [71] also exploits depth priors, it attempts to directly employ depth prior information by using depth correlation loss. It does not yield accurate scene geometry, instead, inaccurate depth prior information exacerbates scene geometry. In contrast, our method utilizes self-supervised view consistency loss to obtain more precise geometry, thereby recovering more structural details in novel views.DTU. Table 2 shows the quantitative results of the DTU dataset under 3, 6, and 9 input views respectively. MuRF [52] performs better on the DTU dataset when using 3 views as input. However, generalizable methods require large scale datasets and time-consuming training to learn a prior. Figure 4 illustrates the visual comparison of novel view synthesis for three scenes in the DTU dataset. To distinctly discern the differences among various methods, we select the synthesized images far from the training views for comparison in each scene. From the rendered results of novel views, it can be observed that NeRF-based methods produce blurry results, while FSGS [71] and DNGaussian [22] exhibit numerous artifacts due to insufficient constraints on Gaussians. Our method outperforms previous state-of-the-art methods on most evaluation metrics and achieves better visual quality as well.Blender. We evaluate our method on 360-degree scenes using the Blender dataset. Table 3 presents the quantitative results under 8 input views. Our approach outperforms all baselines in the PSNR score, and achieves comparable SSIM and LPIPS. We could not reproduce the results reported in the FSGS [71] on the Blender dataset and used the own test results in Table 3 . Figure 5 illustrates the visual comparison results of several 3DGS-based methods. It can be observed that FSGS [71] exhibits noticeable artifacts in the synthesized images due to insufficient Gaussian constraints, and the rendering results of DNGaussian [22] also has some distortions. In contrast, our method performs better in terms of detail preservation. To verify the effectiveness of the view consistency loss, opacity decay strategy and the dense initialization for Gaussians, we isolate each of these modules separately while keeping the other modules unchanged. We then evaluate the metrics and illustrate the visual results.As shown in Tables 4 and 5 , the performance decreases when removing any of the modules we proposed.Effectiveness of view consistency loss. To verify the effectiveness of the view consistency loss, we compare the depth maps from novel views. Figure 6 shows the visual comparison of rendered depth maps. The comparison in the figure shows that it is evident that the view consistency loss significantly improves the estimation of scene geometry.Effectiveness of opacity decay. Figure 7 (b) and (c) show the comparison of novel view synthesis results and the visualization of Gaussian centers for the trex scene in LLFF before and after applying the opacity decay. It can be seen that without the opacity decay, Gaussians appear far from the surfaces and there is noise near the surfaces. After applying the opacity decay strategy, the Gaussians far from the scene surfaces are eliminated, and the noisy point clouds are significantly reduced, thereby further improving the rendering quality of novel views.Effectiveness of dense initialization.  
conclusion: 
 In this paper, we propose a novel method for novel view synthesis from sparse views with 3DGS. We construct a self-supervised multi-view consistency constraint using the rendered and input images, and introduce a Gaussian opacity decay and a dense point cloud initialization strategy. These constraints ensure that the Gaussians are distributed as closely as possible to the scene surfaces and filter out those far from the surfaces. Our approach enables the unstructured Gaussians to accurately represent scene geometry even with sparse input views, resulting in high-quality novel rendering images. Extensive experiments on the LLFF, DTU and Blender datasets demonstrate that our method outperforms existing state-of-the-art sparse view synthesis methods.Limitation. Since our method utilizes the view consistency constraints for estimating scene depth, some scene areas with low texture may lead to the inaccurate depth estimation (e.g. the white background areas in the DTU dataset), thus failing to constrain the corresponding Gaussians. This results in the white Gaussians potentially occluding the object in the novel views. In contrast, DNGaussian [22] uses the depth prior estimated by the pre-trained network to constrain the Gaussians, preventing this scenario from happening. The visual comparisons are presented in the Appendix. A More Visualizations 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify hyperparameters in the "Implementation Details" section of the paper, with any unspecified hyperparameters being the same as the baselines. The supplementary materials include detailed information on data splits.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We report the average performance in the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide the information of GPU resource we used in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper proposes a method for synthesizing novel view images from sparse input views. The aim is to observe the objects in the input images from new perspectives. There is no generation of new or fake data involved.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original papers of the code and datasets we used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Figure A shows the visualization comparison of our method and DNGaussian [22] across all scenes in the LLFF [27] dataset. Each scene is trained using the same 3 input views. Although both methods achieve perceptually acceptable results, the novel view rendering results of our method are closer to the Ground Truth. Figure B shows an additional visual comparison between our method and DNGaussian [22] on the DTU [18] dataset. Each scene is trained using the same 3 input views. To clearly distinguish the differences between the two methods, we select a novel unseen view that is far from the training views for comparison. As analyzed in the limitations section of the main text, method cannot effectively constrain the white background in training views, resulting in objects being occluded by white Gaussians in some scenes. Even so, the rendering quality of the novel views from our method is still better than the results from DNGaussian.The Sugar [13] method applies entropy regularization to the opacity of Gaussians to bring Gaussians closer to the surface, similar to our proposed opacity decay strategy. Therefore, we also evaluate the performance of opacity entropy regularization on the LLFF dataset. Following Sugar [13] , the opacity threshold for Gaussians pruning is set to 0.5. We perform ablation studies on the LLFF [27] and DTU [18] datasets with the value of λ ranging from 0.96 to 1.0, and the results are shown in Table C . We find that the best performance is achieved when the value of λ is 0.995 on both datasets. Therefore we set λ to 0.995 for all datasets. We perform ablation studies for d max on the LLFF [27] and DTU [18] datasets. The value of d max gradually increases from 0.1 to 0.8, we obtain the results as shown in Table D . We find that there is almost no performance increase on both the LLFF and DTU datasets when d max is greater than 0.4, therefore we set the d max to 0.4 for all datasets.We conduct experiments on the LLFF [27] and DTU [18] datasets with several different initialization, including random initialization, sparse initialization, and using the LoFTR [41] matching network to construct initial point clouds. The results are shown in Table E . PDCNet+ [42] can obtain more matching points than LoFTR [41] , so the initialized point cloud obtained by PDCNet+ results in better performance. For the LLFF [27] dataset, the input views have large overlap, SfM methods can generate point clouds with better quality than LoFTR. Therefore, using sparse point clouds generated by SfM as initialization yields better performance than LoFTR. However, for the DTU [18] dataset, where there is small overlap among input views, SfM-generated point clouds are too sparse. Thus, the performance is even worse than random initialization. Since our method uses an opacity decay strategy, we do not use the opacity reset strategy from the original 3DGS, and we also remove the scale threshold for Gaussians pruning from the original 3DGS.We train our model on an RTX 3090 GPU, running 30,000 iterations per scene on the LLFF and DTU datasets, which takes approximately 10 minutes each. For the Blender dataset, we train for 7,000 iterations per scene, which takes approximately 3 minutes each. The storage space required for the Gaussian point clouds is about one-third of that required by the original 3DGS.LLFF. The LLFF [27] is a forward-facing dataset with 8 scenes. Following previous works [30, 54] , we take every 8th image as the novel views for evaluation. The input views are evenly sampled from the remaining views. During training and evaluation, the images are downsampled by a factor of 8, resulting in a resolution of 378×504 for each image.DTU. The DTU [18] dataset consists of 124 scenes. We follow previous works [30, 54] The first 3 and 6 image IDs correspond to the input views in 3-view and 6-view settings respectively. We use 25 images as novel views for evaluation, with the following IDs:During training and evaluation, the images are downsampled by a factor of 4, resulting in a resolution of 300 × 400 for each image.Blender. The Blender [28] dataset consists of 8 synthetic scenes. For fair comparison, we follow previous works [30, 54] and use 8 images as input views for each scene, with the following IDs: 26, 86, 2, 55, 75, 93, 16, 73, 8. The 25 test views are sampled evenly from the testing images for evaluation. During training and evaluation, the images are downsampled by a factor of 2, resulting in a resolution of 400 × 400 for each image.NeurIPS Paper ChecklistJustification: Our research conform with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper received overall positive reviewers with engaged author and reviewer discussion after the author rebuttal.  The reviewers appreciated the novelty of the paper and the results.  The rebuttal and discussion addressed most concerns.  

The remaining negative reviewers expressed the following concerns:

I mainly have three concerns: (1) there are methodological errors, e.g., the backward warping process. (2) the advantage of binocular sampling don't convince me enough, as the binocular sampling is a simplification of existing sampling method, and relative analysis is not appear in the paper. (3) from the results in Tab. K, it seems that the method is sensitive to the type of matching model, especially in DTU dataset, and I am not sure does this method have enough generalization.... I recommend the author including the experiments reported in the rebuttal, doing more sensitivity analysis about the type of matching models in different scenes, and giving more unique advantage of binocular sampling.

The AC reviewed all of this and this there is a nice contribution and that the remaining concerns can be addressed in the final revision.  Thus the AC recommends acceptance of the paper and that the authors very carefully include all experiments in the rebuttal and address the reviewers' comments.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel method for 3D Gaussian-based sparse view synthesis. Specifically, initialized from dense point clouds, the depth-warping loss and  the opacity penalty strategy are introduced to obtain accurate 3D Gaussians. Extensive experiments on the Blender, LLFF and DTU dataset have demonstrated the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Since depth-warping loss is common in many NeRF or 3D Gaussian-based papers, i'm not sure whether this can be viewed as a contribution.
2. Since the dense point clouds are generated from pretrained models, i do not think that this method can be claimed as prior free, please check the definition of what is prior free.
questions: questions: Please see weaknesses. Since the initialization is based on dense point clouds, i think it would be better to do more ablation studies on more dataset, such as the DTU and the Blender dataset. However, in general, i think this is a relatively good work and i would recommond a borderline accept for this paper. I'm glad to improve rating if the weaknesses can be addrssed.
limitations: limitations: Please see weaknesses.
paper_summary: paper_summary: This paper proposes 3D Gaussian splatting from sparse views aided by pre-trained key points matching initialization, binocular stereo constraints, and opacity regularization. Binocular stereo constraints utilize perspective projection to warp synthetic stereo images into the training images for self-supervision. Opacity regularization guides the densification/pruning process to drop non-active Gaussians.
strength_weakness: strength_weakness: 1. Adding an opacity regularization does not seem enough to consider it a contribution.
2. Overstatements. The authors claim their method is prior-free, however, they use a very strong prior for dense initialization.
3. Unclear results. How is your method without any of your contributions (Table 4) 1.5 dB better than the baseline 3DGS?
4. Missing ablation studies. What is the performance if only stereo loss or only density regularization is applied?
5. I am afraid the main performance gains come from the dense keypoint initialization.
6. Inconsistent performance improvements in Table 4. Dense keypoint initialization provides ~3dB improvements, and stereo consistency and opacity regularization provides ~4dB. How is the combination of both only providing ~5dB? I understand improvements cannot be linearly added, but the gap seems unreasonable.
questions: questions: See weaknesses
limitations: limitations: The authors provided limitations. However, the occlusions due to stereoscopic synthesis where not addressed (even though they are small due to small camera baselines).
paper_summary: paper_summary: This paper proposes a new method related to the problem of novel view synthesis in a sparse input setting. The authors propose to exploit stereo consistency as a self-supervision signal in contrast to the use of priors such as diffusion which tends to produce less precise geometry. Specifically, the work proposes the use of binocular stereo consistency as a guiding signal, i.e. with a horizontal translation a disparity between two images can be calculated given intrinsics and depth, and this can be leveraged to warp the translated view to input image and assess the consistency between images. Furthermore, the authors address the issue of filtering the redundant gaussians by introducing a decay scheme based on penalising the opacity leading to pruning gaussians far from the surface of the object. The paper evaluates the proposed approach in a commonly used in literature sparse novel view scenario, using 3 common datasets. The authors present competitive results in their evaluation. An ablation of system elements/contributions is also included.
strength_weakness: strength_weakness: - The main weakness I see is the lack of comparison to generalisable methods [1, 2, 3, 4, 5, 6, 7]. DTU dataset which is already used in the paper is also commonly used to benchmark methods that perform a training step on a selection of scenes. While the setup of generalisable, and per-scene optimisation is favourable towards the former, I believe the proposed method would be competitive. This would potentially be an interesting comparison, both quantitatively (I would expect competitive scores), and qualitatively (varying artefacts between the two approaches), and would additionally increase the value of the benchmark. 
- I believe that computational performance comparison is a missed opportunity. Proposing one of the first methods for sparse novel view synthesis would benefit greatly by emphasising the speed differences between the methods. It would be a great addition to show the training and inference speed with respect to NeRF-based methods (to show improvement in quality metrics and speed), and other GS-based methods (to show probably similar speed, but increased reconstruction quality).
- Both disparity consistency loss and opacity-based pruning rely on the choice of hyperparameters - namely camera shift $d_{max}$ and decay coefficient $\lambda$. It would be great if this choice was motivated and its influence on the performance analysed (including analysis of whether the same values should be applied across tested datasets).
- The authors show estimated depth maps for their method with and without the use of view consistency loss. It would be good to see the depth maps from other methods (e.g. DNGaussian) - it would be interesting to see a comparison with methods that use depth priors as supervision. It would also be nice to see the comparison to ground truth depth (where not available, an interesting comparison would be the reconstruction method in a dense setup scenario - as a proxy for ground truth)
- The authors mention in the checklist that results in the paper are reported as an average performance - it would be good to specify whether experiments were run multiple times, and investigate how the performance varies across the runs. 

[1] Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa, *pixelNeRF: Neural Radiance Fields from One or Few Images*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021

[2] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang
Jingyi Yu, Hao Su, *MVSNeRF: Fast Generalizable Radiance Field Reconstruction
from Multi-View Stereo*, IEEE/CVF International Conference on Computer Vision, 2021

[3] Mohammed Suhail, Carlos Esteves, Leonid Sigal, Ameesh Makadia, *Generalizable Patch-Based Neural Rendering*, European Conference on Computer Vision, 2022

[4] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou, *Efficient Neural Radiance Fields for Interactive Free-viewpoint Video*, SIGGRAPH Asia, 2022

[5] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, *Is Attention All That NeRF Needs?*, International Conference on Learning Representations, 2023

[6] Thomas Tanay, Matteo Maggioni, *Global Latent Neural Rendering*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

[7] Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Marc Pollefeys, Andreas Geiger, Fisher Yu, *MuRF: Multi-Baseline Radiance Fields*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024
questions: questions: - How was the camera shift $d_{max}$ and decay coefficient $\lambda$ chosen? Is the same value appropriate to all datasets? I would imagine it would be increased and decreased based on the distance of the object to the camera. Also, are the values of camera shift sampled uniformly?
- How would the application of disparity consistency loss to the existing methods look like? I would imagine it would be rather straightforward. Did the authors try that?
- Previous methods, as pointed out by the authors, use depth priors (using pre-trained models) as the supervision. Have the authors tried adding such supervision to their approach? I understand that the method is designed to be prior-free but it would be interesting to see an indication whether this would improve the performance. 
- Zoom-in areas could be put side-by-side (possibly in the appendix, or another row in the figure with zoom-ins). It is a bit hard to compare small elements the authors want the reader to focus on when zooming the document - the focus areas (red boxes) are a bit far away and hard to notice small differences.
limitations: limitations: The authors provide a brief limitation section. I believe the issue with consistency constraints in textureless areas is a very good mention. Also, the experiment with masked training shows that the limitation is identified correctly.
paper_summary: paper_summary: This paper proposed a 3D Gaussian Splatting method to render novel views with sparse inputs. This paper first use a pre-trained keypoints matching network to generate dense point initializations, and propose a consistent loss between the warped binocular stereo image. Meanwhile, they introduce a opacity penalty strategy to further learn the accurate gaussian points. The proposed method achieves the state-of-the-art performance on LLFF, DTU and Blender datasets.
strength_weakness: strength_weakness: 1. While the reported results show that the opacity penalty strategy brings the best improvement, it's motivation is not clear.  
- Why add a coefficient to the opacity can guide the gaussian point to be closer to the scene surface? (L52. "guiding the remaining Gaussians to be closer to the scene surface"), especially under the setting of $\lambda=0.995$ (has a minor difference with 1.0), and are there any ablation results with different value configurations of $\lambda$. Because the coefficient only works on the opacity, the opinion of guiding the position of Gaussian to be closer to the scene surface does not convince me enough.
- The statement of "the gaussian close to surface has larger gradient and the gaussian far from the surface has smaller gradient" (Line176-179) seems unconvincing. The accurate Gaussian near the surface can render accurate colors with relatively small computational loss and should have small gradients. And on the contrary, the wrong Gaussian may has a larger gradient, like the situation of "wall" in the overfitting scenario.
- Such a small modification can bring nearly 2dB improvement is shocking, and the explanation and content of this part is not adequate. Adding more analysis might help make it more convincing.
2. The proposed binocular stereo consistency is actually a method to regularize the model using unseen virtual views. 
- And using the virtual unseen view is a demonstrated existing strategy that has been adapted by many methods like [1,2,3], and the difference of this method is using the different sampling space of virtual views (the sampling space in this paper only has horizontal translation and no rotation).
- Intuitively, using more diverse sampling space like [1,2,3] may be more powerful, because it can use more unseen views from different angles to regularize the model. So what are the advantages of sampling only in the horizontal direction and are there any experimental comparisons?
- The disparity-based or depth-based warping will introduce the occlusion, which has a great impact of the performance. And how to mitigate this is not clarified in this paper.
- There might be an inaccuracy about the method. It seems like you want to use the backward warping to warp the right image to the left camera coordinate (Line148-160). Thus the depth needs to be pixel-aligned with the left image in Eq. (3), but you just use the depth of right image. This just confuses me a lot.
3. It seems that the dense initialization using the keypoints matching network plays an important role in the overall performance. 
- How about the sensitivity of the method to different matching network?
- How about the performance of the method w/o dense or sparse initialization and only w/ the random initialization like DNGaussian. Because in the sparse scenario, popular SFM methods are hard to work and always fail to generate accurate points.
4. There are also some confusions in the experiment.
- The results reported by FSGS differ significantly from those in the paper, and there seems to be no explanation for this in the paper.
- Some visualization results (e.g., Fig. 3, Fig. 4 and Fig. 9) are not labeled with the corresponding methods, which makes comparison difficult.
- Since the runtime of  3DGS-based method is relatively fast, reporting the error bar or standard deviations of multiple experiments can help to enhance the credibility of the paper.

[1] SPARF: Neural Radiance Fields from Sparse and Noisy Poses, CVPR2023.

[2] GeCoNeRF: Few-Shot Neural Radiance Fields via Geometric Consistency, ICML2023.

[3] RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs, CVPR2022.
questions: questions: Refer to weaknesses for details. And due to these doubts, I tend to give the borderline and hope to see the author's response.
limitations: limitations: Have declared in the paper.

Comments: title: Responses to Reviewer UVce
Thanks for detailed response
Thanks to Reviewer sng2
Thanks for your reply
Responses to Reviewer gh5j
Responses to Reviewer gh5j
We will be happy to take any questions
comment: Dear reviewer gh5j,

As the reviewer-author discussion period is about to end, can you please let us know if our rebuttal addressed your concerns? If it is not the case, we are looking forward to taking the last minute to make further explanation and clarification.

Thanks,

The authors
Dear Reviewer UVce,

Thanks for your feedback.  

**_Q1: Whether the choice of $\lambda$ is empirically easy._**

We conduct ablation studies for $\lambda$ on the DTU and Blender datasets, the results are as follows:

Table G2: Ablation studies for $\lambda$ on DTU dataset with 3 input views.

| $\lambda$ | 0.960 | 0.970 | 0.975 | 0.980 | 0.985 | 0.990 | 0.995 | 0.998 | 1.0   |
|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| PSNR   | 17.06 | 18.11 | 19.60 | 19.63 | 19.77 | 20.49 | 20.71 | 20.67 | 19.08 |
| SSIM   | 0.785 | 0.813 | 0.846 | 0.848 | 0.853 | 0.861 | 0.862 | 0.862 | 0.832 |
| LPIPS  | 0.223 | 0.196 | 0.158 | 0.152 | 0.139 | 0.121 | 0.111 | 0.109 | 0.131 |

Table G3: Ablation studies for $\lambda$ on Blender dataset with 8 input views.

| $\lambda$ | 0.960 | 0.970 | 0.975 | 0.980 | 0.985 | 0.990 | 0.995 | 0.998 | 1.0   |
|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| PSNR   | 23.68 | 24.24 | 24.37 | 24.48 | 24.57 | 24.69 | 24.71 | 24.59 | 23.47 |
| SSIM   | 0.851 | 0.864 | 0.869 | 0.871 | 0.873 | 0.872 | 0.872 | 0.870 | 0.861 |
| LPIPS  | 0.143 | 0.125 | 0.118 | 0.114 | 0.107 | 0.103 | 0.101 | 0.101 | 0.112 |

We can see that the effect of $\lambda$ on performance is generally consistent across different datasets.
Dear Authors,

Thank you for the amount of information provided in response to my review. I have read your rebuttal. 

I find the experiment with using your additional supervision with DNGaussian particularly convincing. I saw that performance is very competitive against MuRF (except 3 views DTU but that may be due to how DTU is evaluated, i.e. same views for all the scenes).

One last worry I have is whether the choice of $\lambda$ is empirically easy. The method seems to be rather sensitive to small changes. Unless the value of $0.995$ is the best or close to best for all dataset.
Dear Reviewer sng2,

Many thanks for all the helpful comments and positive assessment. We really appreciate your expertise and the score upgrade.

Best,

Authors
Thank you for addressing my comments. I am willing to raise the score, provided the clarifications and ablation studies are included in the final paper version.
Ablation studies for the pruning threshold on LLFF dataset with 3 input views.

| **pruning threshold** | 0.005 (baseline) | 0.010 | 0.015 | 0.020 | 0.025 | 0.030 | 0.035 | 0.040 | 0.045 | 0.050 |
|-----------------|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| **PSNR**        | 20.48            | 20.30 | 20.23 | 20.05 | 19.93 | 19.93 | 20.09 | 19.96 | 20.08 | 19.91 |
| **SSIM**        | 0.715            | 0.711 | 0.708 | 0.707 | 0.705 | 0.706 | 0.707 | 0.706 | 0.707 | 0.706 |
| **LPIPS**       | 0.218            | 0.212 | 0.203 | 0.205 | 0.206 | 0.205 | 0.205 | 0.205 | 0.205 | 0.204 |

We perform ablation studies for the pruning threshold, using dense initialization and view consistency loss by default. The experimental results show that directly increasing the pruning threshold leads to a performance decline.
Dear Reviewer gh5j,

Thanks for  your feedback.

**_Q1: How about directly adjusting the threshold of the pruning operation._**

It is clear that the opacity penalty strategy is different from setting a fixed pruning threshold. The opacity penalty strategy is applied to all Gaussians and acts as a global constraint. In contrast, directly adjusting the pruning threshold only affects Gaussians with opacity lower than the threshold.  For Gaussians with opacity higher than the threshold but not on the scene surface, a fixed pruning threshold is ineffective. 

We are conducting experiments on the LLFF dataset with different pruning threshold, the results will be provided later.

**_Q2: The range control of sampling space can be achieved by any existing sampling methods and the camera can be translated in all direction._**

Yes, the existing methods might be able to change the position and angle of the unseen view to achieve the same effect as ours, but they didn't do it. To the best of our knowledge, our method is the first to attempt using viewpoint translation based on Gaussian Splatting to optimize the problem of sparse view synthesis, and it has been proven that it is more effective than other sampling methods.

Indeed, the camera can be translated in any direction, but that still falls under translation without rotation, which we believe is essentially no different from horizontal movement.

**_Q3: The method is relatively sensitive to the matching model and have a great reliance of the matching model._**

Dense initialization is an indispensable part of our method, as we mentioned in Section 3.4 of our paper, our method needs a dense initialization to achieve better geometry initialization, thereby improving the quality of 3DGS. Additionally, experiments have shown that the matching model we selected is effective across different datasets. 

Although our method performs poor when using random initialization, it still outperforms the baseline 3DGS, especially on the DTU dataset, which proves that the other two components of our method are effective.
Thanks for authors' response. 
1. I'm glad to see the ablation studies of the opacity penalty strategy. It would make the effectiveness of this module more convincing. But the motivation is still not clear for me. To my knowledge, this strategy adds a factor (less than 1.0) to weight down the original opacity and make a part of Gaussians below the pruning threshold, and how about directly adjusting the threshold of the pruning operation.
2. The explanation of the binocular stereo consistency do not convince me enough. The authors explain that the advantage of only sampling the virtual view in the horizontal direction is the small rotation and less occlusion. And I think this is just the range control of the virtual view sampling space and can be achieved by any existing sampling methods, and don't show the advantage of the binocular sampling, after all, I can translate the camera in all direction but no rotation or small rotation.
3. The results in Tab. K show that the dense matching model plays a very important role in the overall performance, and this method is relatively sensitive to the matching model. The effectiveness of the proposed modules seems have a great reliance of the matching model, and the random initialization performs poor especially on DTU dataset. And all these results should be analyzed and declared in the paper.
4. Methodological errors and additional explanations about experiments and comparisons should be corrected and stated in the paper.

Therefore, I still think my original rating is fair and this paper requires major revision on the method and experiment.
Dear reviewers,

We appreciate your comments and expertise. Please let us know if there is anything we can clarify further. We would be happy to take this opportunity to discuss with you.

Thanks,

The authors